On Sun, Jan 5, 2020 at 10:18 AM Zbigniew Jędrzejewski-Szmek
On Sat, Jan 04, 2020 at 04:38:19PM -0700, Chris Murphy wrote:
> On Sat, Jan 4, 2020 at 2:51 AM Aleksandra Fedorova <alpha(a)bookwar.info>
> > Since in the Change we are not introducing just the earlyoom tool but enable it
with a specific profile I would add those details here. Smth like:
> > "earlyoom service will choose the offending process based on the same
oom_score as kernel uses. It will send a SIGTERM signal on 10% of RAM left, and SIGKILL on
> I add this information to the summary. Also, I think these numbers may
> need to change to avoid prematurely sending SIGTERM when the system
> has no swap device.
> > As I understand in the current setup we are looking more for a controlled
failure scenario rather than for a solution.
> Yes, it's fair to say this proposal is to make things "less bad". It
> doesn't improve system responsiveness. Once heavy swap starts, the
> system is sluggish, stutters, and briefly stalls. This proposal
> doesn't fix that. There is a lot of room for improvement.
> > Can we get a specific manual, what users supposed to do, once they trigger the
earlyoom? Does earlyoom help in reporting? Which logs we need to look at?
> > Maybe add a section in UX part of the change, or setup a dedicated wiki page?
> The user shouldn't need to do anything differently than if the kernel
> oom-killer had triggered. The system journal will contain messages
> showing what was killed and why:
> Jan 04 16:05:42 fmac.local earlyoom: low memory! at or below
> SIGTERM limits: mem 10 %, swap 10 %
> Jan 04 16:05:42 fmac.local earlyoom: sending SIGTERM to process
> 27421 "chrome": badness 305, VmRSS 42 MiB
> > Additionally, there was a question during the chat discussion: how the earlyoom
setup will work together with OOMPolicy and any other related options of systemd units?
Will systemd recognize the OOM event?
> My understanding of systemd OOMPolicy= behavior, is it looks for the
> kernel's oom-killer messages and acts upon those. Whereas earlyoom
> uses the same metric (oom_score) as the oom-killer, it does not invoke
> the oom-killer. Therefore systemd probably does not get the proper
> hint to implement OOMPolicy=
Yes. The kernel reports oom events in the cgroup file memory.events,
and systemd waits for an inotify event on that file; OOMPolicy=stop is
implemented that way. And the OOMPolicy=kill option is "implemented"
by setting memory.oom.group=1 in the kernel  and having the kernel
kill all the processes. So systemd is providing a thin wrapper around
the kernel functionality.
If processes are not killed by the kernel but through a signal from
userspace, all of this will not work.
I grepped /usr/lib/systemd and /etc/systemd for "OOM" on my
workstation and it seems that we have only OOMScoreAdjust option used
in the installed systemd units. And this option will be respected by
Since on workstation we don't use tweaking of the OOMPolicy on the
unit level, I'd say we can leave the tweaking to the system
administrators: when there is need to adjust OOMPolicy of a service,
administrators would need to tweak or disable earlyoom service as
But I'd like to understand better the difference between _default_
OOM-event and _default_ earlyoom-event:
Afaik DefaultOOMPolicy is set to "stop", which means if one of the
processes in the service is killed by OOM, other processes from the
same service are gracefully stopped by systemd.
What is the default behavior of the systemd service on external
SIGTERM/SIGKILL signal sent to the process by earlyoom?