Where's my load average coming from?

[Sam Varshavchik]

One of my servers was a bit "unresponsive". After waiting about 20 seconds  
for an ssh connection, the root shell seemed fine, but top showed this:

top - 06:31:36 up 3 days, 21:37,  2 users,  load average: 6.00, 6.00, 6.00
Tasks: 294 total,   1 running, 277 sleeping,   0 stopped,  16 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0  
st
KiB Mem :  4045896 total,  1200824 free,   346588 used,  2498484 buff/cache
KiB Swap:  2096112 total,  2093448 free,     2664 used.  3288268 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+  
COMMAND
22031 root      20   0  156776   4092   3480 R   0.9  0.1   0:00.04  
top
    1 root      20   0  147032   7660   5616 D   0.0  0.2   0:14.18  
systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.05  
kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.01  
ksoftirqd/0


The load average was 6. But nothing was burning CPU.

After poking around, all signs were pointing to systemd doing what systemd  
does best:

# systemctl status
Failed to read server status: Connection timed out

And the 16 zombie processes were system daemons, that should've been reaped  
by systemd.

"reboot" did nothing, of course. "reboot --force" did the trick.

Setting aside yet another systemd fiasco (on a mostly idle server that did  
absolutely nothing for the last ten hours) I'm curious as to how  
/proc/loadavg could end up reporting a load average of 6, without any  
processes being seeming to be doing anything.

[Joe Zeff]

On 01/18/2017 03:41 AM, Sam Varshavchik wrote:
>  top - 06:31:36 up 3 days, 21:37,  2 users,  load average: 6.00, 6.00, 6.00
>  Tasks: 294 total,   1 running, 277 sleeping,   0 stopped,  16 zombie 
Why do you have 16 zombies after just under 4 days?

[Rick Stevens]

On 01/18/2017 03:41 AM, Sam Varshavchik wrote:
>  One of my servers was a bit "unresponsive". After waiting
> about 20
>  seconds for an ssh connection, the root shell seemed fine, but top
>  showed this:
>  
>  top - 06:31:36 up 3 days, 21:37,  2 users,  load average: 6.00, 6.00, 6.00
>  Tasks: 294 total,   1 running, 277 sleeping,   0 stopped,  16 zombie
>  %Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si, 
>  0.0 st
>  KiB Mem :  4045896 total,  1200824 free,   346588 used,  2498484 buff/cache
>  KiB Swap:  2096112 total,  2093448 free,     2664 used.  3288268 avail Mem
>  
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
>  22031 root      20   0  156776   4092   3480 R   0.9  0.1   0:00.04 top
>     1 root      20   0  147032   7660   5616 D   0.0  0.2   0:14.18 systemd
>     2 root      20   0       0      0      0 S   0.0  0.0   0:00.05 kthreadd
>     3 root      20   0       0      0      0 S   0.0  0.0   0:00.01
>  ksoftirqd/0
>  
>  
>  The load average was 6. But nothing was burning CPU.
>  
>  After poking around, all signs were pointing to systemd doing what
>  systemd does best:
>  
>  # systemctl status
>  Failed to read server status: Connection timed out
>  
>  And the 16 zombie processes were system daemons, that should've been
>  reaped by systemd.
>  
>  "reboot" did nothing, of course. "reboot --force" did the trick.
>  
>  Setting aside yet another systemd fiasco (on a mostly idle server that
>  did absolutely nothing for the last ten hours) I'm curious as to how
>  /proc/loadavg could end up reporting a load average of 6, without any
>  processes being seeming to be doing anything. 
Your systemd was in a D (I/O wait) state and any process in that state
will drive the load up very high. This can be caused by devices that
aren't responding (e.g. a boatload of disk I/O), a ton of interrupts or
possibly context switches. In your case it isn't interrupts (the "hi"
and "si" fields of top), so it's probably I/O or context switches. You
need to run something like "vmstat 5" to see I/O and context switches.
The "bi" (block in) and "bo" (block out) fields under
"--io--" in the
vmstat output show disk I/O, the "cs" field under "--system--" shows
the
context switches.

And yes, I agree...systemd is a spectacular failure.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
-   Let us think the unthinkable. Let us do the undoable. Let us     -
-   prepare to grapple with the ineffable itself, and see if we may  -
-                      not eff it up after all.                      -
-                                                 -- Douglas Adams   -
----------------------------------------------------------------------

[Joe Zeff]

On 01/18/2017 09:20 AM, Rick Stevens wrote:
>  And yes, I agree...systemd is a spectacular failure. 
This March, systemd will celebrate its 7th birthday.  Complaints like 
this are about as useful as King Canute's attempt to hold back the tide. 
  Don't you think it's time that you learned to stop worrying and love 
systemd?

[Rick Stevens]

On 01/18/2017 09:33 AM, Joe Zeff wrote:
>  On 01/18/2017 09:20 AM, Rick Stevens wrote:
> > And yes, I agree...systemd is a spectacular failure.
>  
>  This March, systemd will celebrate its 7th birthday.  Complaints like
>  this are about as useful as King Canute's attempt to hold back the tide.
>   Don't you think it's time that you learned to stop worrying and love
>  systemd? 
Joe, we've had this conversation before. I may be forced to use this
overly complex, broken piece of crud, but I sure as hell don't have to
"love" it. And the discussion ends there.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital    ricks(a)alldigital.com -
- AIM/Skype: therps2        ICQ: 226437340           Yahoo: origrps2 -
-                                                                    -
- If at first you don't succeed, quit. No sense being a damned fool! -
----------------------------------------------------------------------

[Joe Zeff]

On 01/18/2017 10:18 AM, Rick Stevens wrote:
>  On 01/18/2017 09:33 AM, Joe Zeff wrote:
> > On 01/18/2017 09:20 AM, Rick Stevens wrote:
> >> And yes, I agree...systemd is a spectacular failure.
> >
> > This March, systemd will celebrate its 7th birthday.  Complaints like
> > this are about as useful as King Canute's attempt to hold back the tide.
> >   Don't you think it's time that you learned to stop worrying and love
> > systemd?
>
>  Joe, we've had this conversation before. I may be forced to use this
>  overly complex, broken piece of crud, but I sure as hell don't have to
>  "love" it. And the discussion ends there. 
I made that comment just to get in a reference to a movie from the '60s, 
and that's all the response I get?

[Patrick O'Callaghan]

On Wed, 2017-01-18 at 10:29 -0800, Joe Zeff wrote:
>  On 01/18/2017 10:18 AM, Rick Stevens wrote:
>  > On 01/18/2017 09:33 AM, Joe Zeff wrote:
>  > > On 01/18/2017 09:20 AM, Rick Stevens wrote:
>  > > > And yes, I agree...systemd is a spectacular failure.
>  > > 
>  > > This March, systemd will celebrate its 7th birthday.  Complaints like
>  > > this are about as useful as King Canute's attempt to hold back the tide.
>  > >   Don't you think it's time that you learned to stop worrying and love
>  > > systemd?
>  > 
>  > Joe, we've had this conversation before. I may be forced to use this
>  > overly complex, broken piece of crud, but I sure as hell don't have to
>  > "love" it. And the discussion ends there.
>  
>  I made that comment just to get in a reference to a movie from the '60s, 
>  and that's all the response I get? 
You can't fight in here, this is the War Room!

poc

[Tim]

Allegedly, on or about 18 January 2017, Joe Zeff sent:
>  I made that comment just to get in a reference to a movie from the
>  '60s, and that's all the response I get?  
I'm tempted to make some comment about Bat Guano...

-- 


Boilerplate:  All mail to my mailbox is automatically deleted, there is
no point trying to privately email me, I only get to see the messages
posted to the mailing list.

[Chris Adams]

Once upon a time, Tim <ignored_mailbox(a)yahoo.com.au&gt; said:
>  Allegedly, on or about 18 January 2017, Joe Zeff sent:
>  > I made that comment just to get in a reference to a movie from the
>  > '60s, and that's all the response I get? 
>  
>  I'm tempted to make some comment about Bat Guano... 
If you do that, you'll have to answer to the Coca-Cola company!
-- 
Chris Adams <linux(a)cmadams.net&gt;

[Sam Varshavchik]

Joe Zeff writes:

>  On 01/18/2017 03:41 AM, Sam Varshavchik wrote:
> >
> > top - 06:31:36 up 3 days, 21:37,  2 users,  load average: 6.00, 6.00, 6.00
> > Tasks: 294 total,   1 running, 277 sleeping,   0 stopped,  16 zombie
>
>  Why do you have 16 zombies after just under 4 days? 
Good question. Why is systemd not reaping zombies, as PID 1 should be doing?

[Sam Varshavchik]

Joe Zeff writes:

>  On 01/18/2017 10:18 AM, Rick Stevens wrote:
> > On 01/18/2017 09:33 AM, Joe Zeff wrote:
> >> On 01/18/2017 09:20 AM, Rick Stevens wrote:
> >>> And yes, I agree...systemd is a spectacular failure.
> >>
> >> This March, systemd will celebrate its 7th birthday.  Complaints like
> >> this are about as useful as King Canute's attempt to hold back the tide.
> >>   Don't you think it's time that you learned to stop worrying and love
> >> systemd?
> >
> > Joe, we've had this conversation before. I may be forced to use this
> > overly complex, broken piece of crud, but I sure as hell don't have to
> > "love" it. And the discussion ends there.
>
>  I made that comment just to get in a reference to a movie from the '60s, and  
>  that's all the response I get? 
Yeah, I'll have to give you that one. The reference was rather obvious.

[Gordon Messmer]

On 01/19/2017 04:02 AM, Sam Varshavchik wrote:
>  Good question. Why is systemd not reaping zombies, as PID 1 should be
>
>  doing?  

Since it was in D state, it probably issued a blocking system call that 
didn't return as expected and hung the application. Unless that 
condition is reproducible, it'll probably be hard to track down and fix 
the bug.  If it is reproducible, see if you can get a backtrace.

[Tom Horsley]

On Thu, 19 Jan 2017 07:06:23 -0800
Gordon Messmer wrote:

>  > Good question. Why is systemd not reaping zombies, as PID 1
> should be 
>  > doing?   
>  
>  
>  Since it was in D state, it probably issued a blocking system call that 
>  didn't return as expected and hung the application. 
Better question: Why is PID 1 the monolithic insanely complex "systemd"
program? Why isn't PID 1 just an "init" process that does nothing but
reap zombies and is much less likely to break due to some complex
operation going wrong, updates modifying libraries out from under it, etc?
Why not let all the complex stuff in systemd happen in PID 2?

[Matthew Miller]

On Thu, Jan 19, 2017 at 10:16:13AM -0500, Tom Horsley wrote:
>  > Since it was in D state, it probably issued a blocking system
> call that 
>  > didn't return as expected and hung the application.
>  
>  Better question: Why is PID 1 the monolithic insanely complex "systemd"
>  program? Why isn't PID 1 just an "init" process that does nothing but
>  reap zombies and is much less likely to break due to some complex
>  operation going wrong, updates modifying libraries out from under it, etc?
>  Why not let all the complex stuff in systemd happen in PID 2? 
I don't think that's really a better question, because actually most of
the complex other stuff _does_ happen in other processes. Take a look
at /usr/lib/systemd for a variety of discrete parts.

There are plenty of things which could be improved in systemd, but I
don't see what tilting at entirely imaginary windmills accomplishes.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Tom Horsley]

On Thu, 19 Jan 2017 10:41:41 -0500
Matthew Miller wrote:

>  I don't think that's really a better question, because
> actually most of
>  the complex other stuff _does_ happen in other processes. Take a look
>  at /usr/lib/systemd for a variety of discrete parts.
>  
>  There are plenty of things which could be improved in systemd, but I
>  don't see what tilting at entirely imaginary windmills accomplishes. 
An old system:

root@ubuntu8d04x:~# ldd /sbin/init
        linux-vdso.so.1 =>  (0x00007fff8b3ff000)
        libc.so.6 => /lib/libc.so.6 (0x00007ff582d2a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ff58308c000)

A new system:

[root@tomh ~]# ldd /usr/lib/systemd/systemd
        linux-vdso.so.1 (0x00007ffc78d2b000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe219ddd000)
        libcap.so.2 => /lib64/libcap.so.2 (0x00007fe219bd8000)
        librt.so.1 => /lib64/librt.so.1 (0x00007fe2199cf000)
        libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007fe21978e000)
        libpam.so.0 => /lib64/libpam.so.0 (0x00007fe21957f000)
        libaudit.so.1 => /lib64/libaudit.so.1 (0x00007fe219356000)
        libkmod.so.2 => /lib64/libkmod.so.2 (0x00007fe219140000)
        libmount.so.1 => /lib64/libmount.so.1 (0x00007fe218ef6000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe218cde000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe218ac2000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe218700000)
        /lib64/ld-linux-x86-64.so.2 (0x0000563248d57000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe21848c000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe218288000)
        libattr.so.1 => /lib64/libattr.so.1 (0x00007fe218083000)
        libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fe217e7c000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fe217c56000)
        libz.so.1 => /lib64/libz.so.1 (0x00007fe217a40000)
        libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fe2177fc000)
        libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fe2175f7000)

Doesn't look imaginary to me.

[Matthew Miller]

On Thu, Jan 19, 2017 at 12:48:33PM -0500, Tom Horsley wrote:
>  An old system:
>  
>  root@ubuntu8d04x:~# ldd /sbin/init
>          linux-vdso.so.1 =>  (0x00007fff8b3ff000)
>          libc.so.6 => /lib/libc.so.6 (0x00007ff582d2a000)
>          /lib64/ld-linux-x86-64.so.2 (0x00007ff58308c000)
>  
>  A new system:
>  
>  [root@tomh ~]# ldd /usr/lib/systemd/systemd
>          linux-vdso.so.1 (0x00007ffc78d2b000)
>          libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe219ddd000)
>          libcap.so.2 => /lib64/libcap.so.2 (0x00007fe219bd8000)
>          librt.so.1 => /lib64/librt.so.1 (0x00007fe2199cf000)
>          libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007fe21978e000)
>          libpam.so.0 => /lib64/libpam.so.0 (0x00007fe21957f000)
>          libaudit.so.1 => /lib64/libaudit.so.1 (0x00007fe219356000)
>          libkmod.so.2 => /lib64/libkmod.so.2 (0x00007fe219140000)
>          libmount.so.1 => /lib64/libmount.so.1 (0x00007fe218ef6000)
>          libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe218cde000)
>          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe218ac2000)
>          libc.so.6 => /lib64/libc.so.6 (0x00007fe218700000)
>          /lib64/ld-linux-x86-64.so.2 (0x0000563248d57000)
>          libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe21848c000)
>          libdl.so.2 => /lib64/libdl.so.2 (0x00007fe218288000)
>          libattr.so.1 => /lib64/libattr.so.1 (0x00007fe218083000)
>          libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fe217e7c000)
>          liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fe217c56000)
>          libz.so.1 => /lib64/libz.so.1 (0x00007fe217a40000)
>          libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fe2177fc000)
>          libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fe2175f7000)
>  
>  Doesn't look imaginary to me. 
"Makes use of shared libraries" isn't "monolithic insanely complex
[...] program".



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Sam Varshavchik]

Tom Horsley writes:

>  On Thu, 19 Jan 2017 10:41:41 -0500
>  Matthew Miller wrote:
>
>  > I don't think that's really a better question, because actually most of
>  > the complex other stuff _does_ happen in other processes. Take a look
>  > at /usr/lib/systemd for a variety of discrete parts.
>  >
>  > There are plenty of things which could be improved in systemd, but I
>  > don't see what tilting at entirely imaginary windmills accomplishes.
>
>  An old system:
>
>  root@ubuntu8d04x:~# ldd /sbin/init
>          linux-vdso.so.1 =>  (0x00007fff8b3ff000)
>          libc.so.6 => /lib/libc.so.6 (0x00007ff582d2a000)
>          /lib64/ld-linux-x86-64.so.2 (0x00007ff58308c000)
>
>  A new system:
>
>  [root@tomh ~]# ldd /usr/lib/systemd/systemd
>          linux-vdso.so.1 (0x00007ffc78d2b000)
>          libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe219ddd000)
>          libcap.so.2 => /lib64/libcap.so.2 (0x00007fe219bd8000)
>          librt.so.1 => /lib64/librt.so.1 (0x00007fe2199cf000)
>          libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007fe21978e000)
>          libpam.so.0 => /lib64/libpam.so.0 (0x00007fe21957f000)
>          libaudit.so.1 => /lib64/libaudit.so.1 (0x00007fe219356000)
>          libkmod.so.2 => /lib64/libkmod.so.2 (0x00007fe219140000)
>          libmount.so.1 => /lib64/libmount.so.1 (0x00007fe218ef6000)
>          libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe218cde000)
>          libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe218ac2000)
>          libc.so.6 => /lib64/libc.so.6 (0x00007fe218700000)
>          /lib64/ld-linux-x86-64.so.2 (0x0000563248d57000)
>          libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fe21848c000)
>          libdl.so.2 => /lib64/libdl.so.2 (0x00007fe218288000)
>          libattr.so.1 => /lib64/libattr.so.1 (0x00007fe218083000)
>          libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fe217e7c000)
>          liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fe217c56000)
>          libz.so.1 => /lib64/libz.so.1 (0x00007fe217a40000)
>          libblkid.so.1 => /lib64/libblkid.so.1 (0x00007fe2177fc000)
>          libuuid.so.1 => /lib64/libuuid.so.1 (0x00007fe2175f7000)
>
>  Doesn't look imaginary to me. 
And before someone starts claiming that /usr/lib/systemd/systemd is not  
/sbin/init (it is), or that most of these are indirect dependencies: nope,  
about half of these are direct required for PID 1. Scary.

[root@octopus ~]# objdump -p /sbin/init | grep NEED
  NEEDED               libsystemd-shared-231.so
  NEEDED               libselinux.so.1
  NEEDED               librt.so.1
  NEEDED               libseccomp.so.2
  NEEDED               libpam.so.0
  NEEDED               libaudit.so.1
  NEEDED               libkmod.so.2
  NEEDED               libmount.so.1
  NEEDED               libgcc_s.so.1
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  VERNEED              0x000000000000a3fc
  VERNEEDNUM           0x0000000000000007

A bug in any of those (including the indirect dependencies) is going to  
bring down the whole house of cards.

And it's going to be FUN when an update to libpcre results in a completely  
unbootable brick (not meant to diss libpcre, it's a very dependable library,  
but crap happens, and when it does, it won't be libpcre that'll be held  
responsible for the fallout).