An infection seems to be spreading in systemd. First I saw dhcpd taking forever to shut down:
https://bugzilla.redhat.com/show_bug.cgi?id=1768604
Now I just saw the exact same thing with the apache httpd service.
I found the systemctl --no-ask-password option, so I tried it, now it no longer forks and execs the password-agent process, it merely sits like a lump timing out for the same amount of time.
Anyone have any idea what leads systemd to decide it needs to wait for some mysterious something when stopping some services? And why is the number of these mysterious services growing? (Does this have something to do with me not using NetworkManager?)
On Sun, 16 Feb 2020 13:41:06 -0500 Tom Horsley horsley1953@gmail.com wrote:
An infection seems to be spreading in systemd. First I saw dhcpd taking forever to shut down:
https://bugzilla.redhat.com/show_bug.cgi?id=1768604
Now I just saw the exact same thing with the apache httpd service.
I found the systemctl --no-ask-password option, so I tried it, now it no longer forks and execs the password-agent process, it merely sits like a lump timing out for the same amount of time.
Anyone have any idea what leads systemd to decide it needs to wait for some mysterious something when stopping some services? And why is the number of these mysterious services growing? (Does this have something to do with me not using NetworkManager?)
I run a custom daemon to gather entropy, and it sleeps most of the time. If I don't kill -9 it before I shut down, it is sleeping, and I get the behavior you describe (it doesn't catch the kill signal, presumably -15, that systemd sends). When I researched the issue, I got as far as finding that there is a timer that can be set that defaults to 1 minute 30 seconds. I'm sure it is in the documentation where this timer is configured, but I never got that far. Setting it to a lower number for this, and other things, would reduce your aggravation.
On Sun, Feb 16, 2020 at 10:25 PM stan via users users@lists.fedoraproject.org wrote:
On Sun, 16 Feb 2020 13:41:06 -0500 Tom Horsley horsley1953@gmail.com wrote:
An infection seems to be spreading in systemd. First I saw dhcpd taking forever to shut down:
https://bugzilla.redhat.com/show_bug.cgi?id=1768604
Now I just saw the exact same thing with the apache httpd service.
I found the systemctl --no-ask-password option, so I tried it, now it no longer forks and execs the password-agent process, it merely sits like a lump timing out for the same amount of time.
Anyone have any idea what leads systemd to decide it needs to wait for some mysterious something when stopping some services? And why is the number of these mysterious services growing? (Does this have something to do with me not using NetworkManager?)
I run a custom daemon to gather entropy, and it sleeps most of the time. If I don't kill -9 it before I shut down, it is sleeping, and I get the behavior you describe (it doesn't catch the kill signal, presumably -15, that systemd sends). When I researched the issue, I got as far as finding that there is a timer that can be set that defaults to 1 minute 30 seconds. I'm sure it is in the documentation where this timer is configured, but I never got that far. Setting it to a lower number for this, and other things, would reduce your aggravation.
"DefaultTimeoutStartSec" and "DefaultTimeoutStopSec" in "/etc/systemd/system.conf"
On Mon, 17 Feb 2020 11:39:25 +0100 Tom H tomh0665@gmail.com wrote:
On Sun, Feb 16, 2020 at 10:25 PM stan via users users@lists.fedoraproject.org wrote:
defaults to 1 minute 30 seconds. I'm sure it is in the documentation where this timer is configured, but I never got that far. Setting it to a lower number for this, and other things, would reduce your aggravation.
"DefaultTimeoutStartSec" and "DefaultTimeoutStopSec" in "/etc/systemd/system.conf"
Thanks, now when I forget to kill the process, I won't have to wait as long.
On Tue, Feb 18, 2020 at 8:29 PM stan via users users@lists.fedoraproject.org wrote:
On Mon, 17 Feb 2020 11:39:25 +0100 Tom H tomh0665@gmail.com wrote:
On Sun, Feb 16, 2020 at 10:25 PM stan via users users@lists.fedoraproject.org wrote:
defaults to 1 minute 30 seconds. I'm sure it is in the documentation where this timer is configured, but I never got that far. Setting it to a lower number for this, and other things, would reduce your aggravation.
"DefaultTimeoutStartSec" and "DefaultTimeoutStopSec" in "/etc/systemd/system.conf"
Thanks, now when I forget to kill the process, I won't have to wait as long.
You're welcome. It's too bad that there isn't a ctrl-something combination to "kill -9" reluctant daemons at shutdown.
On Sun, 16 Feb 2020 at 18:41, Tom Horsley horsley1953@gmail.com wrote:
An infection seems to be spreading in systemd. First I saw dhcpd taking forever to shut down:
https://bugzilla.redhat.com/show_bug.cgi?id=1768604
Now I just saw the exact same thing with the apache httpd service.
I found the systemctl --no-ask-password option, so I tried it, now it no longer forks and execs the password-agent process, it merely sits like a lump timing out for the same amount of time.
Anyone have any idea what leads systemd to decide it needs to wait for some mysterious something when stopping some services? And why is the number of these mysterious services growing? (Does this have something to do with me not using NetworkManager?) _______________________________________________
Just a shot in the dark, do you have selinux enabled?
On Thu, 20 Feb 2020 at 23:27, Tom Horsley horsley1953@gmail.com wrote:
On Thu, 20 Feb 2020 23:06:41 +0000 Anthony F McInerney wrote:
Just a shot in the dark, do you have selinux enabled?
Selinux is completely disabled. Maybe it is mad because I don't have it turned on :-).
I guess at this point uname -a systemctl --version cat /proc/cmdline Would be interesting. (Apologies if you posted this or attached it to the bug report - i don't appear to see anything for these)
On Fri, 21 Feb 2020 00:55:00 +0000 Anthony F McInerney wrote:
I guess at this point uname -a systemctl --version cat /proc/cmdline Would be interesting. (Apologies if you posted this or attached it to the bug report - i don't appear to see anything for these)
Not sure why it would be interesting, but it is easy to get:
[root@zooty ~]# uname -a Linux zooty.my.lan 5.4.19-200.fc31.x86_64 #1 SMP Wed Feb 12 15:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@zooty ~]# systemctl --version systemd 243 (v243.7-1.fc31) +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified [root@zooty ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,msdos5)/boot/vmlinuz-5.4.19-200.fc31.x86_64 root=UUID=47a0b43c-2a9d-4929-81e5-860ece806911 ro selinux=0 audit=0 net.ifnames=0 biosdevname=0 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1
What would really be interesting is a way to get a backtrace from systemd when it is waiting for the service to see what in the sam hill made it decide to wait :-).
On Fri, 21 Feb 2020 at 01:16, Tom Horsley horsley1953@gmail.com wrote:
On Fri, 21 Feb 2020 00:55:00 +0000 Anthony F McInerney wrote:
I guess at this point uname -a systemctl --version cat /proc/cmdline Would be interesting. (Apologies if you posted this or attached it to the bug report - i don't appear to see anything for these)
Not sure why it would be interesting, but it is easy to get:
[root@zooty ~]# uname -a Linux zooty.my.lan 5.4.19-200.fc31.x86_64 #1 SMP Wed Feb 12 15:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@zooty ~]# systemctl --version systemd 243 (v243.7-1.fc31) +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified [root@zooty ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,msdos5)/boot/vmlinuz-5.4.19-200.fc31.x86_64 root=UUID=47a0b43c-2a9d-4929-81e5-860ece806911 ro selinux=0 audit=0 net.ifnames=0 biosdevname=0 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1
What would really be interesting is a way to get a backtrace from systemd when it is waiting for the service to see what in the sam hill made it decide to wait :-).
On the bug report, you ran dhcp -f (without using systemd?),same issue,
SIGINT being ignored, and again with httpd (with or without systemd i guess it doesn't matter).
So it seems, that SIGINT is also being ignored when systemd sends it to the process? (And then you ran into the infamous 1 min 30 sec+ timer, since the process won't exit gracefully?).
So rather, you have some odd, kernel issue maybe?
Lotta questions marks there, as it seems you still think the issue is sytemd, and i'm wondering if i misread something...
-
Anthony F McInerney -
stan -
Tom H -
Tom Horsley