HTTPD shutting down every night

[Scott van Looy]

Hi,

Since the upgrade to F29 my HTTPD has been shutting down around 3am every night. I’d like
to try and investigate why, but the only thing in the log is:
[mpm_event:notice] [pid 1096508:tid 140027704744192] AH00492: caught SIGWINCH, shutting
down gracefully

These all happen at roughly the same time, but not the exact same time - around 3am

It’s strange

Scott

[Andy Blanchard]

On Wed, 7 Nov 2018 at 10:40, Scott van Looy via users
<users(a)lists.fedoraproject.org&gt; wrote:
>  Since the upgrade to F29 my HTTPD has been shutting down around 3am
> every night. I’d like to try and investigate why, but the only thing in the log is:
>  [mpm_event:notice] [pid 1096508:tid 140027704744192] AH00492: caught SIGWINCH, shutting
> down gracefully
>
>  These all happen at roughly the same time, but not the exact same time - around 3am
Semi-regular times like "around 3am" generally imply something being
governed by Anacron's variable task scheduler which will randomise the
time of cron jobs.  Since this is daily, I'd start by taking a look at
your daily crons and see if any of them might relate to the issue -
most likely a log rotation tool or something along those lines.

SIGWINCH isn't one of the more common signals you'll interact with
like TERM, but it supposedly relates to "WINdow CHange", or a change
in the size of the controlling terminal's window size - e.g. to tell a
shell like bash that the terminal emulator window has been resized and
it now needs to wrap at a different number of characters.  However it
seems Apache doesn't use it entirely correctly so it may be that your
log rotation tool is actually stopping the daemon rather than HUPing
it as it should.

Andy

[Roger Heflin]

this is the daily logrotate process.  it rotates the log files and
reloads httpd.  It is normal.   A number of other process will also
restart at the same time.

See:
cat /etc/logrotate.d/httpd
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    delaycompress
    postrotate
        /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
    endscript
}
On Wed, Nov 7, 2018 at 5:53 AM Andy Blanchard <zocalo(a)gmail.com&gt; wrote:
>
> On Wed, 7 Nov 2018 at 10:40, Scott van Looy via users
> <users(a)lists.fedoraproject.org&gt; wrote:
> > Since the upgrade to F29 my HTTPD has been shutting down around 3am every night.
I’d like to try and investigate why, but the only thing in the log is:
> > [mpm_event:notice] [pid 1096508:tid 140027704744192] AH00492: caught SIGWINCH,
shutting down gracefully
> >
> > These all happen at roughly the same time, but not the exact same time - around
3am
>
> Semi-regular times like "around 3am" generally imply something being
> governed by Anacron's variable task scheduler which will randomise the
> time of cron jobs.  Since this is daily, I'd start by taking a look at
> your daily crons and see if any of them might relate to the issue -
> most likely a log rotation tool or something along those lines.
>
> SIGWINCH isn't one of the more common signals you'll interact with
> like TERM, but it supposedly relates to "WINdow CHange", or a change
> in the size of the controlling terminal's window size - e.g. to tell a
> shell like bash that the terminal emulator window has been resized and
> it now needs to wrap at a different number of characters.  However it
> seems Apache doesn't use it entirely correctly so it may be that your
> log rotation tool is actually stopping the daemon rather than HUPing
> it as it should.
>
> Andy
> _______________________________________________
> users mailing list -- users(a)lists.fedoraproject.org
> To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]

[Scott van Looy]

OK thanks for this.

It’s not normal though, it doesn’t actually restart - it just crashes.

/bin/systemctl reload httpd seems to work OK when I run it as root, the server restarts.

So I’m still a little confused. Does anyone have any suggestions?

>  On 7. Nov 2018, at 15:52, Roger Heflin <rogerheflin(a)gmail.com&gt;
> wrote:
>  
>  this is the daily logrotate process.  it rotates the log files and
>  reloads httpd.  It is normal.   A number of other process will also
>  restart at the same time.
>  
>  See:
>  cat /etc/logrotate.d/httpd
>  /var/log/httpd/*log {
>     missingok
>     notifempty
>     sharedscripts
>     delaycompress
>     postrotate
>         /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true
>     endscript
>  }
>  On Wed, Nov 7, 2018 at 5:53 AM Andy Blanchard <zocalo(a)gmail.com&gt; wrote:
> > 
> > On Wed, 7 Nov 2018 at 10:40, Scott van Looy via users
> > <users(a)lists.fedoraproject.org&gt; wrote:
> >> Since the upgrade to F29 my HTTPD has been shutting down around 3am every night.
> I’d like to try and investigate why, but the only thing in the log is:
> >> [mpm_event:notice] [pid 1096508:tid 140027704744192] AH00492: caught SIGWINCH,
> shutting down gracefully
> >> 
> >> These all happen at roughly the same time, but not the exact same time - around
> 3am
> > 
> > Semi-regular times like "around 3am" generally imply something being
> > governed by Anacron's variable task scheduler which will randomise the
> > time of cron jobs.  Since this is daily, I'd start by taking a look at
> > your daily crons and see if any of them might relate to the issue -
> > most likely a log rotation tool or something along those lines.
> > 
> > SIGWINCH isn't one of the more common signals you'll interact with
> > like TERM, but it supposedly relates to "WINdow CHange", or a change
> > in the size of the controlling terminal's window size - e.g. to tell a
> > shell like bash that the terminal emulator window has been resized and
> > it now needs to wrap at a different number of characters.  However it
> > seems Apache doesn't use it entirely correctly so it may be that your
> > log rotation tool is actually stopping the daemon rather than HUPing
> > it as it should.
> > 
> > Andy
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Andy Blanchard]

Just upgraded a box to 29, and I've spotted the following in the httpd
section of my Logwatch email:

 Unescaped left brace in regex is deprecated here (and will be fatal
in Perl 5.32), passed through in regex; marked by <-- HERE in m/\G%({
<-- HERE .*?})?./ at /usr/share/logwatch/scripts/services/http line
319.

It does, however, appear to have successfully rotated the httpd
logfiles and restated httpd without incident, but may be a pointer as
to what might be going wrong on your server.

Andy

[Scott van Looy]

Thanks for this.

I noticed last night that both https and bind had been killed and failed to restart so I’m
beginning to suspect maybe there’s a memory problem somewhere in the log rotate stuff -
but I’m not sure.

Lots of updates came down today, hoping it might magically fix itself :)

Fun!

Scott

>  On 11. Nov 2018, at 11:27, Andy Blanchard <zocalo(a)gmail.com&gt;
> wrote:
>  
>  Just upgraded a box to 29, and I've spotted the following in the httpd
>  section of my Logwatch email:
>  
>  Unescaped left brace in regex is deprecated here (and will be fatal
>  in Perl 5.32), passed through in regex; marked by <-- HERE in m/\G%({
>  <-- HERE .*?})?./ at /usr/share/logwatch/scripts/services/http line
>  319.
>  
>  It does, however, appear to have successfully rotated the httpd
>  logfiles and restated httpd without incident, but may be a pointer as
>  to what might be going wrong on your server.
>  
>  Andy
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Scott van Looy]

Still restarting.

Today the log reads:

[Mon Nov 12 03:54:57.152381 2018] [mpm_event:notice] [pid 9721:tid 140481219127552]
AH00492: caught SIGWINCH, shutting down gracefully

Checking the cron log:

Nov 12 03:54:49 novak run-parts[81241]: (/etc/cron.daily) finished 0logwatch
Nov 12 03:54:49 novak run-parts[81241]: (/etc/cron.daily) starting autodld
Nov 12 03:54:49 novak run-parts[81241]: (/etc/cron.daily) finished autodld
Nov 12 03:54:49 novak run-parts[81241]: (/etc/cron.daily) starting certwatch
Nov 12 03:54:55 novak run-parts[81241]: (/etc/cron.daily) finished certwatch
Nov 12 03:54:55 novak run-parts[81241]: (/etc/cron.daily) starting logrotate
Nov 12 03:54:58 novak run-parts[81241]: (/etc/cron.daily) finished logrotate
Nov 12 03:54:58 novak run-parts[81241]: (/etc/cron.daily) starting rkhunter

Tried running them individually, nothing seems to stop httpd. I’ve disabled a few so I’ll
see what happens tonight. Again, if anyone has any suggestions, it’d be great to try some
stuff!

Regards,

Scott

>  On 11. Nov 2018, at 11:50, Scott van Looy via users
> <users(a)lists.fedoraproject.org&gt; wrote:
>  
>  Thanks for this.
>  
>  I noticed last night that both https and bind had been killed and failed to restart so
> I’m beginning to suspect maybe there’s a memory problem somewhere in the log rotate stuff
> - but I’m not sure.
>  
>  Lots of updates came down today, hoping it might magically fix itself :)
>  
>  Fun!
>  
>  Scott
>  
> > On 11. Nov 2018, at 11:27, Andy Blanchard <zocalo(a)gmail.com&gt; wrote:
> > 
> > Just upgraded a box to 29, and I've spotted the following in the httpd
> > section of my Logwatch email:
> > 
> > Unescaped left brace in regex is deprecated here (and will be fatal
> > in Perl 5.32), passed through in regex; marked by <-- HERE in m/\G%({
> > <-- HERE .*?})?./ at /usr/share/logwatch/scripts/services/http line
> > 319.
> > 
> > It does, however, appear to have successfully rotated the httpd
> > logfiles and restated httpd without incident, but may be a pointer as
> > to what might be going wrong on your server.
> > 
> > Andy
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Andy Blanchard]

Of that bunch, autodld (downloading RPMs) and rkhunter (scanning the
filesystem, taking checksums, and looking for changes) are both
passive, certwatch probably shouldn't be updating certificates every
24 hours (but could do, so worth checking!), and logwatch just parses
the existing logfiles - it shouldn't touch the daemons.  That leaves
logrotate as the prime suspect, IMHO, as that definitely HUPs/restarts
daemons after it rotates logfiles.  However, it also tracks when it
was last run and won't cycle a logfile and HUP a daemon if it doesn't
need to - since it's already cycled the logs at 03:54, it won't do so
again when you're trying to run it manually unless you tweak the file
"/var/lib/logrotate.status" first.

Note also that httpd exited within the window for logrotate, which
also seems to point the finger at logrotate, so I'd definitely make
sure that was in tonight's list of disabled scripts, and it might be
worth double checking what update cycle certwatch is using too.

Andy

[Scott van Looy]

Thanks for this.

Certwatch is uninstalled now, it just tells me if any of my certs have expired and as I
don’t have any on that box any more it’s kinda pointless :)

I’ve removed the stuff to hide the output from the apache restart in logrotate so I’m
going to let it run tonight and see if anything exciting gets logged

Regards,

Scott

>  On 12. Nov 2018, at 16:08, Andy Blanchard <zocalo(a)gmail.com&gt;
> wrote:
>  
>  Of that bunch, autodld (downloading RPMs) and rkhunter (scanning the
>  filesystem, taking checksums, and looking for changes) are both
>  passive, certwatch probably shouldn't be updating certificates every
>  24 hours (but could do, so worth checking!), and logwatch just parses
>  the existing logfiles - it shouldn't touch the daemons.  That leaves
>  logrotate as the prime suspect, IMHO, as that definitely HUPs/restarts
>  daemons after it rotates logfiles.  However, it also tracks when it
>  was last run and won't cycle a logfile and HUP a daemon if it doesn't
>  need to - since it's already cycled the logs at 03:54, it won't do so
>  again when you're trying to run it manually unless you tweak the file
>  "/var/lib/logrotate.status" first.
>  
>  Note also that httpd exited within the window for logrotate, which
>  also seems to point the finger at logrotate, so I'd definitely make
>  sure that was in tonight's list of disabled scripts, and it might be
>  worth double checking what update cycle certwatch is using too.
>  
>  Andy
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Scott van Looy]

So httpd survived Tuesday without crashing but crashed again yesterday and today.

In /var/log/messages I can see:

Nov 14 03:26:57 novak systemd[1]: Reloading The Apache HTTP Server.
Nov 14 03:26:57 novak systemd[739346]: httpd.service: Failed to set up mount namespacing:
No such file or directory
Nov 14 03:26:57 novak systemd[739346]: httpd.service: Failed at step NAMESPACE spawning
/usr/sbin/httpd: No such file or directory
Nov 14 03:26:57 novak systemd[1]: httpd.service: Control process exited, code=exited
status=226

And then a little later

Nov 14 03:26:58 novak systemd[1]: httpd.service: Killing process 230630 (nss_pcache) with
signal SIGKILL.
Nov 14 03:26:58 novak systemd[1]: httpd.service: Failed with result 'exit-code'.
Nov 14 03:26:58 novak systemd[1]: Reload failed for The Apache HTTP Server.
Nov 14 03:26:58 novak audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295
subj=system_u:system_r:init_t:s0 msg='unit=httpd comm="systemd"
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Nov 14 03:26:58 novak logrotate[739362]: ALERT exited abnormally with [1]

Googling the NAMESPACE error I find a post about /tmp or /var/tmp being symbolic links.
Neither are.

Yesterday I reinstalled httpd from scratch, last night it crashed again in the same way.

Does anyone know what user logrotate runs as? I’ve tried /bin/systemctl reload
httpd.service as root and it reloads as expected and am wondering if there’s something
else weird going on here?

Regards,

Scott

>  On 12. Nov 2018, at 16:32, Scott van Looy via users
> <users(a)lists.fedoraproject.org&gt; wrote:
>  
>  Thanks for this.
>  
>  Certwatch is uninstalled now, it just tells me if any of my certs have expired and as I
> don’t have any on that box any more it’s kinda pointless :)
>  
>  I’ve removed the stuff to hide the output from the apache restart in logrotate so I’m
> going to let it run tonight and see if anything exciting gets logged
>  
>  Regards,
>  
>  Scott
>  
> > On 12. Nov 2018, at 16:08, Andy Blanchard <zocalo(a)gmail.com&gt; wrote:
> > 
> > Of that bunch, autodld (downloading RPMs) and rkhunter (scanning the
> > filesystem, taking checksums, and looking for changes) are both
> > passive, certwatch probably shouldn't be updating certificates every
> > 24 hours (but could do, so worth checking!), and logwatch just parses
> > the existing logfiles - it shouldn't touch the daemons.  That leaves
> > logrotate as the prime suspect, IMHO, as that definitely HUPs/restarts
> > daemons after it rotates logfiles.  However, it also tracks when it
> > was last run and won't cycle a logfile and HUP a daemon if it doesn't
> > need to - since it's already cycled the logs at 03:54, it won't do so
> > again when you're trying to run it manually unless you tweak the file
> > "/var/lib/logrotate.status" first.
> > 
> > Note also that httpd exited within the window for logrotate, which
> > also seems to point the finger at logrotate, so I'd definitely make
> > sure that was in tonight's list of disabled scripts, and it might be
> > worth double checking what update cycle certwatch is using too.
> > 
> > Andy
> > _______________________________________________
> > users mailing list -- users(a)lists.fedoraproject.org
> > To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  _______________________________________________
>  users mailing list -- users(a)lists.fedoraproject.org
>  To unsubscribe send an email to users-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Berend De Schouwer]

On Thu, 2018-11-15 at 06:12 +0100, Scott van Looy via users wrote:
>  So httpd survived Tuesday without crashing but crashed again
>  yesterday and today.
>  
>  In /var/log/messages I can see:
>  
>  Nov 14 03:26:57 novak systemd[1]: Reloading The Apache HTTP Server.
>  Nov 14 03:26:57 novak systemd[739346]: httpd.service: Failed to set
>  up mount namespacing: No such file or directory
>  Nov 14 03:26:57 novak systemd[739346]: httpd.service: Failed at step
>  NAMESPACE spawning /usr/sbin/httpd: No such file or directory 
These two lines are bad.

Do you have custom /etc/systemd/system/httpd* ?  It likely comes from
"PrivateTmp=..." in httpd.service.  The stock file *does* have this.


>  Googling the NAMESPACE error I find a post about /tmp or /var/tmp
>  being symbolic links. Neither are. 
Does ls -laZd /tmp or /var/tmp look like:

drwxrwxrwt. 18 root root system_u:object_r:tmp_t:s0               16384
Nov 15 10:30 tmp

Does ls -laZd /run /run/httpd/ look like:

drwxr-xr-x. 65 root root   system_u:object_r:var_run_t:s0       1660
Nov 15 09:41 /run/
drwx--x---.  3 root apache system_u:object_r:httpd_var_run_t:s0  100
Nov 13 13:06 /run/httpd/

Maybe there's a cron process that messes with /tmp/ ?  Some
RedHat/Fedora systems come with tmpwatch (rpm -q tmpwatch ; ls
/etc/cron*/tmpwatch*) which removes stale files from /tmp/ and
/var/tmp/.  One of these could be biting.


>  Yesterday I reinstalled httpd from scratch, last night it crashed
>  again in the same way. 
Seems more like a bad selinux or permissions thing.  Or something that
deletes files.


>  Does anyone know what user logrotate runs as? I’ve tried
>  /bin/systemctl reload httpd.service as root and it reloads as
>  expected and am wondering if there’s something else weird going on
>  here? 
root, except where specified otherwise.  Look for "su someuser" in
/etc/logrotate.d/*

/etc/logrotate.conf/httpd on my machine runs as root (some conf files
run as a different user)  httpd *startup* needs root.

Note that logrotate should reload (not restart) httpd.  Reloading
should not kill httpd.  This happens to provide an uninterrupted
service on tcp port 80.

*That* "needs" (not really) a consistent file like /run/httpd.pid and
that might go missing.

If you don't need 100% uptime you could replace reload with restart in
/etc/logrotate.d/httpd, but you'll lose your webserver for a few
seconds every day.


Personally I might make a new cron script like
(
   ls -laZd /tmp/ /var/tmp/ /run/ /run/httpd/
   ... something else?
) 2>&1 | logger -t "myscript"

and see if the files go missing.