HTTP returns non-2xx status code

[Tania Hagan]

Hi freeipa-users,

We're currently using Freeipa version 4.11.0 on Rocky 9.4 with Apache/2.4.57 and every so often our HTTP fails with our monitoring returning non-2xx status code.  Looking at the error log for httpd I see:

[mpm_event:error] [pid 1800:tid 1800] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.

Researching this error, it appears to recommend adding to the httpd configuration https://forum.directadmin.com/threads/ah03490-scoreboard-is-full-note-at-max....  Has anyone had to do this or can recommend a "freeipa" way to resolve this? (Reluctant to mess around with the httpd configuration as unsure on the effects of the client connections with freeipa)

Many Thanks, 
Tania

[Rob Crittenden]

Tania Hagan via FreeIPA-users wrote:
> Hi freeipa-users,
>
> We're currently using Freeipa version 4.11.0 on Rocky 9.4 with Apache/2.4.57 and every so often our HTTP fails with our monitoring returning non-2xx status code.  Looking at the error log for httpd I see:
>
> [mpm_event:error] [pid 1800:tid 1800] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
>
> Researching this error, it appears to recommend adding to the httpd configuration https://forum.directadmin.com/threads/ah03490-scoreboard-is-full-note-at-max....  Has anyone had to do this or can recommend a "freeipa" way to resolve this? (Reluctant to mess around with the httpd configuration as unsure on the effects of the client connections with freeipa)

What is the frequency you are seeing this? Does the timing correlate to
other things happening on the system? High load, log rotation, etc?

How are you addressing it now? I assume an Apache restart?

rob

[Tania Hagan]

Hi,

I would say it happens around every 5 days, but we have 7 replicas and it seems to be a different replica each time. We have been seeing high CPU and increased the spec of some of our aws instances which has seemed to settle down the CPU a bit (slapd is usually hogging the CPU).  (our average instance has 16Gb ram, 2 sockets, 2 cores, and plenty of disk space).   Other than the apache error I've not found anything useful in the logs.

To address we reboot the server. which isn't ideal.

Many Thanks, 
Tania

[Rob Crittenden]

Tania Hagan via FreeIPA-users wrote:
> Hi,
>
> I would say it happens around every 5 days, but we have 7 replicas and it seems to be a different replica each time. We have been seeing high CPU and increased the spec of some of our aws instances which has seemed to settle down the CPU a bit (slapd is usually hogging the CPU).  (our average instance has 16Gb ram, 2 sockets, 2 cores, and plenty of disk space).   Other than the apache error I've not found anything useful in the logs.
>
> To address we reboot the server. which isn't ideal.

Can you correlate that to log rotation?

You might try https://access.redhat.com/solutions/7093830

rob

[Tania Hagan]

Hi,

Looks like we do have a reload of httpd on log rotate, so I'll turn that off and see how that goes.

Many Thanks, 
Tania

[Alex Crow]

Hi Tania,

Didn't expect to see you here! Did you ever fix it? Happened to us last night when a colleague added a Nessus scan to our IPA cluster.

Cheers

Alex

[Tania Hagan]

Hi,

Yes sorry, bad me, I should have posted the fix here.

Basically check your /etc/logrotate.d/httpd it will reload the logs, so you can amend the logrotate settings or amend your timeout on your monitoring.

Also doublecheck pkidebuglog see https://access.redhat.com/solutions/4049411 as I found instances filling up with space, but change the line /var/log/pki/*/*/debug { to /var/log/pki/*/*/debug* { otherwise it will still fill up the disk

Cheers, 
Tania