I recently reinstalled a couple of our freeipa replicas and they're both falling over with the same error. They run for a few minutes - as little as one, or up to an hour, and then fall over with thousands of errors like this:
> ERR - accept_and_configure - PR_Accept() failed, Netscape Portable Runtime error -5971 (Process open FD table is full.)
Eventually we get some db errors and dirsrv crashes (which presumably is related to the FD table being full). I'm unable to determine why the table is filling.
While they're up, the 2 failing replicas work fine (web, ldap) and seem to have all the appropriate info. The server they're being replicated from is fine, no notable errors, but it only has about 180 servers hitting it. There's another replica with no clients that works fine, no issues. These two servers used to work fine before I reinstalled the replicas (I have not changed the OS). It's a pretty small directory and we've got a couple thousand servers hitting the two failing replicas. They were recently updated from ipa-server 4.4 to 4.6, and used to work fine, the errors started after I rebuilt them. (BTW, I'm not the person who built the originals.)
The logs are not very giving me much in the way of clues. File handles are set in /etc/sysconfig/dirsrv to 32k, which should be more than enough, it was previously set to 16k. It hasn't gotten any busier than it used to be, so I don't feel like the file handle issue is simply due to being sized wrong. Debug logging generated a lot of output but nothing I could clearly identify as being an issue.
I'm at a loss. What should I be checking?
Versions:
CentOS Linux release 7.6.1810 (Core)
ipa-common-4.6.4-10.el7.centos.noarch
ipa-server-4.6.4-10.el7.centos.x86_64
ipa-client-4.6.4-10.el7.centos.x86_64
python2-ipalib-4.6.4-10.el7.centos.noarch
ipa-client-common-4.6.4-10.el7.centos.noarch
python2-ipaserver-4.6.4-10.el7.centos.noarch
ipa-server-common-4.6.4-10.el7.centos.noarch
sssd-ipa-1.16.2-13.el7.x86_64
389-ds-base-libs-1.3.8.4-18.el7_6.x86_64
389-ds-base-1.3.8.4-18.el7_6.x86_64
Thanks,
--Adam