ns-slapd uses 100% CPU, replications crash source servers, everything is terrible, send help

[Nerd Invert]

I hardly know where to start describing this. I'm having tons of trouble adding a simple fourth replica to my setup, something I've done effortlessly in the past.

The setup: Rocky Linux 8.10 all around, FreeIPA 4.9.13.
Three servers: idm0, idm1 and idm2. I wish to add idm3.
Very small database, maybe 50 users and 200 hosts. Has been going for 7-8 years so many hosts have a couple of expired certificates, but again, very small. I did an "ipa backup" recently and the resulting file was 80MB. I also checked one of my old logs, and then the replication took literally 10 seconds.

Problem 1: replication doesn't work. It takes forever (hours) then either crashes or crashes the source server with an OOM killer fault. It should be noted that some of the source servers had very little memory, 4GB and the same for swap, but I have since upgraded.

My install procedure for idm3 is:
Turn off firewall on idm3 completely, to take that out of the equation.
ipa-client-install --principal admin --no-ntp --no-nisdomain --force-join
ipa-replica-install --debug --principal admin --setup-ca --setup-dns --forwarder=8.8.8.8 --forwarder=8.8.4.4

It makes it to step 26 where it's "replicating" and sits there forever.

Problem 2: ns-slapd consumes 100% CPU for days on end, on some of the existing servers. The pattern of this behavior escapes me.

It should be noted that I might have created an "idm3" host and tried to promote it to replica years ago. I deleted the host, but perhaps some replication agreements are lingering?

Input on which log files I should follow and what I should try greatly appreciated.

[Rob Crittenden]

On 9/6/26 12:28 AM, Nerd Invert via FreeIPA-users wrote:
> I hardly know where to start describing this. I'm having tons of trouble adding a simple fourth replica to my setup, something I've done effortlessly in the past.
>
> The setup: Rocky Linux 8.10 all around, FreeIPA 4.9.13.
> Three servers: idm0, idm1 and idm2. I wish to add idm3.
> Very small database, maybe 50 users and 200 hosts. Has been going for 7-8 years so many hosts have a couple of expired certificates, but again, very small. I did an "ipa backup" recently and the resulting file was 80MB. I also checked one of my old logs, and then the replication took literally 10 seconds.
>
> Problem 1: replication doesn't work. It takes forever (hours) then either crashes or crashes the source server with an OOM killer fault. It should be noted that some of the source servers had very little memory, 4GB and the same for swap, but I have since upgraded.
>
> My install procedure for idm3 is:
> Turn off firewall on idm3 completely, to take that out of the equation.
> ipa-client-install --principal admin --no-ntp --no-nisdomain --force-join
> ipa-replica-install --debug --principal admin --setup-ca --setup-dns --forwarder=8.8.8.8 --forwarder=8.8.4.4
>
> It makes it to step 26 where it's "replicating" and sits there forever.
>
> Problem 2: ns-slapd consumes 100% CPU for days on end, on some of the existing servers. The pattern of this behavior escapes me.
>
> It should be noted that I might have created an "idm3" host and tried to promote it to replica years ago. I deleted the host, but perhaps some replication agreements are lingering?
>
> Input on which log files I should follow and what I should try greatly appreciated.

The 389-ds team will want to see a pstack of the process(es) that are 
pegged, so you'll need to reproduce this again.

I'd recommend installing the debuginfo 389-ds packages to increase the 
chance that they can tell what is going on. There is a 389 FAQ page 
which describes how to get the pstack.

And the package version of 389-ds will likely be wanted as well.

rob

[Florence Blanc-Renaud]

Hi,

*Replica installation issue*
Could you share the content of /var/log/ipareplica-install.log and
/var/log/pki/pki-ca-spawn.$DATE.log?
PKI recently fixed an issue where the replica installation was stuck during
the step installing the CA component. This occurred when the master had a
HTTP/LDAP server certificate signed by an external CA instead of the IPA CA.

The fix is available in https://access.redhat.com/errata/RHBA-2026:65135
with pki-core-10.15.1-2.module+el8.10.0+24257+6ba56fa4

*ns-slapd consumes CPU*
Which version of 389-ds is installed? Do you see any errors in
/var/log/dirsrv/slapd-xxx/errors?
You may want to reach out to [e-mail]
https://lists.fedoraproject.org/admin/lists/389-users.lists.fedoraproject.org
for
help.
flo

On Sun, Sep 6, 2026 at 7:05 PM Rob Crittenden via FreeIPA-users <
[e-mail]> wrote:
> On 9/6/26 12:28 AM, Nerd Invert via FreeIPA-users wrote:
> > I hardly know where to start describing this. I'm having tons of trouble
>
> adding a simple fourth replica to my setup, something I've done
> effortlessly in the past.
> > The setup: Rocky Linux 8.10 all around, FreeIPA 4.9.13.
> > Three servers: idm0, idm1 and idm2. I wish to add idm3.
> > Very small database, maybe 50 users and 200 hosts. Has been going for
>
> 7-8 years so many hosts have a couple of expired certificates, but again,
> very small. I did an "ipa backup" recently and the resulting file was 80MB.
> I also checked one of my old logs, and then the replication took literally
> 10 seconds.
> > Problem 1: replication doesn't work. It takes forever (hours) then
>
> either crashes or crashes the source server with an OOM killer fault. It
> should be noted that some of the source servers had very little memory, 4GB
> and the same for swap, but I have since upgraded.
> > My install procedure for idm3 is:
> > Turn off firewall on idm3 completely, to take that out of the equation.
> > ipa-client-install --principal admin --no-ntp --no-nisdomain --force-join
> > ipa-replica-install --debug --principal admin --setup-ca --setup-dns
>
> --forwarder=8.8.8.8 --forwarder=8.8.4.4
> > It makes it to step 26 where it's "replicating" and sits there forever.
> >
> > Problem 2: ns-slapd consumes 100% CPU for days on end, on some of the
>
> existing servers. The pattern of this behavior escapes me.
> > It should be noted that I might have created an "idm3" host and tried to
>
> promote it to replica years ago. I deleted the host, but perhaps some
> replication agreements are lingering?
> > Input on which log files I should follow and what I should try greatly
>
> appreciated.
>
> The 389-ds team will want to see a pstack of the process(es) that are
> pegged, so you'll need to reproduce this again.
>
> I'd recommend installing the debuginfo 389-ds packages to increase the
> chance that they can tell what is going on. There is a 389 FAQ page
> which describes how to get the pstack.
>
> And the package version of 389-ds will likely be wanted as well.
> rob
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://forge.fedoraproject.org/infra/tickets/issues/new

[Mark Reynolds]

On 9/9/26 4:03 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
> Hi,
>
> *Replica installation issue*
> Could you share the content of /var/log/ipareplica-install.log and 
> /var/log/pki/pki-ca-spawn.$DATE.log?
> PKI recently fixed an issue where the replica installation was stuck 
> during the step installing the CA component. This occurred when the 
> master had a HTTP/LDAP server certificate signed by an external CA 
> instead of the IPA CA.
>
> The fix is available in 
> https://access.redhat.com/errata/RHBA-2026:65135 with 
> pki-core-10.15.1-2.module+el8.10.0+24257+6ba56fa4
>
> *ns-slapd consumes CPU*
> Which version of 389-ds is installed? Do you see any errors in 
> /var/log/dirsrv/slapd-xxx/errors?
> You may want to reach out to [e-mail] 
> https://lists.fedoraproject.org/admin/lists/389-users.lists.fedoraproject.org for 
> help.

Also, like Rob stated we need more info to figure out why the CPU is 
high.  A few consecutive pstacks of the "ns-slapd" process would be 
helpful (make sure to install 389-ds-base-debuginfo first).  Also the 
"access" log could help identify what is going on.  Most high CPU issues 
are related to unindexed searches (notes=U or notes=A) in the access log.

Mark
> flo
>
> On Sun, Sep 6, 2026 at 7:05 PM Rob Crittenden via FreeIPA-users 
>  wrote:
> On 9/6/26 12:28 AM, Nerd Invert via FreeIPA-users wrote:
> > I hardly know where to start describing this. I'm having tons of
> trouble adding a simple fourth replica to my setup, something I've
> done effortlessly in the past.
> >
> > The setup: Rocky Linux 8.10 all around, FreeIPA 4.9.13.
> > Three servers: idm0, idm1 and idm2. I wish to add idm3.
> > Very small database, maybe 50 users and 200 hosts. Has been
> going for 7-8 years so many hosts have a couple of expired
> certificates, but again, very small. I did an "ipa backup"
> recently and the resulting file was 80MB. I also checked one of my
> old logs, and then the replication took literally 10 seconds.
> >
> > Problem 1: replication doesn't work. It takes forever (hours)
> then either crashes or crashes the source server with an OOM
> killer fault. It should be noted that some of the source servers
> had very little memory, 4GB and the same for swap, but I have
> since upgraded.
> >
> > My install procedure for idm3 is:
> > Turn off firewall on idm3 completely, to take that out of the
> equation.
> > ipa-client-install --principal admin --no-ntp --no-nisdomain
> --force-join
> > ipa-replica-install --debug --principal admin --setup-ca
> --setup-dns --forwarder=8.8.8.8 --forwarder=8.8.4.4
> >
> > It makes it to step 26 where it's "replicating" and sits there
> forever.
> >
> > Problem 2: ns-slapd consumes 100% CPU for days on end, on some
> of the existing servers. The pattern of this behavior escapes me.
> >
> > It should be noted that I might have created an "idm3" host and
> tried to promote it to replica years ago. I deleted the host, but
> perhaps some replication agreements are lingering?
> >
> > Input on which log files I should follow and what I should try
> greatly appreciated.
>
> The 389-ds team will want to see a pstack of the process(es) that are
> pegged, so you'll need to reproduce this again.
>
> I'd recommend installing the debuginfo 389-ds packages to increase
> the
> chance that they can tell what is going on. There is a 389 FAQ page
> which describes how to get the pstack.
>
> And the package version of 389-ds will likely be wanted as well.
>
> rob
> -- 
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://forge.fedoraproject.org/infra/tickets/issues/new

[Bo Lind]

Hi again, sorry for the delayed response.

With some help from Claude AI, I traced the high CPU load to a corrupted database, once this was fixed, the replication progressed further and the CPU load bug wasn't seen again. (The reason I say "further" is that I ran into other, unrelated issues.)

I did do a ptrace, ten seconds apart, and the difference was:

9c9
< #0  0x00007faf030160de in idl_id_is_in_idlist_ranges () at target:/usr/lib64/dirsrv/plugins/libback-ldbm.so
---
> #0  0x00007faf030160d0 in idl_id_is_in_idlist_ranges () at target:/usr/lib64/dirsrv/plugins/libback-ldbm.so

Also, running dbverify revealed:

[root@idm1 ~]# dsctl slapd-EXAMPLE-COM dbverify userRoot
[07/Sep/2026:05:25:59.263146525 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[07/Sep/2026:05:25:59.358908886 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[07/Sep/2026:05:25:59.409645646 -0400] - INFO - ldbm_instance_config_cachememsize_set - force a minimal value 512000
[07/Sep/2026:05:25:59.492751146 -0400] - ERR - libdb - BDB1066 Page 6: out-of-order key at entry 444
[07/Sep/2026:05:25:59.514091385 -0400] - ERR - libdb - /var/lib/dirsrv/slapd-EXAMPLE-COM/db/userRoot/parentid.db: BDB0090 DB_VERIFY_BAD: Database verification failed
[07/Sep/2026:05:25:59.539081256 -0400] - ERR - dbverify_ext - verify failed(-30970): /var/lib/dirsrv/slapd-EXAMPLE-COM/db/userRoot/parentid.db
dbverify failed

This was fixed with

dsctl slapd-EXAMPLE-COM db2index userRoot