Fix bad host ============ First fix the issue that caused the issuing of too many certificates. Make sure it successfully issued and saved the cert on the client and that it's in status "MONITORING", "stuck: no". Or if you don't need the certificates that are managed by certmonger on that host, you can stop and disable the certmonger service. You can leave the certificate in stuck state, even if it expires, it can be renewed later when the certmonger bug is fixed. All of the below commands must be run as root, so 'sudo -i' first, then 'kinit', your user should have rights to run ipa cert-revoke. Create a directory for this host/process and cd into it. mkdir cert_delete_badhost && cd cert_delete_badhost Prepare a file containing the directory manager password named dirmgr_pass in CWD. Make sure it's owned by root and chmod 600. Delete it after you're done. If you don't want to do that, you can have it always prompt for the password, if you replace '-y dirmgr_pass' with '-W'. Delete unused certificates ========================== Find the serial number of the cert currently present on the client. 'sudo getcert list', look at "certificate:" in my case it was in "/etc/ssl/private/hostname-ipa-cert.crt" openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text In my case it was 268369940. If it's in /etc/ipa/nssdb, use: certutil -L -d /etc/ipa/nssdb/ -n 'Local IPA host' Save commonly used values in variables. goodcn=268369940 subjectname="badhost.example.com" Make a list of certificate serials to revoke. Since ipa cert-find doesn't show all certificates present (weird, but I confirmed it misses some present in LDAP), build list directly from LDAP. Filter for certs that don't match the current cert serial and are in status VALID. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(!(cn=$goodcn))(certStatus=VALID))" dn | grep -o '[[:digit:]]*' > cert_to_revoke You can view the above list of certs with this ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(!(cn=$goodcn))(certStatus=VALID))" Revoke the certs while read -r; do ipa cert-revoke "$REPLY"; done < cert_to_revoke You can view all the revoked cert cn's with this command before deleting them. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(certStatus=REVOKED))" dn certStatus|less Make a list of all cert cn's not matching the used cert, save output into a file, ready to be read by ldapdelete later. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(!(cn=$goodcn)))" | grep -o 'cn=.*' > cert_to_delete Make a list of all the requestId for all the certs to be deleted. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(!(cn=$goodcn)))" metaInfo|grep -oP 'requestId:\K.*' > cert_request_to_delete_from_metaInfo In my case there were a couple more requests than issued certs, I'm not sure why. I made a list of all requests for this host excluding the requestId of the correct cert. First find the correct/used cert requestId. In my case it was 9990026. Save it in a variable. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" "(&(subjectName~=$subjectname)(cn=$goodcn))" metaInfo|grep -oP 'requestId:\K.*' requestid=9990026 We must enable filtering on unindexed attributes or the next step won't work. cat > config_warn.ldif <<-'EOF' dn: cn=config changetype: modify replace: nsslapd-verify-filter-schema nsslapd-verify-filter-schema: warn-invalid EOF ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f config_warn.ldif Use this ldif later to return it to how it was before. cat > config_process-safe.ldif <<-'EOF' dn: cn=config changetype: modify replace: nsslapd-verify-filter-schema nsslapd-verify-filter-schema: process-safe EOF Then get a list of all requests for that host, excluding that one requestId. ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=ca,ou=requests,o=ipaca" "(&(extdata-req--005fsubject--005fname--002ecn=$subjectname)(!(cn=$requestid)))" dn|grep -o 'cn=.*' > cert_request_to_delete Count the number of certs/requests from the previous operations. The first two must match, the 3rd shows how many extra requests there are. wc -l cert_* 3982 cert_to_delete 3982 cert_request_to_delete_from_metaInfo 3990 cert_request_to_delete 3982 cert_to_revoke So there are 8 extra requests without corresponding certs. I chose to ignore them for now. They're probably fine to delete in the future. If all your numbers are equal, everything is fine and you can proceed. If not, investigate. Before deleting the requests, make a file in a format ldapdelete expects. while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY"; done < cert_request_to_delete_from_metaInfo > cert_request_to_delete_from_metaInfo_ldapdelete Now the actual deletion steps. Delete the certs. ldapdelete -x -D "cn=directory manager" -y dirmgr_pass -f cert_to_delete Delete the requests. ldapdelete -x -D "cn=directory manager" -y dirmgr_pass -f cert_request_to_delete_from_metaInfo_ldapdelete (you can add '-nv' to test ldapdelete) Delete entries from userCertificates ==================================== Save your domain suffix in a variable: domainsuffix="dc=example,dc=com" First save a list of all current userCertificate entries to a file ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -o ldif-wrap=no -b "fqdn=$subjectname,cn=computers,cn=accounts,$domainsuffix" userCertificate > userCertificate.ldif Copy the certificate you want to keep from where it's stored on the host to where you are working. Put it in $subjectname.crt in PEM format. If it's stored in /etc/ipa/nssdb, use this command: certutil -L -d /etc/ipa/nssdb -n 'Local IPA host' -a > $subjectname.crt Create a new ldif without that certificate. sed -e '/^dn: .*$/d' -e "s#userCertificate:: $(cat $subjectname.crt|grep -v -e '-----'|tr -d '\n')##" userCertificate.ldif > userCertificate_to_delete.ldif Check that the operation deleted exactly one certificate from the file. # wc -l userCertificate* 3980 userCertificate.ldif 3979 userCertificate_to_delete.ldif The number of userCertificate entries should be equal to or smaller than the number of deleted certificates in the previous step in cert_to_delete. In my case it was smaller by 2, meaning those certificates were issued but not added to userCertificates. If the number of userCertificates is larger, some may have already been deleted from the CA but not from userCertificates in the past, or some were added manually by the user. Investigate if any certificates need to be saved and also delete them from the to_delete ldif. You can delete other certificates that you also want to keep from this ldif. If you know the serial numbers of them, you can construct a shell one-liner script with a loop that iterates over each certificate in each line of the file, uses openssl x509 to retrieve its serial from base64-encoded certificate and deletes the certificate from the file if it matches. I didn't need that so I don't have the script. Format the ldif so it can be consumed by ldapmodify. Note that you have to remove the newlines that the mailing list adds so that everything is on one line, especially printf's format strings! { printf "dn: fqdn=$subjectname,cn=computers,cn=accounts,$domainsuffix\nchangetype: modify\ndelete: userCertificate\n"; while read -r; do if [ -z "$REPLY" ]; then continue; fi; printf '%s\n' "$REPLY"; done < userCertificate_to_delete.ldif; } > userCertificate_to_delete_ldapmodify.ldif Test if the ldif is correct ldapmodify -nv -x -D "cn=directory manager" -y dirmgr_pass -f userCertificate_to_delete_ldapmodify.ldif echo $? should say 0, if not, there's an error in the ldif you need to correct. Then remove the -nv and run the ldapmodify again to delete the entries for real. Trim changelog ============== NOTE: I'm not sure the process below works as intended. The nsds5 attributes might need to be set under a different base dn or in addition to cn=o\3ipaca. After this, I decided to trim the changelog and tombstones from ldap. Roughly followed the info from https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html Do the steps on all servers. cat > changelog_short.ldif <<-'EOF' dn: cn=changelog5,cn=config changetype: modify replace: nsslapd-changelogmaxage nsslapd-changelogmaxage: 300 - replace: nsslapd-changelogcompactdb-interval nsslapd-changelogcompactdb-interval: 300 - replace: nsslapd-changelogtrim-interval nsslapd-changelogtrim-interval: 30 - dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5ReplicaPurgeDelay nsds5ReplicaPurgeDelay: 300 - replace: nsds5ReplicaTombstonePurgeInterval nsds5ReplicaTombstonePurgeInterval: 300 - 'EOF' cat > changelog_normal.ldif <<-'EOF' dn: cn=changelog5,cn=config changetype: modify replace: nsslapd-changelogmaxage nsslapd-changelogmaxage: 30d - delete: nsslapd-changelogcompactdb-interval - delete: nsslapd-changelogtrim-interval - dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5ReplicaPurgeDelay nsds5ReplicaPurgeDelay: 604800 - replace: nsds5ReplicaTombstonePurgeInterval nsds5ReplicaTombstonePurgeInterval: 86400 - 'EOF' Check the current values for the above settings: ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "cn=changelog5,cn=config" nsslapd-changelogmaxage nsslapd-changelogcompactdb-interval nsslapd-changelogtrim-interval ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nsds5ReplicaPurgeDelay nsds5ReplicaTombstonePurgeInterval If any of the values differ from the above changelog_normal.ldif, edit it to match your current config. The two nsds5 attributes weren't present before, but the server refused to delete them, so I had to look up what the default values were. I'm not sure they're correct since I copied them from docs for Red Hat Directory Server. https://github.com/389ds/389-ds-base/issues/6348 ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f changelog_short.ldif systemctl restart dirsrv@EXAMPLE-COM.service Make a change in the webUI or ipa cli. There must be a write/change operation in LDAP. I added a TXT record in DNS. Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all servers, there should be no errors. ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f changelog_normal.ldif systemctl restart dirsrv@EXAMPLE-COM.service Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the purge 295M. This was only on the server I executed the deletions on, not on any replicas. So not a drastic difference. After the latest deletions (2024) this purging didn't free up any considerable space (less than 10% in /var/lib/dirsrv), I suspect because the "rogue" hosts were disabled a long time ago (months) so 389-ds already compacted its changelogs. Cleanup ======= Return nsslapd-verify-filter-schema back to default (process-safe). See above. Delete the dirmgr_pass file. Clean up log files from /var/log/pki/pki-tomcat (if required). du -ah /var/log/pki/pki-tomcat/|sort -hr|less