Fix bad host
============

First fix the issue that caused the issuing of too many certificates.
Make sure it successfully issued and saved the cert on the client and
that it's in status "MONITORING", "stuck: no".

Create a directory for this host/process and cd into it.
Prepare a file containing the directory manager password named dirmgr_pass in CWD. Make sure it's owned by root and chmod 600. Delete it after you're done.
If you don't want to do that, you can have it always prompt for the password, if you replace '-y dirmgr_pass' with '-W'.

Delete unused certificates
==========================

Find the serial number of the cert currently present on the client.
'sudo getcert list', look at "certificate:" in my case it was in
"/etc/ssl/private/hostname-ipa-cert.crt"
openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
In my case it was 268369940.
If it's in /etc/ipa/nssdb, use:
certutil -L -d /etc/ipa/nssdb/ -n 'Local IPA host'

Make a list of certificate serials to revoke.
Since ipa cert-find doesn't show all certificates present (weird, but I confirmed it misses some present in LDAP), build list directly from LDAP.
Filter for certs that don't match the current cert serial and are in status VALID.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940))(certStatus=VALID))' dn | grep -o 'cn=[[:digit:]]*' > cert_to_revoke_badhost

You can view the above list of certs with this
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940))(certStatus=VALID))'

Revoke the certs
while read -r; do ipa cert-revoke "$REPLY"; done < cert_to_revoke_badhost

You can view all the revoked cert cn's with this command before
deleting them.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less

Make a list of all cert cn's not matching the used cert, save output
into a file, ready to be read by ldapdelete later.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' > cert_to_delete_not_used_badhost

Make a list of all the requestId for all the certs to be deleted.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost

In my case there were a couple more requests than issued certs, I'm not
sure why. I made a list of all requests for this host excluding the
requestId of the correct cert. First find the correct/used cert
requestId. In my case it was 9990026.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP 'requestId:\K.*'

We must enable filtering on unindexed attributes or the next step won't work.
cat > config_warn.ldif <<-'EOF'
dn: cn=config
changetype: modify
replace: nsslapd-verify-filter-schema
nsslapd-verify-filter-schema: warn-invalid
EOF
ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f config_warn.ldif

Use this ldif later to return it to how it was before.
cat > config_process-safe.ldif <<-'EOF'
dn: cn=config
changetype: modify
replace: nsslapd-verify-filter-schema
nsslapd-verify-filter-schema: process-safe
EOF

Then get a list of all requests for that host, excluding that one
requestId.
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "ou=ca,ou=requests,o=ipaca" '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))' dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost

Count the number of certs/requests from the previous operations. The
first two must match, the 3rd shows how many extra requests there are.
wc -l cert_to_delete_not_used_badhost
cert_request_to_delete_not_used_from_metaInfo_badhost cert_request_to_delete_not_used_badhost
  3982 cert_to_delete_not_used_badhost
  3982 cert_request_to_delete_not_used_from_metaInfo_badhost
  3990 cert_request_to_delete_not_used_badhost
  3982 cert_to_revoke

So there are 8 extra requests without corresponding certs. I chose to
ignore them for now. They're probably fine to delete in the future.
If all your numbers are equal, everything is fine and you can proceed.
If not, investigate.

Before deleting the requests, make a file in a format ldapdelete
expects.
while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY"; done < cert_request_to_delete_not_used_from_metaInfo_badhost > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost

Now the actual deletion steps.
Delete the certs.
ldapdelete -x -D "cn=directory manager" -y dirmgr_pass -f cert_to_delete_not_used_badhost

Delete the requests.
ldapdelete -x -D "cn=directory manager" -y dirmgr_pass -f cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost

(you can add '-nv' to test ldapdelete)

Delete entries from userCertificates
====================================

First save a list of all current userCertificate entries to a file

ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -o ldif-wrap=no -b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com" userCertificate > userCertificate_badhost.ldif

Copy the certificate you want to keep from where it's stored on the
host to where you are working. Put it in badhost.crt in PEM format.
If it's stored in /etc/ipa/nssdb, use this command:
certutil -L -d /etc/ipa/nssdb -n 'Local IPA host' -a > badhost.crt

Create a new ldif without that certificate.
sed -e '/^dn: .*$/d' -e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-----'|tr -d '\n')##" userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif

Check that the operation deleted exactly one certificate from the file.
# wc -l userCertificate_badhost*
    5611 userCertificate_badhost.ldif
    5610 userCertificate_badhost_to_delete.ldif

You can delete other certificates that you also want to keep from this
ldif. If you know the serial numbers of them, you can construct a shell
one-liner script with a loop that iterates over each certificate in each
line of the file, uses openssl x509 to retrieve its serial from
base64-encoded certificate and deletes the certificate from the file if
it matches.
I didn't need that so I don't have the script.

Format the ldif so it can be consumed by ldapmodify. Note that you have
to remove the newlines that the mailing list adds so that everything is
on one line, especially printf's format strings!

{ printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\nchangetype: modify\ndelete: userCertificate\n'; while read -r; do if [ -z "$REPLY" ]; then continue; fi; printf '%s\n' "$REPLY"; done < userCertificate_badhost_to_delete.ldif; } > userCertificate_badhost_to_delete_ldapmodify.ldif

Test if the ldif is correct
ldapmodify -nv -x -D "cn=directory manager" -y dirmgr_pass -f userCertificate_badhost_to_delete_ldapmodify.ldif

echo $? should say 0, if not, there's an error in the ldif you need to
correct.

Then remove the -nv and run the ldapmodify again to delete the entries
for real.

Trim changelog
==============

After this, I decided to trim the changelog and tombstones from ldap.
Roughly followed the info from
https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
Do the steps on all servers.


cat > changelog_short.ldif <<-'EOF'
dn: cn=changelog5,cn=config
changetype: modify
replace: nsslapd-changelogmaxage
nsslapd-changelogmaxage: 300
-
replace: nsslapd-changelogcompactdb-interval
nsslapd-changelogcompactdb-interval: 300
-
replace: nsslapd-changelogtrim-interval
nsslapd-changelogtrim-interval: 30
-

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 300
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 300
-
'EOF'

cat > changelog_normal.ldif <<-'EOF'
dn: cn=changelog5,cn=config
changetype: modify
replace: nsslapd-changelogmaxage
nsslapd-changelogmaxage: 30d
-
delete: nsslapd-changelogcompactdb-interval
-
delete: nsslapd-changelogtrim-interval
-

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 604800
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 86400
-
'EOF'

Check the current values for the above settings:
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "cn=changelog5,cn=config" nsslapd-changelogmaxage nsslapd-changelogcompactdb-interval nsslapd-changelogtrim-interval
ldapsearch -LLL -x -D "cn=directory manager" -y dirmgr_pass -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nsds5ReplicaPurgeDelay nsds5ReplicaTombstonePurgeInterval

If any of the values differ from the above changelog_normal.ldif, edit it to match your current config. The two nsds5 attributes weren't present before, but the server refused to delete them, so I had to look up what the default values were. I'm not sure they're correct since I copied them from docs for Red Hat Directory Server.

ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f changelog_short.ldif
systemctl restart dirsrv@EXAMPLE-COM.service

Make a change in the webUI or ipa cli. There must be a write/change operation in LDAP. I added a TXT record in DNS.
Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all servers, there should be no errors.

ldapmodify -x -D "cn=directory manager" -y dirmgr_pass -f changelog_normal.ldif
systemctl restart dirsrv@EXAMPLE-COM.service

Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
purge 295M. This was only on the server I executed the deletions on,
not on any replicas. So not a drastic difference.

After the latest deletions (2024) this purging didn't free up any considerable space (less than 10% in /var/lib/dirsrv), I suspect because the "rogue" hosts were disabled a long time ago (months) so 389-ds already compacted its changelogs.

Cleanup
=======
Return nsslapd-verify-filter-schema back to default (process-safe). See above.
Delete the dirmgr_pass file.