Removal & clean up certificates from o=ipaca

[David Goudet]

Hello all,

I have to clean up lot of useless certificate in dirsrv database.
Because of resubmit loop on Certmonger client, i have 99,9% of certificate in dirsrv database that are useless and not obsolete (expiration in 2020) (it represent ~85 000 certificates).

These useless certificates produce some issues on FreeIPA:
 - decrease FreeIPA performances on CLI and GUI
 - increase the LDAP size
 - increase size and time of FreeIPA backup
...

Is it possible to purge these certificates in dirsrv database and how?

I found two branches in LDAP directory about these certificates:
dn: cn=xxx,ou=ca,ou=requests,o=ipaca
dn: cn=yyy,ou=certificateRepository,ou=ca,o=ipaca

I can remove all requests and certificates entry from dirsrv database but how it is supported by PKI manager Dogtag (CRL, certificate generation, OCSP)?

(This topic has already been discuss on https://lists.fedorahosted.org/archives/list/[e-mail]...)

Thank you for you help

[Fraser Tweedale]

On Wed, Nov 07, 2018 at 04:29:36PM +0100, David Goudet via FreeIPA-users wrote:
> Hello all,

Hi David,
> I have to clean up lot of useless certificate in dirsrv database.
> Because of resubmit loop on Certmonger client, i have 99,9% of certificate in dirsrv database that are useless and not obsolete (expiration in 2020) (it represent ~85 000 certificates).

Did you already resolve the Certmonger resubmit loop?
> These useless certificates produce some issues on FreeIPA:
> decrease FreeIPA performances on CLI and GUI
> increase the LDAP size
> increase size and time of FreeIPA backup
>
> ...
>
> Is it possible to purge these certificates in dirsrv database and how?

Yes.  You can remove them manually.
> I found two branches in LDAP directory about these certificates:
>
> dn: cn=xxx,ou=ca,ou=requests,o=ipaca
> dn: cn=yyy,ou=certificateRepository,ou=ca,o=ipaca

The certificateRepository contains the issued certificates, the
ou=ca,ou=requests contains data about the certificate requests.
Each certificateRepository entry contains a reference to the request
that produced it.

You'll have to manually work out which certs you don't want, delete
its certificateRepository entry (cn is the serial number), and
delete the corresponding request entry.
> I can remove all requests and certificates entry from dirsrv
> database but how it is supported by PKI manager Dogtag (CRL,
> certificate generation, OCSP)?

CRLs and OCSP responses are generated using the data from the
certificateRepository.  Forgetting about non-expired certificates is
not valid under X.509, but since you have an operational issue, just
choose carefully which ones you keep and which ones you delete.

Don't delete the entry for any certificates in active use, OR any
non-expired but revoked certificate where you want it to appear in
CRLs or want valid OCSP responses for that certificate.

Also, whatever certificate has the highest serial number, do not
delete it.  When using sequential serial number (which is how Dogtag
gets configured by FreeIPA) upon startup Dogtag looks for the
highest serial number to work out what is the next serial number to
use.  So keep the cert with the highest serial number otherwise
serial numbers will be re-used.

Cheers,
Fraser

[David Goudet]

Hello,

I did request and certificate suppression test and restart IPA stack. It works!

ldapdelete -x -D "cn=directory manager" -W  "cn=87289,ou=ca,ou=requests,o=ipaca"
ldapdelete -x -D "cn=directory manager" -W  "cn=87273,ou=certificateRepository,ou=ca,o=ipaca"

I am going to generate the list of request and certificate entries that are useless.

Hereafter a little procedure:

(cn (in ou=certificateRepository,ou=ca,o=ipaca) is equal to serialId decimal in x509 certificate)

Backup IPA and save the ipaca tree (sudo ldapsearch -x -h localhost -D "cn=directory manager" -W -b o=ipaca > /var/lib/ipa/backup/all)

Certificate tree purge (ou=certificateRepository,ou=ca,o=ipaca):
1. Identify entry that have to be excluded (non garbage certificate: used & expired certificates)
 - Get serial ID of certificate used: sudo openssl x509 -in xxx.crt -text -noout | grep "Seria|Not|Sub"
2. Get garbage certificate list (used & expired certificates are excluded): ldapsearch -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~=<host>)(!(cn=<used certificate(s) serial>))(certStatus=VALID))' dn | grep "cn=" | sed -e "s/dn: //" -e "/#/d" > cert_<host>

Request tree purge (ou=ca,ou=requests,o=ipaca):
1.  Identify entry that have to be excluded (non garbage certificate: used & expired certificates)
 - Get requestID of certificate used: sudo ldapsearch -x -D "cn=directory manager" -W -b "cn=<used certificate(s) serial>,ou=certificateRepository,ou=ca,o=ipaca" '(subjectName~=<host>)'  "metaInfo"
 - Get requestID of certificate expired: sudo ldapsearch -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~=<host>)(!(certStatus=VALID)))' "metaInfo"
2. Get garbage certificate request list (used & expired certificates are excluded):  sudo ldapsearch -x -D "cn=directory manager" -W  -b "ou=ca,ou=requests,o=ipaca" '(&(extdata-req--005fsubject--005fname--002ecn=<host>)(&(!(cn=<used certificate(s) request Id>))(!(cn=<expired certificate(s) request Id>))))' dn | grep "cn=" | sed -e "s/dn: //" -e "/#/d" > req_<host>

Check that number of request and certificate entry to purge are equal:
grep -c cn= cert_<host>
grep -c cn= req_<host>

(I hope this will help)

Thank you for your response,

----- Original Message -----
From: "Fraser Tweedale" 
To: "freeipa-users" 
Cc: "David Goudet" 
Sent: Thursday, November 8, 2018 2:28:03 AM
Subject: Re: [Freeipa-users] Removal & clean up certificates from o=ipaca

On Wed, Nov 07, 2018 at 04:29:36PM +0100, David Goudet via FreeIPA-users wrote:
> Hello all,

Hi David,
> I have to clean up lot of useless certificate in dirsrv database.
> Because of resubmit loop on Certmonger client, i have 99,9% of certificate in dirsrv database that are useless and not obsolete (expiration in 2020) (it represent ~85 000 certificates).

Did you already resolve the Certmonger resubmit loop?
> These useless certificates produce some issues on FreeIPA:
> decrease FreeIPA performances on CLI and GUI
> increase the LDAP size
> increase size and time of FreeIPA backup
>
> ...
>
> Is it possible to purge these certificates in dirsrv database and how?

Yes.  You can remove them manually.
> I found two branches in LDAP directory about these certificates:
>
> dn: cn=xxx,ou=ca,ou=requests,o=ipaca
> dn: cn=yyy,ou=certificateRepository,ou=ca,o=ipaca

The certificateRepository contains the issued certificates, the
ou=ca,ou=requests contains data about the certificate requests.
Each certificateRepository entry contains a reference to the request
that produced it.

You'll have to manually work out which certs you don't want, delete
its certificateRepository entry (cn is the serial number), and
delete the corresponding request entry.
> I can remove all requests and certificates entry from dirsrv
> database but how it is supported by PKI manager Dogtag (CRL,
> certificate generation, OCSP)?

CRLs and OCSP responses are generated using the data from the
certificateRepository.  Forgetting about non-expired certificates is
not valid under X.509, but since you have an operational issue, just
choose carefully which ones you keep and which ones you delete.

Don't delete the entry for any certificates in active use, OR any
non-expired but revoked certificate where you want it to appear in
CRLs or want valid OCSP responses for that certificate.

Also, whatever certificate has the highest serial number, do not
delete it.  When using sequential serial number (which is how Dogtag
gets configured by FreeIPA) upon startup Dogtag looks for the
highest serial number to work out what is the next serial number to
use.  So keep the cert with the highest serial number otherwise
serial numbers will be re-used.

Cheers,
Fraser

[Jernej Jakob]

Hi David. I had the same issue here and found your writeup to be very
helpful.

I used more or less the same ldap actions to delete the certificates
and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
just the one "used"/"correct" certificate for the host, but the main
issue is not fixed. The webUI still displays all the old certificates
that I have deleted from LDAP. Opening the "Hosts" tab or a host page
takes very long, around 1-2 minutes.

So I want to know what else needs to be done to make the webUI "forget"
about the wrongly issued certificates?

Where does the webUI get its list of certificates?
I did some searching through the code and could only find the JS
code that invokes a RPC call. But I could not find the code that
handles that call.

IIRC my issue that caused certmonger to request a certificate over and
over was caused by a bug after upgrading a client from Ubuntu 14.04 to
16.04. The path to ca_external_helper changed but it was not changed in
/var/lib/certmonger/cas/* which caused certmonger to fail running
ca_external_helper. To fix it I did:

sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g' /var/lib/certmonger/cas/*

Below is the exact procedure I used to delete the certificates from
LDAP.

First fix the issue that caused the issuing of too many certificates.
Make sure it successfully issued and saved the cert on the client and
that it's in status "MONITORING", "stuck: no".
Find the serial number of the cert currently present on the client.
'sudo getcert list', look at "certificate:" in my case it was in
"/etc/ssl/private/hostname-ipa-cert.crt"
openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
In my case it was 268369940.

Used the following shell script to revoke all the certificates with
serial not matching. I did this before I knew howo to get the cert
serials from ldap so it uses ipa cert-find. It's a slow process.
for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s" ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done

You can view all the revoked cert cn's with this command before
deleting them.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less

Make a list of all cert cn's not matching the used cert, save output
into a file, ready to be read by ldapdelete later.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' > cert_to_delete_not_used_badhost

Make a list of all the requestId for all the certs to be deleted.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost

In my case there were a couple more requests than issued certs, I'm not
sure why. I made a list of all requests for this host excluding the
requestId of the correct cert. First find the correct/used cert
requestId. In my case it was 9990026.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=certificateRepository,ou=ca,o=ipaca" '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP 'requestId:\K.*'

Then get a list of all requests for that host, excluding that one
requestId.
ldapsearch -LLL -x -D "cn=directory manager" -W -b "ou=ca,ou=requests,o=ipaca" '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))' dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost

Count the number of certs/requests from the previous operations. The
first two must match, the 3rd shows how many extra requests there are.
wc -l cert_to_delete_not_used_badhost
cert_request_to_delete_not_used_from_metaInfo_badhost cert_request_to_delete_not_used_badhost
  3982 cert_to_delete_not_used_badhost
  3982 cert_request_to_delete_not_used_from_metaInfo_badhost
  3990 cert_request_to_delete_not_used_badhost

So there are 8 extra requests without corresponding certs. I chose to
ignore them for now. They're probably fine to delete in the future.

Before deleting the requests, make a file in a format ldapdelete
expects.
while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY"; done < cert_request_to_delete_not_used_from_metaInfo_badhost > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost

Now the actual deletion steps.
Delete the certs.
ldapdelete -x -D "cn=directory manager" -W -f cert_to_delete_not_used_badhost

Delete the requests.
ldapdelete -x -D "cn=directory manager" -W -f cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost

(you can add '-nv' to test ldapdelete)

After this, I decided to trim the changelog and tombstones from ldap.
Roughly followed the info from
https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
Do the steps on all servers.

cat > changelog_short.ldif <<-'EOF'
dn: cn=changelog5,cn=config
changetype: modify
replace: nsslapd-changelogmaxage
nsslapd-changelogmaxage: 300
-
replace: nsslapd-changelogcompactdb-interval
nsslapd-changelogcompactdb-interval: 300
-
replace: nsslapd-changelogtrim-interval
nsslapd-changelogtrim-interval: 30
-

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 300
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 300
-
'EOF'

cat > changelog_normal.ldif <<-'EOF'
dn: cn=changelog5,cn=config
changetype: modify
replace: nsslapd-changelogmaxage
nsslapd-changelogmaxage: 7d
-
delete: nsslapd-changelogcompactdb-interval
-
delete: nsslapd-changelogtrim-interval
-

dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaPurgeDelay
nsds5ReplicaPurgeDelay: 604800
-
replace: nsds5ReplicaTombstonePurgeInterval
nsds5ReplicaTombstonePurgeInterval: 86400
-
'EOF'

ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
systemctl restart [e-mail]

Make a change in the webUI or ipa cli. There must be a write/change operation in LDAP.
Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all servers, there should be no errors.

ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
systemctl restart [e-mail]

Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
purge 295M. This was only on the server I executed the deletions on,
not on any replicas. So not a drastic difference.

[Florence Blanc-Renaud]

Hi,

On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
[e-mail]> wrote:
> Hi David. I had the same issue here and found your writeup to be very
> helpful.
>
> I used more or less the same ldap actions to delete the certificates
> and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> just the one "used"/"correct" certificate for the host, but the main
> issue is not fixed. The webUI still displays all the old certificates
> that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> takes very long, around 1-2 minutes.
>
> So I want to know what else needs to be done to make the webUI "forget"
> about the wrongly issued certificates?
>
> Where does the webUI get its list of certificates?
> I did some searching through the code and could only find the JS
> code that invokes a RPC call. But I could not find the code that
> handles that call.
>
> The webui is making a call equivalent to "ipa cert-find" which is handled

by the following code:
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L14...

The call looks for certificates in multiple locations:
- in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
- in the suffix "dc=example,dc=com", in the users/hosts/services entries

You cleaned the certificates from the cert repository but there may be many
entries (users/hosts/services) containing a userCertificate attribute. To
avoid seeing those certs you would have to delete the corresponding
userCertificate values.

HTH,
flo
> IIRC my issue that caused certmonger to request a certificate over and
> over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> 16.04. The path to ca_external_helper changed but it was not changed in
> /var/lib/certmonger/cas/* which caused certmonger to fail running
> ca_external_helper. To fix it I did:
>
> sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> /var/lib/certmonger/cas/*
>
> Below is the exact procedure I used to delete the certificates from
> LDAP.
>
> First fix the issue that caused the issuing of too many certificates.
> Make sure it successfully issued and saved the cert on the client and
> that it's in status "MONITORING", "stuck: no".
> Find the serial number of the cert currently present on the client.
> 'sudo getcert list', look at "certificate:" in my case it was in
> "/etc/ssl/private/hostname-ipa-cert.crt"
> openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> In my case it was 268369940.
>
> Used the following shell script to revoke all the certificates with
> serial not matching. I did this before I knew howo to get the cert
> serials from ldap so it uses ipa cert-find. It's a slow process.
> for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s"
> ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
>
> You can view all the revoked cert cn's with this command before
> deleting them.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
>
> Make a list of all cert cn's not matching the used cert, save output
> into a file, ready to be read by ldapdelete later.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
> cert_to_delete_not_used_badhost
>
> Make a list of all the requestId for all the certs to be deleted.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
> 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
>
> In my case there were a couple more requests than issued certs, I'm not
> sure why. I made a list of all requests for this host excluding the
> requestId of the correct cert. First find the correct/used cert
> requestId. In my case it was 9990026.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=certificateRepository,ou=ca,o=ipaca"
> '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> 'requestId:\K.*'
>
> Then get a list of all requests for that host, excluding that one
> requestId.
> ldapsearch -LLL -x -D "cn=directory manager" -W -b
> "ou=ca,ou=requests,o=ipaca"
> '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
>
> Count the number of certs/requests from the previous operations. The
> first two must match, the 3rd shows how many extra requests there are.
> wc -l cert_to_delete_not_used_badhost
> cert_request_to_delete_not_used_from_metaInfo_badhost
> cert_request_to_delete_not_used_badhost
>   3982 cert_to_delete_not_used_badhost
>   3982 cert_request_to_delete_not_used_from_metaInfo_badhost
>   3990 cert_request_to_delete_not_used_badhost
>
> So there are 8 extra requests without corresponding certs. I chose to
> ignore them for now. They're probably fine to delete in the future.
>
> Before deleting the requests, make a file in a format ldapdelete
> expects.
> while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY";
> done < cert_request_to_delete_not_used_from_metaInfo_badhost >
> cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
>
> Now the actual deletion steps.
> Delete the certs.
> ldapdelete -x -D "cn=directory manager" -W -f
> cert_to_delete_not_used_badhost
>
> Delete the requests.
> ldapdelete -x -D "cn=directory manager" -W -f
> cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
>
> (you can add '-nv' to test ldapdelete)
>
> After this, I decided to trim the changelog and tombstones from ldap.
> Roughly followed the info from
> https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
> Do the steps on all servers.
>
> cat > changelog_short.ldif <<-'EOF'
> dn: cn=changelog5,cn=config
> changetype: modify
> replace: nsslapd-changelogmaxage
> nsslapd-changelogmaxage: 300
>
> replace: nsslapd-changelogcompactdb-interval
> nsslapd-changelogcompactdb-interval: 300
>
> replace: nsslapd-changelogtrim-interval
> nsslapd-changelogtrim-interval: 30
>
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5ReplicaPurgeDelay
> nsds5ReplicaPurgeDelay: 300
>
> replace: nsds5ReplicaTombstonePurgeInterval
> nsds5ReplicaTombstonePurgeInterval: 300
>
> 'EOF'
>
> cat > changelog_normal.ldif <<-'EOF'
> dn: cn=changelog5,cn=config
> changetype: modify
> replace: nsslapd-changelogmaxage
> nsslapd-changelogmaxage: 7d
>
> delete: nsslapd-changelogcompactdb-interval
>
> delete: nsslapd-changelogtrim-interval
>
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5ReplicaPurgeDelay
> nsds5ReplicaPurgeDelay: 604800
>
> replace: nsds5ReplicaTombstonePurgeInterval
> nsds5ReplicaTombstonePurgeInterval: 86400
>
> 'EOF'
>
> ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
> systemctl restart [e-mail]
>
> Make a change in the webUI or ipa cli. There must be a write/change
> operation in LDAP.
> Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all
> servers, there should be no errors.
>
> ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
> systemctl restart [e-mail]
>
> Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
> purge 295M. This was only on the server I executed the deletions on,
> not on any replicas. So not a drastic difference.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Jernej Jakob]

On Tue, 7 Feb 2023 10:35:35 +0100
Florence Blanc-Renaud  wrote:
> Hi,
>
> On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
> [e-mail]> wrote:
> > Hi David. I had the same issue here and found your writeup to be very
> > helpful.
> >
> > I used more or less the same ldap actions to delete the certificates
> > and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> > just the one "used"/"correct" certificate for the host, but the main
> > issue is not fixed. The webUI still displays all the old certificates
> > that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> > takes very long, around 1-2 minutes.
> >
> > So I want to know what else needs to be done to make the webUI "forget"
> > about the wrongly issued certificates?
> >
> > Where does the webUI get its list of certificates?
> > I did some searching through the code and could only find the JS
> > code that invokes a RPC call. But I could not find the code that
> > handles that call.
> >
> > The webui is making a call equivalent to "ipa cert-find" which is handled
>
> by the following code:
> https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L14...
>
> The call looks for certificates in multiple locations:
> in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
> in the suffix "dc=example,dc=com", in the users/hosts/services entries
>
> You cleaned the certificates from the cert repository but there may be many
> entries (users/hosts/services) containing a userCertificate attribute. To
> avoid seeing those certs you would have to delete the corresponding
> userCertificate values.
>
> HTH,
> flo

Thanks flo, this was exactly what I needed, I managed to delete them
from userCertificate under
"fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
and now the webui is fast again!
In retrospect I think making the webui work fast even with such a large
number of certificates would be better, since this cleanup is not easy
and it's possible to delete too much. That would probably mean adding a
pager and filtering to the certificate list which is displayed on the
host page. Right now they are all displayed at once on one page and
there is no way to filter out e.g. all revoked certificates. Also find
and fix why loading the hosts list takes so long even though it doesn't
display any certificates on that page.

Here are the commands to delete the certificates from userCertificate
of the host. Add them to those in the previous message, after o=ipaca
cleaning but before changelog purging.

First save a list of all current userCertificate entries to a file
ldapsearch -LLL -x -D "cn=directory manager" -W -o ldif-wrap=no
-b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
userCertificate > userCertificate_badhost.ldif

Copy the certificate you want to keep from where it's stored on the
host to where you are working. Put it in badhost.crt in PEM format.
Create a new ldif without that certificate.
sed -e '/^dn: .*$/d'
-e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-----'|tr -d '\n')##"
userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif

You can delete other certificates that you also want to keep from this
ldif. If you know the serial numbers of them, you can construct a shell
one-liner script with a loop that iterates over each certificate in each
line of the file, uses openssl x509 to retrieve its serial from
base64-encoded certificate and deletes the certificate from the file if
it matches.
I didn't need that so I don't have the script.

Format the ldif so it can be consumed by ldapmodify. Note that you have
to remove the newlines that the mailing list adds so that everything is
on one line, especially printf's format strings!
{ printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\n
changetype: modify\ndelete: userCertificate\n';
while read -r; do if [ -z "$REPLY" ]; then continue; fi;
printf '%s\n' "$REPLY"; done < userCertificate_badhost_to_delete.ldif;
} > userCertificate_badhost_to_delete_ldapmodify.ldif

Test if the ldif is correct
ldapmodify -nv -x -D "cn=directory manager" -W
-f userCertificate_badhost_to_delete_ldapmodify.ldif

echo $? should say 0, if not, there's an error in the ldif you need to
correct.

Then remove the -nv and run the ldapmodify again to delete the entries
for real.
> > IIRC my issue that caused certmonger to request a certificate over and
> > over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> > 16.04. The path to ca_external_helper changed but it was not changed in
> > /var/lib/certmonger/cas/* which caused certmonger to fail running
> > ca_external_helper. To fix it I did:
> >
> > sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> > /var/lib/certmonger/cas/*
> >
> > Below is the exact procedure I used to delete the certificates from
> > LDAP.
> >
> > First fix the issue that caused the issuing of too many certificates.
> > Make sure it successfully issued and saved the cert on the client and
> > that it's in status "MONITORING", "stuck: no".
> > Find the serial number of the cert currently present on the client.
> > 'sudo getcert list', look at "certificate:" in my case it was in
> > "/etc/ssl/private/hostname-ipa-cert.crt"
> > openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> > In my case it was 268369940.
> >
> > Used the following shell script to revoke all the certificates with
> > serial not matching. I did this before I knew howo to get the cert
> > serials from ldap so it uses ipa cert-find. It's a slow process.
> > for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> > '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s"
> > ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
> >
> > You can view all the revoked cert cn's with this command before
> > deleting them.
> > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > "ou=certificateRepository,ou=ca,o=ipaca"
> > '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
> >
> > Make a list of all cert cn's not matching the used cert, save output
> > into a file, ready to be read by ldapdelete later.
> > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > "ou=certificateRepository,ou=ca,o=ipaca"
> > '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
> > cert_to_delete_not_used_badhost
> >
> > Make a list of all the requestId for all the certs to be deleted.
> > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > "ou=certificateRepository,ou=ca,o=ipaca"
> > '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
> > 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
> >
> > In my case there were a couple more requests than issued certs, I'm not
> > sure why. I made a list of all requests for this host excluding the
> > requestId of the correct cert. First find the correct/used cert
> > requestId. In my case it was 9990026.
> > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > "ou=certificateRepository,ou=ca,o=ipaca"
> > '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> > 'requestId:\K.*'
> >
> > Then get a list of all requests for that host, excluding that one
> > requestId.
> > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > "ou=ca,ou=requests,o=ipaca"
> > '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> > dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
> >
> > Count the number of certs/requests from the previous operations. The
> > first two must match, the 3rd shows how many extra requests there are.
> > wc -l cert_to_delete_not_used_badhost
> > cert_request_to_delete_not_used_from_metaInfo_badhost
> > cert_request_to_delete_not_used_badhost
> >   3982 cert_to_delete_not_used_badhost
> >   3982 cert_request_to_delete_not_used_from_metaInfo_badhost
> >   3990 cert_request_to_delete_not_used_badhost
> >
> > So there are 8 extra requests without corresponding certs. I chose to
> > ignore them for now. They're probably fine to delete in the future.
> >
> > Before deleting the requests, make a file in a format ldapdelete
> > expects.
> > while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY";
> > done < cert_request_to_delete_not_used_from_metaInfo_badhost >
> > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> >
> > Now the actual deletion steps.
> > Delete the certs.
> > ldapdelete -x -D "cn=directory manager" -W -f
> > cert_to_delete_not_used_badhost
> >
> > Delete the requests.
> > ldapdelete -x -D "cn=directory manager" -W -f
> > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> >
> > (you can add '-nv' to test ldapdelete)
> >
> > After this, I decided to trim the changelog and tombstones from ldap.
> > Roughly followed the info from
> > https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
> > Do the steps on all servers.
> >
> > cat > changelog_short.ldif <<-'EOF'
> > dn: cn=changelog5,cn=config
> > changetype: modify
> > replace: nsslapd-changelogmaxage
> > nsslapd-changelogmaxage: 300
> >
> > replace: nsslapd-changelogcompactdb-interval
> > nsslapd-changelogcompactdb-interval: 300
> >
> > replace: nsslapd-changelogtrim-interval
> > nsslapd-changelogtrim-interval: 30
> >
> > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > changetype: modify
> > replace: nsds5ReplicaPurgeDelay
> > nsds5ReplicaPurgeDelay: 300
> >
> > replace: nsds5ReplicaTombstonePurgeInterval
> > nsds5ReplicaTombstonePurgeInterval: 300
> >
> > 'EOF'
> >
> > cat > changelog_normal.ldif <<-'EOF'
> > dn: cn=changelog5,cn=config
> > changetype: modify
> > replace: nsslapd-changelogmaxage
> > nsslapd-changelogmaxage: 7d
> >
> > delete: nsslapd-changelogcompactdb-interval
> >
> > delete: nsslapd-changelogtrim-interval
> >
> > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > changetype: modify
> > replace: nsds5ReplicaPurgeDelay
> > nsds5ReplicaPurgeDelay: 604800
> >
> > replace: nsds5ReplicaTombstonePurgeInterval
> > nsds5ReplicaTombstonePurgeInterval: 86400
> >
> > 'EOF'
> >
> > ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
> > systemctl restart [e-mail]
> >
> > Make a change in the webUI or ipa cli. There must be a write/change
> > operation in LDAP.
> > Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all
> > servers, there should be no errors.
> >
> > ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
> > systemctl restart [e-mail]
> >
> > Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
> > purge 295M. This was only on the server I executed the deletions on,
> > not on any replicas. So not a drastic difference.
> > _______________________________________________
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Jernej Jakob]

After the last client that went haywire, two more decided to do the
same (both after upgrading to Ubuntu 20.04). I still haven't diagnosed
the issue why they did that, I just disabled certmonger. I know, not
good, after that their host certs already expired, but I don't really
care since neither uses their host cert for anything. Diagnosing the
issue is my next priority, or maybe I'll upgrade to Ubuntu 24.04
(latest LTS) to see if the bug has been fixed in a newer version. (I
have 3 bad hosts to test on now, so I can still keep one on 20.04 if I
want to)

I decided to go through the procedure once again for these two hosts,
but since I upgraded the IPA servers in the meantime, some things in
the procedure changed, so I'm attaching the latest procedure as an
attachment.

Significant changes:
I found that ipa cert-find didn't find all certificates that were
actually present in ldap! so I changed the step that used it, to
instead pull the serial numbers directly from ldap. It found 12 more
certificates than ipa cert-find did.

When searching for cert request id's in ou=ca,ou=requests,o=ipaca
the search returned 0 results even though it returned success. A
look in 389-ds error log found that it refused the search becase
the extdata field was not indexed. This required changing the config
setting for nsslapd-verify-filter-schema from process-safe to
warn-invalid. I changed it back after the process was complete.

389-ds didn't want to delete the nsds5 config attribtes for changing
replication intervals, even though they weren't present before. This
left me scrambling to find what the defaults were, the closest I could
find was from Red Hat Directory Server 11 docs so I'm not sure they're
correct. If there's a way to delete them and return them to defaults,
I'd want to know.

5k certs deleted for one host, on to the next.

On Mon, 13 Feb 2023 09:48:29 +0100
Jernej Jakob via FreeIPA-users 
wrote:
> On Tue, 7 Feb 2023 10:35:35 +0100
> Florence Blanc-Renaud  wrote:
> > Hi,
> >
> > On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
> > [e-mail]> wrote:
> > > Hi David. I had the same issue here and found your writeup to be very
> > > helpful.
> > >
> > > I used more or less the same ldap actions to delete the certificates
> > > and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> > > just the one "used"/"correct" certificate for the host, but the main
> > > issue is not fixed. The webUI still displays all the old certificates
> > > that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> > > takes very long, around 1-2 minutes.
> > >
> > > So I want to know what else needs to be done to make the webUI "forget"
> > > about the wrongly issued certificates?
> > >
> > > Where does the webUI get its list of certificates?
> > > I did some searching through the code and could only find the JS
> > > code that invokes a RPC call. But I could not find the code that
> > > handles that call.
> > >
> > > The webui is making a call equivalent to "ipa cert-find" which is handled
> >
> > by the following code:
> > https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L14...
> >
> > The call looks for certificates in multiple locations:
> > in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
> > in the suffix "dc=example,dc=com", in the users/hosts/services entries
> >
> > You cleaned the certificates from the cert repository but there may be many
> > entries (users/hosts/services) containing a userCertificate attribute. To
> > avoid seeing those certs you would have to delete the corresponding
> > userCertificate values.
> >
> > HTH,
> > flo
>
> Thanks flo, this was exactly what I needed, I managed to delete them
> from userCertificate under
> "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
> and now the webui is fast again!
> In retrospect I think making the webui work fast even with such a large
> number of certificates would be better, since this cleanup is not easy
> and it's possible to delete too much. That would probably mean adding a
> pager and filtering to the certificate list which is displayed on the
> host page. Right now they are all displayed at once on one page and
> there is no way to filter out e.g. all revoked certificates. Also find
> and fix why loading the hosts list takes so long even though it doesn't
> display any certificates on that page.
>
> Here are the commands to delete the certificates from userCertificate
> of the host. Add them to those in the previous message, after o=ipaca
> cleaning but before changelog purging.
>
> First save a list of all current userCertificate entries to a file
> ldapsearch -LLL -x -D "cn=directory manager" -W -o ldif-wrap=no
> -b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
> userCertificate > userCertificate_badhost.ldif
>
> Copy the certificate you want to keep from where it's stored on the
> host to where you are working. Put it in badhost.crt in PEM format.
> Create a new ldif without that certificate.
> sed -e '/^dn: .*$/d'
> -e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-----'|tr -d '\n')##"
> userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif
>
> You can delete other certificates that you also want to keep from this
> ldif. If you know the serial numbers of them, you can construct a shell
> one-liner script with a loop that iterates over each certificate in each
> line of the file, uses openssl x509 to retrieve its serial from
> base64-encoded certificate and deletes the certificate from the file if
> it matches.
> I didn't need that so I don't have the script.
>
> Format the ldif so it can be consumed by ldapmodify. Note that you have
> to remove the newlines that the mailing list adds so that everything is
> on one line, especially printf's format strings!
> { printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\n
> changetype: modify\ndelete: userCertificate\n';
> while read -r; do if [ -z "$REPLY" ]; then continue; fi;
> printf '%s\n' "$REPLY"; done < userCertificate_badhost_to_delete.ldif;
> } > userCertificate_badhost_to_delete_ldapmodify.ldif
>
> Test if the ldif is correct
> ldapmodify -nv -x -D "cn=directory manager" -W
> -f userCertificate_badhost_to_delete_ldapmodify.ldif
>
> echo $? should say 0, if not, there's an error in the ldif you need to
> correct.
>
> Then remove the -nv and run the ldapmodify again to delete the entries
> for real.
> > > IIRC my issue that caused certmonger to request a certificate over and
> > > over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> > > 16.04. The path to ca_external_helper changed but it was not changed in
> > > /var/lib/certmonger/cas/* which caused certmonger to fail running
> > > ca_external_helper. To fix it I did:
> > >
> > > sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> > > /var/lib/certmonger/cas/*
> > >
> > > Below is the exact procedure I used to delete the certificates from
> > > LDAP.
> > >
> > > First fix the issue that caused the issuing of too many certificates.
> > > Make sure it successfully issued and saved the cert on the client and
> > > that it's in status "MONITORING", "stuck: no".
> > > Find the serial number of the cert currently present on the client.
> > > 'sudo getcert list', look at "certificate:" in my case it was in
> > > "/etc/ssl/private/hostname-ipa-cert.crt"
> > > openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> > > In my case it was 268369940.
> > >
> > > Used the following shell script to revoke all the certificates with
> > > serial not matching. I did this before I knew howo to get the cert
> > > serials from ldap so it uses ipa cert-find. It's a slow process.
> > > for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> > > '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s"
> > > ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
> > >
> > > You can view all the revoked cert cn's with this command before
> > > deleting them.
> > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
> > >
> > > Make a list of all cert cn's not matching the used cert, save output
> > > into a file, ready to be read by ldapdelete later.
> > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
> > > cert_to_delete_not_used_badhost
> > >
> > > Make a list of all the requestId for all the certs to be deleted.
> > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
> > > 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
> > >
> > > In my case there were a couple more requests than issued certs, I'm not
> > > sure why. I made a list of all requests for this host excluding the
> > > requestId of the correct cert. First find the correct/used cert
> > > requestId. In my case it was 9990026.
> > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> > > 'requestId:\K.*'
> > >
> > > Then get a list of all requests for that host, excluding that one
> > > requestId.
> > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > "ou=ca,ou=requests,o=ipaca"
> > > '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> > > dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
> > >
> > > Count the number of certs/requests from the previous operations. The
> > > first two must match, the 3rd shows how many extra requests there are.
> > > wc -l cert_to_delete_not_used_badhost
> > > cert_request_to_delete_not_used_from_metaInfo_badhost
> > > cert_request_to_delete_not_used_badhost
> > >   3982 cert_to_delete_not_used_badhost
> > >   3982 cert_request_to_delete_not_used_from_metaInfo_badhost
> > >   3990 cert_request_to_delete_not_used_badhost
> > >
> > > So there are 8 extra requests without corresponding certs. I chose to
> > > ignore them for now. They're probably fine to delete in the future.
> > >
> > > Before deleting the requests, make a file in a format ldapdelete
> > > expects.
> > > while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY";
> > > done < cert_request_to_delete_not_used_from_metaInfo_badhost >
> > > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> > >
> > > Now the actual deletion steps.
> > > Delete the certs.
> > > ldapdelete -x -D "cn=directory manager" -W -f
> > > cert_to_delete_not_used_badhost
> > >
> > > Delete the requests.
> > > ldapdelete -x -D "cn=directory manager" -W -f
> > > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> > >
> > > (you can add '-nv' to test ldapdelete)
> > >
> > > After this, I decided to trim the changelog and tombstones from ldap.
> > > Roughly followed the info from
> > > https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
> > > Do the steps on all servers.
> > >
> > > cat > changelog_short.ldif <<-'EOF'
> > > dn: cn=changelog5,cn=config
> > > changetype: modify
> > > replace: nsslapd-changelogmaxage
> > > nsslapd-changelogmaxage: 300
> > >
> > > replace: nsslapd-changelogcompactdb-interval
> > > nsslapd-changelogcompactdb-interval: 300
> > >
> > > replace: nsslapd-changelogtrim-interval
> > > nsslapd-changelogtrim-interval: 30
> > >
> > > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > > changetype: modify
> > > replace: nsds5ReplicaPurgeDelay
> > > nsds5ReplicaPurgeDelay: 300
> > >
> > > replace: nsds5ReplicaTombstonePurgeInterval
> > > nsds5ReplicaTombstonePurgeInterval: 300
> > >
> > > 'EOF'
> > >
> > > cat > changelog_normal.ldif <<-'EOF'
> > > dn: cn=changelog5,cn=config
> > > changetype: modify
> > > replace: nsslapd-changelogmaxage
> > > nsslapd-changelogmaxage: 7d
> > >
> > > delete: nsslapd-changelogcompactdb-interval
> > >
> > > delete: nsslapd-changelogtrim-interval
> > >
> > > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > > changetype: modify
> > > replace: nsds5ReplicaPurgeDelay
> > > nsds5ReplicaPurgeDelay: 604800
> > >
> > > replace: nsds5ReplicaTombstonePurgeInterval
> > > nsds5ReplicaTombstonePurgeInterval: 86400
> > >
> > > 'EOF'
> > >
> > > ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
> > > systemctl restart [e-mail]
> > >
> > > Make a change in the webUI or ipa cli. There must be a write/change
> > > operation in LDAP.
> > > Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all
> > > servers, there should be no errors.
> > >
> > > ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
> > > systemctl restart [e-mail]
> > >
> > > Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
> > > purge 295M. This was only on the server I executed the deletions on,
> > > not on any replicas. So not a drastic difference.
> > > _______________________________________________
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > Do not reply to spam, report it:
> > > https://pagure.io/fedora-infrastructure/new_issue
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Jernej Jakob]

Attached are the latest version of my notes, some bugs fixed, improved,
and made easier to use by putting common values into variables.
They might come useful to someone. I deleted 5k+4k+8k=17k certificates
from 3 previously misbehaving hosts (all after upgrading to Ubuntu
20.04). There are now just 69 certs left and the webui is fast again.

On Sun, 29 Sep 2024 23:37:55 +0200
Jernej Jakob via FreeIPA-users 
wrote:
> After the last client that went haywire, two more decided to do the
> same (both after upgrading to Ubuntu 20.04). I still haven't diagnosed
> the issue why they did that, I just disabled certmonger. I know, not
> good, after that their host certs already expired, but I don't really
> care since neither uses their host cert for anything. Diagnosing the
> issue is my next priority, or maybe I'll upgrade to Ubuntu 24.04
> (latest LTS) to see if the bug has been fixed in a newer version. (I
> have 3 bad hosts to test on now, so I can still keep one on 20.04 if I
> want to)
>
> I decided to go through the procedure once again for these two hosts,
> but since I upgraded the IPA servers in the meantime, some things in
> the procedure changed, so I'm attaching the latest procedure as an
> attachment.
>
> Significant changes:
> I found that ipa cert-find didn't find all certificates that were
> actually present in ldap! so I changed the step that used it, to
> instead pull the serial numbers directly from ldap. It found 12 more
> certificates than ipa cert-find did.
>
> When searching for cert request id's in ou=ca,ou=requests,o=ipaca
> the search returned 0 results even though it returned success. A
> look in 389-ds error log found that it refused the search becase
> the extdata field was not indexed. This required changing the config
> setting for nsslapd-verify-filter-schema from process-safe to
> warn-invalid. I changed it back after the process was complete.
>
> 389-ds didn't want to delete the nsds5 config attribtes for changing
> replication intervals, even though they weren't present before. This
> left me scrambling to find what the defaults were, the closest I could
> find was from Red Hat Directory Server 11 docs so I'm not sure they're
> correct. If there's a way to delete them and return them to defaults,
> I'd want to know.
>
> 5k certs deleted for one host, on to the next.
>
> On Mon, 13 Feb 2023 09:48:29 +0100
> Jernej Jakob via FreeIPA-users 
> wrote:
> > On Tue, 7 Feb 2023 10:35:35 +0100
> > Florence Blanc-Renaud  wrote:
> > > Hi,
> > >
> > > On Tue, Feb 7, 2023 at 1:28 AM Jernej Jakob via FreeIPA-users <
> > > [e-mail]> wrote:
> > > > Hi David. I had the same issue here and found your writeup to be very
> > > > helpful.
> > > >
> > > > I used more or less the same ldap actions to delete the certificates
> > > > and requests (~3.6k) from LDAP. This did make 'ipa cert-find' display
> > > > just the one "used"/"correct" certificate for the host, but the main
> > > > issue is not fixed. The webUI still displays all the old certificates
> > > > that I have deleted from LDAP. Opening the "Hosts" tab or a host page
> > > > takes very long, around 1-2 minutes.
> > > >
> > > > So I want to know what else needs to be done to make the webUI "forget"
> > > > about the wrongly issued certificates?
> > > >
> > > > Where does the webUI get its list of certificates?
> > > > I did some searching through the code and could only find the JS
> > > > code that invokes a RPC call. But I could not find the code that
> > > > handles that call.
> > > >
> > > > The webui is making a call equivalent to "ipa cert-find" which is handled
> > >
> > > by the following code:
> > > https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/cert.py#L14...
> > >
> > > The call looks for certificates in multiple locations:
> > > in the subtree "ou=certificateRepository,ou=ca,o=ipaca"
> > > in the suffix "dc=example,dc=com", in the users/hosts/services entries
> > >
> > > You cleaned the certificates from the cert repository but there may be many
> > > entries (users/hosts/services) containing a userCertificate attribute. To
> > > avoid seeing those certs you would have to delete the corresponding
> > > userCertificate values.
> > >
> > > HTH,
> > > flo
> >
> > Thanks flo, this was exactly what I needed, I managed to delete them
> > from userCertificate under
> > "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
> > and now the webui is fast again!
> > In retrospect I think making the webui work fast even with such a large
> > number of certificates would be better, since this cleanup is not easy
> > and it's possible to delete too much. That would probably mean adding a
> > pager and filtering to the certificate list which is displayed on the
> > host page. Right now they are all displayed at once on one page and
> > there is no way to filter out e.g. all revoked certificates. Also find
> > and fix why loading the hosts list takes so long even though it doesn't
> > display any certificates on that page.
> >
> > Here are the commands to delete the certificates from userCertificate
> > of the host. Add them to those in the previous message, after o=ipaca
> > cleaning but before changelog purging.
> >
> > First save a list of all current userCertificate entries to a file
> > ldapsearch -LLL -x -D "cn=directory manager" -W -o ldif-wrap=no
> > -b "fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com"
> > userCertificate > userCertificate_badhost.ldif
> >
> > Copy the certificate you want to keep from where it's stored on the
> > host to where you are working. Put it in badhost.crt in PEM format.
> > Create a new ldif without that certificate.
> > sed -e '/^dn: .*$/d'
> > -e "s#userCertificate:: $(cat badhost.crt|grep -v -e '-----'|tr -d '\n')##"
> > userCertificate_badhost.ldif > userCertificate_badhost_to_delete.ldif
> >
> > You can delete other certificates that you also want to keep from this
> > ldif. If you know the serial numbers of them, you can construct a shell
> > one-liner script with a loop that iterates over each certificate in each
> > line of the file, uses openssl x509 to retrieve its serial from
> > base64-encoded certificate and deletes the certificate from the file if
> > it matches.
> > I didn't need that so I don't have the script.
> >
> > Format the ldif so it can be consumed by ldapmodify. Note that you have
> > to remove the newlines that the mailing list adds so that everything is
> > on one line, especially printf's format strings!
> > { printf 'dn: fqdn=badhost,cn=computers,cn=accounts,dc=example,dc=com\n
> > changetype: modify\ndelete: userCertificate\n';
> > while read -r; do if [ -z "$REPLY" ]; then continue; fi;
> > printf '%s\n' "$REPLY"; done < userCertificate_badhost_to_delete.ldif;
> > } > userCertificate_badhost_to_delete_ldapmodify.ldif
> >
> > Test if the ldif is correct
> > ldapmodify -nv -x -D "cn=directory manager" -W
> > -f userCertificate_badhost_to_delete_ldapmodify.ldif
> >
> > echo $? should say 0, if not, there's an error in the ldif you need to
> > correct.
> >
> > Then remove the -nv and run the ldapmodify again to delete the entries
> > for real.
> > > > IIRC my issue that caused certmonger to request a certificate over and
> > > > over was caused by a bug after upgrading a client from Ubuntu 14.04 to
> > > > 16.04. The path to ca_external_helper changed but it was not changed in
> > > > /var/lib/certmonger/cas/* which caused certmonger to fail running
> > > > ca_external_helper. To fix it I did:
> > > >
> > > > sed -i -e 's#x86_64-linux-gnu/##g; s#certmonger/certmonger#certmonger#g'
> > > > /var/lib/certmonger/cas/*
> > > >
> > > > Below is the exact procedure I used to delete the certificates from
> > > > LDAP.
> > > >
> > > > First fix the issue that caused the issuing of too many certificates.
> > > > Make sure it successfully issued and saved the cert on the client and
> > > > that it's in status "MONITORING", "stuck: no".
> > > > Find the serial number of the cert currently present on the client.
> > > > 'sudo getcert list', look at "certificate:" in my case it was in
> > > > "/etc/ssl/private/hostname-ipa-cert.crt"
> > > > openssl x509 -in /etc/ssl/private/hostname-ipa-cert.crt -noout -text
> > > > In my case it was 268369940.
> > > >
> > > > Used the following shell script to revoke all the certificates with
> > > > serial not matching. I did this before I knew howo to get the cert
> > > > serials from ldap so it uses ipa cert-find. It's a slow process.
> > > > for s in $(ipa cert-find --hosts=badhost --pkey-only --sizelimit=0|awk
> > > > '/Serial number/{print $3 ;}'); do if [ "$s" = "268369940" ] || [ -z "$s"
> > > > ]; then continue; fi; echo "revoking $s"; ipa cert-revoke "$s"; done
> > > >
> > > > You can view all the revoked cert cn's with this command before
> > > > deleting them.
> > > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > > '(&(subjectName~="badhost")(certStatus=REVOKED))' dn certStatus|less
> > > >
> > > > Make a list of all cert cn's not matching the used cert, save output
> > > > into a file, ready to be read by ldapdelete later.
> > > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > > '(&(subjectName~="badhost")(!(cn=268369940)))' | grep -o 'cn=.*' >
> > > > cert_to_delete_not_used_badhost
> > > >
> > > > Make a list of all the requestId for all the certs to be deleted.
> > > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > > '(&(subjectName~="badhost")(!(cn=268369940)))' metaInfo|grep -oP
> > > > 'requestId:\K.*' > cert_request_to_delete_not_used_from_metaInfo_badhost
> > > >
> > > > In my case there were a couple more requests than issued certs, I'm not
> > > > sure why. I made a list of all requests for this host excluding the
> > > > requestId of the correct cert. First find the correct/used cert
> > > > requestId. In my case it was 9990026.
> > > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > > "ou=certificateRepository,ou=ca,o=ipaca"
> > > > '(&(subjectName~="badhost")(cn=268369940))' metaInfo|grep -oP
> > > > 'requestId:\K.*'
> > > >
> > > > Then get a list of all requests for that host, excluding that one
> > > > requestId.
> > > > ldapsearch -LLL -x -D "cn=directory manager" -W -b
> > > > "ou=ca,ou=requests,o=ipaca"
> > > > '(&(extdata-req--005fsubject--005fname--002ecn=badhost)(!(cn=9990026)))'
> > > > dn|grep -o 'cn=.*' > cert_request_to_delete_not_used_badhost
> > > >
> > > > Count the number of certs/requests from the previous operations. The
> > > > first two must match, the 3rd shows how many extra requests there are.
> > > > wc -l cert_to_delete_not_used_badhost
> > > > cert_request_to_delete_not_used_from_metaInfo_badhost
> > > > cert_request_to_delete_not_used_badhost
> > > >   3982 cert_to_delete_not_used_badhost
> > > >   3982 cert_request_to_delete_not_used_from_metaInfo_badhost
> > > >   3990 cert_request_to_delete_not_used_badhost
> > > >
> > > > So there are 8 extra requests without corresponding certs. I chose to
> > > > ignore them for now. They're probably fine to delete in the future.
> > > >
> > > > Before deleting the requests, make a file in a format ldapdelete
> > > > expects.
> > > > while read -r; do printf 'cn=%s,ou=ca,ou=requests,o=ipaca\n' "$REPLY";
> > > > done < cert_request_to_delete_not_used_from_metaInfo_badhost >
> > > > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> > > >
> > > > Now the actual deletion steps.
> > > > Delete the certs.
> > > > ldapdelete -x -D "cn=directory manager" -W -f
> > > > cert_to_delete_not_used_badhost
> > > >
> > > > Delete the requests.
> > > > ldapdelete -x -D "cn=directory manager" -W -f
> > > > cert_request_to_delete_not_used_from_metaInfo_ldapdelete_badhost
> > > >
> > > > (you can add '-nv' to test ldapdelete)
> > > >
> > > > After this, I decided to trim the changelog and tombstones from ldap.
> > > > Roughly followed the info from
> > > > https://www.port389.org/docs/389ds/FAQ/changelog-trimming.html
> > > > Do the steps on all servers.
> > > >
> > > > cat > changelog_short.ldif <<-'EOF'
> > > > dn: cn=changelog5,cn=config
> > > > changetype: modify
> > > > replace: nsslapd-changelogmaxage
> > > > nsslapd-changelogmaxage: 300
> > > >
> > > > replace: nsslapd-changelogcompactdb-interval
> > > > nsslapd-changelogcompactdb-interval: 300
> > > >
> > > > replace: nsslapd-changelogtrim-interval
> > > > nsslapd-changelogtrim-interval: 30
> > > >
> > > > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > > > changetype: modify
> > > > replace: nsds5ReplicaPurgeDelay
> > > > nsds5ReplicaPurgeDelay: 300
> > > >
> > > > replace: nsds5ReplicaTombstonePurgeInterval
> > > > nsds5ReplicaTombstonePurgeInterval: 300
> > > >
> > > > 'EOF'
> > > >
> > > > cat > changelog_normal.ldif <<-'EOF'
> > > > dn: cn=changelog5,cn=config
> > > > changetype: modify
> > > > replace: nsslapd-changelogmaxage
> > > > nsslapd-changelogmaxage: 7d
> > > >
> > > > delete: nsslapd-changelogcompactdb-interval
> > > >
> > > > delete: nsslapd-changelogtrim-interval
> > > >
> > > > dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> > > > changetype: modify
> > > > replace: nsds5ReplicaPurgeDelay
> > > > nsds5ReplicaPurgeDelay: 604800
> > > >
> > > > replace: nsds5ReplicaTombstonePurgeInterval
> > > > nsds5ReplicaTombstonePurgeInterval: 86400
> > > >
> > > > 'EOF'
> > > >
> > > > ldapmodify -x -D "cn=directory manager" -W -f changelog_short.ldif
> > > > systemctl restart [e-mail]
> > > >
> > > > Make a change in the webUI or ipa cli. There must be a write/change
> > > > operation in LDAP.
> > > > Wait >5 minutes. Monitor /var/log/dirsrv/slapd-EXAMPLE-COM/errors on all
> > > > servers, there should be no errors.
> > > >
> > > > ldapmodify -x -D "cn=directory manager" -W -f changelog_normal.ldif
> > > > systemctl restart [e-mail]
> > > >
> > > > Before the purge /var/lib/dirsrv/slapd-EXAMPLE-COM was 360M, after the
> > > > purge 295M. This was only on the server I executed the deletions on,
> > > > not on any replicas. So not a drastic difference.
> > > > _______________________________________________
> > > > FreeIPA-users mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > > > Do not reply to spam, report it:
> > > > https://pagure.io/fedora-infrastructure/new_issue
> >
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue