Recovering from certificate exparation issues

[Cristian Le]

I have tried my luck around with all the helpers: `pki-server cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but each one is failing on me for multiple reasons.
- `ipa-cacert-manage` Cannot update the CA with `--external-cert-file` because the root ca is not detected to be in the trust list
- `ipa-cert-fix` Was run without overlapping validity time, and the certificate were re-created, so now it is not recoverable, neither back in time, nor in current time
- `pki-tomcat` is failing

It is quite a mess and I would like to ask for some guidance on how one could recover manually from  such dependency issues:
- Is it possible to do a `ipa-server-install` and keep the user data?
- If I sign all of the service's certificates manually, what are all of the manual steps needed to get the services back up so that the helpers can be run.
  - I've tried to install the CA certificate in the nssdb database, ldap, and /etc/ipa/ca.crt. Are there other locations?
  - I've recreated an httpd certificate signed by the root, but I can't figure how to do the same with the ones located in the nssdb database, i.e. to recreate a csr with the same data as one of the certificates there
- What is the order of services that should be updated. My understanding is CA -> `certutil`'s CA -> httpd + slapd + pki-tomcat (not sure where the last one is or how to edit it) -> `ipa-certupdate`

[Florence Blanc-Renaud]

Hi,

On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users <
[e-mail]> wrote:
> I have tried my luck around with all the helpers: `pki-server cert-fix`,
> `ipa-cacert-manage`, `ipa-certupdate`, etc. but each one is failing on me
> for multiple reasons.
> `ipa-cacert-manage` Cannot update the CA with `--external-cert-file`
>
> because the root ca is not detected to be in the trust list

This command is useful if you need to trust a new external CA or renew IPA
CA. Is your IPA CA expired?

- `ipa-cert-fix` Was run without overlapping validity time, and the
> certificate were re-created, so now it is not recoverable, neither back in
> time, nor in current time

It is recommended to do a backup before running ipa-cert-fix. If you
didn't, and want to try the back-in-time method, you can still find the
original certificates in the LDAP database (below
ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching.
You would need to restore the expired certificates, go back in time and
force the renewal.

- `pki-tomcat` is failing
> What is the current situation? Which certs are expired (getcert list)? If

you start the services with "ipactl start --ignore-service-failures", is
pki-tomcat the only service failing?
flo

It is quite a mess and I would like to ask for some guidance on how one
> could recover manually from  such dependency issues:
> Is it possible to do a `ipa-server-install` and keep the user data?

- If I sign all of the service's certificates manually, what are all of the
> manual steps needed to get the services back up so that the helpers can be
> run.
> I've tried to install the CA certificate in the nssdb database, ldap,
>
> and /etc/ipa/ca.crt. Are there other locations?
> I've recreated an httpd certificate signed by the root, but I can't
>
> figure how to do the same with the ones located in the nssdb database, i.e.
> to recreate a csr with the same data as one of the certificates there
> What is the order of services that should be updated. My understanding
>
> is CA -> `certutil`'s CA -> httpd + slapd + pki-tomcat (not sure where the
> last one is or how to edit it) -> `ipa-certupdate`
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Cristian Le]

Hi Florence,

Thanks for the feedback, let me clarify the situation on the certificates:
- External CA is still valid and it is a self-signed certificate that we 
use for other services. So we can manually sign any service certificates 
to get them back up and running
- IPA CA is expired, let's say Aug/10
- I have managed to import a renewed IPA CA and ran `ipa-cert-fix` (and 
also seemed to have run `ipa-certupdate`) on a current date, let's say 
Sep/20. But not all services were recovered and now there is no overlap 
between earliest date in service certificates and the original IPA CA
- I have run a backup, but also did some system upgrades to get the 
`ipa-cacert-manage prune` command, but when I've tried to recover it, 
I've found that the backup was not there.

> you can still find the original certificates in the LDAP database 
(below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of 
searching. You would need to restore the expired certificates, go back 
in time and force the renewal.

I suspect we cannot do this all within LDAP right? If we get back the 
expired certificates, how do we restore them in each service? `httpd` is 
straightforward, and I guess `nssdb` should be doable, assuming the same 
key is used, but is there another database type where the certificates 
are located? Are all the certificates tracked by `getcert list`? Is it 
safe to assume that after running something like `ipa-cert-fix`, they 
are using the same private key?

Some symptoms in the current setup:
- When we are forward in time, `pki-tomcatd` is able to run, but then I 
can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`. From what I've 
read, all of these commands (or at least `ipa-cacert-manage renew`) must 
be done backwards in time.
- When we are backwards in time `pki-tomcatd` is unable to run, failing 
to access `:8080/ca/admin/ca/getStatus`. This then blocks various other 
services to be run. But about `ipactl restart`, only `pki-tomcatd` 
service is actually failing (and ipa service itself of course).

I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and indeed 
there are still a bunch of certificates there in linear order. What are 
the services I should look for in there? I am using Apache Directory 
Studio and I can download the `userCertificate`. Should I just run 
`certutil -A` with those values with corresponding `subjectName`?

BTW, I want to document this process on the website, should I make a PR 
on the github repo or is there somewhere else?

Kind regards,
Cristian

On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
> Hi,
>
> On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users 
>  wrote:
> I have tried my luck around with all the helpers: `pki-server
> cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but each
> one is failing on me for multiple reasons.
> - `ipa-cacert-manage` Cannot update the CA with
> `--external-cert-file` because the root ca is not detected to be
> in the trust list
>
> This command is useful if you need to trust a new external CA or renew 
> IPA CA. Is your IPA CA expired?
> - `ipa-cert-fix` Was run without overlapping validity time, and
> the certificate were re-created, so now it is not recoverable,
> neither back in time, nor in current time
>
> It is recommended to do a backup before running ipa-cert-fix. If you 
> didn't, and want to try the back-in-time method, you can still find 
> the original certificates in the LDAP database (below 
> ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of 
> searching. You would need to restore the expired certificates, go back 
> in time and force the renewal.
> - `pki-tomcat` is failing
>
> What is the current situation? Which certs are expired (getcert list)? 
> If you start the services with "ipactl start 
> --ignore-service-failures", is pki-tomcat the only service failing?
> flo
> It is quite a mess and I would like to ask for some guidance on
> how one could recover manually from  such dependency issues:
> - Is it possible to do a `ipa-server-install` and keep the user data?
>
> - If I sign all of the service's certificates manually, what are
> all of the manual steps needed to get the services back up so that
> the helpers can be run.
>   - I've tried to install the CA certificate in the nssdb
> database, ldap, and /etc/ipa/ca.crt. Are there other locations?
>   - I've recreated an httpd certificate signed by the root, but I
> can't figure how to do the same with the ones located in the nssdb
> database, i.e. to recreate a csr with the same data as one of the
> certificates there
> - What is the order of services that should be updated. My
> understanding is CA -> `certutil`'s CA -> httpd + slapd +
> pki-tomcat (not sure where the last one is or how to edit it) ->
> `ipa-certupdate`
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to
> [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Florence Blanc-Renaud]

Hi,

On Fri, Sep 22, 2023 at 12:36 PM Cristian Le  wrote:
> Hi Florence,
>
> Thanks for the feedback, let me clarify the situation on the certificates:
> External CA is still valid and it is a self-signed certificate that we
>
> use for other services. So we can manually sign any service certificates to
> get them back up and running
> IPA CA is expired, let's say Aug/10
> I have managed to import a renewed IPA CA and ran `ipa-cert-fix` (and
>
> also seemed to have run `ipa-certupdate`) on a current date, let's say
> Sep/20. But not all services were recovered and now there is no overlap
> between earliest date in service certificates and the original IPA CA

Which are the not-recovered services? Can you provide the output of
"getcert list" at the current date?
flo
> I have run a backup, but also did some system upgrades to get the
>
> `ipa-cacert-manage prune` command, but when I've tried to recover it, I've
> found that the backup was not there.
> > you can still find the original certificates in the LDAP database
>
> (below ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
> searching. You would need to restore the expired certificates, go back in
> time and force the renewal.
>
> I suspect we cannot do this all within LDAP right? If we get back the
> expired certificates, how do we restore them in each service? `httpd` is
> straightforward, and I guess `nssdb` should be doable, assuming the same
> key is used, but is there another database type where the certificates are
> located? Are all the certificates tracked by `getcert list`? Is it safe to
> assume that after running something like `ipa-cert-fix`, they are using the
> same private key?
>
> Some symptoms in the current setup:
> When we are forward in time, `pki-tomcatd` is able to run, but then I
>
> can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`. From what I've
> read, all of these commands (or at least `ipa-cacert-manage renew`) must be
> done backwards in time.
> When we are backwards in time `pki-tomcatd` is unable to run, failing to
>
> access `:8080/ca/admin/ca/getStatus`. This then blocks various other
> services to be run. But about `ipactl restart`, only `pki-tomcatd` service
> is actually failing (and ipa service itself of course).
>
> I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and indeed
> there are still a bunch of certificates there in linear order. What are the
> services I should look for in there? I am using Apache Directory Studio and
> I can download the `userCertificate`. Should I just run `certutil -A` with
> those values with corresponding `subjectName`?
>
> BTW, I want to document this process on the website, should I make a PR on
> the github repo or is there somewhere else?
>
> Kind regards,
> Cristian
> On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
>
> Hi,
>
> On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users <
> [e-mail]> wrote:
> > I have tried my luck around with all the helpers: `pki-server cert-fix`,
> > `ipa-cacert-manage`, `ipa-certupdate`, etc. but each one is failing on me
> > for multiple reasons.
> > `ipa-cacert-manage` Cannot update the CA with `--external-cert-file`
> >
> > because the root ca is not detected to be in the trust list
>
> This command is useful if you need to trust a new external CA or renew IPA
> CA. Is your IPA CA expired?
> `ipa-cert-fix` Was run without overlapping validity time, and the
> > certificate were re-created, so now it is not recoverable, neither back in
> > time, nor in current time
>
> It is recommended to do a backup before running ipa-cert-fix. If you
> didn't, and want to try the back-in-time method, you can still find the
> original certificates in the LDAP database (below
> ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of searching.
> You would need to restore the expired certificates, go back in time and
> force the renewal.
> `pki-tomcat` is failing
> > What is the current situation? Which certs are expired (getcert list)? If
>
> you start the services with "ipactl start --ignore-service-failures", is
> pki-tomcat the only service failing?
> flo
>
> It is quite a mess and I would like to ask for some guidance on how one
> > could recover manually from  such dependency issues:
> > Is it possible to do a `ipa-server-install` and keep the user data?
> If I sign all of the service's certificates manually, what are all of
> > the manual steps needed to get the services back up so that the helpers can
> > be run.
> > I've tried to install the CA certificate in the nssdb database, ldap,
> >
> > and /etc/ipa/ca.crt. Are there other locations?
> > I've recreated an httpd certificate signed by the root, but I can't
> >
> > figure how to do the same with the ones located in the nssdb database, i.e.
> > to recreate a csr with the same data as one of the certificates there
> > What is the order of services that should be updated. My understanding
> >
> > is CA -> `certutil`'s CA -> httpd + slapd + pki-tomcat (not sure where the
> > last one is or how to edit it) -> `ipa-certupdate`
> > _______________________________________________
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to
> > [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Cristian Le]

Ok, let me walk through some of the specific errors, and I will also 
censor out some of the output since this is going to the public 
mail-list as well.

Starting from the beginning.
- I have set the date to `1 month` before certificate expired with `sudo 
date`
- I ran `ipactl restart --force` and `pki-tomcatd` failed
- I try to run `ipa-cacert-manage renew` with `--external-ca` and 
`--external-cert-file`, with the CSR signed with `openssl x509 -req 
-copy_extensions=copyall`, but this one failed because of `ERROR: CA 
certificate chain in is incomplete: missing certificate with subject`, 
even though I passed both the root and new CA

Then I have tried to manually add the new CA, but then, that did not 
work. So moving on to the next layer, when I run to `sudo getcert list`, 
the certificates I get:
```
Request ID '20220810070559':
         status: CA_UNREACHABLE
         ca-error: Error setting up ccache for "host" service on client 
using default keytab: Cannot contact any KDC for requested realm.
         stuck: no
         key pair storage: 
type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
         CA: IPA
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 14:21:40 UTC
         expires: 2025-09-20 14:21:40 UTC
Request ID '20230920203122':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=CA Audit,O=My Lab
         issued: 2023-09-20 14:21:35 UTC
         expires: 2025-09-09 14:21:35 UTC
Request ID '20230920203123':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=OCSP Subsystem,O=My Lab
         issued: 2023-09-20 14:21:33 UTC
         expires: 2025-09-09 14:21:33 UTC
Request ID '20230920203125':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=CA Subsystem,O=My Lab
         issued: 2023-09-20 14:21:31 UTC
         expires: 2025-09-09 14:21:31 UTC
Request ID '20230920203128':
         status: MONITORING
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 11:29:21 UTC
         expires: 2023-12-20 11:29:21 UTC
Request ID '20230920203130':
         status: MONITORING
         stuck: no
         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=IPA RA,O=My Lab
         issued: 2023-09-20 14:21:36 UTC
         expires: 2025-09-09 14:21:36 UTC
Request ID '20230920203131':
         status: CA_UNREACHABLE
         ca-error: Error setting up ccache for "host" service on client 
using default keytab: Cannot contact any KDC for requested realm.
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS 
Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
         certificate: 
type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS 
Certificate DB'
         CA: IPA
         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-09-20 14:21:39 UTC
         expires: 2025-09-20 14:21:39 UTC
Request ID '20230920203134':
         status: CA_UNREACHABLE
         stuck: no
         key pair storage: 
type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
         certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
         CA: IPA
         issuer: CN=My Lab Root CA,O=My Lab
         subject: CN=ipa1.my.lab,O=My Lab
         issued: 2023-07-10 04:31:16 UTC
         expires: 2023-08-09 04:31:16 UTC
Request ID '20230510042436':
         status: MONITORING
         ca-error: Updated certificate not available
         stuck: no
         key pair storage: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB',pin set
         certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
         CA: dogtag-ipa-ca-renew-agent
         issuer: CN=My Lab Root CA,O=My Lab
         subject: CN=FreeIPA Root Certificate Authority,O=My Lab
         issued: 2022-08-10 04:29:31 UTC
         expires: 2023-08-10 04:29:31 UTC
```
Note that even the `MONITORING` ones are unusable due to the 
non-overlapping date. So when I do something like `ipa-certupdate` I get:
`The ipa-certupdate command failed, exception: DatabaseError: Connect 
error: error:0A000086:SSL routines::certificate verify failed (unable to 
get issuer certificate)` when trying to use `ipaldap.py`.

It is a mess, that's why I think it is better to have a way to simply 
re-bootstrap all certificates.

On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
> Hi,
>
> On Fri, Sep 22, 2023 at 12:36 PM Cristian Le  wrote:
> Hi Florence,
>
> Thanks for the feedback, let me clarify the situation on the
> certificates:
> - External CA is still valid and it is a self-signed certificate
> that we use for other services. So we can manually sign any
> service certificates to get them back up and running
> - IPA CA is expired, let's say Aug/10
> - I have managed to import a renewed IPA CA and ran `ipa-cert-fix`
> (and also seemed to have run `ipa-certupdate`) on a current date,
> let's say Sep/20. But not all services were recovered and now
> there is no overlap between earliest date in service certificates
> and the original IPA CA
>
> Which are the not-recovered services? Can you provide the output of 
> "getcert list" at the current date?
> flo
> - I have run a backup, but also did some system upgrades to get
> the `ipa-cacert-manage prune` command, but when I've tried to
> recover it, I've found that the backup was not there.
>
>  > you can still find the original certificates in the LDAP
> database (below ou=certificateRepository,ou=ca,o=ipaca) but it
> requires a bit of searching. You would need to restore the expired
> certificates, go back in time and force the renewal.
>
> I suspect we cannot do this all within LDAP right? If we get back
> the expired certificates, how do we restore them in each service?
> `httpd` is straightforward, and I guess `nssdb` should be doable,
> assuming the same key is used, but is there another database type
> where the certificates are located? Are all the certificates
> tracked by `getcert list`? Is it safe to assume that after running
> something like `ipa-cert-fix`, they are using the same private key?
>
> Some symptoms in the current setup:
> - When we are forward in time, `pki-tomcatd` is able to run, but
> then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`.
> From what I've read, all of these commands (or at least
> `ipa-cacert-manage renew`) must be done backwards in time.
> - When we are backwards in time `pki-tomcatd` is unable to run,
> failing to access `:8080/ca/admin/ca/getStatus`. This then blocks
> various other services to be run. But about `ipactl restart`, only
> `pki-tomcatd` service is actually failing (and ipa service itself
> of course).
>
> I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and
> indeed there are still a bunch of certificates there in linear
> order. What are the services I should look for in there? I am
> using Apache Directory Studio and I can download the
> `userCertificate`. Should I just run `certutil -A` with those
> values with corresponding `subjectName`?
>
> BTW, I want to document this process on the website, should I make
> a PR on the github repo or is there somewhere else?
>
> Kind regards,
> Cristian
>
> On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
> > Hi,
> >
> > On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users
> > <[e-mail]> wrote:
> >
> >     I have tried my luck around with all the helpers: `pki-server
> >     cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but
> >     each one is failing on me for multiple reasons.
> >     - `ipa-cacert-manage` Cannot update the CA with
> >     `--external-cert-file` because the root ca is not detected to
> >     be in the trust list
> >
> > This command is useful if you need to trust a new external CA or
> > renew IPA CA. Is your IPA CA expired?
> >
> >     - `ipa-cert-fix` Was run without overlapping validity time,
> >     and the certificate were re-created, so now it is not
> >     recoverable, neither back in time, nor in current time
> >
> > It is recommended to do a backup before running ipa-cert-fix. If
> > you didn't, and want to try the back-in-time method, you can
> > still find the original certificates in the LDAP database (below
> > ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
> > searching. You would need to restore the expired certificates, go
> > back in time and force the renewal.
> >
> >     - `pki-tomcat` is failing
> >
> > What is the current situation? Which certs are expired (getcert
> > list)? If you start the services with "ipactl start
> > --ignore-service-failures", is pki-tomcat the only service failing?
> > flo
> >
> >     It is quite a mess and I would like to ask for some guidance
> >     on how one could recover manually from  such dependency issues:
> >     - Is it possible to do a `ipa-server-install` and keep the
> >     user data?
> >
> >     - If I sign all of the service's certificates manually, what
> >     are all of the manual steps needed to get the services back
> >     up so that the helpers can be run.
> >       - I've tried to install the CA certificate in the nssdb
> >     database, ldap, and /etc/ipa/ca.crt. Are there other locations?
> >       - I've recreated an httpd certificate signed by the root,
> >     but I can't figure how to do the same with the ones located
> >     in the nssdb database, i.e. to recreate a csr with the same
> >     data as one of the certificates there
> >     - What is the order of services that should be updated. My
> >     understanding is CA -> `certutil`'s CA -> httpd + slapd +
> >     pki-tomcat (not sure where the last one is or how to edit it)
> >     -> `ipa-certupdate`
> >     _______________________________________________
> >     FreeIPA-users mailing list --
> >     [e-mail]
> >     To unsubscribe send an email to
> >     [e-mail]
> >     Fedora Code of Conduct:
> >     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >     List Guidelines:
> >     https://fedoraproject.org/wiki/Mailing_list_guidelines
> >     List Archives:
> >     https://lists.fedorahosted.org/archives/list/[e-mail]
> >     Do not reply to spam, report it:
> >     https://pagure.io/fedora-infrastructure/new_issue

[Rob Crittenden]

Cristian Le via FreeIPA-users wrote:
> Ok, let me walk through some of the specific errors, and I will also
> censor out some of the output since this is going to the public
> mail-list as well.
>
> Starting from the beginning.
> I have set the date to `1 month` before certificate expired with `sudo
>
> date`
> I ran `ipactl restart --force` and `pki-tomcatd` failed

If tomcat fails to start then you need to stop and figure out why. Any
further poking at certificates won't yield anything useful without a
working CA.
> I try to run `ipa-cacert-manage renew` with `--external-ca` and
>
> `--external-cert-file`, with the CSR signed with `openssl x509 -req
> -copy_extensions=copyall`, but this one failed because of `ERROR: CA
> certificate chain in is incomplete: missing certificate with subject`,
> even though I passed both the root and new CA

What failed?
> Then I have tried to manually add the new CA, but then, that did not
> work. So moving on to the next layer, when I run to `sudo getcert list`,
> the certificates I get:

Manually add how?
> Request ID '20220810070559':
>         status: CA_UNREACHABLE
>         ca-error: Error setting up ccache for "host" service on client
> using default keytab: Cannot contact any KDC for requested realm.

The KDC isn't running.
> stuck: no
>         key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>         certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>         CA: IPA
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=ipa1.my.lab,O=My Lab
>         issued: 2023-09-20 14:21:40 UTC
>         expires: 2025-09-20 14:21:40 UTC
> Request ID '20230920203122':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=CA Audit,O=My Lab
>         issued: 2023-09-20 14:21:35 UTC
>         expires: 2025-09-09 14:21:35 UTC
> Request ID '20230920203123':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=OCSP Subsystem,O=My Lab
>         issued: 2023-09-20 14:21:33 UTC
>         expires: 2025-09-09 14:21:33 UTC
> Request ID '20230920203125':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=CA Subsystem,O=My Lab
>         issued: 2023-09-20 14:21:31 UTC
>         expires: 2025-09-09 14:21:31 UTC
> Request ID '20230920203128':
>         status: MONITORING
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=ipa1.my.lab,O=My Lab
>         issued: 2023-09-20 11:29:21 UTC
>         expires: 2023-12-20 11:29:21 UTC
> Request ID '20230920203130':
>         status: MONITORING
>         stuck: no
>         key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>         certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=IPA RA,O=My Lab
>         issued: 2023-09-20 14:21:36 UTC
>         expires: 2025-09-09 14:21:36 UTC
> Request ID '20230920203131':
>         status: CA_UNREACHABLE
>         ca-error: Error setting up ccache for "host" service on client
> using default keytab: Cannot contact any KDC for requested realm.
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
>         certificate:
> type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
> Certificate DB'
>         CA: IPA
>         issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
>         subject: CN=ipa1.my.lab,O=My Lab
>         issued: 2023-09-20 14:21:39 UTC
>         expires: 2025-09-20 14:21:39 UTC
> Request ID '20230920203134':
>         status: CA_UNREACHABLE
>         stuck: no
>         key pair storage:
> type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
>         certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
>         CA: IPA
>         issuer: CN=My Lab Root CA,O=My Lab
>         subject: CN=ipa1.my.lab,O=My Lab
>         issued: 2023-07-10 04:31:16 UTC
>         expires: 2023-08-09 04:31:16 UTC
> Request ID '20230510042436':
>         status: MONITORING
>         ca-error: Updated certificate not available
>         stuck: no
>         key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB',pin set
>         certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> cert-pki-ca',token='NSS Certificate DB'
>         CA: dogtag-ipa-ca-renew-agent
>         issuer: CN=My Lab Root CA,O=My Lab
>         subject: CN=FreeIPA Root Certificate Authority,O=My Lab
>         issued: 2022-08-10 04:29:31 UTC
>         expires: 2023-08-10 04:29:31 UTC
> Note that even the `MONITORING` ones are unusable due to the
> non-overlapping date. So when I do something like `ipa-certupdate` I get:
> `The ipa-certupdate command failed, exception: DatabaseError: Connect
> error: error:0A000086:SSL routines::certificate verify failed (unable to
> get issuer certificate)` when trying to use `ipaldap.py`.
>
> It is a mess, that's why I think it is better to have a way to simply
> re-bootstrap all certificates.

There isn't really a good way to start over.

rob
> On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
> > Hi,
> >
> > On Fri, Sep 22, 2023 at 12:36 PM Cristian Le  wrote:
> > Hi Florence,
> >
> > Thanks for the feedback, let me clarify the situation on the
> > certificates:
> > - External CA is still valid and it is a self-signed certificate
> > that we use for other services. So we can manually sign any
> > service certificates to get them back up and running
> > - IPA CA is expired, let's say Aug/10
> > - I have managed to import a renewed IPA CA and ran `ipa-cert-fix`
> > (and also seemed to have run `ipa-certupdate`) on a current date,
> > let's say Sep/20. But not all services were recovered and now
> > there is no overlap between earliest date in service certificates
> > and the original IPA CA
> >
> > Which are the not-recovered services? Can you provide the output of
> > "getcert list" at the current date?
> > flo
> > - I have run a backup, but also did some system upgrades to get
> > the `ipa-cacert-manage prune` command, but when I've tried to
> > recover it, I've found that the backup was not there.
> >
> >  > you can still find the original certificates in the LDAP
> > database (below ou=certificateRepository,ou=ca,o=ipaca) but it
> > requires a bit of searching. You would need to restore the expired
> > certificates, go back in time and force the renewal.
> >
> > I suspect we cannot do this all within LDAP right? If we get back
> > the expired certificates, how do we restore them in each service?
> > `httpd` is straightforward, and I guess `nssdb` should be doable,
> > assuming the same key is used, but is there another database type
> > where the certificates are located? Are all the certificates
> > tracked by `getcert list`? Is it safe to assume that after running
> > something like `ipa-cert-fix`, they are using the same private key?
> >
> > Some symptoms in the current setup:
> > - When we are forward in time, `pki-tomcatd` is able to run, but
> > then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`.
> > From what I've read, all of these commands (or at least
> > `ipa-cacert-manage renew`) must be done backwards in time.
> > - When we are backwards in time `pki-tomcatd` is unable to run,
> > failing to access `:8080/ca/admin/ca/getStatus`. This then blocks
> > various other services to be run. But about `ipactl restart`, only
> > `pki-tomcatd` service is actually failing (and ipa service itself
> > of course).
> >
> > I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and
> > indeed there are still a bunch of certificates there in linear
> > order. What are the services I should look for in there? I am
> > using Apache Directory Studio and I can download the
> > `userCertificate`. Should I just run `certutil -A` with those
> > values with corresponding `subjectName`?
> >
> > BTW, I want to document this process on the website, should I make
> > a PR on the github repo or is there somewhere else?
> >
> > Kind regards,
> > Cristian
> >
> > On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
> > > Hi,
> > >
> > > On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users
> > > <[e-mail]> wrote:
> > >
> > >     I have tried my luck around with all the helpers: `pki-server
> > >     cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but
> > >     each one is failing on me for multiple reasons.
> > >     - `ipa-cacert-manage` Cannot update the CA with
> > >     `--external-cert-file` because the root ca is not detected to
> > >     be in the trust list
> > >
> > > This command is useful if you need to trust a new external CA or
> > > renew IPA CA. Is your IPA CA expired?
> > >
> > >     - `ipa-cert-fix` Was run without overlapping validity time,
> > >     and the certificate were re-created, so now it is not
> > >     recoverable, neither back in time, nor in current time
> > >
> > > It is recommended to do a backup before running ipa-cert-fix. If
> > > you didn't, and want to try the back-in-time method, you can
> > > still find the original certificates in the LDAP database (below
> > > ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
> > > searching. You would need to restore the expired certificates, go
> > > back in time and force the renewal.
> > >
> > >     - `pki-tomcat` is failing
> > >
> > > What is the current situation? Which certs are expired (getcert
> > > list)? If you start the services with "ipactl start
> > > --ignore-service-failures", is pki-tomcat the only service failing?
> > > flo
> > >
> > >     It is quite a mess and I would like to ask for some guidance
> > >     on how one could recover manually from  such dependency issues:
> > >     - Is it possible to do a `ipa-server-install` and keep the
> > >     user data?
> > >
> > >     - If I sign all of the service's certificates manually, what
> > >     are all of the manual steps needed to get the services back
> > >     up so that the helpers can be run.
> > >       - I've tried to install the CA certificate in the nssdb
> > >     database, ldap, and /etc/ipa/ca.crt. Are there other locations?
> > >       - I've recreated an httpd certificate signed by the root,
> > >     but I can't figure how to do the same with the ones located
> > >     in the nssdb database, i.e. to recreate a csr with the same
> > >     data as one of the certificates there
> > >     - What is the order of services that should be updated. My
> > >     understanding is CA -> `certutil`'s CA -> httpd + slapd +
> > >     pki-tomcat (not sure where the last one is or how to edit it)
> > >     -> `ipa-certupdate`
> > >     _______________________________________________
> > >     FreeIPA-users mailing list --
> > >     [e-mail]
> > >     To unsubscribe send an email to
> > >     [e-mail]
> > >     Fedora Code of Conduct:
> > >     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > >     List Guidelines:
> > >     https://fedoraproject.org/wiki/Mailing_list_guidelines
> > >     List Archives:
> > >     https://lists.fedorahosted.org/archives/list/[e-mail]
> > >     Do not reply to spam, report it:
> > >     https://pagure.io/fedora-infrastructure/new_issue
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Cristian Le]

> If tomcat fails to start then you need to stop and figure out why.

Any further poking at certificates won't yield anything useful without a 
working CA.

So at all times I should make sure `pki-tomcatd` is working. Well it is 
often failing with:
`ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:  
for url: http://ipa1.my.lab:8080/ca/admin/ca/getStatus%60

Presumably this is meant to be fixed by `ipa-cacert-manage renew`, but 
as indicated it fails with `ERROR: CA
certificate chain in is incomplete`, specifically the traceback:
```
ipapython.admintool: DEBUG:   File 
"/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in 
execute
     return_value = self.run()
                    ^^^^^^^^^^
   File 
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", 
line 138, in run
     return self.renew()
            ^^^^^^^^^^^^
   File 
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", 
line 201, in renew
     return self.renew_external_step_2(ca, cert)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/usr/lib/python3.11/site-packages/ipaserver/install/ipa_cacert_manage.py", 
line 277, in renew_external_step_2
     cert_file, ca_file = installutils.load_external_cert(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/usr/lib/python3.11/site-packages/ipaserver/install/installutils.py", 
line 1194, in load_external_cert
     raise ScriptError(
```
(disregard the slightly offset line numbers, I've added some debugging 
to confirm that in the loop at [1] the trust chain has indeed only 1 
item and it is not self-signed).

One thing suspicious is that the root CA is not set to `CT,C,C`:
```
ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 
'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=FreeIPA Root Certificate 
Authority,O=My Lab', '-t', 'C,,', '-f', '/tmp/tmprpz4cur1/pwdfile.txt']
ipapython.ipautil: DEBUG: args=['/usr/bin/certutil', '-d', 
'sql:/tmp/tmprpz4cur1', '-M', '-n', 'CN=My Lab Root CA,O=My Lab', '-t', 
'C,,', '-f', '/tmp/tmprpz4cur1/pwdfile.txt']
```

Do you think it is worth delving more into this issue and finding a way 
to recover `ipa-cacert-manage renew`?
> Manually add how?

Using `certutil -A -d` on the `nssdb` and individual files like in 
`/etc/httpd`. I get the locations as reported in `getcert list`.
> The KDC isn't running.

The KDC is running, but the `kdc.crt` certificates are busted.

[1] 
https://pagure.io/freeipa/blob/master/f/ipaserver/install/installutils.py#_1...

On 2023/09/25 21:06, Rob Crittenden wrote:
> Cristian Le via FreeIPA-users wrote:
> > Ok, let me walk through some of the specific errors, and I will also
> > censor out some of the output since this is going to the public
> > mail-list as well.
> >
> > Starting from the beginning.
> > I have set the date to `1 month` before certificate expired with `sudo
> >
> > date`
> > I ran `ipactl restart --force` and `pki-tomcatd` failed
>
> If tomcat fails to start then you need to stop and figure out why. Any
> further poking at certificates won't yield anything useful without a
> working CA.
> > I try to run `ipa-cacert-manage renew` with `--external-ca` and
> >
> > `--external-cert-file`, with the CSR signed with `openssl x509 -req
> > -copy_extensions=copyall`, but this one failed because of `ERROR: CA
> > certificate chain in is incomplete: missing certificate with subject`,
> > even though I passed both the root and new CA
>
> What failed?
> > Then I have tried to manually add the new CA, but then, that did not
> > work. So moving on to the next layer, when I run to `sudo getcert list`,
> > the certificates I get:
>
> Manually add how?
> > Request ID '20220810070559':
> >          status: CA_UNREACHABLE
> >          ca-error: Error setting up ccache for "host" service on client
> > using default keytab: Cannot contact any KDC for requested realm.
>
> The KDC isn't running.
> > stuck: no
> >          key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
> >          certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
> >          CA: IPA
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=ipa1.my.lab,O=My Lab
> >          issued: 2023-09-20 14:21:40 UTC
> >          expires: 2025-09-20 14:21:40 UTC
> > Request ID '20230920203122':
> >          status: MONITORING
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> > cert-pki-ca',token='NSS Certificate DB',pin set
> >          certificate:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
> > cert-pki-ca',token='NSS Certificate DB'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=CA Audit,O=My Lab
> >          issued: 2023-09-20 14:21:35 UTC
> >          expires: 2025-09-09 14:21:35 UTC
> > Request ID '20230920203123':
> >          status: MONITORING
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> > Certificate DB',pin set
> >          certificate:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS
> > Certificate DB'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=OCSP Subsystem,O=My Lab
> >          issued: 2023-09-20 14:21:33 UTC
> >          expires: 2025-09-09 14:21:33 UTC
> > Request ID '20230920203125':
> >          status: MONITORING
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> > cert-pki-ca',token='NSS Certificate DB',pin set
> >          certificate:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
> > cert-pki-ca',token='NSS Certificate DB'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=CA Subsystem,O=My Lab
> >          issued: 2023-09-20 14:21:31 UTC
> >          expires: 2025-09-09 14:21:31 UTC
> > Request ID '20230920203128':
> >          status: MONITORING
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> > cert-pki-ca',token='NSS Certificate DB',pin set
> >          certificate:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> > cert-pki-ca',token='NSS Certificate DB'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=ipa1.my.lab,O=My Lab
> >          issued: 2023-09-20 11:29:21 UTC
> >          expires: 2023-12-20 11:29:21 UTC
> > Request ID '20230920203130':
> >          status: MONITORING
> >          stuck: no
> >          key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
> >          certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=IPA RA,O=My Lab
> >          issued: 2023-09-20 14:21:36 UTC
> >          expires: 2025-09-09 14:21:36 UTC
> > Request ID '20230920203131':
> >          status: CA_UNREACHABLE
> >          ca-error: Error setting up ccache for "host" service on client
> > using default keytab: Cannot contact any KDC for requested realm.
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
> > Certificate DB',pinfile='/etc/dirsrv/slapd-MY-LAB/pwdfile.txt'
> >          certificate:
> > type=NSSDB,location='/etc/dirsrv/slapd-MY-LAB',nickname='Server-Cert',token='NSS
> > Certificate DB'
> >          CA: IPA
> >          issuer: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          subject: CN=ipa1.my.lab,O=My Lab
> >          issued: 2023-09-20 14:21:39 UTC
> >          expires: 2025-09-20 14:21:39 UTC
> > Request ID '20230920203134':
> >          status: CA_UNREACHABLE
> >          stuck: no
> >          key pair storage:
> > type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.my.lab-443-RSA'
> >          certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
> >          CA: IPA
> >          issuer: CN=My Lab Root CA,O=My Lab
> >          subject: CN=ipa1.my.lab,O=My Lab
> >          issued: 2023-07-10 04:31:16 UTC
> >          expires: 2023-08-09 04:31:16 UTC
> > Request ID '20230510042436':
> >          status: MONITORING
> >          ca-error: Updated certificate not available
> >          stuck: no
> >          key pair storage:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> > cert-pki-ca',token='NSS Certificate DB',pin set
> >          certificate:
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
> > cert-pki-ca',token='NSS Certificate DB'
> >          CA: dogtag-ipa-ca-renew-agent
> >          issuer: CN=My Lab Root CA,O=My Lab
> >          subject: CN=FreeIPA Root Certificate Authority,O=My Lab
> >          issued: 2022-08-10 04:29:31 UTC
> >          expires: 2023-08-10 04:29:31 UTC
> > Note that even the `MONITORING` ones are unusable due to the
> > non-overlapping date. So when I do something like `ipa-certupdate` I get:
> > `The ipa-certupdate command failed, exception: DatabaseError: Connect
> > error: error:0A000086:SSL routines::certificate verify failed (unable to
> > get issuer certificate)` when trying to use `ipaldap.py`.
> >
> > It is a mess, that's why I think it is better to have a way to simply
> > re-bootstrap all certificates.
>
> There isn't really a good way to start over.
>
> rob
> > On 2023/09/22 15:57, Florence Blanc-Renaud wrote:
> > > Hi,
> > >
> > > On Fri, Sep 22, 2023 at 12:36 PM Cristian Le  wrote:
> > >  Hi Florence,
> > >
> > >  Thanks for the feedback, let me clarify the situation on the
> > >  certificates:
> > >  - External CA is still valid and it is a self-signed certificate
> > >  that we use for other services. So we can manually sign any
> > >  service certificates to get them back up and running
> > >  - IPA CA is expired, let's say Aug/10
> > >  - I have managed to import a renewed IPA CA and ran `ipa-cert-fix`
> > >  (and also seemed to have run `ipa-certupdate`) on a current date,
> > >  let's say Sep/20. But not all services were recovered and now
> > >  there is no overlap between earliest date in service certificates
> > >  and the original IPA CA
> > >
> > > Which are the not-recovered services? Can you provide the output of
> > > "getcert list" at the current date?
> > > flo
> > >  - I have run a backup, but also did some system upgrades to get
> > >  the `ipa-cacert-manage prune` command, but when I've tried to
> > >  recover it, I've found that the backup was not there.
> > >
> > >   > you can still find the original certificates in the LDAP
> > >  database (below ou=certificateRepository,ou=ca,o=ipaca) but it
> > >  requires a bit of searching. You would need to restore the expired
> > >  certificates, go back in time and force the renewal.
> > >
> > >  I suspect we cannot do this all within LDAP right? If we get back
> > >  the expired certificates, how do we restore them in each service?
> > >  `httpd` is straightforward, and I guess `nssdb` should be doable,
> > >  assuming the same key is used, but is there another database type
> > >  where the certificates are located? Are all the certificates
> > >  tracked by `getcert list`? Is it safe to assume that after running
> > >  something like `ipa-cert-fix`, they are using the same private key?
> > >
> > >  Some symptoms in the current setup:
> > >  - When we are forward in time, `pki-tomcatd` is able to run, but
> > >  then I can't do any `ipa-cert-fix` or `ipa-cacert-manage renew`.
> > >  From what I've read, all of these commands (or at least
> > >  `ipa-cacert-manage renew`) must be done backwards in time.
> > >  - When we are backwards in time `pki-tomcatd` is unable to run,
> > >  failing to access `:8080/ca/admin/ca/getStatus`. This then blocks
> > >  various other services to be run. But about `ipactl restart`, only
> > >  `pki-tomcatd` service is actually failing (and ipa service itself
> > >  of course).
> > >
> > >  I have navigated to `ou=certificateRepository,ou=ca,o=ipaca` and
> > >  indeed there are still a bunch of certificates there in linear
> > >  order. What are the services I should look for in there? I am
> > >  using Apache Directory Studio and I can download the
> > >  `userCertificate`. Should I just run `certutil -A` with those
> > >  values with corresponding `subjectName`?
> > >
> > >  BTW, I want to document this process on the website, should I make
> > >  a PR on the github repo or is there somewhere else?
> > >
> > >  Kind regards,
> > >  Cristian
> > >
> > >  On 2023/09/22 9:00, Florence Blanc-Renaud wrote:
> > > >  Hi,
> > > >
> > > >  On Thu, Sep 21, 2023 at 5:04 PM Cristian Le via FreeIPA-users
> > > >  <[e-mail]> wrote:
> > > >
> > > >      I have tried my luck around with all the helpers: `pki-server
> > > >      cert-fix`, `ipa-cacert-manage`, `ipa-certupdate`, etc. but
> > > >      each one is failing on me for multiple reasons.
> > > >      - `ipa-cacert-manage` Cannot update the CA with
> > > >      `--external-cert-file` because the root ca is not detected to
> > > >      be in the trust list
> > > >
> > > >  This command is useful if you need to trust a new external CA or
> > > >  renew IPA CA. Is your IPA CA expired?
> > > >
> > > >      - `ipa-cert-fix` Was run without overlapping validity time,
> > > >      and the certificate were re-created, so now it is not
> > > >      recoverable, neither back in time, nor in current time
> > > >
> > > >  It is recommended to do a backup before running ipa-cert-fix. If
> > > >  you didn't, and want to try the back-in-time method, you can
> > > >  still find the original certificates in the LDAP database (below
> > > >  ou=certificateRepository,ou=ca,o=ipaca) but it requires a bit of
> > > >  searching. You would need to restore the expired certificates, go
> > > >  back in time and force the renewal.
> > > >
> > > >      - `pki-tomcat` is failing
> > > >
> > > >  What is the current situation? Which certs are expired (getcert
> > > >  list)? If you start the services with "ipactl start
> > > >  --ignore-service-failures", is pki-tomcat the only service failing?
> > > >  flo
> > > >
> > > >      It is quite a mess and I would like to ask for some guidance
> > > >      on how one could recover manually from  such dependency issues:
> > > >      - Is it possible to do a `ipa-server-install` and keep the
> > > >      user data?
> > > >
> > > >      - If I sign all of the service's certificates manually, what
> > > >      are all of the manual steps needed to get the services back
> > > >      up so that the helpers can be run.
> > > >        - I've tried to install the CA certificate in the nssdb
> > > >      database, ldap, and /etc/ipa/ca.crt. Are there other locations?
> > > >        - I've recreated an httpd certificate signed by the root,
> > > >      but I can't figure how to do the same with the ones located
> > > >      in the nssdb database, i.e. to recreate a csr with the same
> > > >      data as one of the certificates there
> > > >      - What is the order of services that should be updated. My
> > > >      understanding is CA -> `certutil`'s CA -> httpd + slapd +
> > > >      pki-tomcat (not sure where the last one is or how to edit it)
> > > >      -> `ipa-certupdate`
> > > >      _______________________________________________
> > > >      FreeIPA-users mailing list --
> > > >      [e-mail]
> > > >      To unsubscribe send an email to
> > > >      [e-mail]
> > > >      Fedora Code of Conduct:
> > > >      https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > >      List Guidelines:
> > > >      https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > >      List Archives:
> > > >      https://lists.fedorahosted.org/archives/list/[e-mail]
> > > >      Do not reply to spam, report it:
> > > >      https://pagure.io/fedora-infrastructure/new_issue
> >
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue