ipacerts expired

[Omar Pagan]

All my certs in IPA are expired and no matter what I do I can't get `getcert` to renew them.  I have changed the date back to before they expired but when I try to restart IPA is trying to do an upgrade and fails.
I'm able to start kdc, directory services, http, pki-tomcat and certmonger, but when I try to resubmit a cert for renewal it complains about not connecting to dbus.
Please help, I need to get this IPA service up and running and I can't figure out what's wrong.

[Florence Blanc-Renaud]

Hi,

which version are you using? ipa-cert-fix is available since IPA 4.6.6 and
can help you renew expired certs.
The doc is available at
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...

flo

On Mon, Mar 20, 2023 at 2:23 PM Omar Pagan via FreeIPA-users <
[e-mail]> wrote:
> All my certs in IPA are expired and no matter what I do I can't get
> `getcert` to renew them.  I have changed the date back to before they
> expired but when I try to restart IPA is trying to do an upgrade and fails.
> I'm able to start kdc, directory services, http, pki-tomcat and
> certmonger, but when I try to resubmit a cert for renewal it complains
> about not connecting to dbus.
> Please help, I need to get this IPA service up and running and I can't
> figure out what's wrong.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Omar Pagan]

I'm running version 4.6.8 and it does have the ipa-cert-fix.  But when I run it, I get this errors:
cannot connect to 'ldapi:<URL>.socket':
The api-cert-fix command failed.

Thoughts?  Thank you

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> I'm running version 4.6.8 and it does have the ipa-cert-fix.  But when I run it, I get this errors:
> cannot connect to 'ldapi:<URL>.socket':
> The api-cert-fix command failed.
>
> Thoughts?  Thank you

It means that 389-ds is not running.

rob

[Omar Pagan]

okay, now I am getting the following error:

Command: `pki-server cert-fix --ldapi-socket /var/run/slapd-<hsotname>.socket --agent-uid ipara --cert sslserver --cert subsystem --cert ca_ocsp_signing --cert ca_audit_signing --extra-cert 6' returned non-zero exit status 1
The ipa-cert-fix command failed.

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> okay, now I am getting the following error:
>
> Command: `pki-server cert-fix --ldapi-socket /var/run/slapd-<hsotname>.socket --agent-uid ipara --cert sslserver --cert subsystem --cert ca_ocsp_signing --cert ca_audit_signing --extra-cert 6' returned non-zero exit status 1
> The ipa-cert-fix command failed.

We need more context to understand what is happening. I'd recommend
running with the -v option as well (verbose).

rob

[Omar Pagan]

I'm trying to clean up the verbose logs, but I see four issues:
1. certutil: Could not find cert: trasnportCert cert-pki-kra
2. certutil: Could not find cert: storageCert cert-pki-kra
3. certutil: Could not find cert: auditSigningCert cert-pki-kra
4. Failed to update password 
    This one is right before it shows the following error:
ERROR:  Command '['ldappasswd', '-H', 'ldapi://<URL>.socket', '-Y', 'EXTERNAL', '-T', '/tmp/tmp5VRd4o', 'uid=pkidbuser,ou=people,o=ipaca']' returned non-zero exit status 1

Thoughts?

[Florence Blanc-Renaud]

Hi,

On Tue, Mar 21, 2023 at 2:53 PM Omar Pagan via FreeIPA-users <
[e-mail]> wrote:
> I'm trying to clean up the verbose logs, but I see four issues:
> certutil: Could not find cert: trasnportCert cert-pki-kra
> certutil: Could not find cert: storageCert cert-pki-kra
> certutil: Could not find cert: auditSigningCert cert-pki-kra

You can ignore the above 3 warnings if you didn't install the KRA on this
server.
> Failed to update password
>  This one is right before it shows the following error:
>
> ERROR:  Command '['ldappasswd', '-H', 'ldapi://<URL>.socket', '-Y',
> 'EXTERNAL', '-T', '/tmp/tmp5VRd4o', 'uid=pkidbuser,ou=people,o=ipaca']'
> returned non-zero exit status 1
>
> Thoughts?

Is the directory server running? You can run "ipactl status" to check if
all IPA services are running and launch them with "ipactl start
--ignore-service-failures".

flo
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Omar Pagan]

Hello flo,

Thanks everyone for the support.  I have tried to start the service and I will like to attach the errors I'm getting.  Please review attachments.  Let me know what you think I should do.

[Omar Pagan]

Sorry, here is the link for the paste errors:

https://justpaste.it/57k4t

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> Sorry, here is the link for the paste errors:
>
> https://justpaste.it/57k4t

Add --skip-version-check to the ipactl invocation to skip the upgrade.

rob

[Omar Pagan]

Thanks, I got all the services up and running, yet I can't get the certs to renew.

When I look at certmonger it seems to be having dbus connection issues.  Are those normal?  I have tried to use the `resubmit` option for the certs ID but that doesn't seem to work.

Thoughts?

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> Thanks, I got all the services up and running, yet I can't get the certs to renew.
>
> When I look at certmonger it seems to be having dbus connection issues.  Are those normal?  I have tried to use the `resubmit` option for the certs ID but that doesn't seem to work.
>
> Thoughts?

It's hard to have any without any sort of logs or output. We need to see
what you're seeing to understand what is happening.

The clearer the steps of what you've done and what you're seeing the
easier it is to help.

rob

[Omar Pagan]

Hello guys,
The team was trying some new things and we got some errors we would like to share:
ERR - _csngen_adjust_local_time - Adjustment limit exceeded; value - ####, limit - ####  (I'm not sure if you care to see the actual numbers)

ERR - ldbm_back_modify - failed to generate modify CSN for entry (cn=MasterCRL,ou=crlIssuingPoints,ou=ca,o=ipaca), aborting operation

After some google searches we found the following links, but want to validate with you guys the steps are what we need.  Here are some of those links we have found:

We have perform the following steps following this link:  https://lists.fedoraproject.org/archives/list/[e-mail]...

The steps are (for the case where your certs are still valid):

1. Stop certmonger
2. grep dogtag-ipa-ca-renew-agent /var/lib/certmonger/cas/*
3. There should be two. You want the one with "id=dogtag-ipa-ca-renew-agent"
4. Modify that file and add -N to ca_external_helper. It needs to look like:

ca_external_helper=/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -N

We have also found the following link, but not perform the suggested steps.

https://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-tim...

Since the only way to get the service back is to set the local time and date back to a time before the certs expired, do you know of any way to resolve the clock skew problem with the directory service?  Other than what is suggested in the link above?

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> Hello guys,
> The team was trying some new things and we got some errors we would like to share:
> ERR - _csngen_adjust_local_time - Adjustment limit exceeded; value - ####, limit - ####  (I'm not sure if you care to see the actual numbers)
>
> ERR - ldbm_back_modify - failed to generate modify CSN for entry (cn=MasterCRL,ou=crlIssuingPoints,ou=ca,o=ipaca), aborting operation
>
> After some google searches we found the following links, but want to validate with you guys the steps are what we need.  Here are some of those links we have found:
>
> We have perform the following steps following this link:  https://lists.fedoraproject.org/archives/list/[e-mail]...
>
> The steps are (for the case where your certs are still valid):
> Stop certmonger
> grep dogtag-ipa-ca-renew-agent /var/lib/certmonger/cas/*
> There should be two. You want the one with "id=dogtag-ipa-ca-renew-agent"
> Modify that file and add -N to ca_external_helper. It needs to look like:
>
> ca_external_helper=/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit -N

Yes.
> We have also found the following link, but not perform the suggested steps.
>
> https://directory.fedoraproject.org/docs/389ds/howto/howto-fix-and-reset-tim...
>
> Since the only way to get the service back is to set the local time and date back to a time before the certs expired, do you know of any way to resolve the clock skew problem with the directory service?  Other than what is suggested in the link above?

I'd worry about the certificates first. Worst case is you re-initialize
the other replicas from the data on the renewal master.

rob

[Omar Pagan]

not sure I follow your answers, can you clarify what I should be doing to get those Errors or the `clock skew` issue resolved?

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> not sure I follow your answers, can you clarify what I should be doing to get those Errors or the `clock skew` issue resolved?

Focus on one problem at a time. The expired certificates are more
critical IMHO.

rob

[Omar Pagan]

but it seems that I'm getting the clock skew error for the directory service every time I try to resubmit the cert renewal because the rolling back of the date/time to the local server is affecting the clock for the directory service.  I think that's causing my renewals to fail.

[Florence Blanc-Renaud]

Hi Omar,
can you give us more information? How many servers/replicas do you have,
and on how many do you have expired certs?

The repair procedure must start on the server that is currently CA master.
You can find which one is CA master by using "ipa config-show | grep
renewal". Warning, if the replication is broken the result may be different
on different servers. In this case, pick the server that you want to use as
source of data and perform the repair steps on this server.

I am not sure if you tried ipa-cert-fix or the method changing the date
into the past. In any case, try to repair one server first and the replicas
can be re-initialized later with the data from this server.

Can you provide the output of "getcert list" on this server? It will help
us identify which certs need to be renewed.

flo

On Fri, Mar 31, 2023 at 10:55 PM Omar Pagan via FreeIPA-users <
[e-mail]> wrote:
> but it seems that I'm getting the clock skew error for the directory
> service every time I try to resubmit the cert renewal because the rolling
> back of the date/time to the local server is affecting the clock for the
> directory service.  I think that's causing my renewals to fail.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Omar Pagan]

Hello Flo,

We have three (3) servers and two of them are replicas.

From the cli:  
# `ipa-getcert list` shows two certs both expired,
# `getcert list` shows 8 certs, 7 of those expired.

We are working from the CA master and trying everything we have listed above.  We tried the ipa-cert-fix too, time rolled back and everything done on the CA master, but nothing worked.

[Rob Crittenden]

Omar Pagan via FreeIPA-users wrote:
> Hello Flo,
>
> We have three (3) servers and two of them are replicas.
>
> From the cli:  
> # `ipa-getcert list` shows two certs both expired,
> # `getcert list` shows 8 certs, 7 of those expired.
>
> We are working from the CA master and trying everything we have listed above.  We tried the ipa-cert-fix too, time rolled back and everything done on the CA master, but nothing worked.

We need to see what you are seeing in order to help. The getcert output,
the journal output after resubmitting (and failing), any related
logging, the status of the services prior to doing the resubmit and/or
ipa-cert-fix, ipa config-show output, etc.

rob

PS ipa-getcert is shorthand for getcert -c IPA which is a subset of the
certificates. It is a subset of the getcert output.