Comments in-line.
Dmitry Krasov via FreeIPA-users wrote:
it's ubuntu 16.04.7. Freeipa 4.3.1-0ubuntu1 which packages do you need else?
That's enough.
I'm was under the impression that Ubuntu never worked with renewal though your certificates seem to have been renewed at least once so maybe there is a glimmer of hope.
I forget if you tried ipa-cert-fix or not. If not I'd give that a shot. It will attempt to renew the CA subsystem certificates off-line. I assume you tried going back in time to November 18, 2024 and that seems to have renewed two certificates but no the CA subsystem certificates.
If you want to try that again you can. You need to stop any time service, go back in time, restart all of IPA, then certmonger, then give certmonger a chance to try to renew the certificates. If it fails then I'd need to see the journal and PKI debug log.
If it ends up being unrecoverable there is no "get a new CA" option. The only option is a re-install which will be very intrusive. For that you have three main options.
1. Use ipa migrate-ds to migrate only users and groups to a new IPA server. This is documented in the official docs but it isn't ideal because you lose all HBAC, sudo rules, private groups become POSIX groups and more.
2. Export to LDIF, manually massage the data and re-import into a newly installed IPA server. This requires pretty deep understanding of the data but mostly you need to remove any private key material and need to be careful not to overwrite certain entries. It can be prone to error and it's unlikely something we would work out over an e-mail.
3. Install a replacement server in Fedora 41 and use the ipa-migrate command to pull all the data over that way. It is also overwhelming because you'll need to re-enroll all clients, migrate all user passwords and depending on how custom your environment is potentially re-create some manual keytabs and certificates.
#3 is the recommendation if you can't get your server working.
If you do somehow get it working then the recommendation would be to get off Ubuntu as quickly as possible. It was not well supported in 2016 much less today.
getcert list:
Request ID '20221130052539': status: MONITORING ca-error: Server at "https://ipa.dom.loc:8443/ca/agent/ca/profileProcess" replied: 1: You did not provide a valid certificate for this operation stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=CA Audit,O=DOM.LOC expires: 2024-11-19 05:25:15 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20221130052540': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=OCSP Subsystem,O=DOM.LOC expires: 2024-11-19 05:25:14 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20221130052541': status: MONITORING ca-error: Server at "https://ipa.dom.loc:8443/ca/agent/ca/profileProcess" replied: 1: You did not provide a valid certificate for this operation stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=CA Subsystem,O=DOM.LOC expires: 2024-11-19 05:25:14 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca" track: yes auto-renew: yes Request ID '20221130052542': status: MONITORING ca-error: Server at "https://ipa.dom.loc:8443/ca/agent/ca/profileProcess" replied: 1: You did not provide a valid certificate for this operation stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=Certificate Authority,O=DOM.LOC expires: 2042-11-30 05:25:14 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20221130052543': status: MONITORING ca-error: Server at "https://ipa.dom.loc:8443/ca/agent/ca/profileProcess" replied: 1: You did not provide a valid certificate for this operation stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=IPA RA,O=DOM.LOC expires: 2024-11-19 05:25:36 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/lib/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/lib/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20221130052544': status: MONITORING ca-error: Server at "https://ipa.dom.loc:8443/ca/agent/ca/profileProcess" replied: 1: You did not provide a valid certificate for this operation stuck: no key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=ipa.dom.loc,O=DOM.LOC expires: 2024-11-19 05:25:14 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth pre-save command: /usr/lib/ipa/certmonger/stop_pkicad post-save command: /usr/lib/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20221130052605': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOM-LOC',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOM-LOC/pwdfile.txt' certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOM-LOC',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=ipa.dom.loc,O=DOM.LOC expires: 2026-11-18 20:02:32 UTC principal name: ldap/ipa.dom.loc@DOM.LOC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_dirsrv DOM-LOC track: yes auto-renew: yes Request ID '20221130052625': status: MONITORING stuck: no key pair storage: type=NSSDB,location='/etc/apache2/nssdb',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt' certificate: type=NSSDB,location='/etc/apache2/nssdb',nickname='Server-Cert',token='NSS Certificate DB' CA: IPA issuer: CN=Certificate Authority,O=DOM.LOC subject: CN=ipa.dom.loc,O=DOM.LOC expires: 2026-11-18 20:02:42 UTC principal name: HTTP/ipa.dom.loc@DOM.LOC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib/ipa/certmonger/restart_httpd track: yes auto-renew: yes
ipa-cacert-manage renew -v:
ipa: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' ipa.ipaserver.install.ipa_cacert_manage.CACertManage: DEBUG: Not logging to a file ipa: DEBUG: importing all plugin modules in ipalib.plugins... ipa: DEBUG: importing plugin module ipalib.plugins.aci ipa: DEBUG: importing plugin module ipalib.plugins.automember ipa: DEBUG: importing plugin module ipalib.plugins.automount ipa: DEBUG: importing plugin module ipalib.plugins.baseldap ipa: DEBUG: importing plugin module ipalib.plugins.baseuser ipa: DEBUG: importing plugin module ipalib.plugins.batch ipa: DEBUG: importing plugin module ipalib.plugins.caacl ipa: DEBUG: importing plugin module ipalib.plugins.cert ipa: DEBUG: importing plugin module ipalib.plugins.certprofile ipa: DEBUG: importing plugin module ipalib.plugins.config ipa: DEBUG: importing plugin module ipalib.plugins.delegation ipa: DEBUG: importing plugin module ipalib.plugins.dns ipa: DEBUG: importing plugin module ipalib.plugins.domainlevel ipa: DEBUG: importing plugin module ipalib.plugins.group ipa: DEBUG: importing plugin module ipalib.plugins.hbacrule ipa: DEBUG: importing plugin module ipalib.plugins.hbacsvc ipa: DEBUG: importing plugin module ipalib.plugins.hbacsvcgroup ipa: DEBUG: importing plugin module ipalib.plugins.hbactest ipa: DEBUG: importing plugin module ipalib.plugins.host ipa: DEBUG: importing plugin module ipalib.plugins.hostgroup ipa: DEBUG: importing plugin module ipalib.plugins.idrange ipa: DEBUG: importing plugin module ipalib.plugins.idviews ipa: DEBUG: importing plugin module ipalib.plugins.internal ipa: DEBUG: importing plugin module ipalib.plugins.krbtpolicy ipa: DEBUG: importing plugin module ipalib.plugins.migration ipa: DEBUG: importing plugin module ipalib.plugins.misc ipa: DEBUG: importing plugin module ipalib.plugins.netgroup ipa: DEBUG: importing plugin module ipalib.plugins.otpconfig ipa: DEBUG: importing plugin module ipalib.plugins.otptoken ipa: DEBUG: importing plugin module ipalib.plugins.otptoken_yubikey ipa: DEBUG: importing plugin module ipalib.plugins.passwd ipa: DEBUG: importing plugin module ipalib.plugins.permission ipa: DEBUG: importing plugin module ipalib.plugins.ping ipa: DEBUG: importing plugin module ipalib.plugins.pkinit ipa: DEBUG: importing plugin module ipalib.plugins.privilege ipa: DEBUG: importing plugin module ipalib.plugins.pwpolicy ipa: DEBUG: Starting external process ipa: DEBUG: args=klist -V ipa: DEBUG: Process finished, return code=0 ipa: DEBUG: stdout=Kerberos 5 version 1.13.2
ipa: DEBUG: stderr= ipa: DEBUG: importing plugin module ipalib.plugins.radiusproxy ipa: DEBUG: importing plugin module ipalib.plugins.realmdomains ipa: DEBUG: importing plugin module ipalib.plugins.role ipa: DEBUG: importing plugin module ipalib.plugins.rpcclient ipa: DEBUG: importing plugin module ipalib.plugins.selfservice ipa: DEBUG: importing plugin module ipalib.plugins.selinuxusermap ipa: DEBUG: importing plugin module ipalib.plugins.server ipa: DEBUG: importing plugin module ipalib.plugins.service ipa: DEBUG: importing plugin module ipalib.plugins.servicedelegation ipa: DEBUG: importing plugin module ipalib.plugins.session ipa: DEBUG: importing plugin module ipalib.plugins.stageuser ipa: DEBUG: importing plugin module ipalib.plugins.sudocmd ipa: DEBUG: importing plugin module ipalib.plugins.sudocmdgroup ipa: DEBUG: importing plugin module ipalib.plugins.sudorule ipa: DEBUG: importing plugin module ipalib.plugins.topology ipa: DEBUG: importing plugin module ipalib.plugins.trust ipa: DEBUG: importing plugin module ipalib.plugins.user ipa: DEBUG: importing plugin module ipalib.plugins.vault ipa: DEBUG: importing plugin module ipalib.plugins.virtual ipa: DEBUG: importing all plugin modules in ipaserver.plugins... ipa: DEBUG: importing plugin module ipaserver.plugins.dogtag ipa: DEBUG: importing plugin module ipaserver.plugins.join ipa: DEBUG: importing plugin module ipaserver.plugins.ldap2 ipa: DEBUG: importing plugin module ipaserver.plugins.rabase ipa: DEBUG: importing plugin module ipaserver.plugins.xmlserver ipa.ipalib.session.SessionAuthManager: DEBUG: SessionAuthManager.register: name=jsonserver_session_140159754316752 ipa.ipalib.session.SessionAuthManager: DEBUG: SessionAuthManager.register: name=xmlserver_session_140159754359568 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.xmlserver() at '/xml' ipa.ipaserver.rpcserver.xmlserver: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.xmlserver_session() at '/session/xml' ipa.ipaserver.rpcserver.xmlserver_session: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.xmlserver_session: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.login_password() at '/session/login_password' ipa.ipaserver.rpcserver.login_password: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.change_password() at '/session/change_password' ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.jsonserver_session() at '/session/json' ipa.ipaserver.rpcserver.jsonserver_session: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.sync_token() at '/session/sync_token' ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.jsonserver_kerb() at '/json' ipa.ipaserver.rpcserver.jsonserver_kerb: DEBUG: session_auth_duration: 0:20:00 ipa.ipaserver.rpcserver.wsgi_dispatch: DEBUG: Mounting ipaserver.rpcserver.login_kerberos() at '/session/login_kerberos' ipa.ipaserver.rpcserver.login_kerberos: DEBUG: session_auth_duration: 0:20:00 ipa: DEBUG: Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' ipa.ipaserver.install.ipa_cacert_manage.CACertManage: DEBUG: Found certmonger request id dbus.String(u'20221130052542', variant_level=1) ipa: DEBUG: Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' ipa: DEBUG: Starting external process ipa: DEBUG: args=/usr/bin/certutil -d /etc/pki/pki-tomcat/alias -L -n caSigningCert cert-pki-ca -a ipa: DEBUG: Process finished, return code=0 ipa: DEBUG: stdout=-----BEGIN CERTIFICATE----- MIIDfzCCAmegAwIBAgIBATANBgkqhkiG9w0BAQsFADAyMRAwDgYDVQQKDAdET00u TE9DMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMjIxMTMwMDUy NTE0WhcNNDIxMTMwMDUyNTE0WjAyMRAwDgYDVQQKDAdET00uTE9DMR4wHAYDVQQD DBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAw ggEKAoIBAQDScx6Ah9lD3MZ9Y/FnmC2BuM1l5mbaDo6n8ke07So+J2ryG13kKWf6 eGyaMiFf3o6bi9zTB2gDlIWDAgjsjYeVo7dz3dO+DM4o57C8OYGecySsJ3VSsYTs utNNKxqMprOxqNB2ascwLiR6Oy2NWzOFtg0ZP4GBW1uqv26cYl0s28CcL1xU+Rnh FsXTtn5yGdkUKPj9vBFxiQI11ILV+mp58NmIddqjjzsXzHrAJ7+v7EcVS1tlZvLA bfgWVgaHE1GNdmL7DzkBtrIX6nwzVhbVFhKpYAAGJUPHFS9yMxgwGFejkVmyFOzG o/cwikq699YHujpgPLej98BM6e9VIpxvAgMBAAGjgZ8wgZwwHwYDVR0jBBgwFoAU CBaGdFi3XREanbDOr1fXZH4KKakwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8E BAMCAcYwHQYDVR0OBBYEFAgWhnRYt10RGp2wzq9X12R+CimpMDkGCCsGAQUFBwEB BC0wKzApBggrBgEFBQcwAYYdaHR0cDovL2lwYS5kb20ubG9jOjgwL2NhL29jc3Aw DQYJKoZIhvcNAQELBQADggEBAEDtgTehcANC+hTvgxXsV6tboYBAza6+Gvs+jQd4 2LfBwZNJClqTL0F2u2vUBH6m4gaUMWmPoP6bwqFJ7Yw+ZT04DlGpt0JyaVfP8zAU FV3k9fygY9Qk6+WGyIi172uB+7GR7CIDT90cGftq3RqF5kapnbRXmT46RHNIC2gB /Ld/fG4SPWwmSB91YPbiaRJcWdCC2QZsn7i2pikqyOfn7m9Oim8HZhd4/t1TMezD +AJcfwCkWyqaLZPGwvdt8gf6vk7DR+FYIvmLxGbhrmS3yfuBmcJ8LgCKK5QtMXUo FNc869oM4O6QoH87gzef9Lu9LrbWH23V7LH33G0aY1v5Jxs= -----END CERTIFICATE-----
ipa: DEBUG: stderr= Renewing CA certificate, please wait ipa.ipapython.ipaldap.SchemaCache: DEBUG: flushing ldapi://%2fvar%2frun%2fslapd-DOM-LOC.socket from SchemaCache ipa.ipapython.ipaldap.SchemaCache: DEBUG: retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-DOM-LOC.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f797c741248> ipa.ipaserver.install.ipa_cacert_manage.CACertManage: DEBUG: resubmitting certmonger request '20221130052542' ipa: DEBUG: certmonger request is in state dbus.String(u'GENERATING_CSR', variant_level=1) ipa: DEBUG: certmonger request is in state dbus.String(u'MONITORING', variant_level=1) ipa.ipaserver.install.ipa_cacert_manage.CACertManage: DEBUG: File "/usr/lib/python2.7/dist-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/dist-packages/ipaserver/install/ipa_cacert_manage.py", line 114, in run rc = self.renew() File "/usr/lib/python2.7/dist-packages/ipaserver/install/ipa_cacert_manage.py", line 172, in renew return self.renew_self_signed(ca) File "/usr/lib/python2.7/dist-packages/ipaserver/install/ipa_cacert_manage.py", line 184, in renew_self_signed self.resubmit_request(ca, 'caCACert') File "/usr/lib/python2.7/dist-packages/ipaserver/install/ipa_cacert_manage.py", line 314, in resubmit_request "please check the request manually" % self.request_id)
ipa.ipaserver.install.ipa_cacert_manage.CACertManage: DEBUG: The ipa-cacert-manage command failed, exception: ScriptError: Error resubmitting certmonger request '20221130052542', please check the request manually ipa.ipaserver.install.ipa_cacert_manage.CACertManage: ERROR: Error resubmitting certmonger request '20221130052542', please check the request manually ipa.ipaserver.install.ipa_cacert_manage.CACertManage: ERROR: The ipa-cacert-manage command failed.
This was the wrong command to run. It does not renew the subsystem certs. It attempts to renew the CA. It is lucky that it failed.
rob