Hi,

On Wed, Feb 19, 2025 at 11:57 AM Boris <bb@kervyn.de> wrote:

Hi flo,

`ipa cert-show 1` works on both IPA hosts and returns correct data, from what I can tell. 

That's strange because cert-show is also authenticating to the CA REST API.

`ipa config-show` gies the following:

  IPA masters: ipa1.redacted, ipa2.redacted
  IPA master capable of PKINIT: ipa2.redacted
  IPA CA servers: ipa1.redacted
  IPA CA renewal master: ipa1.redacted
  IPA DNS servers: ipa1.redacted, ipa2.redacted

regarding the named crashes: I think the problem might be related to ldap. The last time the named daemons were in a restart/crash loop I restarted the ipa2 host which immediately resolved the problem.

ipa1:

bind-9.18.19-1.fc37.x86_64

bind-dyndb-ldap-11.10-17.fc37.x86_64

ipa2:

bind-9.16.28-1.fc35.x86_64

bind-dyndb-ldap-11.9-12.fc35.x86_64

for the coredump I would need your guidance what to do, because I am not that firm with named debugging.

Here are the last couple of line from the /var/log/ipaupgrade.log file. The update seems to go through, but it fails when it needs to authenticate with the CA REST API

2025-01-21T20:24:14Z DEBUG stderr=
2025-01-21T20:24:14Z DEBUG Starting external process
2025-01-21T20:24:14Z DEBUG args=['/bin/systemctl', 'start', 'certmonger.service']
2025-01-21T20:24:15Z DEBUG Process finished, return code=0
2025-01-21T20:24:15Z DEBUG stdout=
2025-01-21T20:24:15Z DEBUG stderr=
2025-01-21T20:24:15Z DEBUG Starting external process
2025-01-21T20:24:15Z DEBUG args=['/bin/systemctl', 'is-active', 'certmonger.service']
2025-01-21T20:24:15Z DEBUG Process finished, return code=0
2025-01-21T20:24:15Z DEBUG stdout=active

2025-01-21T20:24:15Z DEBUG stderr=
2025-01-21T20:24:15Z DEBUG Start of certmonger.service complete
2025-01-21T20:24:15Z DEBUG Starting external process
2025-01-21T20:24:15Z DEBUG args=['pki-server', 'subsystem-show', 'kra']
2025-01-21T20:24:15Z DEBUG Process finished, return code=1
2025-01-21T20:24:15Z DEBUG stdout=
2025-01-21T20:24:15Z DEBUG stderr=ERROR: ERROR: No kra subsystem in instance pki-tomcat.

2025-01-21T20:24:15Z INFO [Update certmonger certificate renewal configuration]
2025-01-21T20:24:15Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2025-01-21T20:24:15Z DEBUG Starting external process
2025-01-21T20:24:15Z DEBUG args=['/usr/bin/certutil', '-d', 'sql:/etc/dirsrv/slapd-redacted/', '-L', '-n', 'Server-Cert', '-a', '-f', '/etc/dirsrv/slapd-redacted/pwdfile.txt']
2025-01-21T20:24:15Z DEBUG Process finished, return code=0
2025-01-21T20:24:15Z DEBUG stdout=-----BEGIN CERTIFICATE-----
redacted
-----END CERTIFICATE-----

2025-01-21T20:24:15Z DEBUG stderr=
2025-01-21T20:24:15Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
2025-01-21T20:24:15Z DEBUG Starting external process
2025-01-21T20:24:15Z DEBUG args=['/usr/bin/certutil', '-d', 'sql:/etc/pki/pki-tomcat/alias', '-L', '-f', '/etc/pki/pki-tomcat/alias/pwdfile.txt']
2025-01-21T20:24:15Z DEBUG Process finished, return code=0
2025-01-21T20:24:15Z DEBUG stdout=
Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

caSigningCert cert-pki-ca                                    CTu,Cu,Cu
caSigningCert cert-pki-ca 6148bb27-6bd6-4a0a-b607-6ba538a6c401 u,u,u
ocspSigningCert cert-pki-ca                                  u,u,u
subsystemCert cert-pki-ca                                    u,u,u
auditSigningCert cert-pki-ca                                 u,u,Pu
Server-Cert cert-pki-ca                                      u,u,u

2025-01-21T20:24:15Z DEBUG stderr=
2025-01-21T20:24:15Z INFO Certmonger certificate renewal configuration already up-to-date
2025-01-21T20:24:15Z INFO [Enable PKIX certificate path discovery and validation]
2025-01-21T20:24:15Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state'
2025-01-21T20:24:15Z INFO PKIX already enabled
2025-01-21T20:24:15Z INFO [Authorizing RA Agent to modify profiles]
2025-01-21T20:24:15Z INFO [Authorizing RA Agent to manage lightweight CAs]
2025-01-21T20:24:15Z INFO [Ensuring Lightweight CAs container exists in Dogtag database]
2025-01-21T20:24:15Z INFO [Adding default OCSP URI configuration]
2025-01-21T20:24:15Z INFO [Disabling cert publishing]
2025-01-21T20:24:15Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2025-01-21T20:24:15Z INFO [Migrating certificate profiles to LDAP]
2025-01-21T20:24:15Z DEBUG Profile 'AdminCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'DomainController' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'ECAdminCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'acmeServerCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caAdminCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caAgentFileSigning' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caAgentServerCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caAuditSigningCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCACert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCMCECUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCMCECserverCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCMCECsubsystemCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCMCUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caCrossSignedCACert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDirBasedDualCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDirPinUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDirUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDirUserRenewal' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDualCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caDualRAuserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caECAdminCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caECAgentServerCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caECDirPinUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caECDirUserCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z DEBUG Profile 'caECDualCert' is already in LDAP and enabled; skipping
2025-01-21T20:24:15Z INFO Migrating profile 'caECFullCMCSharedTokenCert'
2025-01-21T20:24:15Z DEBUG request GET https://ipa1.redacted:8443/ca/rest/account/login
2025-01-21T20:24:15Z DEBUG request body ''
2025-01-21T20:24:16Z DEBUG response status 404
2025-01-21T20:24:16Z DEBUG response headers Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 784
Date: Tue, 21 Jan 2025 20:24:16 GMT


2025-01-21T20:24:16Z DEBUG response body (decoded): b'<!doctype html><html lang="en"><head><title>HTTP Status 404 \xe2\x80\x93 Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 \xe2\x80\x93 Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> The requested resource [&#47;ca&#47;rest&#47;account&#47;login] is not available</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/9.0.82</h3></body></html>'
2025-01-21T20:24:16Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2025-01-21T20:24:16Z DEBUG   File "/usr/lib/python3.11/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
                   ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
    server.upgrade()
  File "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", line 2061, in upgrade
    upgrade_configuration()
  File "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", line 1914, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py", line 458, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", line 2155, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py", line 2209, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python3.11/site-packages/ipaserver/plugins/dogtag.py", line 1211, in __enter__
    raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))

Can you check if the CA subsystem is enabled?

# pki-server subsystem-show ca
  Subsystem ID: ca
  Instance ID: pki-tomcat
  Enabled: True

If yes, try to authenticate to the rest API with curl:

# curl  --cert /var/lib/ipa/ra-agent.pem --key /var/lib/ipa/ra-agent.key https://`hostname`:8443/ca/rest/account/login
{"id":"ipara","FullName":"ipara","Roles":["Certificate Manager Agents","Enterprise ACME Administrators","Registration Manager Agents","Security Domain Administrators"],"Attributes":{"Attribute":[]}}

If the above commands are working, retry the upgrade with

# ipa-server-upgrade

and send us the full /var/log/ipaupgrade.log and /var/log/pki/pki-tomcat/ca/debug.$DATE.log.

flo

2025-01-21T20:24:16Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2025-01-21T20:24:16Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
2025-01-21T20:24:16Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Am Mi., 19. Feb. 2025 um 10:56 Uhr schrieb Florence Blanc-Renaud <flo@redhat.com>:

Hi,

in a previous message you mentioned that the directory manager password is lost. You can follow this article to reset the DM password: https://access.redhat.com/solutions/203473

Named crashes could be related to multiple issues:

- inconsistent versions between bind and bind-dyndb-ldap. Which versions do you have?

- an insufficient number of threads

- an issue when reloading the zones

If you can gather a coredump and install the debug packages it could help identify if you're hitting a known issue.

You mentioned that ipa1 needs to be started with --force, can you tell which service is failing and provide the logs? There should be also more information in /var/log/ipaupgrade.log.

In order to check the CA state, a useful command is 'ipa cert-show 1' as it communicates with the CA to gather the certificate details. If this command is failing (likely with "Failed to Authenticate to CA rest API") you need to understand where the config is broken.

Start by checking which system is the CA renewal master:

ipa config-show

The CA renewal master will be your priority.

flo

On Wed, Feb 19, 2025 at 10:25 AM Boris via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

I think the CA is working, but I don't know for sure and how to verify it. At least there are no expired certs on both ipa hosts

[root@ipa1 ~]# getcert list | grep expires
expires: 2025-11-29 13:19:40 CET
expires: 2025-04-15 16:27:34 CEST
expires: 2025-04-15 16:26:44 CEST
expires: 2025-04-15 16:27:14 CEST
expires: 2037-08-19 16:11:12 CEST
expires: 2025-04-15 16:27:54 CEST
expires: 2025-04-15 16:27:04 CEST
expires: 2040-02-12 12:46:50 CET
expires: 2025-05-29 16:12:51 CEST
expires: 2026-01-26 13:48:23 CET

[root@ipa2 ~]# getcert list | grep expires
expires: 2027-02-16 10:42:29 CET
expires: 2027-02-16 10:42:51 CET
expires: 2025-04-15 16:27:04 CEST
expires: 2027-02-16 10:43:26 CET

The healthcheck showed some "group is not correct" and "files are to permissive" which I resolved.

Now I have these to checks which do not tell me anything

      "msg": "certmonger tracking request {key} found and is not expected on an IPA master."
      "msg": "No KDC workers defined in {sysconfig}"

Am Di., 18. Feb. 2025 um 15:22 Uhr schrieb Rob Crittenden via FreeIPA-users <freeipa-users@lists.fedorahosted.org>:

Boris wrote:
> Hi Rob,
>
> I have two hosts: ipa1 and ipa2
>
> ipa1:
> Fedora 37
> freeipa-server-4.10.1-1.fc37.x86_64
> Managed suffixes: domain, ca
> running with ipactl start --force because the update is not working (The
> ipa-server-upgrade command failed, exception: RemoteRetrieveError:
> Failed to authenticate to CA REST API).
> I tried to upgrade, but the upgrade did not go through.

Your existing CA is having issues. I'd start by checking that your CA
certificates are still valid: getcert list | grep expires

You might also try installing the freeipa-healthcheck package and
running ipa-healthcheck. Expect a lot of errors since it won't be able
to connect to the CA but it will also check the validity dates, etc.

> ipa2:
> Fedora 35
> freeipa-server-4.9.11-1.fc35.x86_64
> Managed suffixes: domain
>
> So my thought process was: if it can not authenticate against the CA
> REST API, I need to add the CA capability to ipa2

You need to authenticate to the CA to create a clone of it. You can't
install another CA until you get your existing one working.

rob

--
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

--

Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groüen Saal.