Russell Long wrote:
Here is the obfuscated sosreport.
It looks like the CA restart that happened immediately before the first 404 was successful. At least it doesn't report any errors beyond the usual LDAP connection failures at startup.
The startup looks to be basically done around 2025-02-03 14:37:56
The CA returned 404's at 2025-02-03T19:39:34Z in the upgrade log.
We don't have the tomcat access logs in the sosreport so we can't see the requests but they would likely also report 404 and no other details.
If you manually start things can you communicate with the CA?
# ipactl --skip-version-check
Does the CA start? If not add --ignore-service-failures
Once everything else is up and settled, if the CA start failed run: systemctl restart pki-tomcatd@pki-tomcat
And see if that is successful. I think it should succeed since it appears to have done so in the recent past.
If so try a basic cert command: # openssl x509 -serial -noout -in /etc/ipa/ca.crt # ipa cert-show <that serial number>
Does it give you data or a connection error?
rob
On Wed, Feb 5, 2025 at 2:33 AM Alexander Bokovoy <abokovoy@redhat.com mailto:abokovoy@redhat.com> wrote:
On Пан, 03 лют 2025, Russell Long via FreeIPA-users wrote: >Here is the log, sorry for the delay. Logs are redacted, but the only thing >changed was the domain names and DNs. The upgrade log chokes on the CA application not being registered in tomcat container (the corresponding /ca/rest/... path is giving 404 error). So we get back to the same point as before. An upgrade has been in progress but somehow was interrupted. Directory server was having some of listeners disabled to avoid external communication during the upgrade and those listeners weren't recovered due to an interruption. You recovered some of them but it looks like there is still something that messes up. If you are saying all services are working fine, just the upgrade kicks in every time 'ipactl restart' is run (which is part of ipa.service machinery), it means the logged IPA data version is older than what IPA sees in the RPM database. Temporarily, this can be fixed by looking at /var/lib/ipa/sysupgrade/sysupgrade.state and changing ipa.data_version value to be exact same as the RPM package version-release values. However, it would help to understand why an upgrade causes CA apps to fail to register with the tomcat container. It looks like we have at least three such cases on this list over past week or so, all on CentOS 9 Stream, so there might be something? May be you can install sos report tool and collect a larger amount of data altogether so that we can see a greater picture? # dnf install sos # sos report --profile={identity,security,system,services,network} --clean -a This should produce logs with consistently obfuscated hostnames and domains across all files. You can add more domains to obfuscate with `--domains={domain1,domain2,..}` to `sos report` tool. > > > >On Wed, Jan 29, 2025 at 4:51 PM Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com>> wrote: > >> Russ Long via FreeIPA-users wrote: >> > Things are functional, however IPA still thinks it needs an upgrade, so >> any time the service restarts, it breaks again. >> > >> >> If you have time to run the upgrade again and send us a compressed >> /var/log/ipaupgrade.log we can see if we can identify the root cause. >> >> rob >> >> -- / Alexander Bokovoy Sr. Principal Software Engineer Security / Identity Management Engineering Red Hat Limited, Finland
freeipa-users@lists.fedorahosted.org
-
Rob Crittenden