I just upgraded, using dnf, from 4.12.2-1.el9 to 4.12.2-7.el9 on a Centos Stream 9 IPA server. During the DNF upgrade, I received the following error:
Job for ipa.service failed because the control process exited with error code. See "systemctl status ipa.service" and "journalctl -xeu ipa.service" for details.
After the upgrade I attempted a manual ipa-server-upgrade which was met with many errors. I attempted a reboot of the host as it needed it for a kernel update anyways. Post reboot, none of the IPA Services will start, so ipa-server-upgrade fails nearly instantly.
ipactl start fails as it attempts to run the server-upgrade. ipactl start --skip-version-check fails with an error about dirsrv:
[root@ipa-primary sysrestore]# ipactl start --skip-version-check Skipping version check Starting Directory Service Failed to start Directory Service: CalledProcessError(Command ['/bin/systemctl', 'start', 'dirsrv@REDACTED.service'] returned non-zero exit status 1)
That service shows multiple errors: Jan 24 10:08:01 REDACTED ns-slapd[4763]: [24/Jan/2025:10:08:01.238214824 -0500] - ERR - plugin_setup - Required attribute nsslapd-pluginInitFunc is missing from entry "cn=Case Exact String Syntax,cn=plugins,cn=config" Jan 24 10:08:01 REDACTED ns-slapd[4763]: [24/Jan/2025:10:08:01.240364122 -0500] - ERR - slapd_bootstrap_config - The plugin entry [cn=Case Exact String Syntax,cn=plugins,cn=config]in the configfile /etc/dirsrv/slapd-REDACTED/dse.ldif was invalid. Required attribute nsslapd-pluginInitFunc is missing from entry. Jan 24 10:08:01 REDACTED ns-slapd[4763]: [24/Jan/2025:10:08:01.241806986 -0500] - EMERG - main - The configuration files in directory /etc/dirsrv/slapd-REDACTED could not be read or were not found. Please refer to the error log or output for more information. Jan 24 10:08:01 REDACTED systemd[1]: dirsrv@REDACTED.service: Main process exited, code=exited, status=1/FAILURE Jan 24 10:08:01 REDACTED systemd[1]: dirsrv@REDACTED.service: Failed with result 'exit-code'. Jan 24 10:08:01 REDACTED systemd[1]: Failed to start 389 Directory Server REDACTED..
Is there any way to save this?
I found the above errors were due to issues with dse.ldif. Something went wrong in the upgrading of that file it appears and it ruined the syntax removed required options in multiple places. I have reverted this file to the backup that was auto-created. Now getting errors that the connection to dirsrv is failing with a timeout (which is expected since it is not running). Trying to work through that now.
Well now dirsrv is running, but it is not listening on 389 or 636, so everything else is still broken.
Fixed the broken listeners, but Tomcat says it cannot connect to LDAP:
2025-01-24 11:04:13 [AuthorityMonitor] WARNING: AuthorityMonitor: Failed to get LDAPConnection: LdapBoundConnFactory: Unable to create master connection. LDAP server is unavailable: REDACTED:636 LdapBoundConnFactory: Unable to create master connection. LDAP server is unavailable: REDACTED:636
I am out of ideas of where to go from here.
Also seeing this error in the CA Debug log:
2025-01-24T16:17:32Z DEBUG response headers Content-Type: text/html;charset=utf-8 Content-Language: en Content-Length: 784 Date: Fri, 24 Jan 2025 16:17:32 GMT
2025-01-24T16:17:32Z DEBUG response body (decoded): b'<!doctype html><html lang="en"><head><title>HTTP Status 404 \xe2\x80\x93 Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;}h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 \xe2\x80\x93 Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> The requested resource [/ca/rest/account/login] is not available</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/9.0.87</h3></body></html>'
Russ Long via FreeIPA-users wrote:
Fixed the broken listeners, but Tomcat says it cannot connect to LDAP:
2025-01-24 11:04:13 [AuthorityMonitor] WARNING: AuthorityMonitor: Failed to get LDAPConnection: LdapBoundConnFactory: Unable to create master connection. LDAP server is unavailable: REDACTED:636 LdapBoundConnFactory: Unable to create master connection. LDAP server is unavailable: REDACTED:636
I am out of ideas of where to go from here.
How did you fix the listeners?
Does ldapsearch work if you only start the dirsrv.target service?
rob
ldapsearch works as expected.
I fixed the listeners by stopping dirsrv.target, and editing the listener config in dse.ldif as the port had been changed to 0 and the secure was disabled. I set the port appropriately and enabled secure and started dirsrv.target.
Russ Long via FreeIPA-users wrote:
ldapsearch works as expected.
I fixed the listeners by stopping dirsrv.target, and editing the listener config in dse.ldif as the port had been changed to 0 and the secure was disabled. I set the port appropriately and enabled secure and started dirsrv.target.
This has occurred in the past if the upgrade process is interrupted. It tries a fair bit harder to avoid this situation in current code.
Are things functional again?
rob
Things are functional, however IPA still thinks it needs an upgrade, so any time the service restarts, it breaks again.
Russ Long via FreeIPA-users wrote:
Things are functional, however IPA still thinks it needs an upgrade, so any time the service restarts, it breaks again.
If you have time to run the upgrade again and send us a compressed /var/log/ipaupgrade.log we can see if we can identify the root cause.
rob
Here is the log, sorry for the delay. Logs are redacted, but the only thing changed was the domain names and DNs.
On Wed, Jan 29, 2025 at 4:51 PM Rob Crittenden rcritten@redhat.com wrote:
Russ Long via FreeIPA-users wrote:
Things are functional, however IPA still thinks it needs an upgrade, so
any time the service restarts, it breaks again.
If you have time to run the upgrade again and send us a compressed /var/log/ipaupgrade.log we can see if we can identify the root cause.
rob
On Пан, 03 лют 2025, Russell Long via FreeIPA-users wrote:
Here is the log, sorry for the delay. Logs are redacted, but the only thing changed was the domain names and DNs.
The upgrade log chokes on the CA application not being registered in tomcat container (the corresponding /ca/rest/... path is giving 404 error).
So we get back to the same point as before. An upgrade has been in progress but somehow was interrupted. Directory server was having some of listeners disabled to avoid external communication during the upgrade and those listeners weren't recovered due to an interruption. You recovered some of them but it looks like there is still something that messes up.
If you are saying all services are working fine, just the upgrade kicks in every time 'ipactl restart' is run (which is part of ipa.service machinery), it means the logged IPA data version is older than what IPA sees in the RPM database. Temporarily, this can be fixed by looking at /var/lib/ipa/sysupgrade/sysupgrade.state and changing ipa.data_version value to be exact same as the RPM package version-release values.
However, it would help to understand why an upgrade causes CA apps to fail to register with the tomcat container. It looks like we have at least three such cases on this list over past week or so, all on CentOS 9 Stream, so there might be something?
May be you can install sos report tool and collect a larger amount of data altogether so that we can see a greater picture?
# dnf install sos # sos report --profile={identity,security,system,services,network} --clean -a
This should produce logs with consistently obfuscated hostnames and domains across all files. You can add more domains to obfuscate with `--domains={domain1,domain2,..}` to `sos report` tool.
On Wed, Jan 29, 2025 at 4:51 PM Rob Crittenden rcritten@redhat.com wrote:
Russ Long via FreeIPA-users wrote:
Things are functional, however IPA still thinks it needs an upgrade, so
any time the service restarts, it breaks again.
If you have time to run the upgrade again and send us a compressed /var/log/ipaupgrade.log we can see if we can identify the root cause.
rob
We found this issue today, upgrading from Fedora 40 to Fedora 41.
In dogtagpk the /ca/rest/account/login enpoint (actually all /ca/rest enpoints) was moved to /ca/v1 in this commit:
https://github.com/dogtagpki/pki/commit/850e6b9e5128264a471e57b70ab8d970cf91...
We were able to complete the upgrade succesfully modifying the file /usr/lib/python3.13/site-packages/ipaserver/plugins/dogtag.py changing al "/rest" to "/v1" (patch file attached). With this modification ipa-server-upgrade completed successfully.
It seems there is some kind of version mismatch.
Greetings, Vicente Quintáns.
Vicente Quintans via FreeIPA-users wrote:
We found this issue today, upgrading from Fedora 40 to Fedora 41.
In dogtagpk the /ca/rest/account/login enpoint (actually all /ca/rest enpoints) was moved to /ca/v1 in this commit:
https://github.com/dogtagpki/pki/commit/850e6b9e5128264a471e57b70ab8d970cf91...
We were able to complete the upgrade succesfully modifying the file /usr/lib/python3.13/site-packages/ipaserver/plugins/dogtag.py changing al "/rest" to "/v1" (patch file attached). With this modification ipa-server-upgrade completed successfully.
It seems there is some kind of version mismatch.
Thanks for sharing your research!
What is strange is that new installs don't seem to be impact, only upgrades.
Do you happen to have the version of IPA you were running on F40 and are now running on 41? That may help me reproduce the issue.
I think you forgot to attach your patch, though I don't believe changing the endpoints to v1 should be necessary.
rob
Yes... it was my first post and go a little confused creating and validating account before being able to post.
I simply changed this lines in /usr/lib/python3.13/site-packages/ipaserver/plugins/dogtag.py ``` 602c602 < url='/ca/rest/account/login', ---
url='/ca/v1/account/login',
618c618 < url='/ca/rest/account/logout', ---
url='/ca/v1/account/logout',
653c653 < resource = '/ca/rest' ---
resource = '/ca/v1'
1071c1071 < path = "/pki/rest/info" ---
path = "/pki/v1/info"
1410c1410 < url = '/ca/rest/certs/search?size=%d' % ( ---
url = '/ca/v1/certs/search?size=%d' % (
```
In regard to the versions I were running, dnf history info show this:
Upgrade freeipa-server-4.12.2-8.fc41.x86_64 @updates Upgraded freeipa-server-4.12.2-4.fc40.x86_64 @@System
Upgrade python3-ipaserver-4.12.2-8.fc41.noarch @updates Upgraded python3-ipaserver-4.12.2-4.fc40.noarch @@System
Upgrade dogtag-pki-server-11.6.0-1.fc41.2.noarch @updates Upgraded dogtag-pki-server-11.5.0-3.fc40.noarch @@System
Vicente Quintáns.
It looks like the root cause of this was discovered in https://bugzilla.redhat.com/show_bug.cgi?id=2350322 . I would recommend following those steps as the changes will survive server package updates.
rob
Vicente Quintans via FreeIPA-users wrote:
Yes... it was my first post and go a little confused creating and validating account before being able to post.
I simply changed this lines in /usr/lib/python3.13/site-packages/ipaserver/plugins/dogtag.py
602c602 < url='/ca/rest/account/login', --- > url='/ca/v1/account/login', 618c618 < url='/ca/rest/account/logout', --- > url='/ca/v1/account/logout', 653c653 < resource = '/ca/rest' --- > resource = '/ca/v1' 1071c1071 < path = "/pki/rest/info" --- > path = "/pki/v1/info" 1410c1410 < url = '/ca/rest/certs/search?size=%d' % ( --- > url = '/ca/v1/certs/search?size=%d' % (In regard to the versions I were running, dnf history info show this:
Upgrade freeipa-server-4.12.2-8.fc41.x86_64 @updates Upgraded freeipa-server-4.12.2-4.fc40.x86_64 @@System Upgrade python3-ipaserver-4.12.2-8.fc41.noarch @updates Upgraded python3-ipaserver-4.12.2-4.fc40.noarch @@System Upgrade dogtag-pki-server-11.6.0-1.fc41.2.noarch @updates Upgraded dogtag-pki-server-11.5.0-3.fc40.noarch @@SystemVicente Quintáns.
ipa-acme-manage *was not* working with my fix to dogtag.py
I followed the indicated steps in https://bugzilla.redhat.com/show_bug.cgi?id=2350322 and everything looks fine now. This fix makes more sense as well.
The steps enable an existing rewrite.config file so that the old endpoints work again, not only "/pki/rest/info" but also "/acme/*"
Thanks!
On Mon, 7 Apr 2025 at 17:06, Rob Crittenden rcritten@redhat.com wrote:
It looks like the root cause of this was discovered in https://bugzilla.redhat.com/show_bug.cgi?id=2350322 . I would recommend following those steps as the changes will survive server package updates.
rob
Vicente Quintans via FreeIPA-users wrote:
Yes... it was my first post and go a little confused creating and
validating account before being able to post.
I simply changed this lines in
/usr/lib/python3.13/site-packages/ipaserver/plugins/dogtag.py
602c602 < url='/ca/rest/account/login', --- > url='/ca/v1/account/login', 618c618 < url='/ca/rest/account/logout', --- > url='/ca/v1/account/logout', 653c653 < resource = '/ca/rest' --- > resource = '/ca/v1' 1071c1071 < path = "/pki/rest/info" --- > path = "/pki/v1/info" 1410c1410 < url = '/ca/rest/certs/search?size=%d' % ( --- > url = '/ca/v1/certs/search?size=%d' % (In regard to the versions I were running, dnf history info show this:
Upgrade freeipa-server-4.12.2-8.fc41.x86_64@updatesUpgraded freeipa-server-4.12.2-4.fc40.x86_64@@SystemUpgrade python3-ipaserver-4.12.2-8.fc41.noarch@updatesUpgraded python3-ipaserver-4.12.2-4.fc40.noarch@@SystemUpgrade dogtag-pki-server-11.6.0-1.fc41.2.noarch@updatesUpgraded dogtag-pki-server-11.5.0-3.fc40.noarch@@SystemVicente Quintáns.
On Fri, Apr 04, 2025 at 02:45:14PM -0000, Vicente Quintans via FreeIPA-users wrote:
We found this issue today, upgrading from Fedora 40 to Fedora 41.
In dogtagpk the /ca/rest/account/login enpoint (actually all /ca/rest enpoints) was moved to /ca/v1 in this commit:
https://github.com/dogtagpki/pki/commit/850e6b9e5128264a471e57b70ab8d970cf91...
I believe I've reported this as a problem with ACME as https://bugzilla.redhat.com/show_bug.cgi?id=2350322
I sent a reply yesterday with the log gzipped and attached, but it appears the list ate it, please let me know how you would prefer I send it.
On Аўт, 04 лют 2025, Russ Long via FreeIPA-users wrote:
I sent a reply yesterday with the log gzipped and attached, but it appears the list ate it, please let me know how you would prefer I send it.
The mail has most likely stuck in the list moderation. Unfortunately, our list moderator is away on a sick leave.
You can send the upgrade log to me directly and I'll look at it tomorrow as I'm about to sign-off for tonight.
freeipa-users@lists.fedorahosted.org
-
Alexander Bokovoy -
Rob Crittenden -
Russ Long -
Russell Long
-
Tomasz Torcz -
Vicente Quintans