CentOS 7 FreeIPA upgrade, 4.5 to 4.6.8: certmonger hanger

[Melissa Ferreira da Silva Boiko]

Hello all.

I'm trying to replace an ancient FreeIPA 4.5.0 master (and primary CA
master) on CentOS 7.4.  I am having problems trying to make replicas with
FreeIPA 4.11, and past threads suggest the errors are due to
incompatibility of password hash algorithms, which are supposed to be fixed
on the older releases rather than the newer.

Therefore I'm trying to upgrade the old server to the current version in
the CentOS 7 repos, 4.6.8, to try to create fresh replicas from there.  But
I'm having issues with the certmonger systemd service hanging, and breaking
ipa-server-upgrade--whether I update the whole CentOS to 7.9.2009, or just
ipa-server and its dependencies, the result is the same.

This is where ipa-server-upgrade breaks:

[Verifying that root certificate is published]
        [Migrate CRL publish directory]
        CRL tree already moved
        [Verifying that CA proxy configuration is correct]
        IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
        Unexpected error - see /var/log/ipaupgrade.log for details:
        CalledProcessError: Command '/bin/systemctl start
certmonger.service' returned non-zero exit status 1
        The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
for more information

This is because certmonger.service hangs until timeout.  That happens when
starting the service manually, too.  Logs for certmonger.service are not
informative:

-- Subject: Unit certmonger.service has begun start-up
        -- Unit certmonger.service has begun starting up.
        Jan 31 14:38:59 vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
certmonger.service start operation timed out. Terminating.
        Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
certmonger.service stop-sigterm timed out. Killing.
        Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
certmonger.service: main process exited, code=killed, status=9/KILL
        -- Subject: Unit certmonger.service has failed
        -- Unit certmonger.service has failed.
        Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de systemd[1]: Unit
certmonger.service entered failed state.
        Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
certmonger.service failed.
        [e-mail][lxc](e:0,1s)(j:0) ~

Running `certmonger -S -n -d 9` seems to run ok.  The only difference in
the systemd service file is, I think, whatever it is that the BusName
setting does.  dbus is running seemingly without issue, nothing on logs.
Restarting dbus.service doesn't help.

The machine is an LXC container with 4GiB RAM, which doesn't come close to
being exhausted when trying to restart certmonger.  No OOM in logs.

I saw this thread about certmonger problems with ulimit in containers:
https://bugzilla.redhat.com/show_bug.cgi?id=1656519
But the suggested workaround (make sure ulimit -n is the same in container
and host) doesn't apply because it's already the same for us.

How should I proceed from here?

[Rob Crittenden]

Melissa Ferreira da Silva Boiko via FreeIPA-users wrote:
> Hello all.
>
> I'm trying to replace an ancient FreeIPA 4.5.0 master (and primary CA
> master) on CentOS 7.4.  I am having problems trying to make replicas
> with FreeIPA 4.11, and past threads suggest the errors are due to
> incompatibility of password hash algorithms, which are supposed to be
> fixed on the older releases rather than the newer.
>
> Therefore I'm trying to upgrade the old server to the current version in
> the CentOS 7 repos, 4.6.8, to try to create fresh replicas from there. 
> But I'm having issues with the certmonger systemd service hanging, and
> breaking ipa-server-upgrade--whether I update the whole CentOS to
> 7.9.2009, or just ipa-server and its dependencies, the result is the same.
>
> This is where ipa-server-upgrade breaks:
>
> [Verifying that root certificate is published]
>         [Migrate CRL publish directory]
>         CRL tree already moved
>         [Verifying that CA proxy configuration is correct]
>         IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> run command ipa-server-upgrade manually.
>         Unexpected error - see /var/log/ipaupgrade.log for details:
>         CalledProcessError: Command '/bin/systemctl start
> certmonger.service' returned non-zero exit status 1
>         The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>
> This is because certmonger.service hangs until timeout.  That happens
> when starting the service manually, too.  Logs for certmonger.service
> are not informative:
>
> -- Subject: Unit certmonger.service has begun start-up
>         -- Unit certmonger.service has begun starting up.
>         Jan 31 14:38:59 vm-ipa-1.intra.viaboxxsystems.de
> http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> start operation timed out. Terminating.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> stop-sigterm timed out. Killing.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> certmonger.service: main process exited, code=killed, status=9/KILL
>         -- Subject: Unit certmonger.service has failed
>         -- Unit certmonger.service has failed.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: Unit
> certmonger.service entered failed state.
>         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> failed.
>         [e-mail]
> [lxc](e:0,1s)(j:0) ~
>
> Running `certmonger -S -n -d 9` seems to run ok.  The only difference in
> the systemd service file is, I think, whatever it is that the BusName
> setting does.  dbus is running seemingly without issue, nothing on
> logs.  Restarting dbus.service doesn't help.
>
> The machine is an LXC container with 4GiB RAM, which doesn't come close
> to being exhausted when trying to restart certmonger.  No OOM in logs.
>
> I saw this thread about certmonger problems with ulimit in containers:
> https://bugzilla.redhat.com/show_bug.cgi?id=1656519
> But the suggested workaround (make sure ulimit -n is the same in
> container and host) doesn't apply because it's already the same for us.
>
> How should I proceed from here?

Why not create a new CentOS replica from the current one. Then use that
to upgrade to 8 and then 9?

rob

[Melissa Ferreira da Silva Boiko]

Thanks for the suggestion!

I spun a new CentOS 7 image with 7.9.2009 / FreeIPA 4.6.8 (which involved
setting up the incus server to cgroups v1).  Then I tried creating a
replica from the 4.5.  It again broke on pki-tomcatd, but with a somewhat
baffling error that I didn't know what to do about:

[3/30]: creating ACIs for admin
      [4/30]: creating installation admin user
      [5/30]: configuring certificate server instance
      [error] IOError: [Errno 13] Permission denied: '/tmp/tmpRJcwYV'
    Your system may be partly configured.
    Run /usr/sbin/ipa-server-install --uninstall to clean up.

This appears to be related to
https://bugzilla.redhat.com/show_bug.cgi?id=1677027
Which apparently was fixed on FreeIPA 4.7.

I couldn't find a release with 4.7 so I tried again with a VM with CentOS
8-Stream, which is supposed to have 4.9.  Only yum/dnf couldn't find the
ipa-server package at all; they'd list ipa-client, but not server.  I'm not
familiar with RPM-based systems so it took a lot of digging (including
trying my hand at Rocky Linux and localinstall from manual RPM downloads,
which led to problems with dependencies) until I found out I had to add
module_hotfixes=1 in the appstream.repo file.

With FreeIPA 4.9 once again the CA setup failed, with a new error:
  [28/30]: importing IPA certificate profiles
Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does
not provide CA.
Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does
not provide CA.
  [error] RemoteRetrieveError: Failed to authenticate to CA REST API
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

There were some hard-to-read HTTP errors on the install log so I was
already getting ready to give up and try to create a replica from Docker or
something.  But just as a hail-mary I run the uninstall scripts, rebooted,
and tried again, and to my surprise this time pki-tomcatd install worked!
I got a warning then on the KDC step:

Configuring Kerberos KDC (krb5kdc)
  [1/1]: installing X509 Certificate for PKINIT
PKINIT certificate request failed: Certificate issuance failed
(CA_UNREACHABLE: Server at https://vm-ipa-half-3.intra.vi
aboxxsystems.de/ipa/json failed request, will retry: 4035 (Request failed
with status 500: Non-2xx response from CA REST
 API: 500. ).)
Failed to configure PKINIT
Full PKINIT configuration did not succeed
The setup will only install bits essential to the server functionality
You can enable PKINIT after the setup completed using 'ipa-pkinit-manage'

But ipa-replica-install finished with status: successful for the first
time, and "ipa-pkinit-manage enable" worked.  I now have a FreeIPA 4.9.13
replica with CA; hopefully 4.11 will replicate from that without issues and
then I can promote it to primary CA.

---

In summary, these are the issues I found so far trying to upgrade from
FreeIPA 4.5.0:

Upgrade FreeIPA 4.5 / CentOS7 to the last CentOS 7 / FreeIPA 4.6: breaks
due to certmonger timeout.
Replicate to current FreeIPA 4.11 / Fedora 39: breaks due to hash
incompatibility.
Replicate to FreeIPA 4.6 / fresh CentOS 7: breaks due to systemd /tmp
permissions error.
Replicate to FreeIPA 4.7: couldn't find lxc containers with this version.
Replicate to Freeipa 4.9 / CentOS 8-Stream: had to downgrade incus host to
cgroups v1; first couple attempts failed, but eventually it worked.

[]s

Am Do., 1. Feb. 2024 um 17:07 Uhr schrieb Rob Crittenden <
[e-mail]>:
> Melissa Ferreira da Silva Boiko via FreeIPA-users wrote:
> > Hello all.
> >
> > I'm trying to replace an ancient FreeIPA 4.5.0 master (and primary CA
> > master) on CentOS 7.4.  I am having problems trying to make replicas
> > with FreeIPA 4.11, and past threads suggest the errors are due to
> > incompatibility of password hash algorithms, which are supposed to be
> > fixed on the older releases rather than the newer.
> >
> > Therefore I'm trying to upgrade the old server to the current version in
> > the CentOS 7 repos, 4.6.8, to try to create fresh replicas from there.
> > But I'm having issues with the certmonger systemd service hanging, and
> > breaking ipa-server-upgrade--whether I update the whole CentOS to
> > 7.9.2009, or just ipa-server and its dependencies, the result is the
>
> same.
> > This is where ipa-server-upgrade breaks:
> >     [Verifying that root certificate is published]
> >     [Migrate CRL publish directory]
> >     CRL tree already moved
> >     [Verifying that CA proxy configuration is correct]
> >     IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> >
> > run command ipa-server-upgrade manually.
> >         Unexpected error - see /var/log/ipaupgrade.log for details:
> >         CalledProcessError: Command '/bin/systemctl start
> > certmonger.service' returned non-zero exit status 1
> >         The ipa-server-upgrade command failed. See
> > /var/log/ipaupgrade.log for more information
> >
> > This is because certmonger.service hangs until timeout.  That happens
> > when starting the service manually, too.  Logs for certmonger.service
> > are not informative:
> >     -- Subject: Unit certmonger.service has begun start-up
> >     -- Unit certmonger.service has begun starting up.
> >     Jan 31 14:38:59 vm-ipa-1.intra.viaboxxsystems.de
> >
> > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> > start operation timed out. Terminating.
> >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> > stop-sigterm timed out. Killing.
> >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> > certmonger.service: main process exited, code=killed, status=9/KILL
> >         -- Subject: Unit certmonger.service has failed
> >         -- Unit certmonger.service has failed.
> >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: Unit
> > certmonger.service entered failed state.
> >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: certmonger.service
> > failed.
> >         [e-mail]
> > [lxc](e:0,1s)(j:0) ~
> >
> > Running `certmonger -S -n -d 9` seems to run ok.  The only difference in
> > the systemd service file is, I think, whatever it is that the BusName
> > setting does.  dbus is running seemingly without issue, nothing on
> > logs.  Restarting dbus.service doesn't help.
> >
> > The machine is an LXC container with 4GiB RAM, which doesn't come close
> > to being exhausted when trying to restart certmonger.  No OOM in logs.
> >
> > I saw this thread about certmonger problems with ulimit in containers:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1656519
> > But the suggested workaround (make sure ulimit -n is the same in
> > container and host) doesn't apply because it's already the same for us.
> >
> > How should I proceed from here?
>
> Why not create a new CentOS replica from the current one. Then use that
> to upgrade to 8 and then 9?
>
> rob

[Melissa Ferreira da Silva Boiko]

Oh I'm not touching the original instance, it's still up and I'll leave it
there for the foreseeable future.  Attempts to upgrade CentOS in-place all
ran into tricky problems.  (The way I tested this was: I prepared snapshots
*and* backups of the VM, then tried to upgrade, and when it broke I rolled
it all back.)

What I did was to create replicas; as you've seen in my story, the winning
combination was a temporary CentOS 8 Stream / FreeIPA 4.9 replica as a
stepping-stone from 4.5, and from there a definitive replica with Fedora 39
/ FreeIPA 4.11.  This was mostly done with commands like
    ipa-replica-install --server very-old-instance.example.com -n
example.com -r EXAMPLE.COM --setup-ca --no-ntp -w "$adminpass"
(--no-ntp because our instances are all LXC containers and the clock is set
by the host machine.)

Our installation uses the CA feature but not the DNS server.  Once I got a
FreeIPA 4.11 replica working with a functional CA, I promoted it to CA
"renewal master" as instructed in
https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master .
That also seems to have worked without a hitch.

I now changed our FreeIPA client configs, Ansible, and various LDAP clients
to point to the new replica.  So far it appears to work.  If a couple
months go uneventfully, I'll shut down the old instance and probably leave
it like that for a few months more, until finally I'm convinced everything
is working properly, and then unenroll the old instance from the replica
list (but still leave the VM there, shut down but not deleted. just in
case.)

Good luck with your systems!
[]s

Am Mo., 5. Feb. 2024 um 15:15 Uhr schrieb Kevin Vasko :
> Melissa,
>
> I’ve been following your thread here as I also have a 4.5.x system that
> _really_ needs to be updated, but I’m nervous to touch it since it “works”.
>
> How you testing this without breaking the instance? Im nervous I have
> issues like you are experiencing and it breaks everything. I would like to
> test and validate that the upgrade works before going all in. Care to share
> your process?
>
> -Kevin
>
> On Feb 5, 2024, at 7:21 AM, Melissa Ferreira da Silva Boiko via
> FreeIPA-users  wrote:
>
> 
> Thanks for the suggestion!
>
> I spun a new CentOS 7 image with 7.9.2009 / FreeIPA 4.6.8 (which involved
> setting up the incus server to cgroups v1).  Then I tried creating a
> replica from the 4.5.  It again broke on pki-tomcatd, but with a somewhat
> baffling error that I didn't know what to do about:
>   [3/30]: creating ACIs for admin
>   [4/30]: creating installation admin user
>   [5/30]: configuring certificate server instance
>   [error] IOError: [Errno 13] Permission denied: '/tmp/tmpRJcwYV'
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> This appears to be related to
> https://bugzilla.redhat.com/show_bug.cgi?id=1677027
> Which apparently was fixed on FreeIPA 4.7.
>
> I couldn't find a release with 4.7 so I tried again with a VM with CentOS
> 8-Stream, which is supposed to have 4.9.  Only yum/dnf couldn't find the
> ipa-server package at all; they'd list ipa-client, but not server.  I'm not
> familiar with RPM-based systems so it took a lot of digging (including
> trying my hand at Rocky Linux and localinstall from manual RPM downloads,
> which led to problems with dependencies) until I found out I had to add
> module_hotfixes=1 in the appstream.repo file.
>
> With FreeIPA 4.9 once again the CA setup failed, with a new error:
>   [28/30]: importing IPA certificate profiles
> Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does
> not provide CA.
> Lookup failed: Preferred host vm-ipa-half-3.intra.viaboxxsystems.de does
> not provide CA.
>   [error] RemoteRetrieveError: Failed to authenticate to CA REST API
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> There were some hard-to-read HTTP errors on the install log so I was
> already getting ready to give up and try to create a replica from Docker or
> something.  But just as a hail-mary I run the uninstall scripts, rebooted,
> and tried again, and to my surprise this time pki-tomcatd install worked!
> I got a warning then on the KDC step:
>
> Configuring Kerberos KDC (krb5kdc)
>   [1/1]: installing X509 Certificate for PKINIT
> PKINIT certificate request failed: Certificate issuance failed
> (CA_UNREACHABLE: Server at https://vm-ipa-half-3.intra.vi
> aboxxsystems.de/ipa/json failed request, will retry: 4035 (Request failed
> with status 500: Non-2xx response from CA REST
>  API: 500. ).)
> Failed to configure PKINIT
> Full PKINIT configuration did not succeed
> The setup will only install bits essential to the server functionality
> You can enable PKINIT after the setup completed using 'ipa-pkinit-manage'
>
> But ipa-replica-install finished with status: successful for the first
> time, and "ipa-pkinit-manage enable" worked.  I now have a FreeIPA 4.9.13
> replica with CA; hopefully 4.11 will replicate from that without issues and
> then I can promote it to primary CA.
>
> In summary, these are the issues I found so far trying to upgrade from
> FreeIPA 4.5.0:
>
> Upgrade FreeIPA 4.5 / CentOS7 to the last CentOS 7 / FreeIPA 4.6: breaks
> due to certmonger timeout.
> Replicate to current FreeIPA 4.11 / Fedora 39: breaks due to hash
> incompatibility.
> Replicate to FreeIPA 4.6 / fresh CentOS 7: breaks due to systemd /tmp
> permissions error.
> Replicate to FreeIPA 4.7: couldn't find lxc containers with this version.
> Replicate to Freeipa 4.9 / CentOS 8-Stream: had to downgrade incus host to
> cgroups v1; first couple attempts failed, but eventually it worked.
>
> []s
>
> Am Do., 1. Feb. 2024 um 17:07 Uhr schrieb Rob Crittenden <
> [e-mail]>:
> > Melissa Ferreira da Silva Boiko via FreeIPA-users wrote:
> > > Hello all.
> > >
> > > I'm trying to replace an ancient FreeIPA 4.5.0 master (and primary CA
> > > master) on CentOS 7.4.  I am having problems trying to make replicas
> > > with FreeIPA 4.11, and past threads suggest the errors are due to
> > > incompatibility of password hash algorithms, which are supposed to be
> > > fixed on the older releases rather than the newer.
> > >
> > > Therefore I'm trying to upgrade the old server to the current version in
> > > the CentOS 7 repos, 4.6.8, to try to create fresh replicas from there.
> > > But I'm having issues with the certmonger systemd service hanging, and
> > > breaking ipa-server-upgrade--whether I update the whole CentOS to
> > > 7.9.2009, or just ipa-server and its dependencies, the result is the
> >
> > same.
> > > This is where ipa-server-upgrade breaks:
> > >     [Verifying that root certificate is published]
> > >     [Migrate CRL publish directory]
> > >     CRL tree already moved
> > >     [Verifying that CA proxy configuration is correct]
> > >     IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> > >
> > > run command ipa-server-upgrade manually.
> > >         Unexpected error - see /var/log/ipaupgrade.log for details:
> > >         CalledProcessError: Command '/bin/systemctl start
> > > certmonger.service' returned non-zero exit status 1
> > >         The ipa-server-upgrade command failed. See
> > > /var/log/ipaupgrade.log for more information
> > >
> > > This is because certmonger.service hangs until timeout.  That happens
> > > when starting the service manually, too.  Logs for certmonger.service
> > > are not informative:
> > >     -- Subject: Unit certmonger.service has begun start-up
> > >     -- Unit certmonger.service has begun starting up.
> > >     Jan 31 14:38:59 vm-ipa-1.intra.viaboxxsystems.de
> > >
> > > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> >
> > certmonger.service
> > > start operation timed out. Terminating.
> > >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> >
> > certmonger.service
> > > stop-sigterm timed out. Killing.
> > >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> > > certmonger.service: main process exited, code=killed, status=9/KILL
> > >         -- Subject: Unit certmonger.service has failed
> > >         -- Unit certmonger.service has failed.
> > >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]: Unit
> > > certmonger.service entered failed state.
> > >         Jan 31 14:40:29 vm-ipa-1.intra.viaboxxsystems.de
> > > http://vm-ipa-1.intra.viaboxxsystems.de systemd[1]:
> >
> > certmonger.service
> > > failed.
> > >         [e-mail]
> > > [lxc](e:0,1s)(j:0) ~
> > >
> > > Running `certmonger -S -n -d 9` seems to run ok.  The only difference in
> > > the systemd service file is, I think, whatever it is that the BusName
> > > setting does.  dbus is running seemingly without issue, nothing on
> > > logs.  Restarting dbus.service doesn't help.
> > >
> > > The machine is an LXC container with 4GiB RAM, which doesn't come close
> > > to being exhausted when trying to restart certmonger.  No OOM in logs.
> > >
> > > I saw this thread about certmonger problems with ulimit in containers:
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1656519
> > > But the suggested workaround (make sure ulimit -n is the same in
> > > container and host) doesn't apply because it's already the same for us.
> > >
> > > How should I proceed from here?
> >
> > Why not create a new CentOS replica from the current one. Then use that
> > to upgrade to 8 and then 9?
> >
> > rob
> >
> > --
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue