Best practices for upgrading when running dockerized FreeIPA

[Sebastiano Pomata]

Hi all,

I successfully deployed a FreeIPA installation with a master server and two replicas using
podman and the container images provided on docker.io (specifically, those based on fedora
36) on RHEL 8.
Time has passed (indeed flied) and fedora 36 is now about to reach end of security support
and I started thinking about upgrading to either the 4.10 freeipa based on fedora 38 or
the one based on RHEL 9.

Whatever the final choice, I wonder what's the recommended path to follow? I remember
having asked in the past on the freeipa IRC channel and the most common suggestion was to
avoid mounting the same ipa-data directory under a new, upgraded container image, but
rather creating a new replica directly based on the updated container image.

This is very sensible however now I'm faced with a practical issue on the steps to
take: assuming I wanted to upgrade the master and two replicas from 4.9 to 4.10 one by
one, shall I create a temporary replica under a new hostname (and same IP), delete the old
replica from topology and bring its container down, then re-create a new replica with the
proper previous hostname?
Or just give up on the old hostname and stick with the new one for the upgraded replica?
As I manage the installation with SRV records from DNS, ditching the old name for a new
one doesn't seem painful, however we have some services that rely on the LDAP hostname
of the current IPA servers and would still require manual upgrade.

DNS is not managed by FreeIPA but externally on another server, which I fully control.

Hope my question is clear and somebody who dealt with upgrades more often can provide some
feedback.

Thanks
Regards

[Alexander Bokovoy]

On la, 29 huhti 2023, Sebastiano Pomata via FreeIPA-users wrote:
> Hi all,
>
> I successfully deployed a FreeIPA installation with a master server and
> two replicas using podman and the container images provided on
> docker.io (specifically, those based on fedora 36) on RHEL 8.  Time has
> passed (indeed flied) and fedora 36 is now about to reach end of
> security support and I started thinking about upgrading to either the
> 4.10 freeipa based on fedora 38 or the one based on RHEL 9.
>
> Whatever the final choice, I wonder what's the recommended path to
> follow? I remember having asked in the past on the freeipa IRC channel
> and the most common suggestion was to avoid mounting the same ipa-data
> directory under a new, upgraded container image, but rather creating a
> new replica directly based on the updated container image.
>
> This is very sensible however now I'm faced with a practical issue on
> the steps to take: assuming I wanted to upgrade the master and two
> replicas from 4.9 to 4.10 one by one, shall I create a temporary
> replica under a new hostname (and same IP), delete the old replica from
> topology and bring its container down, then re-create a new replica
> with the proper previous hostname?  Or just give up on the old hostname
> and stick with the new one for the upgraded replica? As I manage the
> installation with SRV records from DNS, ditching the old name for a new
> one doesn't seem painful, however we have some services that rely on
> the LDAP hostname of the current IPA servers and would still require
> manual upgrade.
>
> DNS is not managed by FreeIPA but externally on another server, which I
> fully control.
>
> Hope my question is clear and somebody who dealt with upgrades more
> often can provide some feedback. 
FreeIPA container is supposed to run upgrade on the data volume when you
do upgrade images. This is one of scenarios tested by the upstream CI.
This is documented in the upstream documentation:
https://github.com/freeipa/freeipa-container/blob/master/README#L183-L189
----------------------
If you have existing container with data volume, it should be safe to
shut it down and run new one based on newer image, with the same data
directory bind-mounted to /data. The container logic will detect that it
is running with data produced by different image and attempt to upgrade
the configuration and data. Of course, keeping backup of the data
directory for cases when the upgrade process fails is recommended.
----------------------

What probably would be good to do is to simulate incremental version
upgrades here -- if you are going up from Fedora 36, step up to Fedora
36:latest first, then Fedora 37:latest, then Fedora 38:latest.


-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[Sebastiano Pomata]

El 04/05/2023 a las 7:59, Alexander Bokovoy escribió:
>  FreeIPA container is supposed to run upgrade on the data volume when
> you
>  do upgrade images. This is one of scenarios tested by the upstream CI.
>  This is documented in the upstream documentation:
>  https://github.com/freeipa/freeipa-container/blob/master/README#L183-L189
>  ----------------------
>  If you have existing container with data volume, it should be safe to
>  shut it down and run new one based on newer image, with the same data
>  directory bind-mounted to /data. The container logic will detect that it
>  is running with data produced by different image and attempt to upgrade
>  the configuration and data. Of course, keeping backup of the data
>  directory for cases when the upgrade process fails is recommended.
>  ----------------------
>  
>  What probably would be good to do is to simulate incremental version
>  upgrades here -- if you are going up from Fedora 36, step up to Fedora
>  36:latest first, then Fedora 37:latest, then Fedora 38:latest.
>  
>   
I indeed already tried an upgrade of a minor version on the same base 
distro and it worked fine, so probably I can give it a go at jumping 
the base distro version as well, following the path you suggested.

As you said keeping a backup of the ipa-data directory should prevent 
any major headache in case things go wrong.

Thanks for the reply,

Sebastiano

[Alexander Bokovoy]

On to, 04 touko 2023, Sebastiano Pomata wrote:
> El 04/05/2023 a las 7:59, Alexander Bokovoy escribió:
> >FreeIPA container is supposed to run upgrade on the data volume when you
> >do upgrade images. This is one of scenarios tested by the upstream CI.
> >This is documented in the upstream documentation:
> >https://github.com/freeipa/freeipa-container/blob/master/README#L183-L189
> >----------------------
> >If you have existing container with data volume, it should be safe to
> >shut it down and run new one based on newer image, with the same data
> >directory bind-mounted to /data. The container logic will detect that it
> >is running with data produced by different image and attempt to upgrade
> >the configuration and data. Of course, keeping backup of the data
> >directory for cases when the upgrade process fails is recommended.
> >----------------------
> >
> >What probably would be good to do is to simulate incremental version
> >upgrades here -- if you are going up from Fedora 36, step up to Fedora
> >36:latest first, then Fedora 37:latest, then Fedora 38:latest.
> >
> >
>
> I indeed already tried an upgrade of a minor version on the same base 
> distro and it worked fine, so probably I can give it a go at jumping 
> the base distro version as well, following the path you suggested. 
This github action in upstream tests upgrade scenarios:
https://github.com/freeipa/freeipa-container/blob/master/.github/workflow...

There is one specifically for fedora-36 to fedora 38.
Latest run is succeeding:
https://github.com/freeipa/freeipa-container/actions/runs/4868306151/jobs...

You can look at the logs in 'Run master and replica' section on to what
to expect.

-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[Sebastiano Pomata]

>  On to, 04 touko 2023, Sebastiano Pomata wrote:
>  There is one specifically for fedora-36 to fedora 38.
>  Latest run is succeeding:
>  https://github.com/freeipa/freeipa-container/actions/runs/4868306151/jobs...
>  
>  You can look at the logs in 'Run master and replica' section on to what
>  to expect. 
Nice tip, I might even jump straight to f38 from f36 as in the test you linked above.
Worst case I'll have to repeat it by restoring the backup.

Thanks!

[Sebastiano Pomata]

Hi all,

I finally managed to run the upgrade from fedora36 to fedora38, both replicas work fine. A
minor caveat, though: I received an error about the replication topology which I'm not
sure whether it affects the normal functioning of FreeIPA, it looks like it doesn't.
Here's the error:

[snip]
+ read -r PATTERN
+ sed -i.bak -e '/^\s*dynamic-db/,/};/
{s/\(\s*\)arg\s\+\(["'\'']\)\([a-zA-Z_]\+\s\)/\1\3\2/g;s/^dynamic-db/dyndb/;s@\(dyndb
"[^"]\+"\)@\1 "/usr/lib64/bind/ldap.so"@;s@\(dyndb
'\''[^'\'']\+'\''\)@\1
'\''/usr/lib64/bind/ldap.so'\''@;/\s*library[^;]\+;/d;/\s*cache_ttl[^;]\+;/d;/\s*psearch[^;]\+;/d;/\s*serial_autoincrement[^;]\+;/d;/\s*zone_refresh[^;]\+;/d;}'
/data/etc/named.conf
+ test 2 -eq 1
+ test 2 -gt 1
+ /usr/bin/getcert remove-ca -c certmaster
No CA with name "certmaster" found.
+ :
+ test 2 -eq 1
Upgrading IPA:. Estimated time: 1 minute 30 seconds
  [1/9]: saving configuration
  [2/9]: disabling listeners
  [3/9]: enabling DS global lock
  [4/9]: disabling Schema Compat
  [5/9]: starting directory server
  [6/9]: updating schema
  [7/9]: upgrading server
Error caught updating nsDS5ReplicatedAttributeList: Server is unwilling to perform: Entry
and attributes are managed by topology plugin.No direct modifications allowed.
Error caught updating nsDS5ReplicatedAttributeListTotal: Server is unwilling to perform:
Entry and attributes are managed by topology plugin.No direct modifications allowed.
  [8/9]: stopping directory server
  [9/9]: restoring configuration
Done.
Update complete
[snip]

The rest of the log goes on about checks and updates and reported no more errors nor
warnings.

I also checked with the tool "checkipaconsistency" I found on GitHub and replica
state is deemed OK (I added a new posix test group and it propagated immediately). Not too
worried about this but wondering if anyone here has more insight on the error above.

Thanks