RHEL9 Replica Install fail at 22/30 Importing RA key

[Ivars Strazdins]

Hi guys,
I am installing IPA replica on RHEL9 (well, Alma Linux 9 actually) and got exactly the same issue as here: https://access.redhat.com/discussions/6961739 https://access.redhat.com/discussions/6961739
And similarly to the poster of that issue, also my IPA master server is IPA 4.6.8 on Centos7.

I was trying to migrate IPA to a newer version by using Alma Linux 9.
I removed Centos 7 replica and tried to install Alma Linux 9 replica. IPA client was installed without issues.
No SELinux alerts.
Content of /var/lib/ipa folder:
[root@fricka ~]# ls /var/lib/ipa
backup  certs  gssproxy  passwds  pki-ca  private  ra-agent.pem  sysrestore  sysupgrade

Any suggestions how this could be resolved?
Thank you in advance,
Ivars

Log of replica install:
….
Starting replication, please wait until this has completed.
Update in progress, 9 seconds elapsed
Update succeeded

[3/30]: creating ACIs for admin
  [4/30]: creating installation admin user
  [5/30]: configuring certificate server instance
  [6/30]: stopping certificate server instance to update CS.cfg
  [7/30]: backing up CS.cfg
  [8/30]: Add ipa-pki-wait-running
  [9/30]: secure AJP connector
  [10/30]: reindex attributes
  [11/30]: exporting Dogtag certificate store pin
  [12/30]: disabling nonces
  [13/30]: set up CRL publishing
  [14/30]: enable PKIX certificate path discovery and validation
  [15/30]: authorizing RA to modify profiles
  [16/30]: authorizing RA to manage lightweight CAs
  [17/30]: Ensure lightweight CAs container exists
  [18/30]: Ensuring backward compatibility
  [19/30]: destroying installation admin user
  [20/30]: starting certificate server instance
  [21/30]: Finalize replication settings
  [22/30]: configure certmonger for renewals
  [23/30]: Importing RA key
Error storing key "keys/ra/ipaCert": CalledProcessError(Command ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-'] returned non-zero exit status 1: 'Traceback (most recent call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line 8, in <module>\n    main(ra_agent_parser())\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 114, in main\n    common.main(parser, export_key, import_key)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py", line 73, in main\n    func(args, tmpdir, **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 69, in import_key\n    ipautil.run(cmd, umask=0o027)\n  File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run\n    raise CalledProcessError(\nipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-in', '/tmp/tmp5koo8ca2/import.p12', '-clcerts', '-nokeys', '-out', '/var/lib/ipa/ra-agent.pem', '-password', 'file:/tmp/tmp5koo8ca2/passwd'] returned non-zero exit status 1: 'Error outputting keys and certificates\n802B104A807F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()\n')\n')
  [error] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

[Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

[Florence Blanc-Renaud]

Hi,

On Mon, Jul 4, 2022 at 5:07 PM Ivars Strazdins via FreeIPA-users <
[e-mail]> wrote:
> Hi guys,
> I am installing IPA replica on RHEL9 (well, Alma Linux 9 actually) and got
> exactly the same issue as here:
> https://access.redhat.com/discussions/6961739
> And similarly to the poster of that issue, also my IPA master server is
> IPA 4.6.8 on Centos7.
>
> I was trying to migrate IPA to a newer version by using Alma Linux 9.
> I removed Centos 7 replica and tried to install Alma Linux 9 replica. IPA
> client was installed without issues.
> No SELinux alerts.
> Content of /var/lib/ipa folder:
>
> [root@fricka ~]# ls /var/lib/ipa
>
> backup  certs  gssproxy  passwds  pki-ca  private  ra-agent.pem  sysrestore  sysupgrade
>
> Any suggestions how this could be resolved?
> Thank you in advance,
> Ivars
>
> Log of replica install:
> ….
> Starting replication, please wait until this has completed.
> Update in progress, 9 seconds elapsed
> Update succeeded
>
> [3/30]: creating ACIs for admin
>   [4/30]: creating installation admin user
>   [5/30]: configuring certificate server instance
>   [6/30]: stopping certificate server instance to update CS.cfg
>   [7/30]: backing up CS.cfg
>   [8/30]: Add ipa-pki-wait-running
>   [9/30]: secure AJP connector
>   [10/30]: reindex attributes
>   [11/30]: exporting Dogtag certificate store pin
>   [12/30]: disabling nonces
>   [13/30]: set up CRL publishing
>   [14/30]: enable PKIX certificate path discovery and validation
>   [15/30]: authorizing RA to modify profiles
>   [16/30]: authorizing RA to manage lightweight CAs
>   [17/30]: Ensure lightweight CAs container exists
>   [18/30]: Ensuring backward compatibility
>   [19/30]: destroying installation admin user
>   [20/30]: starting certificate server instance
>   [21/30]: Finalize replication settings
>   [22/30]: configure certmonger for renewals
>   [23/30]: Importing RA key
> Error storing key "keys/ra/ipaCert": CalledProcessError(Command
> ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-']
> returned non-zero exit status 1: 'Traceback (most recent
> call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line
> 8, in <module>\n    main(ra_agent_parser())\n  File
> "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py",
> line 114, in main\n    common.main(parser, export_key, import_key)\n  File
> "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py",
> line 73, in main\n    func(args, tmpdir,
> **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py",
> line 69, in import_key\n    ipautil.run(cmd, umask=0o027)\n  File
> "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in
> run\n    raise CalledProcessError(\nipapython.ipautil.CalledProcessError:
> CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-in',
> '/tmp/tmp5koo8ca2/import.p12', '-clcerts', '-nokeys', '-out',
> '/var/lib/ipa/ra-agent.pem', '-password',
> 'file:/tmp/tmp5koo8ca2/passwd'] returned non-zero exit status 1: 'Error
> outputting keys and certificates\n802B104A807F0000:error:0308010C:digital
> envelope
> routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global
> default library context, Algorithm (RC2-40-CBC : 0), Properties ()\n')\n')
>   [error] FileNotFoundError: [Errno 2] No such file or directory:
> '/var/lib/ipa/ra-agent.key'
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
> The ipa-replica-install command failed. See
> /var/log/ipareplica-install.log for more information
>
> This error looks like issue #9101 [1] / BZ #2032806 [2].

To be able to install a RHEL9 replica, I think you will have to install
first a RHEL8 replica (or CentOS8, but a version with the fix for #9101),
then install the RHEL9 replica from the RHEL8 replica.

HTH,
flo

[1] https://pagure.io/freeipa/issue/9101
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2032806
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure

[Ivars Strazdins]

Hi Florence,
followed the advice and installed RHEL 8 replica first (Alma Linux 8.6), then from that went to RHEL 9 (Alma Linux 9.0) and all is good now.
In more detail, I had 3 replicas:

Beginning:
R1 (Centos 7), R2 (Centos 7), R3 (Centos 7)

After Step 1, upgrade R2 to Alma Linux 8.6
R1 (Centos 7), R2 (Alma Linux 8.6), R3 (Centos 7)

After Step 2, upgrade R1 to Alma Linux 9.0
R1 (Alma Linux 9.0), R2 (Alma Linux 8.6), R3 (Centos 7)

After Step 3, upgrade R2 to Alma Linux 9.0
R1 (Alma Linux 9.0), R2 (Alma Linux 9.0), R3 (Centos 7)

After Step 4, drop Centos 7
R1 (Alma Linux 9.0), R2 (Alma Linux 9.0)

Thanks!
Ivars
> On 5 Jul 2022, at 09:33, Florence Blanc-Renaud  wrote:
>
> Hi,
>
> On Mon, Jul 4, 2022 at 5:07 PM Ivars Strazdins via FreeIPA-users <[e-mail] > wrote:
> Hi guys,
> I am installing IPA replica on RHEL9 (well, Alma Linux 9 actually) and got exactly the same issue as here: https://access.redhat.com/discussions/6961739 https://access.redhat.com/discussions/6961739
> And similarly to the poster of that issue, also my IPA master server is IPA 4.6.8 on Centos7.
>
> I was trying to migrate IPA to a newer version by using Alma Linux 9.
> I removed Centos 7 replica and tried to install Alma Linux 9 replica. IPA client was installed without issues.
> No SELinux alerts.
> Content of /var/lib/ipa folder:
> [root@fricka ~]# ls /var/lib/ipa
> backup  certs  gssproxy  passwds  pki-ca  private  ra-agent.pem  sysrestore  sysupgrade
>
> Any suggestions how this could be resolved?
> Thank you in advance,
> Ivars
>
> Log of replica install:
> ….
> Starting replication, please wait until this has completed.
> Update in progress, 9 seconds elapsed
> Update succeeded
>
> [3/30]: creating ACIs for admin
>   [4/30]: creating installation admin user
>   [5/30]: configuring certificate server instance
>   [6/30]: stopping certificate server instance to update CS.cfg
>   [7/30]: backing up CS.cfg
>   [8/30]: Add ipa-pki-wait-running
>   [9/30]: secure AJP connector
>   [10/30]: reindex attributes
>   [11/30]: exporting Dogtag certificate store pin
>   [12/30]: disabling nonces
>   [13/30]: set up CRL publishing
>   [14/30]: enable PKIX certificate path discovery and validation
>   [15/30]: authorizing RA to modify profiles
>   [16/30]: authorizing RA to manage lightweight CAs
>   [17/30]: Ensure lightweight CAs container exists
>   [18/30]: Ensuring backward compatibility
>   [19/30]: destroying installation admin user
>   [20/30]: starting certificate server instance
>   [21/30]: Finalize replication settings
>   [22/30]: configure certmonger for renewals
>   [23/30]: Importing RA key
> Error storing key "keys/ra/ipaCert": CalledProcessError(Command ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-'] returned non-zero exit status 1: 'Traceback (most recent call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line 8, in <module>\n    main(ra_agent_parser())\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 114, in main\n    common.main(parser, export_key, import_key)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py", line 73, in main\n    func(args, tmpdir, **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py", line 69, in import_key\n    ipautil.run(cmd, umask=0o027)\n  File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in run\n    raise CalledProcessError(\nipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-in', '/tmp/tmp5koo8ca2/import.p12', '-clcerts', '-nokeys', '-out', '/var/lib/ipa/ra-agent.pem', '-password', 'file:/tmp/tmp5koo8ca2/passwd'] returned non-zero exit status 1: 'Error outputting keys and certificates\n802B104A807F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()\n')\n')
>   [error] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
> The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
>
> This error looks like issue #9101 [1] / BZ #2032806 [2].
> To be able to install a RHEL9 replica, I think you will have to install first a RHEL8 replica (or CentOS8, but a version with the fix for #9101), then install the RHEL9 replica from the RHEL8 replica.
>
> HTH,
> flo
>
> [1] https://pagure.io/freeipa/issue/9101 https://pagure.io/freeipa/issue/9101
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=2032806 https://bugzilla.redhat.com/show_bug.cgi?id=2032806
>
> FreeIPA-users mailing list -- [e-mail] 
> To unsubscribe send an email to [e-mail] 
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]... https://lists.fedorahosted.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure https://pagure.io/fedora-infrastructure

[Florence Blanc-Renaud]

Great, thanks for the update!

On Wed, Jul 6, 2022 at 4:37 PM Ivars Strazdins 
wrote:
> Hi Florence,
> followed the advice and installed RHEL 8 replica first (Alma Linux 8.6),
> then from that went to RHEL 9 (Alma Linux 9.0) and all is good now.
> In more detail, I had 3 replicas:
>
> Beginning:
> R1 (Centos 7), R2 (Centos 7), R3 (Centos 7)
>
> After Step 1, upgrade R2 to Alma Linux 8.6
> R1 (Centos 7), R2 (Alma Linux 8.6), R3 (Centos 7)
>
> After Step 2, upgrade R1 to Alma Linux 9.0
> R1 (Alma Linux 9.0), R2 (Alma Linux 8.6), R3 (Centos 7)
>
> After Step 3, upgrade R2 to Alma Linux 9.0
> R1 (Alma Linux 9.0), R2 (Alma Linux 9.0), R3 (Centos 7)
>
> After Step 4, drop Centos 7
> R1 (Alma Linux 9.0), R2 (Alma Linux 9.0)
>
> Thanks!
> Ivars
>
> On 5 Jul 2022, at 09:33, Florence Blanc-Renaud  wrote:
>
> Hi,
>
> On Mon, Jul 4, 2022 at 5:07 PM Ivars Strazdins via FreeIPA-users <
> [e-mail]> wrote:
> > Hi guys,
> > I am installing IPA replica on RHEL9 (well, Alma Linux 9 actually) and
> > got exactly the same issue as here:
> > https://access.redhat.com/discussions/6961739
> > And similarly to the poster of that issue, also my IPA master server is
> > IPA 4.6.8 on Centos7.
> >
> > I was trying to migrate IPA to a newer version by using Alma Linux 9.
> > I removed Centos 7 replica and tried to install Alma Linux 9 replica. IPA
> > client was installed without issues.
> > No SELinux alerts.
> > Content of /var/lib/ipa folder:
> >
> > [root@fricka ~]# ls /var/lib/ipa
> >
> > backup  certs  gssproxy  passwds  pki-ca  private  ra-agent.pem  sysrestore  sysupgrade
> >
> > Any suggestions how this could be resolved?
> > Thank you in advance,
> > Ivars
> >
> > Log of replica install:
> > ….
> > Starting replication, please wait until this has completed.
> > Update in progress, 9 seconds elapsed
> > Update succeeded
> >
> > [3/30]: creating ACIs for admin
> >   [4/30]: creating installation admin user
> >   [5/30]: configuring certificate server instance
> >   [6/30]: stopping certificate server instance to update CS.cfg
> >   [7/30]: backing up CS.cfg
> >   [8/30]: Add ipa-pki-wait-running
> >   [9/30]: secure AJP connector
> >   [10/30]: reindex attributes
> >   [11/30]: exporting Dogtag certificate store pin
> >   [12/30]: disabling nonces
> >   [13/30]: set up CRL publishing
> >   [14/30]: enable PKIX certificate path discovery and validation
> >   [15/30]: authorizing RA to modify profiles
> >   [16/30]: authorizing RA to manage lightweight CAs
> >   [17/30]: Ensure lightweight CAs container exists
> >   [18/30]: Ensuring backward compatibility
> >   [19/30]: destroying installation admin user
> >   [20/30]: starting certificate server instance
> >   [21/30]: Finalize replication settings
> >   [22/30]: configure certmonger for renewals
> >   [23/30]: Importing RA key
> > Error storing key "keys/ra/ipaCert": CalledProcessError(Command
> > ['/usr/libexec/ipa/custodia/ipa-custodia-ra-agent', '--import', '-']
> > returned non-zero exit status 1: 'Traceback (most recent
> > call last):\n  File "/usr/libexec/ipa/custodia/ipa-custodia-ra-agent", line
> > 8, in <module>\n    main(ra_agent_parser())\n  File
> > "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py",
> > line 114, in main\n    common.main(parser, export_key, import_key)\n  File
> > "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/common.py",
> > line 73, in main\n    func(args, tmpdir,
> > **kwargs)\n  File "/usr/lib/python3.9/site-packages/ipaserver/secrets/handlers/pemfile.py",
> > line 69, in import_key\n    ipautil.run(cmd, umask=0o027)\n  File
> > "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 598, in
> > run\n    raise CalledProcessError(\nipapython.ipautil.CalledProcessError:
> > CalledProcessError(Command ['/usr/bin/openssl', 'pkcs12', '-in',
> > '/tmp/tmp5koo8ca2/import.p12', '-clcerts', '-nokeys', '-out',
> > '/var/lib/ipa/ra-agent.pem', '-password',
> > 'file:/tmp/tmp5koo8ca2/passwd'] returned non-zero exit status 1: 'Error
> > outputting keys and certificates\n802B104A807F0000:error:0308010C:digital
> > envelope
> > routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global
> > default library context, Algorithm (RC2-40-CBC : 0), Properties ()\n')\n')
> >   [error] FileNotFoundError: [Errno 2] No such file or directory:
> > '/var/lib/ipa/ra-agent.key'
> > Your system may be partly configured.
> > Run /usr/sbin/ipa-server-install --uninstall to clean up.
> >
> > [Errno 2] No such file or directory: '/var/lib/ipa/ra-agent.key'
> > The ipa-replica-install command failed. See
> > /var/log/ipareplica-install.log for more information
> >
> > This error looks like issue #9101 [1] / BZ #2032806 [2].
>
> To be able to install a RHEL9 replica, I think you will have to install
> first a RHEL8 replica (or CentOS8, but a version with the fix for #9101),
> then install the RHEL9 replica from the RHEL8 replica.
>
> HTH,
> flo
>
> [1] https://pagure.io/freeipa/issue/9101
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=2032806
> > FreeIPA-users mailing list -- [e-mail]
> > To unsubscribe send an email to
> > [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam on the list, report it:
> > https://pagure.io/fedora-infrastructure

[glenn w]

Here is a simpler workaround using a three-line Python source patch.

First, the title of this thread is, perhaps, misleading.  The bug occurs when installing a replica running a newer version of freeIPA than what is running on the server you are replicating from.  I don't know the entire range of versions, but in my case, the master server is running freeIPA 4.9.6 and the replica has 4.9.10.  The package for which the fix was tested is ipa-server-4.9.6-10 so I guess that is the boundary.

If only I could upgrade my master server to 4.9.6-10 or later.  But it is running CentOS 8 which, I believe, was dead-ended by CentOS Stream. And I didn't want to go through the version hell associated with trying to put 4.9.6 freeIPA on my Fedora replica/client.

But patching the server turned out to be a piece of cake.  The fix is three lines in one python file: https://github.com/freeipa/freeipa/pull/6155/commits/018720248ab64300d903641...

On the master server, add these lines to /usr/lib/python3.6/site-packages/ipaserver/secrets/handlers/pemfile.py (path might differ):

'-keypbe', 'AES-256-CBC',
         '-certpbe', 'AES-256-CBC',
         '-macalg', 'sha384',

at the end of the call to ipautil.run() in the export_key() function ... around line 34 (see referenced commit delta).

Then delete the contents of the __pycache__ subdirectory (because I am always suspicious of caches).
Then reboot (because I am always suspicious of caches).

It took a while for my rebooted server's time to sync to good sources.  After that the ipa replica install succeeded.