New subject: ipa upgrade failed

22 Jun 2024

Dear Alexander,

On Wednesday, 21 December 2022 10:10 a.m., Martin (Lists)
<https://lists.fedorahosted.org/archives/users/233826406258280183065251682616278220915/>
wrote:

    Hallo all
    I have a strange issue with one of my ipa servers. after an
    upgrade from
    fedora 35 to fedora 37 the ipa-server-upgrade failed on the
    pki-tomcat
    part. The ipaupgrade.log says:
    2022-12-21T15:27:52Z INFO Migrating profile
    'caECFullCMCSharedTokenCert'
    2022-12-21T15:27:52Z DEBUG request GET
    https://ipa1.server.org:8443/ca/rest/account/login
    2022-12-21T15:27:52Z DEBUG request body ''
    2022-12-21T15:27:52Z DEBUG response status 404
    2022-12-21T15:27:52Z DEBUG response headers Content-Type:
    text/html;charset=utf-8
    Content-Language: de
    Content-Length: 795
    Date: Wed, 21 Dec 2022 15:27:52 GMT
    2022-12-21T15:27:52Z DEBUG response body (decoded): b'<!doctype
    html><html lang="de"><head><title>HTTP Status 404
    \xe2\x80\x93 nicht
    gefunden</title><style
    type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1,
    h2, h3,
    b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2
    {font-size:16px;
    } h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line
    {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
    Status 40
    4 \xe2\x80\x93 nicht gefunden</h1><hr class="line"
    /><p><b>Type</b>
    Status Report</p><p><b>Message</b> The requested resource
    [&#47;ca&#47;rest&#47;account
    &#47;login] is not available</p><p><b>Beschreibung</b> The
    origin server
    did not find a current representation for the target resource or
    is not
    willing to
    disclose that one exists.</p><hr class="line" /><h3>Apache
    Tomcat/9.0.68</h3></body></html>'
    2022-12-21T15:27:52Z ERROR IPA server upgrade failed: Inspect
    /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
    2022-12-21T15:27:52Z DEBUG   File
    "/usr/lib/python3.11/site-packages/ipapython/admintool.py", line
    180, in
    execute
        return_value = self.run()
                       ^^^^^^^^^^
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/ipa_server_upgrade.py",

    line 54, in run    server.upgrade()
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",

    line 2061, in upgrade    upgrade_configuration()
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",

    line 1914, in upgrade_configuration  
     ca_enable_ldap_profile_subsystem(ca)
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/server/upgrade.py",

    line 458, in ca_enable_ldap_profile_subsystem    
    cainstance.migrate_profiles_to_ldap()
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py",

    line 2155, in migrate_profiles_to_ldap    
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
      File
    "/usr/lib/python3.11/site-packages/ipaserver/install/cainstance.py",

    line 2209, in _create_dogtag_profile    with
    api.Backend.ra_certprofile
    as profile_api:
      File
    "/usr/lib/python3.11/site-packages/ipaserver/plugins/dogtag.py",
    line 1211, in __enter__    raise
    errors.RemoteRetrieveError(reason=_('Failed to authenticate to
    CA REST
    API'))
    2022-12-21T15:27:52Z DEBUG The ipa-server-upgrade command failed,
    exception: RemoteRetrieveError: Failed to authenticate to CA
    REST API
    The catalina logfile says:
    21-Dec-2022 16:27:26.946 SCHWERWIEGEND [main]
    org.apache.catalina.core.StandardContext.startInternal One or more
    listeners failed to start. Full details will be found in the
    appropriate
    container log file
    21-Dec-2022 16:27:26.948 SCHWERWIEGEND [main]
    org.apache.catalina.core.StandardContext.startInternal Context
    [/ca]
    startup failed due to previous errors
    the CA debug log file says:
    2022-12-21 16:27:26 [main] FINE: LdapBoundConnection: Connecting to
    ipa1.server.org:636 <http://ipa1.server.org:636> with client
    cert auth
    2022-12-21 16:27:26 [main] FINE:
    ldapconn/PKISocketFactory.makeSSLSocket: begins
    2022-12-21 16:27:26 [main] FINE: SignedAuditLogger: event
    CLIENT_ACCESS_SESSION_ESTABLISH
    2022-12-21 16:27:26 [main] SEVERE: Unable to create socket:
    java.net.ConnectException: Verbindungsaufbau abgelehnt
    with many java traceback errors following. directory server is
    running
    at this time and there is no connection reported at the given time.
    ipa-healthceck does not give anny errors or warnings.
    Re-starting the
    pki-tomcat server manually afterwards ist working fine and does
    not give
    any errors. starting ipa in force mode gives no errors as well.
    What can
    I do?
    Regards
    Martin

FWIW, I used the hyperkitty web reply link, which gives a direct
mailto link and no option to add anything, and unfortunately didn't
realize the context would be missing. Sorry about that (and for the
double reply spam).
I was replying to
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/message/5PC566RTR2XDCSV5MYLM7QJZBXART535/
(also quoted at the start of this e-mail this time).

I see the exact same error as above upgrading either F35->F36 or
F35->F37 (and I also tried 37->38 on top in case it is somehow
resolved by some later patch).
I am currently on F35/freeipa-server 4.9.11. And ipa-healthcheck
finds no issues (besides missing URI records). It's a single master
server.

The second error mentioned later in the thread
(55-pbacmemberof.update) for me does in fact show up in an earlier
upgrade to reach Fedora 35/freeipa-server 4.9.11. However, the
upgrade returned exit 0 anyway on F34->F35. And the error does not
occur again on the later attempted updates F35->F36 or F35->F37. So
I think it's probably unrelated.

On F35 everything *appears* to be working fine, no expired certs,
except for the fact that the freeipa web interface seems to be showing 
1) a number of expired certs (old ones)
2) a number of greyed-out certs without name (and higher serial # -
the actual renewed ones)
getcert list shows all certs as normal (I assume it grabs them from
LDAP) - but I see that /etc/pki/pki-tomcat/alias NSSDB last
modification date is from before their issuing - so it seems it was
not updated properly for some reason.

I suspect this might be related to the upgrade failure I am seeing.
Is there some easy way to refresh the pki-tomcat alias NSSDB without
reissuing certs from those stored in LDAP?

Best wishes,
Johannes

On Mon, 24 Jun 2024 at 08:44, Alexander Bokovoy <abokovoy@redhat.com
<mailto:abokovoy@redhat.com>> wrote:

    On Няд, 23 чэр 2024, Johannes Falke via FreeIPA-users wrote:
    >   How did you actually manage to resolve this issue? I'm
    seeing the same       
    >   thing trying to upgrade either f35->f37 or f35->f36 (and NO
    ldap errors).   
    >   On f35, freeipa says it's healthy.                         

    Just a reminder: this is a mailing list, not a forum. If you are
    using
    lists.fedorahosted.org <http://lists.fedorahosted.org> web
    interface, make sure to check the box to
    quote the email you are answering to because otherwise this message
    appears without a reference to the thread you are trying to
    reply to.

    As such, this email contains no reference to existing discussion
    of the
    same topic and it is literally impossible to identify what do
    you mean
    by 'resolve this issue'.

    Also, lists.fedorahosted.org <http://lists.fedorahosted.org> web
    interface gives you a way to search the
    list archives and see the whole discussion there. It should help in
    searching for successful resolutions of many problems. If you still
    cannot find a solution, please make sure to include enough
    details in
    your new email thread to provide information that will allow list
    members to help you.

    -- 
    / Alexander Bokovoy
    Sr. Principal Software Engineer
    Security / Identity Management Engineering
    Red Hat Limited, Finland

Re: ipa upgrade failed