New subject: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start

Wednesday, 20 June 2018

Hello all!

I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...

ipa-server-upgrade fails as below

--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
--

And the log tells me that CA returns status 500

--
DEBUG Waiting for CA to start...
DEBUG request POST
http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus<http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close

DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color
: black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem
unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b> Exception
report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The server
encountered an internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76
logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with
status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command
ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in
execute
    return_value = self.run()
  File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line
48, in run
    raise admintool.ScriptError(str(e))

The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
ERROR CA did not start in 300.0s
--

With command "ipactl start --ignore-service-failures" I can start all the
services but pki-tomcatd.

--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--

Suggested resolution to above problem doesn't help me since the LDAP and NSS DB seem
to have same certificates (some difference in wrapping but the string is same if I take
out the line breaks) and even the serial number matches.

--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD...
...Prh2G
-----END CERTIFICATE-----

certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |grep
Serial
        Serial Number: 4 (0x4)

ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD...
...Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--

And here's where my actual knowledge of things end. I've been trying to figure out
all kind of logs (tomcat, Kerberos, directory server, ...) but haven't found a solid
reason for it. I'm starting to believe this is a certificate issue, because although
"getcert list" tells me that the certificate status is "Monitoring" on
all certificates the expiry date is already in the past (current date 20.6.2018,
certificate expiry 21.03.2018) on 4 certificates and it won't update even if I
resubmit it or delete certificate and manually redo it (it got the same date as the
"old ones"). The "main certs" ("caSigningCert cert-pki-ca",
"Server-Cert cert-pki-ca" and two directory server certs) are valid for years
(until 2020+).

--
Request ID '20160331084234':
        status: MONITORING
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
        certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=<<REALM>>
        subject: CN=OCSP Subsystem,O=<<REALM>>
        expires: 2018-03-21 09:42:04 UTC
        key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
        eku: id-kp-OCSPSigning
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert
cert-pki-ca"
        track: yes
        auto-renew: yes
Request ID '20160331085008':
        status: MONITORING
        stuck: no
        key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=<<REALM>>
        subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
        expires: 2020-03-04 09:58:23 UTC
        principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command:
        post-save command: /usr/lib64/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
--

Has anyone else bumped into same kind of issues? Any ideas where I should continue
looking? I'm starting to run out of ideas...

Eemeli Jokinen

2024

2023

2022

2021

2020

2019

2018

2017

Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start