From Eemeli.Jokinen at cinia.fi Wed Jun 20 11:53:40 2018 Content-Type: multipart/mixed; boundary="===============1363744815246032696==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 20 Jun 2018 11:53:09 +0000 Message-ID: <9dbd45efa82b4a9c943e1e3bd8071d43@cinia.fi> --===============1363744815246032696== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello all! I have very similiar problem as this one: https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedoraho= sted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ ipa-server-upgrade fails as below -- Update complete Upgrading IPA services Upgrading the configuration of the IPA services [Verifying that root certificate is published] [Migrate CRL publish directory] CRL tree already moved [Verifying that CA proxy configuration is correct] IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command = ipa-server-upgrade manually. CA did not start in 300.0s The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more= information -- And the log tells me that CA returns status 500 -- DEBUG Waiting for CA to start... DEBUG request POST http://<>:8080/ca/admin/ca/getStatus DEBUG request body '' DEBUG response status 500 DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=3Dutf-8 Content-Language: en Content-Length: 2208 Date: Fri, 15 Jun 2018 10:05:29 GMT Connection: close DEBUG response body 'Apache Tomcat/7.0.76 - Error report=

HTTP Status 500 - Subsystem unavailable

type Exception report

= message Subsystem unavailable

description The= server encountered an internal error that prevented it from fulfilling thi= s request.

exception

javax.ws.rs.ServiceUnavailableEx=
ception: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSe=
curityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator=
.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalin=
a.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.c=
atalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache=
.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.ap=
ache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.=
java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.=
process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoin=
t$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadP=
oolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent=
.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.t=
omcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tj=
ava.lang.Thread.run(Thread.java:748)\n

note The full = stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs= .

Apache Tomcat/7.0.76

' DEBUG The CA status is: check interrupted due to error: Retrieving CA statu= s failed with status 500 DEBUG Waiting for CA to start... ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run co= mmand ipa-server-upgrade manually. File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, i= n execute return_value =3D self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgra= de.py", line 48, in run raise admintool.ScriptError(str(e)) The ipa-server-upgrade command failed, exception: ScriptError: CA did not s= tart in 300.0s ERROR CA did not start in 300.0s -- With command "ipactl start --ignore-service-failures" I can start all the s= ervices but pki-tomcatd. -- Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: RUNNING httpd Service: RUNNING pki-tomcatd Service: STOPPED smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful -- Suggested resolution to above problem doesn't help me since the LDAP and NS= S DB seem to have same certificates (some difference in wrapping but the st= ring is same if I take out the line breaks) and even the serial number matc= hes. -- certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDjD... ...Prh2G -----END CERTIFICATE----- certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' |gr= ep Serial Serial Number: 4 (0x4) ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b uid=3Dpkidbuser,ou=3Dpeop= le,o=3Dipaca userCertificate description seeAlso Enter LDAP Password: dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate:: MIIDjD... ...Prh2 G description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsystem, O=3D<> seeAlso: CN=3DCA Subsystem,O=3D<> -- And here's where my actual knowledge of things end. I've been trying to fig= ure out all kind of logs (tomcat, Kerberos, directory server, ...) but have= n't found a solid reason for it. I'm starting to believe this is a certific= ate issue, because although "getcert list" tells me that the certificate st= atus is "Monitoring" on all certificates the expiry date is already in the = past (current date 20.6.2018, certificate expiry 21.03.2018) on 4 certifica= tes and it won't update even if I resubmit it or delete certificate and man= ually redo it (it got the same date as the "old ones"). The "main certs" ("= caSigningCert cert-pki-ca", "Server-Cert cert-pki-ca" and two directory ser= ver certs) are valid for years (until 2020+). -- Request ID '20160331084234': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pi= n set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DOCSP Subsystem,O=3D<> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspS= igningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331085008': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nickna= me=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias= /pwdfile.txt' certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D= 'Server-Cert',token=3D'NSS Certificate DB' CA: IPA issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2020-03-04 09:58:23 UTC principal name: HTTP/<>@<> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes -- Has anyone else bumped into same kind of issues? Any ideas where I should c= ontinue looking? I'm starting to run out of ideas... Eemeli Jokinen --===============1363744815246032696== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv VFIvUkVDLWh0bWw0MCI+CjxoZWFkPgo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNv bnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD11cy1hc2NpaSI+CjxtZXRhIG5hbWU9IkdlbmVyYXRv ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPgo8c3R5bGU+ PCEtLQovKiBGb250IERlZmluaXRpb25zICovCkBmb250LWZhY2UKCXtmb250LWZhbWlseToiQ2Ft YnJpYSBNYXRoIjsKCXBhbm9zZS0xOjIgNCA1IDMgNSA0IDYgMyAyIDQ7fQpAZm9udC1mYWNlCgl7 Zm9udC1mYW1pbHk6Q2FsaWJyaTsKCXBhbm9zZS0xOjIgMTUgNSAyIDIgMiA0IDMgMiA0O30KLyog U3R5bGUgRGVmaW5pdGlvbnMgKi8KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05v cm1hbAoJe21hcmdpbjowY207CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7Cglmb250LXNpemU6MTEu MHB0OwoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7Cgltc28tZmFyZWFzdC1sYW5n dWFnZTpFTi1VUzt9CmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsKCXttc28tc3R5bGUtcHJpb3Jp dHk6OTk7Cgljb2xvcjojMDU2M0MxOwoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9CmE6dmlz aXRlZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZAoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsK CWNvbG9yOiM5NTRGNzI7Cgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30Kc3Bhbi5FbWFpbFN0 eWxlMTcKCXttc28tc3R5bGUtdHlwZTpwZXJzb25hbC1jb21wb3NlOwoJZm9udC1mYW1pbHk6IkNh bGlicmkiLHNhbnMtc2VyaWY7Cgljb2xvcjp3aW5kb3d0ZXh0O30KLk1zb0NocERlZmF1bHQKCXtt c28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsKCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNl cmlmOwoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVM7fQpAcGFnZSBXb3JkU2VjdGlvbjEKCXtz aXplOjYxMi4wcHQgNzkyLjBwdDsKCW1hcmdpbjo3Mi4wcHQgNzIuMHB0IDcyLjBwdCA3Mi4wcHQ7 fQpkaXYuV29yZFNlY3Rpb24xCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQotLT48L3N0eWxlPjwhLS1b aWYgZ3RlIG1zbyA5XT48eG1sPgo8bzpzaGFwZWRlZmF1bHRzIHY6ZXh0PSJlZGl0IiBzcGlkbWF4 PSIxMDI2IiAvPgo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4KPG86 c2hhcGVsYXlvdXQgdjpleHQ9ImVkaXQiPgo8bzppZG1hcCB2OmV4dD0iZWRpdCIgZGF0YT0iMSIg Lz4KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0tPgo8L2hlYWQ+Cjxib2R5IGxhbmc9 IkZJIiBsaW5rPSIjMDU2M0MxIiB2bGluaz0iIzk1NEY3MiI+CjxkaXYgY2xhc3M9IldvcmRTZWN0 aW9uMSI+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5IZWxsbyBhbGwh PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+SSBoYXZlIHZlcnkgc2ltaWxpYXIgcHJvYmxlbSBhcyB0aGlzIG9u ZTo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9 IkVOLVVTIj48YSBocmVmPSJodHRwczovL2xpc3RzLmZlZG9yYWhvc3RlZC5vcmcvYXJjaGl2ZXMv bGlzdC9mcmVlaXBhLXVzZXJzQGxpc3RzLmZlZG9yYWhvc3RlZC5vcmcvdGhyZWFkL1lVNlRaSE9K QVY1UUhISFBRV0pIWVgzRlA0T0hBMzdYLyI+aHR0cHM6Ly9saXN0cy5mZWRvcmFob3N0ZWQub3Jn L2FyY2hpdmVzL2xpc3QvZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3JnL3RocmVh ZC9ZVTZUWkhPSkFWNVFISEhQUVdKSFlYM0ZQNE9IQTM3WC88L2E+PG86cD48L286cD48L3NwYW4+ PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8 L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+ aXBhLXNlcnZlci11cGdyYWRlIGZhaWxzIGFzIGJlbG93PG86cD48L286cD48L3NwYW4+PC9wPgo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+LS08bzpw PjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT Ij5VcGRhdGUgY29tcGxldGU8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5VcGdyYWRpbmcgSVBBIHNlcnZpY2VzPG86cD48L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+VXBncmFk aW5nIHRoZSBjb25maWd1cmF0aW9uIG9mIHRoZSBJUEEgc2VydmljZXM8bzpwPjwvbzpwPjwvc3Bh bj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5bVmVyaWZ5aW5n IHRoYXQgcm9vdCBjZXJ0aWZpY2F0ZSBpcyBwdWJsaXNoZWRdPG86cD48L286cD48L3NwYW4+PC9w Pgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+W01pZ3JhdGUgQ1JMIHB1 Ymxpc2ggZGlyZWN0b3J5XTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gbGFuZz0iRU4tVVMiPkNSTCB0cmVlIGFscmVhZHkgbW92ZWQ8bzpwPjwvbzpwPjwv c3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5bVmVyaWZ5 aW5nIHRoYXQgQ0EgcHJveHkgY29uZmlndXJhdGlvbiBpcyBjb3JyZWN0XTxvOnA+PC9vOnA+PC9z cGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPklQQSBzZXJ2 ZXIgdXBncmFkZSBmYWlsZWQ6IEluc3BlY3QgL3Zhci9sb2cvaXBhdXBncmFkZS5sb2cgYW5kIHJ1 biBjb21tYW5kIGlwYS1zZXJ2ZXItdXBncmFkZSBtYW51YWxseS48bzpwPjwvbzpwPjwvc3Bhbj48 L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5DQSBkaWQgbm90IHN0 YXJ0IGluIDMwMC4wczxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiPlRoZSBpcGEtc2VydmVyLXVwZ3JhZGUgY29tbWFuZCBmYWlsZWQu IFNlZSAvdmFyL2xvZy9pcGF1cGdyYWRlLmxvZyBmb3IgbW9yZSBpbmZvcm1hdGlvbjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0t PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+QW5kIHRoZSBsb2cgdGVsbHMgbWUgdGhhdCBDQSByZXR1cm5zIHN0 YXR1cyA1MDA8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4tLTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xh c3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkRFQlVHIFdhaXRpbmcgZm9yIENBIHRv IHN0YXJ0Li4uPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBsYW5nPSJFTi1VUyI+REVCVUcgcmVxdWVzdCBQT1NUIDxhIGhyZWY9Imh0dHA6Ly8lM2MlM2Np cGExLmZxZG4lM2UlM2U6ODA4MC9jYS9hZG1pbi9jYS9nZXRTdGF0dXMiPgpodHRwOi8vJmx0OyZs dDtpcGExLmZxZG4mZ3Q7Jmd0Ozo4MDgwL2NhL2FkbWluL2NhL2dldFN0YXR1czwvYT48bzpwPjwv bzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5E RUJVRyByZXF1ZXN0IGJvZHkgJyc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5ERUJVRyByZXNwb25zZSBzdGF0dXMgNTAwPG86cD48 L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+ REVCVUcgcmVzcG9uc2UgaGVhZGVycyBTZXJ2ZXI6IEFwYWNoZS1Db3lvdGUvMS4xPG86cD48L286 cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Q29u dGVudC1UeXBlOiB0ZXh0L2h0bWw7Y2hhcnNldD11dGYtODxvOnA+PC9vOnA+PC9zcGFuPjwvcD4K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkNvbnRlbnQtTGFuZ3VhZ2U6 IGVuPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5n PSJFTi1VUyI+Q29udGVudC1MZW5ndGg6IDIyMDg8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5EYXRlOiBGcmksIDE1IEp1biAyMDE4 IDEwOjA1OjI5IEdNVDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiPkNvbm5lY3Rpb246IGNsb3NlPG86cD48L286cD48L3NwYW4+PC9w Pgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286 cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+REVC VUcgcmVzcG9uc2UgYm9keSAnJmx0O2h0bWwmZ3Q7Jmx0O2hlYWQmZ3Q7Jmx0O3RpdGxlJmd0O0Fw YWNoZSBUb21jYXQvNy4wLjc2IC0gRXJyb3IgcmVwb3J0Jmx0Oy90aXRsZSZndDsmbHQ7c3R5bGUm Z3Q7Jmx0OyEtLUgxIHtmb250LWZhbWlseTpUYWhvbWEsQXJpYWwsc2Fucy1zZXJpZjtjb2xvcjp3 aGl0ZTtiYWNrZ3JvdW5kLWNvbG9yOiM1MjVENzY7Zm9udC1zaXplOjIycHg7fSBIMiB7Zm9udC1m YW1pbHk6VGFob21hLEFyaWFsLHNhbnMtc2VyaWY7Y29sb3I6d2hpdGU7YmFja2dyb3VuZC1jb2xv cjojNTI1RDc2O2ZvbnQtc2l6ZToxNnB4O30KIEgzIHtmb250LWZhbWlseTpUYWhvbWEsQXJpYWws c2Fucy1zZXJpZjtjb2xvcjp3aGl0ZTtiYWNrZ3JvdW5kLWNvbG9yOiM1MjVENzY7Zm9udC1zaXpl OjE0cHg7fSBCT0RZIHtmb250LWZhbWlseTpUYWhvbWEsQXJpYWwsc2Fucy1zZXJpZjtjb2xvcjpi bGFjaztiYWNrZ3JvdW5kLWNvbG9yOndoaXRlO30gQiB7Zm9udC1mYW1pbHk6VGFob21hLEFyaWFs LHNhbnMtc2VyaWY7Y29sb3I6d2hpdGU7YmFja2dyb3VuZC1jb2xvcjojNTI1RDc2O30gUCB7Zm9u dC1mYW1pbHk6VGFob21hLEFyaWFsLHNhbnMtc2VyaWY7YmFja2dyb3VuZDp3aGl0ZTtjb2xvcjpi bGFjaztmb250LXNpemU6MTJweDt9QQoge2NvbG9yIDogYmxhY2s7fUEubmFtZSB7Y29sb3IgOiBi bGFjazt9SFIge2NvbG9yIDogIzUyNUQ3Njt9LS0mZ3Q7Jmx0Oy9zdHlsZSZndDsgJmx0Oy9oZWFk Jmd0OyZsdDtib2R5Jmd0OyZsdDtoMSZndDtIVFRQIFN0YXR1cyA1MDAgLSBTdWJzeXN0ZW0gdW5h dmFpbGFibGUmbHQ7L2gxJmd0OyZsdDtIUiBzaXplPSZxdW90OzEmcXVvdDsgbm9zaGFkZT0mcXVv dDtub3NoYWRlJnF1b3Q7Jmd0OyZsdDtwJmd0OyZsdDtiJmd0O3R5cGUmbHQ7L2ImZ3Q7IEV4Y2Vw dGlvbiByZXBvcnQmbHQ7L3AmZ3Q7Jmx0O3AmZ3Q7Jmx0O2ImZ3Q7bWVzc2FnZSZsdDsvYiZndDsg Jmx0O3UmZ3Q7U3Vic3lzdGVtIHVuYXZhaWxhYmxlJmx0Oy91Jmd0OyZsdDsvcCZndDsmbHQ7cCZn dDsmbHQ7YiZndDtkZXNjcmlwdGlvbiZsdDsvYiZndDsKICZsdDt1Jmd0O1RoZSBzZXJ2ZXIgZW5j b3VudGVyZWQgYW4gaW50ZXJuYWwgZXJyb3IgdGhhdCBwcmV2ZW50ZWQgaXQgZnJvbSBmdWxmaWxs aW5nIHRoaXMgcmVxdWVzdC4mbHQ7L3UmZ3Q7Jmx0Oy9wJmd0OyZsdDtwJmd0OyZsdDtiJmd0O2V4 Y2VwdGlvbiZsdDsvYiZndDsgJmx0O3ByZSZndDtqYXZheC53cy5ycy5TZXJ2aWNlVW5hdmFpbGFi bGVFeGNlcHRpb246IFN1YnN5c3RlbSB1bmF2YWlsYWJsZVxuXHRjb20ubmV0c2NhcGUuY21zLnRv bWNhdC5Qcm94eVJlYWxtLmZpbmRTZWN1cml0eUNvbnN0cmFpbnRzKFByb3h5UmVhbG0uamF2YTox NDUpXG5cdG9yZy5hcGFjaGUuY2F0YWxpbmEuYXV0aGVudGljYXRvci5BdXRoZW50aWNhdG9yQmFz ZS5pbnZva2UoQXV0aGVudGljYXRvckJhc2UuamF2YTo1MDApXG5cdG9yZy5hcGFjaGUuY2F0YWxp bmEudmFsdmVzLkVycm9yUmVwb3J0VmFsdmUuaW52b2tlKEVycm9yUmVwb3J0VmFsdmUuamF2YTox MDMpXG5cdG9yZy5hcGFjaGUuY2F0YWxpbmEudmFsdmVzLkFjY2Vzc0xvZ1ZhbHZlLmludm9rZShB Y2Nlc3NMb2dWYWx2ZS5qYXZhOjk2Milcblx0b3JnLmFwYWNoZS5jYXRhbGluYS5jb25uZWN0b3Iu Q295b3RlQWRhcHRlci5zZXJ2aWNlKENveW90ZUFkYXB0ZXIuamF2YTo0NDUpXG5cdG9yZy5hcGFj aGUuY295b3RlLmh0dHAxMS5BYnN0cmFjdEh0dHAxMVByb2Nlc3Nvci5wcm9jZXNzKEFic3RyYWN0 SHR0cDExUHJvY2Vzc29yLmphdmE6MTA4Nylcblx0b3JnLmFwYWNoZS5jb3lvdGUuQWJzdHJhY3RQ cm90b2NvbCRBYnN0cmFjdENvbm5lY3Rpb25IYW5kbGVyLnByb2Nlc3MoQWJzdHJhY3RQcm90b2Nv bC5qYXZhOjYzNylcblx0b3JnLmFwYWNoZS50b21jYXQudXRpbC5uZXQuSklvRW5kcG9pbnQkU29j a2V0UHJvY2Vzc29yLnJ1bihKSW9FbmRwb2ludC5qYXZhOjMxNilcblx0amF2YS51dGlsLmNvbmN1 cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yLnJ1bldvcmtlcihUaHJlYWRQb29sRXhlY3V0b3IuamF2 YToxMTQ5KVxuXHRqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2Vy LnJ1bihUaHJlYWRQb29sRXhlY3V0b3IuamF2YTo2MjQpXG5cdG9yZy5hcGFjaGUudG9tY2F0LnV0 aWwudGhyZWFkcy5UYXNrVGhyZWFkJFdyYXBwaW5nUnVubmFibGUucnVuKFRhc2tUaHJlYWQuamF2 YTo2MSlcblx0amF2YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6NzQ4KVxuJmx0Oy9wcmUm Z3Q7Jmx0Oy9wJmd0OyZsdDtwJmd0OyZsdDtiJmd0O25vdGUmbHQ7L2ImZ3Q7CiAmbHQ7dSZndDtU aGUgZnVsbCBzdGFjayB0cmFjZSBvZiB0aGUgcm9vdCBjYXVzZSBpcyBhdmFpbGFibGUgaW4gdGhl IEFwYWNoZSBUb21jYXQvNy4wLjc2IGxvZ3MuJmx0Oy91Jmd0OyZsdDsvcCZndDsmbHQ7SFIgc2l6 ZT0mcXVvdDsxJnF1b3Q7IG5vc2hhZGU9JnF1b3Q7bm9zaGFkZSZxdW90OyZndDsmbHQ7aDMmZ3Q7 QXBhY2hlIFRvbWNhdC83LjAuNzYmbHQ7L2gzJmd0OyZsdDsvYm9keSZndDsmbHQ7L2h0bWwmZ3Q7 JzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0i RU4tVVMiPkRFQlVHIFRoZSBDQSBzdGF0dXMgaXM6IGNoZWNrIGludGVycnVwdGVkIGR1ZSB0byBl cnJvcjogUmV0cmlldmluZyBDQSBzdGF0dXMgZmFpbGVkIHdpdGggc3RhdHVzIDUwMDxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkRF QlVHIFdhaXRpbmcgZm9yIENBIHRvIHN0YXJ0Li4uPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+RVJST1IgSVBBIHNlcnZlciB1cGdy YWRlIGZhaWxlZDogSW5zcGVjdCAvdmFyL2xvZy9pcGF1cGdyYWRlLmxvZyBhbmQgcnVuIGNvbW1h bmQgaXBhLXNlcnZlci11cGdyYWRlIG1hbnVhbGx5LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkZpbGUgJnF1b3Q7L3Vzci9saWIv cHl0aG9uMi43L3NpdGUtcGFja2FnZXMvaXBhcHl0aG9uL2FkbWludG9vbC5weSZxdW90OywgbGlu ZSAxNzIsIGluIGV4ZWN1dGU8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsgcmV0dXJuX3ZhbHVlID0g c2VsZi5ydW4oKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gbGFuZz0iRU4tVVMiPiZuYnNwOyBGaWxlICZxdW90Oy91c3IvbGliL3B5dGhvbjIuNy9zaXRl LXBhY2thZ2VzL2lwYXNlcnZlci9pbnN0YWxsL2lwYV9zZXJ2ZXJfdXBncmFkZS5weSZxdW90Oywg bGluZSA0OCwgaW4gcnVuPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7IHJhaXNlIGFkbWludG9vbC5T Y3JpcHRFcnJvcihzdHIoZSkpPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+VGhlIGlwYS1zZXJ2ZXItdXBncmFk ZSBjb21tYW5kIGZhaWxlZCwgZXhjZXB0aW9uOiBTY3JpcHRFcnJvcjogQ0EgZGlkIG5vdCBzdGFy dCBpbiAzMDAuMHM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIj5FUlJPUiBDQSBkaWQgbm90IHN0YXJ0IGluIDMwMC4wczxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0t PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+V2l0aCBjb21tYW5kICYjODIyMDtpcGFjdGwgc3RhcnQgLS1pZ25v cmUtc2VydmljZS1mYWlsdXJlcyYjODIyMTsgSSBjYW4gc3RhcnQgYWxsIHRoZSBzZXJ2aWNlcyBi dXQgcGtpLXRvbWNhdGQuPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+LS08bzpwPjwvbzpwPjwvc3Bhbj48L3A+ CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5EaXJlY3RvcnkgU2Vydmlj ZTogUlVOTklORzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gbGFuZz0iRU4tVVMiPmtyYjVrZGMgU2VydmljZTogUlVOTklORzxvOnA+PC9vOnA+PC9zcGFu PjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmthZG1pbiBTZXJ2 aWNlOiBSVU5OSU5HPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+bmFtZWQgU2VydmljZTogUlVOTklORzxvOnA+PC9vOnA+PC9zcGFu PjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmh0dHBkIFNlcnZp Y2U6IFJVTk5JTkc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIj5wa2ktdG9tY2F0ZCBTZXJ2aWNlOiBTVE9QUEVEPG86cD48L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+c21iIFNl cnZpY2U6IFJVTk5JTkc8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIj53aW5iaW5kIFNlcnZpY2U6IFJVTk5JTkc8bzpwPjwvbzpwPjwv c3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5pcGEtb3Rw ZCBTZXJ2aWNlOiBSVU5OSU5HPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+aXBhLWRuc2tleXN5bmNkIFNlcnZpY2U6IFJVTk5JTkc8 bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIj5pcGE6IElORk86IFRoZSBpcGFjdGwgY29tbWFuZCB3YXMgc3VjY2Vzc2Z1bDxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0t PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+U3VnZ2VzdGVkIHJlc29sdXRpb24gdG8gYWJvdmUgcHJvYmxlbSBk b2VzbiYjODIxNzt0IGhlbHAgbWUgc2luY2UgdGhlIExEQVAgYW5kIE5TUyBEQiBzZWVtIHRvIGhh dmUgc2FtZSBjZXJ0aWZpY2F0ZXMgKHNvbWUgZGlmZmVyZW5jZSBpbiB3cmFwcGluZyBidXQgdGhl IHN0cmluZyBpcyBzYW1lIGlmIEkgdGFrZSBvdXQgdGhlIGxpbmUgYnJlYWtzKSBhbmQgZXZlbiB0 aGUgc2VyaWFsIG51bWJlcgogbWF0Y2hlcy48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48 L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4tLTxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmNlcnR1 dGlsIC1MIC1kIC9ldGMvcGtpL3BraS10b21jYXQvYWxpYXMgLW4gJ3N1YnN5c3RlbUNlcnQgY2Vy dC1wa2ktY2EnIC1hPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tPG86cD48L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+TUlJRGpE JiM4MjMwOzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g bGFuZz0iRU4tVVMiPiYjODIzMDtQcmgyRzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS08 bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIj5jZXJ0dXRpbCAtTCAtZCAvZXRjL3BraS9wa2ktdG9tY2F0L2FsaWFz IC1uICdzdWJzeXN0ZW1DZXJ0IGNlcnQtcGtpLWNhJyB8Z3JlcCBTZXJpYWw8bzpwPjwvbzpwPjwv c3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsm bmJzcDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7U2VyaWFsIE51bWJlcjogNCAoMHg0 KTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0i RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiPmxkYXBzZWFyY2ggLUxMTCAtRCAnY249ZGlyZWN0b3J5IG1hbmFn ZXInIC1XIC1iIHVpZD1wa2lkYnVzZXIsb3U9cGVvcGxlLG89aXBhY2EgdXNlckNlcnRpZmljYXRl IGRlc2NyaXB0aW9uIHNlZUFsc288bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5FbnRlciBMREFQIFBhc3N3b3JkOjxvOnA+PC9vOnA+ PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmRuOiB1 aWQ9cGtpZGJ1c2VyLG91PXBlb3BsZSxvPWlwYWNhPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+dXNlckNlcnRpZmljYXRlOjogTUlJ RGpEJiM4MjMwOzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gbGFuZz0iRU4tVVMiPiYjODIzMDtQcmgyPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+RzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPmRlc2NyaXB0aW9uOiAyOzQ7 Q049Q2VydGlmaWNhdGUgQXV0aG9yaXR5LE89Jmx0OyZsdDtSRUFMTSZndDsmZ3Q7O0NOPUNBIFN1 YnN5c3RlbSw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IGxhbmc9IkVOLVVTIj5PPSZsdDsmbHQ7UkVBTE0mZ3Q7Jmd0OzxvOnA+PC9vOnA+PC9zcGFuPjwv cD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPnNlZUFsc286IENOPUNB IFN1YnN5c3RlbSxPPSZsdDsmbHQ7UkVBTE0mZ3Q7Jmd0OzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0tPG86cD48L286cD48L3Nw YW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+PG86cD4mbmJz cDs8L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V UyI+QW5kIGhlcmUmIzgyMTc7cyB3aGVyZSBteSBhY3R1YWwga25vd2xlZGdlIG9mIHRoaW5ncyBl bmQuIEkmIzgyMTc7dmUgYmVlbiB0cnlpbmcgdG8gZmlndXJlIG91dCBhbGwga2luZCBvZiBsb2dz ICh0b21jYXQsIEtlcmJlcm9zLCBkaXJlY3Rvcnkgc2VydmVyLCAmIzgyMzA7KSBidXQgaGF2ZW4m IzgyMTc7dCBmb3VuZCBhIHNvbGlkIHJlYXNvbiBmb3IgaXQuIEkmIzgyMTc7bSBzdGFydGluZyB0 byBiZWxpZXZlIHRoaXMgaXMgYSBjZXJ0aWZpY2F0ZQogaXNzdWUsIGJlY2F1c2UgYWx0aG91Z2gg JiM4MjIwO2dldGNlcnQgbGlzdCYjODIyMTsgdGVsbHMgbWUgdGhhdCB0aGUgY2VydGlmaWNhdGUg c3RhdHVzIGlzICYjODIyMDtNb25pdG9yaW5nJiM4MjIxOyBvbiBhbGwgY2VydGlmaWNhdGVzIHRo ZSBleHBpcnkgZGF0ZSBpcyBhbHJlYWR5IGluIHRoZSBwYXN0IChjdXJyZW50IGRhdGUgMjAuNi4y MDE4LCBjZXJ0aWZpY2F0ZSBleHBpcnkgMjEuMDMuMjAxOCkgb24gNCBjZXJ0aWZpY2F0ZXMgYW5k IGl0IHdvbiYjODIxNzt0IHVwZGF0ZSBldmVuIGlmIEkgcmVzdWJtaXQKIGl0IG9yIGRlbGV0ZSBj ZXJ0aWZpY2F0ZSBhbmQgbWFudWFsbHkgcmVkbyBpdCAoaXQgZ290IHRoZSBzYW1lIGRhdGUgYXMg dGhlICYjODIyMDtvbGQgb25lcyYjODIyMTspLiBUaGUgJiM4MjIwO21haW4gY2VydHMmIzgyMjE7 ICgmIzgyMjA7Y2FTaWduaW5nQ2VydCBjZXJ0LXBraS1jYSYjODIyMTssICYjODIyMDtTZXJ2ZXIt Q2VydCBjZXJ0LXBraS1jYSYjODIyMTsgYW5kIHR3byBkaXJlY3Rvcnkgc2VydmVyIGNlcnRzKSBh cmUgdmFsaWQgZm9yIHllYXJzICh1bnRpbCAyMDIwJiM0MzspLjxvOnA+PC9vOnA+PC9zcGFuPjwv cD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9v OnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPi0t PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+UmVxdWVzdCBJRCAnMjAxNjAzMzEwODQyMzQnOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4K PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBzdGF0dXM6IE1PTklUT1JJTkc8bzpwPjwvbzpwPjwv c3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgc3R1Y2s6IG5vPG86cD48L286cD48 L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGtleSBwYWlyIHN0b3JhZ2U6IHR5 cGU9TlNTREIsbG9jYXRpb249Jy9ldGMvcGtpL3BraS10b21jYXQvYWxpYXMnLG5pY2tuYW1lPSdv Y3NwU2lnbmluZ0NlcnQgY2VydC1wa2ktY2EnLHRva2VuPSdOU1MgQ2VydGlmaWNhdGUgREInLHBp biBzZXQ8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxh bmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgY2Vy dGlmaWNhdGU6IHR5cGU9TlNTREIsbG9jYXRpb249Jy9ldGMvcGtpL3BraS10b21jYXQvYWxpYXMn LG5pY2tuYW1lPSdvY3NwU2lnbmluZ0NlcnQgY2VydC1wa2ktY2EnLHRva2VuPSdOU1MgQ2VydGlm aWNhdGUgREInPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 IENBOiBkb2d0YWctaXBhLWNhLXJlbmV3LWFnZW50PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGlzc3VlcjogQ049Q2VydGlmaWNhdGUgQXV0aG9yaXR5LE89 Jmx0OyZsdDtSRUFMTSZndDsmZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7IHN1YmplY3Q6IENOPU9DU1AgU3Vic3lzdGVtLE89Jmx0OyZsdDtSRUFMTSZn dDsmZ3Q7PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs YW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGV4 cGlyZXM6IDIwMTgtMDMtMjEgMDk6NDI6MDQgVVRDPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGtleSB1c2FnZTogZGlnaXRhbFNpZ25hdHVyZSxub25SZXB1 ZGlhdGlvbixrZXlDZXJ0U2lnbixjUkxTaWduPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7IGVrdTogaWQta3AtT0NTUFNpZ25pbmc8bzpwPjwvbzpwPjwvc3Bh bj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgcHJlLXNhdmUgY29tbWFuZDogL3Vzci9s aWJleGVjL2lwYS9jZXJ0bW9uZ2VyL3N0b3BfcGtpY2FkPG86cD48L286cD48L3NwYW4+PC9wPgo8 cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHBvc3Qtc2F2ZSBjb21tYW5kOiAvdXNyL2xpYmV4ZWMv aXBhL2NlcnRtb25nZXIvcmVuZXdfY2FfY2VydCAmcXVvdDtvY3NwU2lnbmluZ0NlcnQgY2VydC1w a2ktY2EmcXVvdDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz cGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgJm5ic3A7Jm5ic3A7Jm5i c3A7dHJhY2s6IHllczxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+ PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyBhdXRvLXJlbmV3OiB5ZXM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29O b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj5SZXF1ZXN0IElEICcyMDE2MDMzMTA4NTAwOCc6PG86 cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V UyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHN0YXR1czogTU9O SVRPUklORzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g bGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBz dHVjazogbm88bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsg a2V5IHBhaXIgc3RvcmFnZTogdHlwZT1OU1NEQixsb2NhdGlvbj0nL2V0Yy9odHRwZC9hbGlhcycs bmlja25hbWU9J1NlcnZlci1DZXJ0Jyx0b2tlbj0nTlNTIENlcnRpZmljYXRlIERCJyxwaW5maWxl PScvZXRjL2h0dHBkL2FsaWFzL3B3ZGZpbGUudHh0JzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBjZXJ0aWZpY2F0ZTogdHlwZT1OU1NEQixsb2NhdGlvbj0n L2V0Yy9odHRwZC9hbGlhcycsbmlja25hbWU9J1NlcnZlci1DZXJ0Jyx0b2tlbj0nTlNTIENlcnRp ZmljYXRlIERCJzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw YW4gbGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyBDQTogSVBBPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh biBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 IGlzc3VlcjogQ049Q2VydGlmaWNhdGUgQXV0aG9yaXR5LE89Jmx0OyZsdDtSRUFMTSZndDsmZ3Q7 PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF Ti1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHN1YmplY3Q6 IENOPSZsdDsmbHQ7aXBhc3J2MS5mcWRuJmd0OyZndDssTz0mbHQ7Jmx0O1JFQUxNJmd0OyZndDs8 bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO LVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgZXhwaXJlczog MjAyMC0wMy0wNCAwOTo1ODoyMyBVVEM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsgcHJpbmNpcGFsIG5hbWU6IEhUVFAvJmx0OyZsdDtpcGFzcnYxLmZxZG4m Z3Q7Jmd0O0AmbHQ7Jmx0O1JFQUxNJmd0OyZndDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNs YXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsga2V5IHVzYWdlOiBkaWdpdGFsU2lnbmF0dXJlLG5vblJlcHVk aWF0aW9uLGtleUVuY2lwaGVybWVudCxkYXRhRW5jaXBoZXJtZW50PG86cD48L286cD48L3NwYW4+ PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGVrdTogaWQta3Atc2VydmVyQXV0aCxpZC1r cC1jbGllbnRBdXRoPG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFsIj48 c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7IHByZS1zYXZlIGNvbW1hbmQ6PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNv Tm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7IHBvc3Qtc2F2ZSBjb21tYW5kOiAvdXNyL2xpYjY0L2lwYS9jZXJ0bW9uZ2Vy L3Jlc3RhcnRfaHR0cGQ8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwi PjxzcGFuIGxhbmc9IkVOLVVTIj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm bmJzcDsgdHJhY2s6IHllczxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1h bCI+PHNwYW4gbGFuZz0iRU4tVVMiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyBhdXRvLXJlbmV3OiB5ZXM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNzPSJN c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj4tLTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9z cGFuPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPkhhcyBhbnlv bmUgZWxzZSBidW1wZWQgaW50byBzYW1lIGtpbmQgb2YgaXNzdWVzPyBBbnkgaWRlYXMgd2hlcmUg SSBzaG91bGQgY29udGludWUgbG9va2luZz8gSSYjODIxNzttIHN0YXJ0aW5nIHRvIHJ1biBvdXQg b2YgaWRlYXMmIzgyMzA7PG86cD48L286cD48L3NwYW4+PC9wPgo8cCBjbGFzcz0iTXNvTm9ybWFs Ij48bzpwPiZuYnNwOzwvbzpwPjwvcD4KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8 L286cD48L3A+CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1s YW5ndWFnZTpGSSI+RWVtZWxpIEpva2luZW48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CjxwIGNsYXNz PSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPgo8L2Rpdj4KPC9ib2R5Pgo8L2h0bWw+ Cg== --===============1363744815246032696==-- From flo at redhat.com Wed Jun 20 16:49:35 2018 Content-Type: multipart/mixed; boundary="===============5130246091497708464==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 20 Jun 2018 18:49:04 +0200 Message-ID: In-Reply-To: 9dbd45efa82b4a9c943e1e3bd8071d43@cinia.fi --===============5130246091497708464== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hello all! > = > I have very similiar problem as this one: > = > https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedora= hosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ > = > ipa-server-upgrade fails as below > = > -- > = > Update complete > = > Upgrading IPA services > = > Upgrading the configuration of the IPA services > = > [Verifying that root certificate is published] > = > [Migrate CRL publish directory] > = > CRL tree already moved > = > [Verifying that CA proxy configuration is correct] > = > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = > command ipa-server-upgrade manually. > = > CA did not start in 300.0s > = > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for = > more information > = > -- > = > And the log tells me that CA returns status 500 > = > -- > = > DEBUG Waiting for CA to start... > = > DEBUG request POST http://<>:8080/ca/admin/ca/getStatus = > > = > DEBUG request body '' > = > DEBUG response status 500 > = > DEBUG response headers Server: Apache-Coyote/1.1 > = > Content-Type: text/html;charset=3Dutf-8 > = > Content-Language: en > = > Content-Length: 2208 > = > Date: Fri, 15 Jun 2018 10:05:29 GMT > = > Connection: close > = > DEBUG response body 'Apache Tomcat/7.0.76 - Error = > report = >

HTTP Status 500 - Subsystem unavailable

size=3D"1" noshade=3D"noshade">

type Exception = > report

message Subsystem = > unavailable

description The server encountered an = > internal error that prevented it from fulfilling this = > request.

exception = >

javax.ws.rs.ServiceUnavailableException: Subsystem =

> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints=
(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBas=
e.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorRep=
ortValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.Ac=
cessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connect=
or.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http1=
1.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg=
.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractP=
rotocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor=
.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWo=
rker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecut=
or$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.thread=
s.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.r=
un(Thread.java:748)\n

note = > The full stack trace of the root cause is available in the Apache = > Tomcat/7.0.76 logs.

Apache = > Tomcat/7.0.76

' > = > DEBUG The CA status is: check interrupted due to error: Retrieving CA = > status failed with status 500 > = > DEBUG Waiting for CA to start... > = > ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = > command ipa-server-upgrade manually. > = > File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line = > 172, in execute > = > =C2=A0=C2=A0=C2=A0 return_value =3D self.run() > = > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py= ", = > line 48, in run > = > =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) > = > The ipa-server-upgrade command failed, exception: ScriptError: CA did = > not start in 300.0s > = > ERROR CA did not start in 300.0s > = > -- > = > With command =E2=80=9Cipactl start --ignore-service-failures=E2=80=9D I c= an start all = > the services but pki-tomcatd. > = > -- > = > Directory Service: RUNNING > = > krb5kdc Service: RUNNING > = > kadmin Service: RUNNING > = > named Service: RUNNING > = > httpd Service: RUNNING > = > pki-tomcatd Service: STOPPED > = > smb Service: RUNNING > = > winbind Service: RUNNING > = > ipa-otpd Service: RUNNING > = > ipa-dnskeysyncd Service: RUNNING > = > ipa: INFO: The ipactl command was successful > = > -- > = > Suggested resolution to above problem doesn=E2=80=99t help me since the L= DAP and = > NSS DB seem to have same certificates (some difference in wrapping but = > the string is same if I take out the line breaks) and even the serial = > number matches. > = > -- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a > = > -----BEGIN CERTIFICATE----- > = > MIIDjD=E2=80=A6 > = > =E2=80=A6Prh2G > = > -----END CERTIFICATE----- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' = > |grep Serial > = > =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) > = > ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b = > uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso > = > Enter LDAP Password: > = > dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca > = > userCertificate:: MIIDjD=E2=80=A6 > = > =E2=80=A6Prh2 > = > G > = > description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsyst= em, > = > O=3D<> > = > seeAlso: CN=3DCA Subsystem,O=3D<> > = > -- > = > And here=E2=80=99s where my actual knowledge of things end. I=E2=80=99ve = been trying to = > figure out all kind of logs (tomcat, Kerberos, directory server, =E2=80= =A6) but = > haven=E2=80=99t found a solid reason for it. I=E2=80=99m starting to beli= eve this is a = > certificate issue, because although =E2=80=9Cgetcert list=E2=80=9D tells = me that the = > certificate status is =E2=80=9CMonitoring=E2=80=9D on all certificates th= e expiry date = > is already in the past (current date 20.6.2018, certificate expiry = > 21.03.2018) on 4 certificates and it won=E2=80=99t update even if I resub= mit it = > or delete certificate and manually redo it (it got the same date as the = > =E2=80=9Cold ones=E2=80=9D). The =E2=80=9Cmain certs=E2=80=9D (=E2=80=9Cc= aSigningCert cert-pki-ca=E2=80=9D, =E2=80=9CServer-Cert = > cert-pki-ca=E2=80=9D and two directory server certs) are valid for years = (until = > 2020+). > = > -- > = > Request ID '20160331084234': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB',pin set > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-agent > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsystem,= O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:04 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: digitalSignature,n= onRepudiation,keyCertSign,cRLSign > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libexe= c/ipa/certmonger/stop_pkicad > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/libex= ec/ipa/certmonger/renew_ca_cert = > "ocspSigningCert cert-pki-ca" > = > =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > Request ID '20160331085008': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'NSS = > Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'NSS = > Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<= >,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:23 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp-c= lientAuth > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib64= /ipa/certmonger/restart_httpd > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > -- > = > Has anyone else bumped into same kind of issues? Any ideas where I = > should continue looking? I=E2=80=99m starting to run out of ideas=E2=80= =A6 > = > Eemeli Jokinen > = > = > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ > = Hi, does your topology include multiple CA instances? You need first to find = which master is the CA renewal master: ipa config-show | grep "renewal master" On this host, check that the certificates are still valid and consistent = with the content of the LDAP entries. If it is not the case, you need to = repair the CA renewal master first. When the CA renewal master is OK, check if the replication is working = with the other CA instances, and repair the other masters. HTH, Flo --===============5130246091497708464==-- From Eemeli.Jokinen at cinia.fi Thu Jun 21 05:35:08 2018 Content-Type: multipart/mixed; boundary="===============8937654680795926842==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 21 Jun 2018 05:34:39 +0000 Message-ID: In-Reply-To: cf5b135a-b5ee-64ea-b5ee-9dddc8900661@redhat.com --===============8937654680795926842== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! We do have 2 IPA nodes configured but the second node has been down for som= e time. Tried to update it to same version as node1: - Won't start tells me to use ipa-server-upgrade - Ipa-server-upgrade fails at start, doesn't start directory server - /var/log/dirsrv/slapd-<>/errors log has some acl_parse errors but= last row tells me -- EMERG - main - Fatal Error---No ports specified. Exiting now. -- On node1 ipa config-show results only -- ipa: ERROR: did not receive Kerberos credentials -- Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: keskiviikko 20. kes=C3=A4kuuta 2018 19.49 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade= doesn't complete, pki-tomcatd won't start On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hello all! > = > I have very similiar problem as this one: > = > https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedor > ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ > = > ipa-server-upgrade fails as below > = > -- > = > Update complete > = > Upgrading IPA services > = > Upgrading the configuration of the IPA services > = > [Verifying that root certificate is published] > = > [Migrate CRL publish directory] > = > CRL tree already moved > = > [Verifying that CA proxy configuration is correct] > = > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = > command ipa-server-upgrade manually. > = > CA did not start in 300.0s > = > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for = > more information > = > -- > = > And the log tells me that CA returns status 500 > = > -- > = > DEBUG Waiting for CA to start... > = > DEBUG request POST http://<>:8080/ca/admin/ca/getStatus > > = > DEBUG request body '' > = > DEBUG response status 500 > = > DEBUG response headers Server: Apache-Coyote/1.1 > = > Content-Type: text/html;charset=3Dutf-8 > = > Content-Language: en > = > Content-Length: 2208 > = > Date: Fri, 15 Jun 2018 10:05:29 GMT > = > Connection: close > = > DEBUG response body 'Apache Tomcat/7.0.76 - Error > report

HTTP Status 500 - Subsystem = > unavailable

type = > Exception report

message Subsystem = > unavailable

description The server encountered an = > internal error that prevented it from fulfilling this = > request.

exception >

javax.ws.rs.ServiceUnavailableException: Subsystem =

> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai
> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti
> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va
> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445
> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr
> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\
> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr
> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread
> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre
> ad.java:748)\n

note > The full stack trace of the root cause is available in the Apache > Tomcat/7.0.76 logs.

Apache = > Tomcat/7.0.76

' > = > DEBUG The CA status is: check interrupted due to error: Retrieving CA = > status failed with status 500 > = > DEBUG Waiting for CA to start... > = > ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and = > run command ipa-server-upgrade manually. > = > File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line = > 172, in execute > = > =C2=A0=C2=A0=C2=A0 return_value =3D self.run() > = > =C2=A0 File > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade > .py", > line 48, in run > = > =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) > = > The ipa-server-upgrade command failed, exception: ScriptError: CA did = > not start in 300.0s > = > ERROR CA did not start in 300.0s > = > -- > = > With command "ipactl start --ignore-service-failures" I can start all = > the services but pki-tomcatd. > = > -- > = > Directory Service: RUNNING > = > krb5kdc Service: RUNNING > = > kadmin Service: RUNNING > = > named Service: RUNNING > = > httpd Service: RUNNING > = > pki-tomcatd Service: STOPPED > = > smb Service: RUNNING > = > winbind Service: RUNNING > = > ipa-otpd Service: RUNNING > = > ipa-dnskeysyncd Service: RUNNING > = > ipa: INFO: The ipactl command was successful > = > -- > = > Suggested resolution to above problem doesn't help me since the LDAP = > and NSS DB seem to have same certificates (some difference in wrapping = > but the string is same if I take out the line breaks) and even the = > serial number matches. > = > -- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert = > cert-pki-ca' -a > = > -----BEGIN CERTIFICATE----- > = > MIIDjD. > = > .Prh2G > = > -----END CERTIFICATE----- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' = > |grep Serial > = > =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) > = > ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b = > uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso > = > Enter LDAP Password: > = > dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca > = > userCertificate:: MIIDjD. > = > .Prh2 > = > G > = > description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsyst= em, > = > O=3D<> > = > seeAlso: CN=3DCA Subsystem,O=3D<> > = > -- > = > And here's where my actual knowledge of things end. I've been trying = > to figure out all kind of logs (tomcat, Kerberos, directory server, .) = > but haven't found a solid reason for it. I'm starting to believe this = > is a certificate issue, because although "getcert list" tells me that = > the certificate status is "Monitoring" on all certificates the expiry = > date is already in the past (current date 20.6.2018, certificate = > expiry > 21.03.2018) on 4 certificates and it won't update even if I resubmit = > it or delete certificate and manually redo it (it got the same date as = > the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", = > "Server-Cert cert-pki-ca" and two directory server certs) are valid = > for years (until > 2020+). > = > -- > = > Request ID '20160331084234': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngC > ert cert-pki-ca',token=3D'NSS Certificate DB',pin set > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngC > ert cert-pki-ca',token=3D'NSS Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-agent > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsystem,= O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:04 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: = > digitalSignature,nonRepudiation,keyCertSign,cRLSign > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libexe= c/ipa/certmonger/stop_pkicad > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/libex= ec/ipa/certmonger/renew_ca_cert > "ocspSigningCert cert-pki-ca" > = > =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > Request ID '20160331085008': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'N > SS Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'N > SS > Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<= >,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:23 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp-c= lientAuth > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib64= /ipa/certmonger/restart_httpd > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > -- > = > Has anyone else bumped into same kind of issues? Any ideas where I = > should continue looking? I'm starting to run out of ideas. > = > Eemeli Jokinen > = > = > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to = > freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: = > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: = > https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo > rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ > = Hi, does your topology include multiple CA instances? You need first to find wh= ich master is the CA renewal master: ipa config-show | grep "renewal master" On this host, check that the certificates are still valid and consistent wi= th the content of the LDAP entries. If it is not the case, you need to repa= ir the CA renewal master first. When the CA renewal master is OK, check if the replication is working with = the other CA instances, and repair the other masters. HTH, Flo --===============8937654680795926842==-- From flo at redhat.com Thu Jun 21 06:43:04 2018 Content-Type: multipart/mixed; boundary="===============7543152557173746782==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 21 Jun 2018 08:42:35 +0200 Message-ID: In-Reply-To: dbde34d57e454d72bd2fdd4e1cf4717f@cinia.fi --===============7543152557173746782== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > We do have 2 IPA nodes configured but the second node has been down for s= ome time. Tried to update it to same version as node1: > - Won't start tells me to use ipa-server-upgrade > - Ipa-server-upgrade fails at start, doesn't start directory server > - /var/log/dirsrv/slapd-<>/errors log has some acl_parse errors b= ut last row tells me > -- > EMERG - main - Fatal Error---No ports specified. Exiting now. > -- > = Hi, in this case let's concentrate on node1. > On node1 ipa config-show results only > -- > ipa: ERROR: did not receive Kerberos credentials > -- You need to run kinit to get kerberos credentials before any ipa * = command. If the ipa stack is stopped because pki-tomcat refused to = start, you can force the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show If node1 is not the renewal master, make it the renewal master from now = on (follow instructions from = https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht= ml/linux_domain_identity_authentication_and_policy_guide/server-roles#serve= r-roles-promote-to-ca). After this step, check the certificates in the NSS databases vs the = certificates stored in LDAP. HTH, Flo > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: keskiviikko 20. kes=C3=A4kuuta 2018 19.49 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgra= de doesn't complete, pki-tomcatd won't start > = > On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: >> Hello all! >> >> I have very similiar problem as this one: >> >> https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedor >> ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ >> >> ipa-server-upgrade fails as below >> >> -- >> >> Update complete >> >> Upgrading IPA services >> >> Upgrading the configuration of the IPA services >> >> [Verifying that root certificate is published] >> >> [Migrate CRL publish directory] >> >> CRL tree already moved >> >> [Verifying that CA proxy configuration is correct] >> >> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run >> command ipa-server-upgrade manually. >> >> CA did not start in 300.0s >> >> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for >> more information >> >> -- >> >> And the log tells me that CA returns status 500 >> >> -- >> >> DEBUG Waiting for CA to start... >> >> DEBUG request POST http://<>:8080/ca/admin/ca/getStatus >> >> >> DEBUG request body '' >> >> DEBUG response status 500 >> >> DEBUG response headers Server: Apache-Coyote/1.1 >> >> Content-Type: text/html;charset=3Dutf-8 >> >> Content-Language: en >> >> Content-Length: 2208 >> >> Date: Fri, 15 Jun 2018 10:05:29 GMT >> >> Connection: close >> >> DEBUG response body 'Apache Tomcat/7.0.76 - Error >> report

HTTP Status 500 - Subsystem >> unavailable

type >> Exception report

message Subsystem >> unavailable

description The server encountered an >> internal error that prevented it from fulfilling this >> request.

exception >>

javax.ws.rs.ServiceUnavailableException: Subsystem
>> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai
>> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti
>> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va
>> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
>> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
>> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445
>> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
>> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr
>> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
>> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\
>> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
>> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr
>> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread
>> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre
>> ad.java:748)\n

note >> The full stack trace of the root cause is available in the Apache >> Tomcat/7.0.76 logs.

Apache >> Tomcat/7.0.76

' >> >> DEBUG The CA status is: check interrupted due to error: Retrieving CA >> status failed with status 500 >> >> DEBUG Waiting for CA to start... >> >> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and >> run command ipa-server-upgrade manually. >> >> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line >> 172, in execute >> >> =C2=A0=C2=A0=C2=A0 return_value =3D self.run() >> >> =C2=A0 File >> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade >> .py", >> line 48, in run >> >> =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) >> >> The ipa-server-upgrade command failed, exception: ScriptError: CA did >> not start in 300.0s >> >> ERROR CA did not start in 300.0s >> >> -- >> >> With command "ipactl start --ignore-service-failures" I can start all >> the services but pki-tomcatd. >> >> -- >> >> Directory Service: RUNNING >> >> krb5kdc Service: RUNNING >> >> kadmin Service: RUNNING >> >> named Service: RUNNING >> >> httpd Service: RUNNING >> >> pki-tomcatd Service: STOPPED >> >> smb Service: RUNNING >> >> winbind Service: RUNNING >> >> ipa-otpd Service: RUNNING >> >> ipa-dnskeysyncd Service: RUNNING >> >> ipa: INFO: The ipactl command was successful >> >> -- >> >> Suggested resolution to above problem doesn't help me since the LDAP >> and NSS DB seem to have same certificates (some difference in wrapping >> but the string is same if I take out the line breaks) and even the >> serial number matches. >> >> -- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' -a >> >> -----BEGIN CERTIFICATE----- >> >> MIIDjD. >> >> .Prh2G >> >> -----END CERTIFICATE----- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' >> |grep Serial >> >> =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) >> >> ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b >> uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso >> >> Enter LDAP Password: >> >> dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca >> >> userCertificate:: MIIDjD. >> >> .Prh2 >> >> G >> >> description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsys= tem, >> >> O=3D<> >> >> seeAlso: CN=3DCA Subsystem,O=3D<> >> >> -- >> >> And here's where my actual knowledge of things end. I've been trying >> to figure out all kind of logs (tomcat, Kerberos, directory server, .) >> but haven't found a solid reason for it. I'm starting to believe this >> is a certificate issue, because although "getcert list" tells me that >> the certificate status is "Monitoring" on all certificates the expiry >> date is already in the past (current date 20.6.2018, certificate >> expiry >> 21.03.2018) on 4 certificates and it won't update even if I resubmit >> it or delete certificate and manually redo it (it got the same date as >> the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", >> "Server-Cert cert-pki-ca" and two directory server certs) are valid >> for years (until >> 2020+). >> >> -- >> >> Request ID '20160331084234': >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSign= ingC >> ert cert-pki-ca',token=3D'NSS Certificate DB',pin set >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSign= ingC >> ert cert-pki-ca',token=3D'NSS Certificate DB' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-age= nt >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Au= thority,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsyste= m,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:0= 4 UTC >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >> digitalSignature,nonRepudiation,keyCertSign,cRLSign >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libe= xec/ipa/certmonger/stop_pkicad >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib= exec/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> >> =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >> >> Request ID '20160331085008': >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',toke= n=3D'N >> SS Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',toke= n=3D'N >> SS >> Certificate DB' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Au= thority,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<>,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:2= 3 UTC >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp= -clientAuth >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib= 64/ipa/certmonger/restart_httpd >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >> >> -- >> >> Has anyone else bumped into same kind of issues? Any ideas where I >> should continue looking? I'm starting to run out of ideas. >> >> Eemeli Jokinen >> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo >> rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ >> > = > Hi, > = > does your topology include multiple CA instances? You need first to find = which master is the CA renewal master: > ipa config-show | grep "renewal master" > = > On this host, check that the certificates are still valid and consistent = with the content of the LDAP entries. If it is not the case, you need to re= pair the CA renewal master first. > = > When the CA renewal master is OK, check if the replication is working wit= h the other CA instances, and repair the other masters. > HTH, > Flo > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/ >=20 --===============7543152557173746782==-- From Eemeli.Jokinen at cinia.fi Thu Jun 21 06:57:59 2018 Content-Type: multipart/mixed; boundary="===============2231608466827512636==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 21 Jun 2018 06:57:32 +0000 Message-ID: <19ab4bb4d5a3477fa34929619f90b29f@cinia.fi> In-Reply-To: bf3a9655-35b1-5c60-e2fb-dc7934594a8e@redhat.com --===============2231608466827512636== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! Forgot kinit: -- kinit admin Password for admin@<>: klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@<> Valid starting Expires Service principal 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<>@<> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<>@<> ipa config-show ipa: ERROR: No valid Negotiate header in server response -- Still no luck getting ipa config Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: torstai 21. kes=C3=A4kuuta 2018 9.43 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > We do have 2 IPA nodes configured but the second node has been down for s= ome time. Tried to update it to same version as node1: > - Won't start tells me to use ipa-server-upgrade > - Ipa-server-upgrade fails at start, doesn't start directory server > - /var/log/dirsrv/slapd-<>/errors log has some acl_parse = > errors but last row tells me > -- > EMERG - main - Fatal Error---No ports specified. Exiting now. > -- > = Hi, in this case let's concentrate on node1. > On node1 ipa config-show results only > -- > ipa: ERROR: did not receive Kerberos credentials > -- You need to run kinit to get kerberos credentials before any ipa * command.= If the ipa stack is stopped because pki-tomcat refused to start, you can f= orce the start up, ignoring failed services: $ sudo ipactl start --ignore-service-failures Then to check who is renewal master: $ sudo kinit admin $ sudo ipa config-show If node1 is not the renewal master, make it the renewal master from now on = (follow instructions from https://access.redhat.com/documentation/en-us/red= _hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_polic= y_guide/server-roles#server-roles-promote-to-ca). After this step, check the certificates in the NSS databases vs the certifi= cates stored in LDAP. HTH, Flo > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: keskiviikko 20. kes=C3=A4kuuta 2018 19.49 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Problems after IPA upgrade: = > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > = > On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: >> Hello all! >> >> I have very similiar problem as this one: >> >> https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedo >> r ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ >> >> ipa-server-upgrade fails as below >> >> -- >> >> Update complete >> >> Upgrading IPA services >> >> Upgrading the configuration of the IPA services >> >> [Verifying that root certificate is published] >> >> [Migrate CRL publish directory] >> >> CRL tree already moved >> >> [Verifying that CA proxy configuration is correct] >> >> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = >> command ipa-server-upgrade manually. >> >> CA did not start in 300.0s >> >> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log = >> for more information >> >> -- >> >> And the log tells me that CA returns status 500 >> >> -- >> >> DEBUG Waiting for CA to start... >> >> DEBUG request POST http://<>:8080/ca/admin/ca/getStatus >> >> >> DEBUG request body '' >> >> DEBUG response status 500 >> >> DEBUG response headers Server: Apache-Coyote/1.1 >> >> Content-Type: text/html;charset=3Dutf-8 >> >> Content-Language: en >> >> Content-Length: 2208 >> >> Date: Fri, 15 Jun 2018 10:05:29 GMT >> >> Connection: close >> >> DEBUG response body 'Apache Tomcat/7.0.76 - Error >> report

HTTP Status 500 - Subsystem = >> unavailable

type = >> Exception report

message Subsystem = >> unavailable

description The server encountered an = >> internal error that prevented it from fulfilling this = >> request.

exception >>

javax.ws.rs.ServiceUnavailableException: Subsystem =

>> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra
>> i =

>> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent
>> i =

>> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v
>> a =

>> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
>> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
>> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44
>> 5 =

>> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac
>> t =

>> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst
>> r =

>> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
>> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>> \ =

>> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
>> o =

>> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th
>> r =

>> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
>> d =

>> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr
>> e ad.java:748)\n

note >> The full stack trace of the root cause is available in the Apache >> Tomcat/7.0.76 logs.

Apache = >> Tomcat/7.0.76

' >> >> DEBUG The CA status is: check interrupted due to error: Retrieving CA = >> status failed with status 500 >> >> DEBUG Waiting for CA to start... >> >> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and = >> run command ipa-server-upgrade manually. >> >> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line = >> 172, in execute >> >> =C2=A0=C2=A0=C2=A0 return_value =3D self.run() >> >> =C2=A0 File >> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad >> e >> .py", >> line 48, in run >> >> =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) >> >> The ipa-server-upgrade command failed, exception: ScriptError: CA did = >> not start in 300.0s >> >> ERROR CA did not start in 300.0s >> >> -- >> >> With command "ipactl start --ignore-service-failures" I can start all = >> the services but pki-tomcatd. >> >> -- >> >> Directory Service: RUNNING >> >> krb5kdc Service: RUNNING >> >> kadmin Service: RUNNING >> >> named Service: RUNNING >> >> httpd Service: RUNNING >> >> pki-tomcatd Service: STOPPED >> >> smb Service: RUNNING >> >> winbind Service: RUNNING >> >> ipa-otpd Service: RUNNING >> >> ipa-dnskeysyncd Service: RUNNING >> >> ipa: INFO: The ipactl command was successful >> >> -- >> >> Suggested resolution to above problem doesn't help me since the LDAP = >> and NSS DB seem to have same certificates (some difference in = >> wrapping but the string is same if I take out the line breaks) and = >> even the serial number matches. >> >> -- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert = >> cert-pki-ca' -a >> >> -----BEGIN CERTIFICATE----- >> >> MIIDjD. >> >> .Prh2G >> >> -----END CERTIFICATE----- >> >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' >> |grep Serial >> >> =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) >> >> ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b = >> uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso >> >> Enter LDAP Password: >> >> dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca >> >> userCertificate:: MIIDjD. >> >> .Prh2 >> >> G >> >> description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA = >> Subsystem, >> >> O=3D<> >> >> seeAlso: CN=3DCA Subsystem,O=3D<> >> >> -- >> >> And here's where my actual knowledge of things end. I've been trying = >> to figure out all kind of logs (tomcat, Kerberos, directory server, = >> .) but haven't found a solid reason for it. I'm starting to believe = >> this is a certificate issue, because although "getcert list" tells me = >> that the certificate status is "Monitoring" on all certificates the = >> expiry date is already in the past (current date 20.6.2018, = >> certificate expiry >> 21.03.2018) on 4 certificates and it won't update even if I resubmit = >> it or delete certificate and manually redo it (it got the same date = >> as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", = >> "Server-Cert cert-pki-ca" and two directory server certs) are valid = >> for years (until >> 2020+). >> >> -- >> >> Request ID '20160331084234': >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSign= ing >> C ert cert-pki-ca',token=3D'NSS Certificate DB',pin set >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSign= ing >> C ert cert-pki-ca',token=3D'NSS Certificate DB' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-age= nt >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Au= thority,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsyste= m,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:0= 4 UTC >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >> digitalSignature,nonRepudiation,keyCertSign,cRLSign >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libe= xec/ipa/certmonger/stop_pkicad >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: = >> /usr/libexec/ipa/certmonger/renew_ca_cert >> "ocspSigningCert cert-pki-ca" >> >> =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >> >> Request ID '20160331085008': >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',toke= n=3D' >> N SS Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',toke= n=3D' >> N >> SS >> Certificate DB' >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Au= thority,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<>,O=3D<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:2= 3 UTC >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp= -clientAuth >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib= 64/ipa/certmonger/restart_httpd >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes >> >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >> >> -- >> >> Has anyone else bumped into same kind of issues? Any ideas where I = >> should continue looking? I'm starting to run out of ideas. >> >> Eemeli Jokinen >> >> >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fed >> o rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ >> > = > Hi, > = > does your topology include multiple CA instances? You need first to find = which master is the CA renewal master: > ipa config-show | grep "renewal master" > = > On this host, check that the certificates are still valid and consistent = with the content of the LDAP entries. If it is not the case, you need to re= pair the CA renewal master first. > = > When the CA renewal master is OK, check if the replication is working wit= h the other CA instances, and repair the other masters. > HTH, > Flo > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to = > freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: = > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: = > https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo > rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/ > = --===============2231608466827512636==-- From flo at redhat.com Fri Jun 22 12:39:42 2018 Content-Type: multipart/mixed; boundary="===============7233827499095487489==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Fri, 22 Jun 2018 14:39:09 +0200 Message-ID: <13a705b6-3605-5247-2331-1fbc580141eb@redhat.com> In-Reply-To: 19ab4bb4d5a3477fa34929619f90b29f@cinia.fi --===============7233827499095487489== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > Forgot kinit: > = > -- > kinit admin > Password for admin@<>: > klist > Ticket cache: KEYRING:persistent:0:0 > Default principal: admin@<> > = > Valid starting Expires Service principal > 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<>@<> > 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<>@<> > ipa config-show > ipa: ERROR: No valid Negotiate header in server response > -- > = Hi, can you check if the service gssproxy is running: # systemctl status gssproxy and if not, restart it. Flo > Still no luck getting ipa config > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: torstai 21. kes=C3=A4kuuta 2018 9.43 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> We do have 2 IPA nodes configured but the second node has been down for = some time. Tried to update it to same version as node1: >> - Won't start tells me to use ipa-server-upgrade >> - Ipa-server-upgrade fails at start, doesn't start directory server >> - /var/log/dirsrv/slapd-<>/errors log has some acl_parse >> errors but last row tells me >> -- >> EMERG - main - Fatal Error---No ports specified. Exiting now. >> -- >> > Hi, > = > in this case let's concentrate on node1. > = >> On node1 ipa config-show results only >> -- >> ipa: ERROR: did not receive Kerberos credentials >> -- > = > You need to run kinit to get kerberos credentials before any ipa * comman= d. If the ipa stack is stopped because pki-tomcat refused to start, you can= force the start up, ignoring failed services: > $ sudo ipactl start --ignore-service-failures > = > Then to check who is renewal master: > $ sudo kinit admin > $ sudo ipa config-show > = > If node1 is not the renewal master, make it the renewal master from now o= n (follow instructions from https://access.redhat.com/documentation/en-us/r= ed_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_pol= icy_guide/server-roles#server-roles-promote-to-ca). > After this step, check the certificates in the NSS databases vs the certi= ficates stored in LDAP. > = > HTH, > Flo > = >> >> >> Eemeli >> >> -----Original Message----- >> From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] >> Sent: keskiviikko 20. kes=C3=A4kuuta 2018 19.49 >> To: FreeIPA users list >> Cc: Jokinen Eemeli >> Subject: Re: [Freeipa-users] Problems after IPA upgrade: >> ipa-server-upgrade doesn't complete, pki-tomcatd won't start >> >> On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: >>> Hello all! >>> >>> I have very similiar problem as this one: >>> >>> https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedo >>> r ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ >>> >>> ipa-server-upgrade fails as below >>> >>> -- >>> >>> Update complete >>> >>> Upgrading IPA services >>> >>> Upgrading the configuration of the IPA services >>> >>> [Verifying that root certificate is published] >>> >>> [Migrate CRL publish directory] >>> >>> CRL tree already moved >>> >>> [Verifying that CA proxy configuration is correct] >>> >>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run >>> command ipa-server-upgrade manually. >>> >>> CA did not start in 300.0s >>> >>> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log >>> for more information >>> >>> -- >>> >>> And the log tells me that CA returns status 500 >>> >>> -- >>> >>> DEBUG Waiting for CA to start... >>> >>> DEBUG request POST http://<>:8080/ca/admin/ca/getStatus >>> >>> >>> DEBUG request body '' >>> >>> DEBUG response status 500 >>> >>> DEBUG response headers Server: Apache-Coyote/1.1 >>> >>> Content-Type: text/html;charset=3Dutf-8 >>> >>> Content-Language: en >>> >>> Content-Length: 2208 >>> >>> Date: Fri, 15 Jun 2018 10:05:29 GMT >>> >>> Connection: close >>> >>> DEBUG response body 'Apache Tomcat/7.0.76 - Error >>> report

HTTP Status 500 - Subsystem >>> unavailable

type >>> Exception report

message Subsystem >>> unavailable

description The server encountered an >>> internal error that prevented it from fulfilling this >>> request.

exception >>>

javax.ws.rs.ServiceUnavailableException: Subsystem
>>> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra
>>> i
>>> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent
>>> i
>>> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v
>>> a
>>> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
>>> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
>>> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44
>>> 5
>>> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac
>>> t
>>> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst
>>> r
>>> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
>>> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>>> \
>>> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
>>> o
>>> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th
>>> r
>>> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
>>> d
>>> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr
>>> e ad.java:748)\n

note >>> The full stack trace of the root cause is available in the Apache >>> Tomcat/7.0.76 logs.

Apache >>> Tomcat/7.0.76

' >>> >>> DEBUG The CA status is: check interrupted due to error: Retrieving CA >>> status failed with status 500 >>> >>> DEBUG Waiting for CA to start... >>> >>> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and >>> run command ipa-server-upgrade manually. >>> >>> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line >>> 172, in execute >>> >>> =C2=A0=C2=A0=C2=A0 return_value =3D self.run() >>> >>> =C2=A0 File >>> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad >>> e >>> .py", >>> line 48, in run >>> >>> =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) >>> >>> The ipa-server-upgrade command failed, exception: ScriptError: CA did >>> not start in 300.0s >>> >>> ERROR CA did not start in 300.0s >>> >>> -- >>> >>> With command "ipactl start --ignore-service-failures" I can start all >>> the services but pki-tomcatd. >>> >>> -- >>> >>> Directory Service: RUNNING >>> >>> krb5kdc Service: RUNNING >>> >>> kadmin Service: RUNNING >>> >>> named Service: RUNNING >>> >>> httpd Service: RUNNING >>> >>> pki-tomcatd Service: STOPPED >>> >>> smb Service: RUNNING >>> >>> winbind Service: RUNNING >>> >>> ipa-otpd Service: RUNNING >>> >>> ipa-dnskeysyncd Service: RUNNING >>> >>> ipa: INFO: The ipactl command was successful >>> >>> -- >>> >>> Suggested resolution to above problem doesn't help me since the LDAP >>> and NSS DB seem to have same certificates (some difference in >>> wrapping but the string is same if I take out the line breaks) and >>> even the serial number matches. >>> >>> -- >>> >>> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>> cert-pki-ca' -a >>> >>> -----BEGIN CERTIFICATE----- >>> >>> MIIDjD. >>> >>> .Prh2G >>> >>> -----END CERTIFICATE----- >>> >>> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' >>> |grep Serial >>> >>> =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) >>> >>> ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b >>> uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAl= so >>> >>> Enter LDAP Password: >>> >>> dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca >>> >>> userCertificate:: MIIDjD. >>> >>> .Prh2 >>> >>> G >>> >>> description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA >>> Subsystem, >>> >>> O=3D<> >>> >>> seeAlso: CN=3DCA Subsystem,O=3D<> >>> >>> -- >>> >>> And here's where my actual knowledge of things end. I've been trying >>> to figure out all kind of logs (tomcat, Kerberos, directory server, >>> .) but haven't found a solid reason for it. I'm starting to believe >>> this is a certificate issue, because although "getcert list" tells me >>> that the certificate status is "Monitoring" on all certificates the >>> expiry date is already in the past (current date 20.6.2018, >>> certificate expiry >>> 21.03.2018) on 4 certificates and it won't update even if I resubmit >>> it or delete certificate and manually redo it (it got the same date >>> as the "old ones"). The "main certs" ("caSigningCert cert-pki-ca", >>> "Server-Cert cert-pki-ca" and two directory server certs) are valid >>> for years (until >>> 2020+). >>> >>> -- >>> >>> Request ID '20160331084234': >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >>> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSig= ning >>> C ert cert-pki-ca',token=3D'NSS Certificate DB',pin set >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >>> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSig= ning >>> C ert cert-pki-ca',token=3D'NSS Certificate DB' >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-a= gent >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate = Authority,O=3D<> >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsys= tem,O=3D<> >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42= :04 UTC >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >>> digitalSignature,nonRepudiation,keyCertSign,cRLSign >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/li= bexec/ipa/certmonger/stop_pkicad >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: >>> /usr/libexec/ipa/certmonger/renew_ca_cert >>> "ocspSigningCert cert-pki-ca" >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >>> >>> Request ID '20160331085008': >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: >>> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',tok= en=3D' >>> N SS Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: >>> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',tok= en=3D' >>> N >>> SS >>> Certificate DB' >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate = Authority,O=3D<> >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<>,O=3D<> >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58= :23 UTC >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: >>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-= kp-clientAuth >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/l= ib64/ipa/certmonger/restart_httpd >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes >>> >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes >>> >>> -- >>> >>> Has anyone else bumped into same kind of issues? Any ideas where I >>> should continue looking? I'm starting to run out of ideas. >>> >>> Eemeli Jokinen >>> >>> >>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >>> To unsubscribe send an email to >>> freeipa-users-leave(a)lists.fedorahosted.org >>> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >>> List Guidelines: >>> https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: >>> https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fed >>> o rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ >>> >> >> Hi, >> >> does your topology include multiple CA instances? You need first to find= which master is the CA renewal master: >> ipa config-show | grep "renewal master" >> >> On this host, check that the certificates are still valid and consistent= with the content of the LDAP entries. If it is not the case, you need to r= epair the CA renewal master first. >> >> When the CA renewal master is OK, check if the replication is working wi= th the other CA instances, and repair the other masters. >> HTH, >> Flo >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo >> rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/ >> > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/7COMUEBZFYX6KV55HWWNFZ2H6JKDTNVB/ >=20 --===============7233827499095487489==-- From Eemeli.Jokinen at cinia.fi Mon Jun 25 05:48:58 2018 Content-Type: multipart/mixed; boundary="===============6784867698702509788==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Mon, 25 Jun 2018 05:48:26 +0000 Message-ID: In-Reply-To: 13a705b6-3605-5247-2331-1fbc580141eb@redhat.com --===============6784867698702509788== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! gssproxy up and running -- systemctl status gssproxy =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vend= or preset: disabled) Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks 2 d= ays ago Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, status=3D= 0/SUCCESS) -- Also seems like there's some default configuration of gssproxy, no ipa.conf= (googling said that there should probably be also ipa.conf?). -- ls /etc/gssproxy/ 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf -- Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: perjantai 22. kes=C3=A4kuuta 2018 15.39 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > Forgot kinit: > = > -- > kinit admin > Password for admin@<>: > klist > Ticket cache: KEYRING:persistent:0:0 > Default principal: admin@<> > = > Valid starting Expires Service principal > 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<>@<> > 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<>@<> = > ipa config-show > ipa: ERROR: No valid Negotiate header in server response > -- > = Hi, can you check if the service gssproxy is running: # systemctl status gssproxy and if not, restart it. Flo > Still no luck getting ipa config > = > = > Eemeli > = --===============6784867698702509788==-- From flo at redhat.com Mon Jun 25 09:53:35 2018 Content-Type: multipart/mixed; boundary="===============7911575561786443612==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Mon, 25 Jun 2018 11:53:09 +0200 Message-ID: In-Reply-To: ae8a28d1a260453f8dcaab042757af5c@cinia.fi --===============7911575561786443612== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > gssproxy up and running > = > -- > systemctl status gssproxy > =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon > Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; v= endor preset: disabled) > Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks = 2 days ago > Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, status= =3D0/SUCCESS) > -- > = > Also seems like there's some default configuration of gssproxy, no ipa.co= nf (googling said that there should probably be also ipa.conf?). > = > -- > ls /etc/gssproxy/ > 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf > -- > = Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file = should be created during ipa-server-upgrade, but after the step = restarting pki-tomcat. So let's go back to our initial goal: finding which master is the = renewal master. You can use a ldapsearch query to find out the renewal = master: # ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b = cn=3Dmasters,cn=3Dipa,cn=3Detc,$BASEDN = '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Enter LDAP Password: dn: cn=3DCA,cn=3Dmyrenewalmaster.domain.com,cn=3Dmasters,cn=3Dipa,cn=3Detc,= $BASEDN (replace BASEDN with your own setting that can be found in = /etc/ipa/default.conf) Flo > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: perjantai 22. kes=C3=A4kuuta 2018 15.39 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > On 06/21/2018 08:57 AM, Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> Forgot kinit: >> >> -- >> kinit admin >> Password for admin@<>: >> klist >> Ticket cache: KEYRING:persistent:0:0 >> Default principal: admin@<> >> >> Valid starting Expires Service principal >> 06/21/2018 09:55:07 06/22/2018 09:54:54 HTTP/<>@<> >> 06/21/2018 09:55:02 06/22/2018 09:54:54 krbtgt/<>@<> >> ipa config-show >> ipa: ERROR: No valid Negotiate header in server response >> -- >> > Hi, > can you check if the service gssproxy is running: > # systemctl status gssproxy > and if not, restart it. > = > Flo > = >> Still no luck getting ipa config >> >> >> Eemeli >> > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/GR4ULROH2LK6PXMDLLD7F4JAVEY34776/ >=20 --===============7911575561786443612==-- From Eemeli.Jokinen at cinia.fi Mon Jun 25 12:01:09 2018 Content-Type: multipart/mixed; boundary="===============8532513825580518400==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Mon, 25 Jun 2018 11:59:55 +0000 Message-ID: In-Reply-To: fdbc1901-10f7-cd40-48c1-0e3596ca9e1d@redhat.com --===============8532513825580518400== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! The node 1 is the Renewal Master -- ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa,cn= =3Detc,BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Enter LDAP Password: dn: cn=3DCA,cn=3D<>,cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN -- Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: maanantai 25. kes=C3=A4kuuta 2018 12.53 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > gssproxy up and running > = > -- > systemctl status gssproxy > =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon > Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; v= endor preset: disabled) > Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 weeks = 2 days ago > Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, = > status=3D0/SUCCESS) > -- > = > Also seems like there's some default configuration of gssproxy, no ipa.co= nf (googling said that there should probably be also ipa.conf?). > = > -- > ls /etc/gssproxy/ > 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf > -- > = Hi, you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file sh= ould be created during ipa-server-upgrade, but after the step restarting pk= i-tomcat. So let's go back to our initial goal: finding which master is the renewal m= aster. You can use a ldapsearch query to find out the renewal master: # ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa,cn= =3Detc,$BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Enter L= DAP Password: dn: cn=3DCA,cn=3Dmyrenewalmaster.domain.com,cn=3Dmasters,cn=3Dipa,cn=3Detc,= $BASEDN (replace BASEDN with your own setting that can be found in /etc/ipa/default.conf) Flo --===============8532513825580518400==-- From flo at redhat.com Tue Jun 26 07:27:22 2018 Content-Type: multipart/mixed; boundary="===============2693316474932010832==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Tue, 26 Jun 2018 09:26:57 +0200 Message-ID: <91935ba2-3485-831a-d8d5-a9e60d47e321@redhat.com> In-Reply-To: fe437c4bce07450dbfecae880aff8266@cinia.fi --===============2693316474932010832== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > The node 1 is the Renewal Master > -- > ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa,cn= =3Detc,BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn > Enter LDAP Password: > dn: cn=3DCA,cn=3D<>,cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN > -- > = OK, so we know that your host node1 is the renewal master and it has 4 = expired certificates. What is the full output of getcert list? The journal will show why it was not able to renew them: # journalctl -u certmonger Can you also provide the version of FreeIPA you are using, and the one = you had before the upgrade? (can be found in /var/log/ipaupgrade.log = with the string "IPA version 4.xx", this file keeps the whole upgrade = history). Flo > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: maanantai 25. kes=C3=A4kuuta 2018 12.53 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> gssproxy up and running >> >> -- >> systemctl status gssproxy >> =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon >> Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled;= vendor preset: disabled) >> Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 week= s 2 days ago >> Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, >> status=3D0/SUCCESS) >> -- >> >> Also seems like there's some default configuration of gssproxy, no ipa.c= onf (googling said that there should probably be also ipa.conf?). >> >> -- >> ls /etc/gssproxy/ >> 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf >> -- >> > Hi, > you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file = should be created during ipa-server-upgrade, but after the step restarting = pki-tomcat. > = > So let's go back to our initial goal: finding which master is the renewal= master. You can use a ldapsearch query to find out the renewal > master: > # ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa,= cn=3Detc,$BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Enter= LDAP Password: > dn: cn=3DCA,cn=3Dmyrenewalmaster.domain.com,cn=3Dmasters,cn=3Dipa,cn=3Det= c,$BASEDN > = > (replace BASEDN with your own setting that can be found in > /etc/ipa/default.conf) > = > Flo > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/ >=20 --===============2693316474932010832==-- From Eemeli.Jokinen at cinia.fi Tue Jun 26 07:58:29 2018 Content-Type: multipart/mixed; boundary="===============2123246860639997923==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Tue, 26 Jun 2018 07:58:02 +0000 Message-ID: In-Reply-To: 91935ba2-3485-831a-d8d5-a9e60d47e321@redhat.com --===============2123246860639997923== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello! Thank you for your answers by the way, seems like we're getting closer and = closer every step although haven't had a breakthrough yet... At least I fee= l like I understand the structure of IPA better alredy! A bit long message = incoming... :) First getcert list. Some sites say that there should be 9 certificates list= ed as of ipa-server 4.5 -- getcert list Number of certificates and requests being tracked: 8. Request ID '20160331084233': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB',p= in set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DCA Audit,O=3D<> expires: 2018-03-21 09:42:06 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "audit= SigningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084234': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pi= n set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DOCSP Subsystem,O=3D<> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspS= igningCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084236': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin = set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DCertificate Authority,O=3D<> expires: 2036-03-31 08:42:02 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSigni= ngCert cert-pki-ca" track: yes auto-renew: yes Request ID '20160331084238': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2020-02-11 09:58:22 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Serve= r-Cert cert-pki-ca=E2=80=9D track: yes auto-renew: yes Request ID '20160331084308': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-<>',nickname=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/= dirsrv/slapd-<>/pwdfile.txt' certificate: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-<>',= nickname=3D'Server-Cert',token=3D'NSS Certificate DB' CA: IPA issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2020-03-04 09:58:32 UTC principal name: ldap/<>@<> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <> track: yes auto-renew: yes Request ID '20160331085008': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nickna= me=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias= /pwdfile.txt' certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D= 'Server-Cert',token=3D'NSS Certificate DB' CA: IPA issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2020-03-04 09:58:23 UTC principal name: HTTP/<>@<> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: post-save command: /usr/lib64/ipa/certmonger/restart_httpd track: yes auto-renew: yes Request ID '20180611071929': status: MONITORING stuck: no key pair storage: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.key' certificate: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DIPA RA,O=3D<> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20180615083528': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB',pin = set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DCA Subsystem,O=3D<> expires: 2018-03-21 09:42:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsy= stemCert cert-pki-ca" track: yes auto-renew: yes -- Next journalctl... I've tried changing the date of the server back to older= days to get certmonger automatically renew them. Should I try this one aga= in? -- journalctl -u certmonger -- Logs begin at Mon 2018-06-25 17:46:25 EEST, end at Tue 2018-06-26 10:43:= 30 EEST. -- Jun 25 17:46:27 <> certmonger[16802]: Certificate named "subsyst= emCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki= -tomcat/alias" is no longer valid. Jun 25 17:46:29 <> dogtag-ipa-ca-renew-agent-submit[16804]: Forw= arding request to dogtag-ipa-renew-agent Jun 25 17:46:29 <> dogtag-ipa-ca-renew-agent-submit[16804]: dogt= ag-ipa-renew-agent returned 2 Jun 25 17:46:36 <> certmonger[16822]: Certificate named "auditSi= gningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/= pki-tomcat/alias" is no longer valid. Jun 25 17:46:39 <> dogtag-ipa-ca-renew-agent-submit[16824]: Forw= arding request to dogtag-ipa-renew-agent Jun 25 17:46:39 <> dogtag-ipa-ca-renew-agent-submit[16824]: dogt= ag-ipa-renew-agent returned 2 Jun 25 17:46:41 <> certmonger[16839]: Certificate in file "/var/= lib/ipa/ra-agent.pem" is no longer valid. Jun 25 17:46:43 <> dogtag-ipa-ca-renew-agent-submit[16841]: Forw= arding request to dogtag-ipa-renew-agent Jun 25 17:46:43 <> dogtag-ipa-ca-renew-agent-submit[16841]: dogt= ag-ipa-renew-agent returned 2 ... Jun 26 10:40:47 <> dogtag-ipa-ca-renew-agent-submit[2530]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:40:47 <> dogtag-ipa-ca-renew-agent-submit[2530]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:40:48 <> certmonger[2546]: Certificate named "ocspSign= ingCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pk= i-tomcat/alias" is no longer valid. Jun 26 10:40:51 <> dogtag-ipa-ca-renew-agent-submit[2548]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:40:51 <> dogtag-ipa-ca-renew-agent-submit[2548]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:15 <> certmonger[2580]: Certificate named "subsyste= mCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-= tomcat/alias" is no longer valid. Jun 26 10:41:17 <> dogtag-ipa-ca-renew-agent-submit[2582]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:17 <> dogtag-ipa-ca-renew-agent-submit[2582]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:18 <> dogtag-ipa-ca-renew-agent-submit[2594]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:18 <> dogtag-ipa-ca-renew-agent-submit[2594]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:20 <> dogtag-ipa-ca-renew-agent-submit[2608]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:20 <> dogtag-ipa-ca-renew-agent-submit[2608]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:21 <> certmonger[2624]: Certificate named "ocspSign= ingCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pk= i-tomcat/alias" is no longer valid. Jun 26 10:41:24 <> dogtag-ipa-ca-renew-agent-submit[2626]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:24 <> dogtag-ipa-ca-renew-agent-submit[2626]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:48 <> certmonger[2667]: Certificate named "subsyste= mCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pki-= tomcat/alias" is no longer valid. Jun 26 10:41:50 <> dogtag-ipa-ca-renew-agent-submit[2669]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2669]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2682]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2682]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:53 <> dogtag-ipa-ca-renew-agent-submit[2697]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:53 <> dogtag-ipa-ca-renew-agent-submit[2697]: dogta= g-ipa-renew-agent returned 2 Jun 26 10:41:54 <> certmonger[2713]: Certificate named "ocspSign= ingCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pk= i-tomcat/alias" is no longer valid. Jun 26 10:41:57 <> dogtag-ipa-ca-renew-agent-submit[2715]: Forwa= rding request to dogtag-ipa-renew-agent Jun 26 10:41:57 <> dogtag-ipa-ca-renew-agent-submit[2715]: dogta= g-ipa-renew-agent returned 2 -- About versions: = OS CentOS 7.5.1804 Current IPA version 4.5.4-10.el7.centos.1 (from ipaupgrade.log) Previous IPA version 4.2.0-15.0.1.el7.centos.6 (from ipaserver-install.log) The date of the ipaserver-install.log is 2016.03.31 so exactly 720 days bef= ore the expire date of those 4 certificates... I tought I had upgraded it once before but probably I just remember it wron= g (we have a test environment also and it might be that I updated that one = as part of troubleshooting process of another problem) because can't find a= ny mark of it. Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: tiistai 26. kes=C3=A4kuuta 2018 10.27 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > The node 1 is the Renewal Master > -- > ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b = > cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRe= newalMaster))' dn Enter LDAP Password: > dn: cn=3DCA,cn=3D<>,cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN > -- > = OK, so we know that your host node1 is the renewal master and it has 4 expi= red certificates. What is the full output of getcert list? The journal will show why it was not able to renew them: # journalctl -u certmonger Can you also provide the version of FreeIPA you are using, and the one you = had before the upgrade? (can be found in /var/log/ipaupgrade.log with the s= tring "IPA version 4.xx", this file keeps the whole upgrade history). Flo > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: maanantai 25. kes=C3=A4kuuta 2018 12.53 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: = > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > = > On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> gssproxy up and running >> >> -- >> systemctl status gssproxy >> =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon >> Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled;= vendor preset: disabled) >> Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 week= s 2 days ago >> Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, >> status=3D0/SUCCESS) >> -- >> >> Also seems like there's some default configuration of gssproxy, no ipa.c= onf (googling said that there should probably be also ipa.conf?). >> >> -- >> ls /etc/gssproxy/ >> 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf >> -- >> > Hi, > you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file = should be created during ipa-server-upgrade, but after the step restarting = pki-tomcat. > = > So let's go back to our initial goal: finding which master is the = > renewal master. You can use a ldapsearch query to find out the renewal > master: > # ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa,= cn=3Detc,$BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Enter= LDAP Password: > dn: = > cn=3DCA,cn=3Dmyrenewalmaster.domain.com,cn=3Dmasters,cn=3Dipa,cn=3Detc,$B= ASEDN > = > (replace BASEDN with your own setting that can be found in > /etc/ipa/default.conf) > = > Flo > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to = > freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: = > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: = > https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo > rahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/ > = --===============2123246860639997923==-- From flo at redhat.com Tue Jun 26 18:28:54 2018 Content-Type: multipart/mixed; boundary="===============0982616755273853236==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Tue, 26 Jun 2018 20:28:23 +0200 Message-ID: <29177035-fa60-9fe3-5656-206ebb5fa55f@redhat.com> In-Reply-To: cd27141d1a11476683eb1da1ddba949a@cinia.fi --===============0982616755273853236== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/26/2018 09:58 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hello! > = > Thank you for your answers by the way, seems like we're getting closer an= d closer every step although haven't had a breakthrough yet... At least I f= eel like I understand the structure of IPA better alredy! A bit long messag= e incoming... :) > = > First getcert list. Some sites say that there should be 9 certificates li= sted as of ipa-server 4.5 > = > -- > getcert list > Number of certificates and requests being tracked: 8. > Request ID '20160331084233': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB= ',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DCA Audit,O=3D<> > expires: 2018-03-21 09:42:06 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "au= ditSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160331084234': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB'= ,pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DOCSP Subsystem,O=3D<> > expires: 2018-03-21 09:42:04 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "oc= spSigningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160331084236': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB',p= in set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DCertificate Authority,O=3D<> > expires: 2036-03-31 08:42:02 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > pre-save command: /usr/lib64/ipa/certmonger/stop_pkicad > post-save command: /usr/lib64/ipa/certmonger/renew_ca_cert "caSi= gningCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20160331084238': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin= set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3D<>,O=3D<> > expires: 2020-02-11 09:58:22 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Se= rver-Cert cert-pki-ca=E2=80=9D > track: yes > auto-renew: yes > Request ID '20160331084308': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-<>',nickname=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/e= tc/dirsrv/slapd-<>/pwdfile.txt' > certificate: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-<= >',nickname=3D'Server-Cert',token=3D'NSS Certificate DB' > CA: IPA > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3D<>,O=3D<> > expires: 2020-03-04 09:58:32 UTC > principal name: ldap/<>@<> > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_dirsrv <> > track: yes > auto-renew: yes > Request ID '20160331085008': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nic= kname=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/al= ias/pwdfile.txt' > certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname= =3D'Server-Cert',token=3D'NSS Certificate DB' > CA: IPA > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3D<>,O=3D<> > expires: 2020-03-04 09:58:23 UTC > principal name: HTTP/<>@<> > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: > post-save command: /usr/lib64/ipa/certmonger/restart_httpd > track: yes > auto-renew: yes > Request ID '20180611071929': > status: MONITORING > stuck: no > key pair storage: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.= key' > certificate: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.pem' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DIPA RA,O=3D<> > expires: 2018-03-21 09:42:29 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20180615083528': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB',p= in set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DCA Subsystem,O=3D<> > expires: 2018-03-21 09:42:05 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "su= bsystemCert cert-pki-ca" > track: yes > auto-renew: yes > -- > = Hi, the journal shows that dogtag-ipa-renew-agent returned 2, it means = "Rejected" (see [1] for the return codes). This probably happens because = the cert for IPA RA is no longer valid (this cert is used to = authenticate to Dogtag, and without proper authentication any renewal op = is refused). The expired certificates all expire on 2018-03-21. On the other hand, = ServerCert cert-pki-ca, slapd and httpd certificates were properly = renewed. You need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' = | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not = Before" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" You need then to find a common date where all the certificates are valid = (ie before 2018-03-21 so that the expired certs are not expired yet, and = after the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger = and look in the journal if the renewal goes smoothly or if there are = errors that could point you in the right direction. You can also find instructions on this blog post [2] to increase the log = level for the renewal. HTH, Flo [1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] = https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues= -with-freeipa/ > Next journalctl... I've tried changing the date of the server back to old= er days to get certmonger automatically renew them. Should I try this one a= gain? > = > -- > journalctl -u certmonger > -- Logs begin at Mon 2018-06-25 17:46:25 EEST, end at Tue 2018-06-26 10:4= 3:30 EEST. -- > Jun 25 17:46:27 <> certmonger[16802]: Certificate named "subsy= stemCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/p= ki-tomcat/alias" is no longer valid. > Jun 25 17:46:29 <> dogtag-ipa-ca-renew-agent-submit[16804]: Fo= rwarding request to dogtag-ipa-renew-agent > Jun 25 17:46:29 <> dogtag-ipa-ca-renew-agent-submit[16804]: do= gtag-ipa-renew-agent returned 2 > Jun 25 17:46:36 <> certmonger[16822]: Certificate named "audit= SigningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pk= i/pki-tomcat/alias" is no longer valid. > Jun 25 17:46:39 <> dogtag-ipa-ca-renew-agent-submit[16824]: Fo= rwarding request to dogtag-ipa-renew-agent > Jun 25 17:46:39 <> dogtag-ipa-ca-renew-agent-submit[16824]: do= gtag-ipa-renew-agent returned 2 > Jun 25 17:46:41 <> certmonger[16839]: Certificate in file "/va= r/lib/ipa/ra-agent.pem" is no longer valid. > Jun 25 17:46:43 <> dogtag-ipa-ca-renew-agent-submit[16841]: Fo= rwarding request to dogtag-ipa-renew-agent > Jun 25 17:46:43 <> dogtag-ipa-ca-renew-agent-submit[16841]: do= gtag-ipa-renew-agent returned 2 > ... > Jun 26 10:40:47 <> dogtag-ipa-ca-renew-agent-submit[2530]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:40:47 <> dogtag-ipa-ca-renew-agent-submit[2530]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:40:48 <> certmonger[2546]: Certificate named "ocspSi= gningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/= pki-tomcat/alias" is no longer valid. > Jun 26 10:40:51 <> dogtag-ipa-ca-renew-agent-submit[2548]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:40:51 <> dogtag-ipa-ca-renew-agent-submit[2548]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:15 <> certmonger[2580]: Certificate named "subsys= temCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pk= i-tomcat/alias" is no longer valid. > Jun 26 10:41:17 <> dogtag-ipa-ca-renew-agent-submit[2582]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:17 <> dogtag-ipa-ca-renew-agent-submit[2582]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:18 <> dogtag-ipa-ca-renew-agent-submit[2594]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:18 <> dogtag-ipa-ca-renew-agent-submit[2594]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:20 <> dogtag-ipa-ca-renew-agent-submit[2608]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:20 <> dogtag-ipa-ca-renew-agent-submit[2608]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:21 <> certmonger[2624]: Certificate named "ocspSi= gningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/= pki-tomcat/alias" is no longer valid. > Jun 26 10:41:24 <> dogtag-ipa-ca-renew-agent-submit[2626]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:24 <> dogtag-ipa-ca-renew-agent-submit[2626]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:48 <> certmonger[2667]: Certificate named "subsys= temCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/pk= i-tomcat/alias" is no longer valid. > Jun 26 10:41:50 <> dogtag-ipa-ca-renew-agent-submit[2669]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2669]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2682]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:51 <> dogtag-ipa-ca-renew-agent-submit[2682]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:53 <> dogtag-ipa-ca-renew-agent-submit[2697]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:53 <> dogtag-ipa-ca-renew-agent-submit[2697]: dog= tag-ipa-renew-agent returned 2 > Jun 26 10:41:54 <> certmonger[2713]: Certificate named "ocspSi= gningCert cert-pki-ca" in token "NSS Certificate DB" in database "/etc/pki/= pki-tomcat/alias" is no longer valid. > Jun 26 10:41:57 <> dogtag-ipa-ca-renew-agent-submit[2715]: For= warding request to dogtag-ipa-renew-agent > Jun 26 10:41:57 <> dogtag-ipa-ca-renew-agent-submit[2715]: dog= tag-ipa-renew-agent returned 2 > -- > = > About versions: > OS CentOS 7.5.1804 > Current IPA version 4.5.4-10.el7.centos.1 (from ipaupgrade.log) > Previous IPA version 4.2.0-15.0.1.el7.centos.6 (from ipaserver-install.lo= g) > The date of the ipaserver-install.log is 2016.03.31 so exactly 720 days b= efore the expire date of those 4 certificates... > = > I tought I had upgraded it once before but probably I just remember it wr= ong (we have a test environment also and it might be that I updated that on= e as part of troubleshooting process of another problem) because can't find= any mark of it. > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: tiistai 26. kes=C3=A4kuuta 2018 10.27 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > On 06/25/2018 01:59 PM, Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> The node 1 is the Renewal Master >> -- >> ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b >> cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaR= enewalMaster))' dn Enter LDAP Password: >> dn: cn=3DCA,cn=3D<>,cn=3Dmasters,cn=3Dipa,cn=3Detc,BASEDN >> -- >> > OK, so we know that your host node1 is the renewal master and it has 4 ex= pired certificates. What is the full output of getcert list? > = > The journal will show why it was not able to renew them: > # journalctl -u certmonger > = > Can you also provide the version of FreeIPA you are using, and the one yo= u had before the upgrade? (can be found in /var/log/ipaupgrade.log with the= string "IPA version 4.xx", this file keeps the whole upgrade history). > = > Flo >> >> Eemeli >> >> -----Original Message----- >> From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] >> Sent: maanantai 25. kes=C3=A4kuuta 2018 12.53 >> To: FreeIPA users list >> Cc: Jokinen Eemeli >> Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: >> ipa-server-upgrade doesn't complete, pki-tomcatd won't start >> >> On 06/25/2018 07:48 AM, Jokinen Eemeli via FreeIPA-users wrote: >>> Hi! >>> >>> gssproxy up and running >>> >>> -- >>> systemctl status gssproxy >>> =E2=97=8F gssproxy.service - GSSAPI Proxy Daemon >>> Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disable= d; vendor preset: disabled) >>> Active: active (running) since Fri 2018-06-15 12:58:24 EEST; 1 we= eks 2 days ago >>> Process: 3807 ExecStart=3D/usr/sbin/gssproxy -D (code=3Dexited, >>> status=3D0/SUCCESS) >>> -- >>> >>> Also seems like there's some default configuration of gssproxy, no ipa.= conf (googling said that there should probably be also ipa.conf?). >>> >>> -- >>> ls /etc/gssproxy/ >>> 24-nfs-server.conf 99-nfs-client.conf gssproxy.conf >>> -- >>> >> Hi, >> you are indeed missing the file /etc/gssproxy/10-ipa.conf, and this file= should be created during ipa-server-upgrade, but after the step restarting= pki-tomcat. >> >> So let's go back to our initial goal: finding which master is the >> renewal master. You can use a ldapsearch query to find out the renewal >> master: >> # ldapsearch -D cn=3Ddirectory\ manager -W -LLL -b cn=3Dmasters,cn=3Dipa= ,cn=3Detc,$BASEDN '(&(cn=3DCA)(ipaConfigString=3DcaRenewalMaster))' dn Ente= r LDAP Password: >> dn: >> cn=3DCA,cn=3Dmyrenewalmaster.domain.com,cn=3Dmasters,cn=3Dipa,cn=3Detc,$= BASEDN >> >> (replace BASEDN with your own setting that can be found in >> /etc/ipa/default.conf) >> >> Flo >> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave(a)lists.fedorahosted.org >> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html >> List Guidelines: >> https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo >> rahosted.org/message/VMQPV3EF4XN2QYAFQEG63KU5YNQW64TX/ >> > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/FHKV7F3U4HEA2STDG64L5LKEYXMJVVES/ >=20 --===============0982616755273853236==-- From Eemeli.Jokinen at cinia.fi Wed Jun 27 06:57:10 2018 Content-Type: multipart/mixed; boundary="===============4330520000191315075==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 27 Jun 2018 06:56:45 +0000 Message-ID: <240aa6dfa895434bb248f72dfc25b959@cinia.fi> In-Reply-To: 29177035-fa60-9fe3-5656-206ebb5fa55f@redhat.com --===============4330520000191315075== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! -- certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |gre= p "Not Before" Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Befor= e" Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC -- So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using https= ://access.redhat.com/solutions/3357261 as a guideline. -- systemctl stop ntpd date 031603162018 Fri Mar 16 03:16:00 EET 2018 systemctl restart certmonger certutil -d /var/lib/pki/pki-tomcat/alias/ -L Certificate Nickname Trust Attribut= es SSL,S/MIME,JAR= /XPI auditSigningCert cert-pki-ca u,u,Pu caSigningCert cert-pki-ca CTu,Cu,Cu ocspSigningCert cert-pki-ca u,u,u subsystemCert cert-pki-ca u,u,u Server-Cert cert-pki-ca u,u,u getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 "expires: 2018-03" | grep ID Request ID '20160331084233': Request ID '20160331084234': Request ID '20180611071929': Request ID '20180615083528': ipa-getcert resubmit -i 20160331084233 -v Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20160331084234 -v Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180611071929 -v Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". ipa-getcert resubmit -i 20180615083528 -v Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". journalctl -n 20 -u certmonger -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:04:= 17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certifi= cate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certifi= cate monitoring and PKI enrollment... Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certific= ate monitoring and PKI enrollment. Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI cli= ent step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI cli= ent step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI cli= ent step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI cli= ent step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI cli= ent step 2 Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5103]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5103]: dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5228]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5228]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5256]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5256]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5296]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5322]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5322]: dogtag-ipa-renew-agent returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-su= bmit[5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET 2018 -- I waited for some time to be sure, no luck on my opinion: -- date Fri Mar 16 03:52:24 EET 2018 getcert list |grep expires expires: 2018-03-21 09:42:06 UTC expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2020-02-11 09:58:22 UTC expires: 2020-03-04 09:58:32 UTC expires: 2020-03-04 09:58:23 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-03-21 09:42:05 UTC -- Also did steps 6 & 8 on the guideline page, certificates match. However ste= p 7 fails to error 500. Still wondering if I'm missing some kind of cert from certmonger since the = site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certifica= tes on "getcert list", I only have 8. However if I try to do the tracking r= equests again as suggested by RHEL, I get no new certificates for my list. Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: tiistai 26. kes=C3=A4kuuta 2018 21.28 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start Hi, the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejecte= d" (see [1] for the return codes). This probably happens because the cert f= or IPA RA is no longer valid (this cert is used to authenticate to Dogtag, = and without proper authentication any renewal op is refused). The expired certificates all expire on 2018-03-21. On the other hand, Serve= rCert cert-pki-ca, slapd and httpd certificates were properly renewed. You = need to find at which date they were renewed: # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' = | grep "Not Before") # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Befor= e" # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" You need then to find a common date where all the certificates are valid (i= e before 2018-03-21 so that the expired certs are not expired yet, and afte= r the 'Not Before' date so that the renewed certs are already valid). Then stop ntpd, change the date to this common date, restart certmonger and= look in the journal if the renewal goes smoothly or if there are errors th= at could point you in the right direction. You can also find instructions on this blog post [2] to increase the log le= vel for the renewal. HTH, Flo [1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 [2] https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issues= -with-freeipa/ --===============4330520000191315075==-- From flo at redhat.com Wed Jun 27 07:40:55 2018 Content-Type: multipart/mixed; boundary="===============5878561224331671601==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 27 Jun 2018 09:40:17 +0200 Message-ID: <56a341bb-d17c-467b-92c4-76449baeaf0c@redhat.com> In-Reply-To: 240aa6dfa895434bb248f72dfc25b959@cinia.fi --===============5878561224331671601== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/27/2018 08:56 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > -- > certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |g= rep "Not Before" > Not Before: Wed Feb 21 09:58:22 2018 > certutil -L -d /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Bef= ore" > Not Before: Sun Mar 04 09:58:32 2018 > certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:23 2018 > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using htt= ps://access.redhat.com/solutions/3357261 as a guideline. > = > -- > systemctl stop ntpd > date 031603162018 > Fri Mar 16 03:16:00 EET 2018 > systemctl restart certmonger > certutil -d /var/lib/pki/pki-tomcat/alias/ -L > = > Certificate Nickname Trust Attrib= utes > SSL,S/MIME,= JAR/XPI > = > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > getcert list |grep -B 8 "expires: 2018-03" | grep ID > Request ID '20160331084233': > Request ID '20160331084234': > Request ID '20180611071929': > Request ID '20180615083528': > ipa-getcert resubmit -i 20160331084233 -v > Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20160331084234 -v > Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180611071929 -v > Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180615083528 -v > Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". > journalctl -n 20 -u certmonger > -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:0= 4:17 EEST. -- > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certif= icate monitoring and PKI enrollment. > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 2 > Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5103]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5103]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5228]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5228]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5256]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5256]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5296]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5296]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5322]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5322]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5676]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5676]: dogtag-ipa-renew-agent returned 2 > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > date > Fri Mar 16 03:26:09 EET 2018 > -- > = > I waited for some time to be sure, no luck on my opinion: > = > -- > date > Fri Mar 16 03:52:24 EET 2018 > getcert list |grep expires > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > Also did steps 6 & 8 on the guideline page, certificates match. However s= tep 7 fails to error 500. > = Error 500 is internal error. Can you check the content of Dogtag log? /var/log/pki/pki-tomcat/localhost_access_log_$date.txt must show the = command getCertChain has been received: 10.37.171.235 - - [date] "GET /ca/ee/ca/getCertChain HTTP/1.1" 200 1534 and /var/log/pki/pki-tomcat/ca/debug may show more information. On a = working system: [date][http-bio-8443-exec-13]: SignedAuditLogger: event = ACCESS_SESSION_ESTABLISH [date][http-bio-8443-exec-13]: according to ccMode, authorization for = servlet: caGetCertChain is LDAP based, not XML {1}, use default authz = mgr: {2}. [date][http-bio-8443-exec-13]: CMSServlet:service() uri =3D = /ca/ee/ca/getCertChain [date][http-bio-8443-exec-13]: CMSServlet: caGetCertChain start to service. [date][http-bio-8443-exec-13]: GetCertChain: certificate chain: [date][http-bio-8443-exec-13]: GetCertChain: - CN=3DCertificate = Authority,O=3DDOMAIN.COM [date][http-bio-8443-exec-13]: CMSServlet: curDate=3DWed Jun 27 09:33:23 = CEST 2018 id=3DcaGetCertChain time=3D22 [date][http-bio-8443-exec-13]: SignedAuditLogger: event = ACCESS_SESSION_TERMINATED Flo > Still wondering if I'm missing some kind of cert from certmonger since th= e site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certifi= cates on "getcert list", I only have 8. However if I try to do the tracking= requests again as suggested by RHEL, I get no new certificates for my list. > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: tiistai 26. kes=C3=A4kuuta 2018 21.28 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > Hi, > = > the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejec= ted" (see [1] for the return codes). This probably happens because the cert= for IPA RA is no longer valid (this cert is used to authenticate to Dogtag= , and without proper authentication any renewal op is refused). > = > The expired certificates all expire on 2018-03-21. On the other hand, Ser= verCert cert-pki-ca, slapd and httpd certificates were properly renewed. Yo= u need to find at which date they were renewed: > # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' > | grep "Not Before") > # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Bef= ore" > # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > = > You need then to find a common date where all the certificates are valid = (ie before 2018-03-21 so that the expired certs are not expired yet, and af= ter the 'Not Before' date so that the renewed certs are already valid). > Then stop ntpd, change the date to this common date, restart certmonger a= nd look in the journal if the renewal goes smoothly or if there are errors = that could point you in the right direction. > = > You can also find instructions on this blog post [2] to increase the log = level for the renewal. > = > HTH, > Flo > = > [1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 > [2] > https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-issu= es-with-freeipa/ > = > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/X6XG7L2WYYIHHT72V2OCRVSKINVRCPMU/ >=20 --===============5878561224331671601==-- From Eemeli.Jokinen at cinia.fi Wed Jun 27 08:04:41 2018 Content-Type: multipart/mixed; boundary="===============5107476643785869952==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 27 Jun 2018 08:04:15 +0000 Message-ID: <7ccaab9f06f54fcabd1ce0b91da7fa44@cinia.fi> In-Reply-To: 56a341bb-d17c-467b-92c4-76449baeaf0c@redhat.com --===============5107476643785869952== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! Checked access log for today date: -- <> - - [27/Jun/2018:10:57:38 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D4&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:57:41 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D7&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:57:51 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D5&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:00 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D2&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:11 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D4&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:14 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D7&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:24 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D5&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:33 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D2&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <>- - [27/Jun/2018:10:58:44 +0300] "GET /ca/ee/ca/profileSubmit?profile= Id=3DcaServerCert&serial_num=3D4&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 <> - - [27/Jun/2018:10:58:47 +0300] "GET /ca/ee/ca/profileSubmit?profil= eId=3DcaServerCert&serial_num=3D7&renewal=3Dtrue&xml=3Dtrue&requestor_name= =3DIPA HTTP/1.1" 500 2208 -- No other kind of responses, only timestamps vary. There's no access_log-file with date 2018-03-16 but there is a Catalina.out= -file with that date -- Mar 16, 2018 3:16:06 AM org.apache.catalina.core.ContainerBase backgroundPr= ocess WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm(a)4a= 53d31b background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.= java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(Contain= erBase.java:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(Stand= ardContext.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProces= sor.processChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProces= sor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProces= sor.processChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProces= sor.run(ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) -- This seems to have gotten date of which I used on my "time travel". The er= ror matches 100% with Catalina.out with timestamp matching today. Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: keskiviikko 27. kes=C3=A4kuuta 2018 10.40 To: FreeIPA users list Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 06/27/2018 08:56 AM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > -- > certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |g= rep "Not Before" > Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d = > /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d = > /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep = > "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using htt= ps://access.redhat.com/solutions/3357261 as a guideline. > = > -- > systemctl stop ntpd > date 031603162018 > Fri Mar 16 03:16:00 EET 2018 > systemctl restart certmonger > certutil -d /var/lib/pki/pki-tomcat/alias/ -L > = > Certificate Nickname Trust Attrib= utes > = > SSL,S/MIME,JAR/XPI > = > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 = > "expires: 2018-03" | grep ID Request ID '20160331084233': > Request ID '20160331084234': > Request ID '20180611071929': > Request ID '20180615083528': > ipa-getcert resubmit -i 20160331084233 -v Resubmitting = > "20160331084233" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20160331084234 -v Resubmitting = > "20160331084234" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180611071929 -v Resubmitting = > "20180611071929" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180615083528 -v Resubmitting = > "20180615083528" to "dogtag-ipa-ca-renew-agent". > journalctl -n 20 -u certmonger > -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 = > 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[= 1]: Stopping Certificate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certif= icate monitoring and PKI enrollment. > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = > GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = > ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 = > fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 = > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = > GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = > ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = > dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = > dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: do= gtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud= .fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ip= a-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-= renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15= fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: For= warding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod= .lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agen= t returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-r= enew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 1= 6 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[= 5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET = > 2018 > -- > = > I waited for some time to be sure, no luck on my opinion: > = > -- > date > Fri Mar 16 03:52:24 EET 2018 > getcert list |grep expires > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > Also did steps 6 & 8 on the guideline page, certificates match. However s= tep 7 fails to error 500. > = Error 500 is internal error. Can you check the content of Dogtag log? /var/log/pki/pki-tomcat/localhost_access_log_$date.txt must show the comman= d getCertChain has been received: 10.37.171.235 - - [date] "GET /ca/ee/ca/getCertChain HTTP/1.1" 200 1534 and /var/log/pki/pki-tomcat/ca/debug may show more information. On a workin= g system: [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_ESTA= BLISH [date][http-bio-8443-exec-13]: according to ccMode, authorization for servlet: caGetCertChain is LDAP based, not XML {1}, use default authz mgr: {2}. [date][http-bio-8443-exec-13]: CMSServlet:service() uri =3D /ca/ee/ca/getCe= rtChain [date][http-bio-8443-exec-13]: CMSServlet: caGetCertChain start to service. [date][http-bio-8443-exec-13]: GetCertChain: certificate chain: [date][http-bio-8443-exec-13]: GetCertChain: - CN=3DCertificate Authority,O= =3DDOMAIN.COM [date][http-bio-8443-exec-13]: CMSServlet: curDate=3DWed Jun 27 09:33:23 CE= ST 2018 id=3DcaGetCertChain time=3D22 [date][http-bio-8443-exec-13]: SignedAuditLogger: event ACCESS_SESSION_TERM= INATED Flo > Still wondering if I'm missing some kind of cert from certmonger since th= e site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certifi= cates on "getcert list", I only have 8. However if I try to do the tracking= requests again as suggested by RHEL, I get no new certificates for my list. > = > = > Eemeli > = > -----Original Message----- > From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] > Sent: tiistai 26. kes=C3=A4kuuta 2018 21.28 > To: FreeIPA users list > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: = > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > = > Hi, > = > the journal shows that dogtag-ipa-renew-agent returned 2, it means "Rejec= ted" (see [1] for the return codes). This probably happens because the cert= for IPA RA is no longer valid (this cert is used to authenticate to Dogtag= , and without proper authentication any renewal op is refused). > = > The expired certificates all expire on 2018-03-21. On the other hand, Ser= verCert cert-pki-ca, slapd and httpd certificates were properly renewed. Yo= u need to find at which date they were renewed: > # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' > | grep "Not Before") > # certutil -L -d /etc/dirsrv/slapd-$DOMAIN -n Server-Cert | grep "Not Bef= ore" > # certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > = > You need then to find a common date where all the certificates are valid = (ie before 2018-03-21 so that the expired certs are not expired yet, and af= ter the 'Not Before' date so that the renewed certs are already valid). > Then stop ntpd, change the date to this common date, restart certmonger a= nd look in the journal if the renewal goes smoothly or if there are errors = that could point you in the right direction. > = > You can also find instructions on this blog post [2] to increase the log = level for the renewal. > = > HTH, > Flo > = > [1] https://pagure.io/certmonger/blob/master/f/doc/submit.txt#_46 > [2] > https://floblanc.wordpress.com/2016/12/19/troubleshooting-certmonger-i > ssues-with-freeipa/ > = > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to = > freeipa-users-leave(a)lists.fedorahosted.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: = > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: = > https://lists.fedoraproject.org/archives/list/freeipa-users(a)lists.fedo > rahosted.org/message/X6XG7L2WYYIHHT72V2OCRVSKINVRCPMU/ > = --===============5107476643785869952==-- From rcritten at redhat.com Wed Jun 27 13:26:34 2018 Content-Type: multipart/mixed; boundary="===============7395770769793532063==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 27 Jun 2018 09:26:05 -0400 Message-ID: <29afe4c9-3a64-fd7d-96db-b935f6ac40f5@redhat.com> In-Reply-To: 240aa6dfa895434bb248f72dfc25b959@cinia.fi --===============7395770769793532063== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > -- > certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |g= rep "Not Before" > Not Before: Wed Feb 21 09:58:22 2018 > certutil -L -d /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Bef= ore" > Not Before: Sun Mar 04 09:58:32 2018 > certutil -L -d /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:23 2018 > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using htt= ps://access.redhat.com/solutions/3357261 as a guideline. > = > -- > systemctl stop ntpd > date 031603162018 > Fri Mar 16 03:16:00 EET 2018 > systemctl restart certmonger > certutil -d /var/lib/pki/pki-tomcat/alias/ -L > = > Certificate Nickname Trust Attrib= utes > SSL,S/MIME,J= AR/XPI > = > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > getcert list |grep -B 8 "expires: 2018-03" | grep ID > Request ID '20160331084233': > Request ID '20160331084234': > Request ID '20180611071929': > Request ID '20180615083528': > ipa-getcert resubmit -i 20160331084233 -v > Resubmitting "20160331084233" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20160331084234 -v > Resubmitting "20160331084234" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180611071929 -v > Resubmitting "20180611071929" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180615083528 -v > Resubmitting "20180615083528" to "dogtag-ipa-ca-renew-agent". > journalctl -n 20 -u certmonger > -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 08:0= 4:17 EEST. -- > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Stopping Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certif= icate monitoring and PKI enrollment. > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 1 > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI c= lient step 2 > Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5103]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:18:38 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5103]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5228]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:19:51 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5228]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5256]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:00 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5256]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5296]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5296]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5322]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:20:15 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5322]: dogtag-ipa-renew-agent returned 2 > Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5676]: Forwarding request to dogtag-ipa-renew-agent > Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-= submit[5676]: dogtag-ipa-renew-agent returned 2 > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > date > Fri Mar 16 03:26:09 EET 2018 > -- > = > I waited for some time to be sure, no luck on my opinion: > = > -- > date > Fri Mar 16 03:52:24 EET 2018 > getcert list |grep expires > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > Also did steps 6 & 8 on the guideline page, certificates match. However s= tep 7 fails to error 500. > = > Still wondering if I'm missing some kind of cert from certmonger since th= e site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certifi= cates on "getcert list", I only have 8. However if I try to do the tracking= requests again as suggested by RHEL, I get no new certificates for my list. Hard to know without seeing the list of certs. Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well. rob --===============7395770769793532063==-- From Eemeli.Jokinen at cinia.fi Thu Jun 28 05:45:17 2018 Content-Type: multipart/mixed; boundary="===============5317590188016675950==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 28 Jun 2018 05:44:50 +0000 Message-ID: In-Reply-To: 29afe4c9-3a64-fd7d-96db-b935f6ac40f5@redhat.com --===============5317590188016675950== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! No I haven't since my guide line didn't tell me to. I tried to set the date back, restart certmonger and then I did "ipactl res= tart" and then it got 2 certs renewed! One of the remaining two certificate= s was on "CA_UNREACHABLE" state, so I ran another certmonger restart and it= did get updated. The last one didn't seem to go anywhere so I resubmitted = the cert request and then that one also got renewed. I time jumped back to = today and did another ipactl restart and All this mess got started with fai= led "ipa-server-upgrade" so I ran it afterwards and it completed successful= ly with no errors. That also increased the number of certmonger tracked cer= tificates to 9 from 8 so I believe that one is fixed too. Thank you a lot! It's a bit complicated mess to understand every aspect of = it (for example I was trying to hunt missing certificate that certmonger di= dn't track even though it wasn't the issue but the outcome of failed server= upgrade) but after this I believe very that I understand it a way better! Eemeli -----Original Message----- From: Rob Crittenden [mailto:rcritten(a)redhat.com] = Sent: keskiviikko 27. kes=C3=A4kuuta 2018 16.26 To: FreeIPA users list ; Florence B= lanc-Renaud Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > -- > certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |g= rep "Not Before" > Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d = > /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d = > /etc/httpd/alias/ -n Server-Cert | grep "Not Before" > Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep = > "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using htt= ps://access.redhat.com/solutions/3357261 as a guideline. > = > -- > systemctl stop ntpd > date 031603162018 > Fri Mar 16 03:16:00 EET 2018 > systemctl restart certmonger > certutil -d /var/lib/pki/pki-tomcat/alias/ -L > = > Certificate Nickname Trust Attrib= utes > = > SSL,S/MIME,JAR/XPI > = > auditSigningCert cert-pki-ca u,u,Pu > caSigningCert cert-pki-ca CTu,Cu,Cu > ocspSigningCert cert-pki-ca u,u,u > subsystemCert cert-pki-ca u,u,u > Server-Cert cert-pki-ca u,u,u > getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 = > "expires: 2018-03" | grep ID Request ID '20160331084233': > Request ID '20160331084234': > Request ID '20180611071929': > Request ID '20180615083528': > ipa-getcert resubmit -i 20160331084233 -v Resubmitting = > "20160331084233" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20160331084234 -v Resubmitting = > "20160331084234" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180611071929 -v Resubmitting = > "20180611071929" to "dogtag-ipa-ca-renew-agent". > ipa-getcert resubmit -i 20180615083528 -v Resubmitting = > "20180615083528" to "dogtag-ipa-ca-renew-agent". > journalctl -n 20 -u certmonger > -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 = > 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[= 1]: Stopping Certificate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Certi= ficate monitoring and PKI enrollment... > Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certif= icate monitoring and PKI enrollment. > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = > GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = > ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 = > fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 = > Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = > GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = > ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = > dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = > dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: = > Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 = > fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: do= gtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud= .fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-ip= a-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-= renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:15= fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: For= warding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.prod= .lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-agen= t returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-r= enew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar 1= 6 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[= 5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET = > 2018 > -- > = > I waited for some time to be sure, no luck on my opinion: > = > -- > date > Fri Mar 16 03:52:24 EET 2018 > getcert list |grep expires > expires: 2018-03-21 09:42:06 UTC > expires: 2018-03-21 09:42:04 UTC > expires: 2036-03-31 08:42:02 UTC > expires: 2020-02-11 09:58:22 UTC > expires: 2020-03-04 09:58:32 UTC > expires: 2020-03-04 09:58:23 UTC > expires: 2018-03-21 09:42:29 UTC > expires: 2018-03-21 09:42:05 UTC > -- > = > Also did steps 6 & 8 on the guideline page, certificates match. However s= tep 7 fails to error 500. > = > Still wondering if I'm missing some kind of cert from certmonger since th= e site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certifi= cates on "getcert list", I only have 8. However if I try to do the tracking= requests again as suggested by RHEL, I get no new certificates for my list. Hard to know without seeing the list of certs. Are you restarting dogtag, Apache and 389-ds when setting the date back? That is necessary as well. rob --===============5317590188016675950==-- From rcritten at redhat.com Thu Jun 28 13:05:25 2018 Content-Type: multipart/mixed; boundary="===============9166914136908747375==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 28 Jun 2018 09:05:00 -0400 Message-ID: <08cbade2-d2f3-c3cc-81e6-1280560b720f@redhat.com> In-Reply-To: e5a767b8b44e4cbc892086209c14a4c3@cinia.fi --===============9166914136908747375== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Jokinen Eemeli wrote: > Hi! > = > No I haven't since my guide line didn't tell me to. > = > I tried to set the date back, restart certmonger and then I did "ipactl r= estart" and then it got 2 certs renewed! One of the remaining two certifica= tes was on "CA_UNREACHABLE" state, so I ran another certmonger restart and = it did get updated. The last one didn't seem to go anywhere so I resubmitte= d the cert request and then that one also got renewed. I time jumped back t= o today and did another ipactl restart and All this mess got started with f= ailed "ipa-server-upgrade" so I ran it afterwards and it completed successf= ully with no errors. That also increased the number of certmonger tracked c= ertificates to 9 from 8 so I believe that one is fixed too. There are layers of dependencies on the certs so sometimes multiple rounds of renewal are needed to sort things out. This normally happens gracefully as expiration approaches but in some cases that we haven't been able to identify this doesn't happen. > = > Thank you a lot! It's a bit complicated mess to understand every aspect o= f it (for example I was trying to hunt missing certificate that certmonger = didn't track even though it wasn't the issue but the outcome of failed serv= er upgrade) but after this I believe very that I understand it a way better! Cool, glad you are back up and running. Note that the cert issues weren't caused by the upgrade, the upgrade just made it more apparent. In order to be sure the upgrade is complete you should run: # ipa-server-upgrade The upgrade will also check all of the certs tracked by certmonger and ensure they are set up correctly. rob > = > = > Eemeli > = > -----Original Message----- > From: Rob Crittenden [mailto:rcritten(a)redhat.com] = > Sent: keskiviikko 27. kes=C3=A4kuuta 2018 16.26 > To: FreeIPA users list ; Florence= Blanc-Renaud > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > Jokinen Eemeli via FreeIPA-users wrote: >> Hi! >> >> -- >> certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca' |= grep "Not Before" >> Not Before: Wed Feb 21 09:58:22 2018 certutil -L -d = >> /etc/dirsrv/slapd-<> -n Server-Cert | grep "Not Before" >> Not Before: Sun Mar 04 09:58:32 2018 certutil -L -d = >> /etc/httpd/alias/ -n Server-Cert | grep "Not Before" >> Not Before: Sun Mar 04 09:58:23 2018 getcert list | grep = >> "expires" >> expires: 2018-03-21 09:42:06 UTC >> expires: 2018-03-21 09:42:04 UTC >> expires: 2036-03-31 08:42:02 UTC >> expires: 2020-02-11 09:58:22 UTC >> expires: 2020-03-04 09:58:32 UTC >> expires: 2020-03-04 09:58:23 UTC >> expires: 2018-03-21 09:42:29 UTC >> expires: 2018-03-21 09:42:05 UTC >> -- >> >> So after 4.3.2018 but before 21.3.2018... let's say 16.03.2018. Using ht= tps://access.redhat.com/solutions/3357261 as a guideline. >> >> -- >> systemctl stop ntpd >> date 031603162018 >> Fri Mar 16 03:16:00 EET 2018 >> systemctl restart certmonger >> certutil -d /var/lib/pki/pki-tomcat/alias/ -L >> >> Certificate Nickname Trust Attri= butes >> = >> SSL,S/MIME,JAR/XPI >> >> auditSigningCert cert-pki-ca u,u,Pu >> caSigningCert cert-pki-ca CTu,Cu,Cu >> ocspSigningCert cert-pki-ca u,u,u >> subsystemCert cert-pki-ca u,u,u >> Server-Cert cert-pki-ca u,u,u >> getcert list | grep "expires" >> expires: 2018-03-21 09:42:06 UTC >> expires: 2018-03-21 09:42:04 UTC >> expires: 2036-03-31 08:42:02 UTC >> expires: 2020-02-11 09:58:22 UTC >> expires: 2020-03-04 09:58:32 UTC >> expires: 2020-03-04 09:58:23 UTC >> expires: 2018-03-21 09:42:29 UTC >> expires: 2018-03-21 09:42:05 UTC getcert list |grep -B 8 = >> "expires: 2018-03" | grep ID Request ID '20160331084233': >> Request ID '20160331084234': >> Request ID '20180611071929': >> Request ID '20180615083528': >> ipa-getcert resubmit -i 20160331084233 -v Resubmitting = >> "20160331084233" to "dogtag-ipa-ca-renew-agent". >> ipa-getcert resubmit -i 20160331084234 -v Resubmitting = >> "20160331084234" to "dogtag-ipa-ca-renew-agent". >> ipa-getcert resubmit -i 20180611071929 -v Resubmitting = >> "20180611071929" to "dogtag-ipa-ca-renew-agent". >> ipa-getcert resubmit -i 20180615083528 -v Resubmitting = >> "20180615083528" to "dogtag-ipa-ca-renew-agent". >> journalctl -n 20 -u certmonger >> -- Logs begin at Tue 2018-06-26 15:18:57 EEST, end at Wed 2018-06-27 = >> 08:04:17 EEST. -- Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd= [1]: Stopping Certificate monitoring and PKI enrollment... >> Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Starting Cert= ificate monitoring and PKI enrollment... >> Mar 16 03:16:04 fi-krv1-ipa1.prod.lioncloud.fi systemd[1]: Started Certi= ficate monitoring and PKI enrollment. >> Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = >> GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = >> ipa-submit[4956]: GSSAPI client step 1 Mar 16 03:16:16 = >> fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: GSSAPI client step 1 = >> Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi ipa-submit[4956]: = >> GSSAPI client step 1 Mar 16 03:16:16 fi-krv1-ipa1.prod.lioncloud.fi = >> ipa-submit[4956]: GSSAPI client step 2 Mar 16 03:18:38 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = >> Forwarding request to dogtag-ipa-renew-agent Mar 16 03:18:38 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5103]: = >> dogtag-ipa-renew-agent returned 2 Mar 16 03:19:51 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = >> Forwarding request to dogtag-ipa-renew-agent Mar 16 03:19:51 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5228]: = >> dogtag-ipa-renew-agent returned 2 Mar 16 03:20:00 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: = >> Forwarding request to dogtag-ipa-renew-agent Mar 16 03:20:00 = >> fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5256]: d= ogtag-ipa-renew-agent returned 2 Mar 16 03:20:09 fi-krv1-ipa1.prod.lionclou= d.fi dogtag-ipa-ca-renew-agent-submit[5296]: Forwarding request to dogtag-i= pa-renew-agent Mar 16 03:20:09 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca= -renew-agent-submit[5296]: dogtag-ipa-renew-agent returned 2 Mar 16 03:20:1= 5 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: Fo= rwarding request to dogtag-ipa-renew-agent Mar 16 03:20:15 fi-krv1-ipa1.pro= d.lioncloud.fi dogtag-ipa-ca-renew-agent-submit[5322]: dogtag-ipa-renew-age= nt returned 2 Mar 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-= renew-agent-submit[5676]: Forwarding request to dogtag-ipa-renew-agent Mar = 16 03:25:12 fi-krv1-ipa1.prod.lioncloud.fi dogtag-ipa-ca-renew-agent-submit= [5676]: dogtag-ipa-renew-agent returned 2 getcert list | grep "expires" >> expires: 2018-03-21 09:42:06 UTC >> expires: 2018-03-21 09:42:04 UTC >> expires: 2036-03-31 08:42:02 UTC >> expires: 2020-02-11 09:58:22 UTC >> expires: 2020-03-04 09:58:32 UTC >> expires: 2020-03-04 09:58:23 UTC >> expires: 2018-03-21 09:42:29 UTC >> expires: 2018-03-21 09:42:05 UTC date Fri Mar 16 03:26:09 EET = >> 2018 >> -- >> >> I waited for some time to be sure, no luck on my opinion: >> >> -- >> date >> Fri Mar 16 03:52:24 EET 2018 >> getcert list |grep expires >> expires: 2018-03-21 09:42:06 UTC >> expires: 2018-03-21 09:42:04 UTC >> expires: 2036-03-31 08:42:02 UTC >> expires: 2020-02-11 09:58:22 UTC >> expires: 2020-03-04 09:58:32 UTC >> expires: 2020-03-04 09:58:23 UTC >> expires: 2018-03-21 09:42:29 UTC >> expires: 2018-03-21 09:42:05 UTC >> -- >> >> Also did steps 6 & 8 on the guideline page, certificates match. However = step 7 fails to error 500. >> >> Still wondering if I'm missing some kind of cert from certmonger since t= he site says that after 7.4 (ok, RHEL, not CentOS) you should have 9 certif= icates on "getcert list", I only have 8. However if I try to do the trackin= g requests again as suggested by RHEL, I get no new certificates for my lis= t. > = > Hard to know without seeing the list of certs. > = > Are you restarting dogtag, Apache and 389-ds when setting the date back? > That is necessary as well. > = > rob >=20 --===============9166914136908747375==-- From Eemeli.Jokinen at cinia.fi Wed Jul 4 13:08:45 2018 Content-Type: multipart/mixed; boundary="===============8622651747056479521==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 04 Jul 2018 13:08:18 +0000 Message-ID: <69e78296a0ae48d5a68b04ddcb735bf9@cinia.fi> In-Reply-To: 08cbade2-d2f3-c3cc-81e6-1280560b720f@redhat.com --===============8622651747056479521== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! I reply to this since there's some data in this message queue already relat= ed to my problem: I had 2 ipa node cluster, where the second node had been offline for some t= ime and at some point we had an error while trying to reboot node1 which wa= s a Renewal Master. The issue was that some certs had expired and after a b= it of special work we got the node1 back on track. I can spot three problem= s and I can't (again) figure out which one is the cause and which one I sho= uld repair first. Now I got assigned the case to get the node2 back on track also. It had som= e certificates expired (obviously) so I did a small time jump and some of t= he certs were renewed. However not all of them were upgraded. "getcert list= " reports 3 certs "CA Unreachable", other 3 certs seem fine. -- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pi= n set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DOCSP Subsystem,O=3D<> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nickna= me=3D'ipaCert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias/pwd= file.txt' certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D= 'ipaCert',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DIPA RA,O=3D<> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<>:8080/ca/ee/ca/= profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection -- Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) b= ut shouldn't node1 be the renewal master? Restarting httpd, certmonger and = pki-tomcat don't seem to help, time traveling helped on other certs but not= on these. Directory service seems to work if I start it manually but ipa-server-upgra= de fails on directory server not starting with "No ports specified" so some= thing wrong with it or is it the certificates? -- ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server -- <> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EME= RG - main - Fatal Error---No ports specified. Exiting now. -- Also certmonger has issues: -- dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submi= t", line 541, in = sys.exit(main()) = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submi= t", line 515, in main = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) = File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.p= y", line 43, in kinit_keytab = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, usa= ge=3D'initiate') = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", li= ne 64, in __new__ = store=3Dstore) = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", li= ne 148, in acquire = usage) = File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_s= tore.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) = GSSError: Major (851968): Unspecified GSS failure. Minor code = may provide more information, Minor (2529639068): Cannot contact any KDC fo= r realm '<>' -- but KDCs should be able to be resolved even from ipa node2 -- nslookup -type=3Dsrv _kerberos._tcp.<> Server: <> Address: <>#53 _kerberos._tcp.<> service =3D 0 100 88 <>. _kerberos._tcp.<> service =3D 0 100 88 <>. -- For testing purposes I turned off firewall on ipa node1 Eemeli -----Original Message----- From: Rob Crittenden [mailto:rcritten(a)redhat.com] = Sent: torstai 28. kes=C3=A4kuuta 2018 16.05 To: Jokinen Eemeli ; FreeIPA users list ; Florence Blanc-Renaud Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start Jokinen Eemeli wrote: > Hi! > = > No I haven't since my guide line didn't tell me to. > = > I tried to set the date back, restart certmonger and then I did "ipactl r= estart" and then it got 2 certs renewed! One of the remaining two certifica= tes was on "CA_UNREACHABLE" state, so I ran another certmonger restart and = it did get updated. The last one didn't seem to go anywhere so I resubmitte= d the cert request and then that one also got renewed. I time jumped back t= o today and did another ipactl restart and All this mess got started with f= ailed "ipa-server-upgrade" so I ran it afterwards and it completed successf= ully with no errors. That also increased the number of certmonger tracked c= ertificates to 9 from 8 so I believe that one is fixed too. There are layers of dependencies on the certs so sometimes multiple rounds = of renewal are needed to sort things out. This normally happens gracefully = as expiration approaches but in some cases that we haven't been able to ide= ntify this doesn't happen. > = > Thank you a lot! It's a bit complicated mess to understand every aspect o= f it (for example I was trying to hunt missing certificate that certmonger = didn't track even though it wasn't the issue but the outcome of failed serv= er upgrade) but after this I believe very that I understand it a way better! Cool, glad you are back up and running. Note that the cert issues weren't caused by the upgrade, the upgrade just m= ade it more apparent. In order to be sure the upgrade is complete you should run: # ipa-server-up= grade The upgrade will also check all of the certs tracked by certmonger and ensu= re they are set up correctly. rob > = > = > Eemeli > = > -----Original Message----- > From: Rob Crittenden [mailto:rcritten(a)redhat.com] > Sent: keskiviikko 27. kes=C3=A4kuuta 2018 16.26 > To: FreeIPA users list ; = > Florence Blanc-Renaud > Cc: Jokinen Eemeli > Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: = > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > = > = > Hard to know without seeing the list of certs. > = > Are you restarting dogtag, Apache and 389-ds when setting the date back? > That is necessary as well. > = > rob > = --===============8622651747056479521==-- From Eemeli.Jokinen at cinia.fi Wed Aug 15 11:21:02 2018 Content-Type: multipart/mixed; boundary="===============7235022160292683669==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 15 Aug 2018 11:20:25 +0000 Message-ID: <94d184a04e854f1c9f2804181b48ae30@cinia.fi> In-Reply-To: 69e78296a0ae48d5a68b04ddcb735bf9@cinia.fi --===============7235022160292683669== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! Anybody can help me with this one? Summary: 2 node freeipa server cluster, node 2 was initially down for other reasons = and node 1 (renewal master) had forgot to update own certificates which re= sulted faulty cluster. With help from mailing list we got the node 1 back o= nline and it's working great! Now I'm trying to get node2 back to working o= rder in cluster but it won't update the certificates even when trying the t= imejump. Seems like it tries to renew certificates locally although somehow= I tought that it should renew the certificates from node 1...? Eemeli -----Original Message----- From: Jokinen Eemeli = Sent: keskiviikko 4. hein=C3=A4kuuta 2018 16.08 To: 'Rob Crittenden' ; FreeIPA users list ; Florence Blanc-Renaud Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start Hi! I reply to this since there's some data in this message queue already relat= ed to my problem: I had 2 ipa node cluster, where the second node had been offline for some t= ime and at some point we had an error while trying to reboot node1 which wa= s a Renewal Master. The issue was that some certs had expired and after a b= it of special work we got the node1 back on track. I can spot three problem= s and I can't (again) figure out which one is the cause and which one I sho= uld repair first. Now I got assigned the case to get the node2 back on track also. It had som= e certificates expired (obviously) so I did a small time jump and some of t= he certs were renewed. However not all of them were upgraded. "getcert list= " reports 3 certs "CA Unreachable", other 3 certs seem fine. -- getcert list |grep -A 10 "CA_UNREACH" status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pi= n set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DOCSP Subsystem,O=3D<> expires: 2018-03-21 09:42:04 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign eku: id-kp-OCSPSigning -- status: CA_UNREACHABLE ca-error: Internal error stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nickna= me=3D'ipaCert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias/pwd= file.txt' certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D= 'ipaCert',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3DIPA RA,O=3D<> expires: 2018-03-21 09:42:29 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth -- status: CA_UNREACHABLE ca-error: Error 7 connecting to http://<>:8080/ca/ee/ca/= profileSubmit: Couldn't connect to server. stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alia= s',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-renew-agent issuer: CN=3DCertificate Authority,O=3D<> subject: CN=3D<>,O=3D<> expires: 2018-06-27 07:01:38 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEnci= pherment eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection -- Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2) b= ut shouldn't node1 be the renewal master? Restarting httpd, certmonger and = pki-tomcat don't seem to help, time traveling helped on other certs but not= on these. Directory service seems to work if I start it manually but ipa-server-upgra= de fails on directory server not starting with "No ports specified" so some= thing wrong with it or is it the certificates? -- ipa-server-upgrade Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: starting directory server -- <> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - EME= RG - main - Fatal Error---No ports specified. Exiting now. -- Also certmonger has issues: -- dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submi= t", line 541, in = sys.exit(main()) = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submi= t", line 515, in main = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) = File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.p= y", line 43, in kinit_keytab = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, usa= ge=3D'initiate') = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", li= ne 64, in __new__ = store=3Dstore) = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", li= ne 148, in acquire = usage) = File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_s= tore.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) = GSSError: Major (851968): Unspecified GSS failure. Minor code = may provide more information, Minor (2529639068): Cannot contact any KDC fo= r realm '<>' -- but KDCs should be able to be resolved even from ipa node2 -- nslookup -type=3Dsrv _kerberos._tcp.<> Server: <> Address: <>#53 _kerberos._tcp.<> service =3D 0 100 88 <>. _kerberos._tcp.<> service =3D 0 100 88 <>. -- For testing purposes I turned off firewall on ipa node1 Eemeli --===============7235022160292683669==-- From flo at redhat.com Thu Aug 16 18:54:26 2018 Content-Type: multipart/mixed; boundary="===============7898349356895374288==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Thu, 16 Aug 2018 20:53:57 +0200 Message-ID: In-Reply-To: 94d184a04e854f1c9f2804181b48ae30@cinia.fi --===============7898349356895374288== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > Anybody can help me with this one? > = > Summary: > = > 2 node freeipa server cluster, node 2 was initially down for other reason= s and node 1 (renewal master) had forgot to update own certificates which = resulted faulty cluster. With help from mailing list we got the node 1 back= online and it's working great! Now I'm trying to get node2 back to working= order in cluster but it won't update the certificates even when trying the= timejump. Seems like it tries to renew certificates locally although someh= ow I tought that it should renew the certificates from node 1...? > = Hi, you probably have a combination of multiple issues on your second node. The ipa-server-upgrade failure may leave your instance in a wrong state, = where dse.ldif has disabled the ports for 389-ds (see BZ = https://bugzilla.redhat.com/show_bug.cgi?id=3D1569011 or pagure ticket = https://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable = the LDAP ports (to prevent ldap modifications during the upgrade). = Sometimes, if the upgrade fails, dse.ldif is not restored and the ports = remain disabled. You will have to stop the ldap server, manually edit = dse.ldif (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on then restart the LDAP server. For the cert renewal, your procedure is the valid one. The kerberos = error is probably linked to 389-ds not being accessible. HTH, flo > = > Eemeli > = > -----Original Message----- > From: Jokinen Eemeli > Sent: keskiviikko 4. hein=C3=A4kuuta 2018 16.08 > To: 'Rob Crittenden' ; FreeIPA users list ; Florence Blanc-Renaud > Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-u= pgrade doesn't complete, pki-tomcatd won't start > = > Hi! > = > I reply to this since there's some data in this message queue already rel= ated to my problem: > = > I had 2 ipa node cluster, where the second node had been offline for some= time and at some point we had an error while trying to reboot node1 which = was a Renewal Master. The issue was that some certs had expired and after a= bit of special work we got the node1 back on track. I can spot three probl= ems and I can't (again) figure out which one is the cause and which one I s= hould repair first. > = > Now I got assigned the case to get the node2 back on track also. It had s= ome certificates expired (obviously) so I did a small time jump and some of= the certs were renewed. However not all of them were upgraded. "getcert li= st" reports 3 certs "CA Unreachable", other 3 certs seem fine. > = > -- > getcert list |grep -A 10 "CA_UNREACH" > status: CA_UNREACHABLE > ca-error: Internal error > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB'= ,pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DOCSP Subsystem,O=3D<> > expires: 2018-03-21 09:42:04 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > -- > status: CA_UNREACHABLE > ca-error: Internal error > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nic= kname=3D'ipaCert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias/= pwdfile.txt' > certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname= =3D'ipaCert',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DIPA RA,O=3D<> > expires: 2018-03-21 09:42:29 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > -- > status: CA_UNREACHABLE > ca-error: Error 7 connecting to http://<>:8080/ca/ee/= ca/profileSubmit: Couldn't connect to server. > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin= set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3D<>,O=3D<> > expires: 2018-06-27 07:01:38 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > -- > = > Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2)= but shouldn't node1 be the renewal master? Restarting httpd, certmonger an= d pki-tomcat don't seem to help, time traveling helped on other certs but n= ot on these. > = > Directory service seems to work if I start it manually but ipa-server-upg= rade fails on directory server not starting with "No ports specified" so so= mething wrong with it or is it the certificates? > -- > ipa-server-upgrade > Upgrading IPA:. Estimated time: 1 minute 30 seconds > [1/10]: stopping directory server > [2/10]: saving configuration > [3/10]: disabling listeners > [4/10]: enabling DS global lock > [5/10]: starting directory server > = > -- > <> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - E= MERG - main - Fatal Error---No ports specified. Exiting now. > -- > = > Also certmonger has issues: > -- > dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-su= bmit", line 541, in > = sys.exit(main()) > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-su= bmit", line 515, in main > = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filena= me) > = File "/usr/lib/python2.7/site-packages/ipalib/install/kini= t.py", line 43, in kinit_keytab > = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, = usage=3D'initiate') > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",= line 64, in __new__ > = store=3Dstore) > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",= line 148, in acquire > = usage) > = File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cre= d_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) > = GSSError: Major (851968): Unspecified GSS failure. Minor co= de may provide more information, Minor (2529639068): Cannot contact any KDC= for realm '<>' > -- > = > but KDCs should be able to be resolved even from ipa node2 > -- > nslookup -type=3Dsrv _kerberos._tcp.<> > Server: <> > Address: <>#53 > = > _kerberos._tcp.<> service =3D 0 100 88 <>. > _kerberos._tcp.<> service =3D 0 100 88 <>. > -- > = > For testing purposes I turned off firewall on ipa node1 > = > = > Eemeli > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/XOUL2VQ26BKQHNY2XB3CDSJRKYQCHJ3X/ >=20 --===============7898349356895374288==-- From Eemeli.Jokinen at cinia.fi Fri Aug 17 11:00:16 2018 Content-Type: multipart/mixed; boundary="===============6470664745692378308==" MIME-Version: 1.0 From: Jokinen Eemeli To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Fri, 17 Aug 2018 10:59:51 +0000 Message-ID: <9a68e5ba756b472fb5bb9677b52e1e9d@cinia.fi> In-Reply-To: dbe7ea7c-1375-245a-2e26-d636a57164dd@redhat.com --===============6470664745692378308== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi! Yes, seems like there was "security: off" but that doesn't seem to do it, I= think I have ended up in the situation that I need to recreate some certif= icates, because: I check the renewal dates. -- getcert list |grep expires: expires: 2018-03-21 09:42:04 UTC expires: 2036-03-31 08:42:02 UTC expires: 2018-03-21 09:42:29 UTC expires: 2018-06-27 07:01:38 UTC expires: 2020-08-17 10:17:32 UTC expires: 2020-06-28 05:49:50 UTC -- I timejump to "before oldest expired" =3D 2018-03-20. Dirsrv seems to start= ok. Certmonger restarts ok. Httpd does not start. Error from /etc/httpd/logs/error_log: -- [Tue Mar 20 07:44:39.500363 2018] [:warn] [pid 11961] NSSSessionCacheTimeou= t is deprecated. Ignoring. [Tue Mar 20 07:44:39.688595 2018] [:error] [pid 11961] SSL Library Error: -= 8181 Certificate has expired [Tue Mar 20 07:44:39.688637 2018] [:error] [pid 11961] Unable to verify cer= tificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the s= erver can start until the problem can be resolved. -- Seems like httpd has managed to renew some certificates at some point: -- certutil -L -d /etc/httpd/alias/ -n Server-Cert |grep Not Not Before: Thu Jun 28 05:49:50 2018 Not After : Sun Jun 28 05:49:50 2020 -- Should I remove httpd certificate to be able to start httpd in "before 21-0= 3-2018? I can't seem to be able to renew these 2 (ipaCert, ocspSigningCert)= without httpd because if I try to resubmit them I get "CA_Unreachable"? Eemeli -----Original Message----- From: Florence Blanc-Renaud [mailto:flo(a)redhat.com] = Sent: torstai 16. elokuuta 2018 21.54 To: FreeIPA users list ; Rob Critte= nden Cc: Jokinen Eemeli Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upg= rade doesn't complete, pki-tomcatd won't start On 08/15/2018 01:20 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hi! > = > Anybody can help me with this one? > = > Summary: > = > 2 node freeipa server cluster, node 2 was initially down for other reason= s and node 1 (renewal master) had forgot to update own certificates which = resulted faulty cluster. With help from mailing list we got the node 1 back= online and it's working great! Now I'm trying to get node2 back to working= order in cluster but it won't update the certificates even when trying the= timejump. Seems like it tries to renew certificates locally although someh= ow I tought that it should renew the certificates from node 1...? > = Hi, you probably have a combination of multiple issues on your second node. The ipa-server-upgrade failure may leave your instance in a wrong state, wh= ere dse.ldif has disabled the ports for 389-ds (see BZ https://bugzilla.redhat.com/show_bug.cgi?id=3D1569011 or pagure ticket http= s://pagure.io/freeipa/issue/7534). During the upgrade, dse.ldif is edited in order to temporarily disable the = LDAP ports (to prevent ldap modifications during the upgrade). = Sometimes, if the upgrade fails, dse.ldif is not restored and the ports rem= ain disabled. You will have to stop the ldap server, manually edit dse.ldif= (in /etc/dirsrv/slapd-DOMxxx) and set: nsslapd-port: 389 nsslapd-security: on then restart the LDAP server. For the cert renewal, your procedure is the valid one. The kerberos error i= s probably linked to 389-ds not being accessible. HTH, flo > = > Eemeli > = > -----Original Message----- > From: Jokinen Eemeli > Sent: keskiviikko 4. hein=C3=A4kuuta 2018 16.08 > To: 'Rob Crittenden' ; FreeIPA users list = > ; Florence Blanc-Renaud = > > Subject: RE: [Freeipa-users] Re: Problems after IPA upgrade: = > ipa-server-upgrade doesn't complete, pki-tomcatd won't start > = > Hi! > = > I reply to this since there's some data in this message queue already rel= ated to my problem: > = > I had 2 ipa node cluster, where the second node had been offline for some= time and at some point we had an error while trying to reboot node1 which = was a Renewal Master. The issue was that some certs had expired and after a= bit of special work we got the node1 back on track. I can spot three probl= ems and I can't (again) figure out which one is the cause and which one I s= hould repair first. > = > Now I got assigned the case to get the node2 back on track also. It had s= ome certificates expired (obviously) so I did a small time jump and some of= the certs were renewed. However not all of them were upgraded. "getcert li= st" reports 3 certs "CA Unreachable", other 3 certs seem fine. > = > -- > getcert list |grep -A 10 "CA_UNREACH" > status: CA_UNREACHABLE > ca-error: Internal error > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB'= ,pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DOCSP Subsystem,O=3D<> > expires: 2018-03-21 09:42:04 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > eku: id-kp-OCSPSigning > -- > status: CA_UNREACHABLE > ca-error: Internal error > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nic= kname=3D'ipaCert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias/= pwdfile.txt' > certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname= =3D'ipaCert',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3DIPA RA,O=3D<> > expires: 2018-03-21 09:42:29 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth > -- > status: CA_UNREACHABLE > ca-error: Error 7 connecting to http://<>:8080/ca/ee/= ca/profileSubmit: Couldn't connect to server. > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/a= lias',nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin= set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias'= ,nickname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-renew-agent > issuer: CN=3DCertificate Authority,O=3D<> > subject: CN=3D<>,O=3D<> > expires: 2018-06-27 07:01:38 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataE= ncipherment > eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection > -- > = > Seems like "Server-Cert cert-pki-ca" is trying to renew on itself (node2)= but shouldn't node1 be the renewal master? Restarting httpd, certmonger an= d pki-tomcat don't seem to help, time traveling helped on other certs but n= ot on these. > = > Directory service seems to work if I start it manually but ipa-server-upg= rade fails on directory server not starting with "No ports specified" so so= mething wrong with it or is it the certificates? > -- > ipa-server-upgrade > Upgrading IPA:. Estimated time: 1 minute 30 seconds > [1/10]: stopping directory server > [2/10]: saving configuration > [3/10]: disabling listeners > [4/10]: enabling DS global lock > [5/10]: starting directory server > = > -- > <> ns-slapd[24503]: [04/Jul/2018:13:43:48.829927675 +0300] - E= MERG - main - Fatal Error---No ports specified. Exiting now. > -- > = > Also certmonger has issues: > -- > dogtag-ipa-ca-renew-agent-submit[1892]: Traceback (most recent call last): > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-su= bmit", line 541, in > = sys.exit(main()) > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-su= bmit", line 515, in main > = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filena= me) > = File "/usr/lib/python2.7/site-packages/ipalib/install/kini= t.py", line 43, in kinit_keytab > = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, = usage=3D'initiate') > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",= line 64, in __new__ > = store=3Dstore) > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py",= line 148, in acquire > = usage) > = File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cre= d_store.acquire_cred_from (gssapi/raw/ext_cred_store.c:1732) > = GSSError: Major (851968): Unspecified GSS failure. Minor co= de may provide more information, Minor (2529639068): Cannot contact any KDC= for realm '<>' > -- > = > but KDCs should be able to be resolved even from ipa node2 > -- > nslookup -type=3Dsrv _kerberos._tcp.<> > Server: <> > Address: <