I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
[root@seattlenfs slapd-BPT-ROCKS]# systemctl status pki-tomcatd@pki-tomcat.service ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-07-28 11:03:34 PDT; 36min ago Process: 14289 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 14406 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─14406 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/...
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: Jul 28, 2017 11:39:50 AM org.apache.catalina.core.ContainerBase backgroundProcess Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@67cf2df background process Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at java.lang.Thread.run(Thread.java:748)
However, the /var/log/ipaupgrade.log is full of trouble. It ends with:
2017-07-28T17:01:19Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2017-07-28T17:01:19Z DEBUG Waiting for CA to start... 2017-07-28T17:01:20Z DEBUG request POST http://seattlenfs.bpt.rocks:8080/ca/admin/ca/getStatus 2017-07-28T17:01:20Z DEBUG request body '' 2017-07-28T17:01:20Z DEBUG response status 500 2017-07-28T17:01:20Z DEBUG response headers {'content-length': '2208', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Fri, 28 Jul 2017 17:01:20 GMT', 'content-type': 'text/html;charset=utf-8'} 2017-07-28T17:01:20Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>' 2017-07-28T17:01:20Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2017-07-28T17:01:20Z DEBUG Waiting for CA to start... 2017-07-28T17:01:21Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-07-28T17:01:21Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run raise admintool.ScriptError(str(e))
2017-07-28T17:01:21Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2017-07-28T17:01:21Z ERROR CA did not start in 300.0s 2017-07-28T17:01:21Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Should I just blindly run ipa-server-upgrade again?
Googling had me look at certificate expirations, they seem to be good.
[root@seattlenfs slapd-BPT-ROCKS]# getcert list | grep expires expires: 2019-05-29 05:54:06 UTC expires: 2019-05-29 05:53:57 UTC expires: 2019-05-29 05:53:16 UTC expires: 2035-07-16 12:51:42 UTC expires: 2019-05-29 05:53:37 UTC expires: 2018-08-15 05:20:24 UTC expires: 2018-08-26 05:01:42 UTC expires: 2018-08-26 05:01:43 UTC
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep ipa- ipa-admintools.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-client.x86_64 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-client-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-python-compat.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server.x86_64 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server-dns.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep pki- pki-base.noarch 10.3.3-19.el7_3 @updates pki-base-java.noarch 10.3.3-19.el7_3 @updates pki-ca.noarch 10.3.3-19.el7_3 @updates pki-kra.noarch 10.3.3-19.el7_3 @updates pki-server.noarch 10.3.3-19.el7_3 @updates pki-tools.x86_64 10.3.3-19.el7_3 @updates
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep tomcat tomcat.noarch 7.0.69-12.el7_3 @updates tomcat-el-2.2-api.noarch 7.0.69-12.el7_3 @updates tomcat-jsp-2.2-api.noarch 7.0.69-12.el7_3 @updates tomcat-lib.noarch 7.0.69-12.el7_3 @updates tomcat-servlet-3.0-api.noarch 7.0.69-12.el7_3 @updates tomcatjss.noarch 7.1.2-3.el7 @base
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep java java-1.7.0-openjdk.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.7.0-openjdk-devel.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.7.0-openjdk-headless.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3 @updates java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3 @updates javamail.noarch 1.4.6-8.el7 @base javapackages-tools.noarch 3.4.1-11.el7 @base javassist.noarch 3.16.1-10.el7 @base nuxwdog-client-java.x86_64 1.0.3-5.el7 @base pki-base-java.noarch 10.3.3-19.el7_3 @updates python-javapackages.noarch 3.4.1-11.el7 @base tzdata-java.noarch 2017a-1.el7 @test-centos7-updates
Any other useful information I can provide?
On Sun, Jul 30, 2017 at 6:53 PM, Ian Harding via FreeIPA-users freeipa-users@lists.fedorahosted.org wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
[root@seattlenfs slapd-BPT-ROCKS]# systemctl status pki-tomcatd@pki-tomcat.service ● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2017-07-28 11:03:34 PDT; 36min ago Process: 14289 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS) Main PID: 14406 (java) CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service └─14406 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/...
Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: Jul 28, 2017 11:39:50 AM org.apache.catalina.core.ContainerBase backgroundProcess Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@67cf2df background process Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521) Jul 28 11:39:50 seattlenfs.bpt.rocks server[14406]: at java.lang.Thread.run(Thread.java:748)
However, the /var/log/ipaupgrade.log is full of trouble. It ends with:
2017-07-28T17:01:19Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2017-07-28T17:01:19Z DEBUG Waiting for CA to start... 2017-07-28T17:01:20Z DEBUG request POST http://seattlenfs.bpt.rocks:8080/ca/admin/ca/getStatus 2017-07-28T17:01:20Z DEBUG request body '' 2017-07-28T17:01:20Z DEBUG response status 500 2017-07-28T17:01:20Z DEBUG response headers {'content-length': '2208', 'content-language': 'en', 'server': 'Apache-Coyote/1.1', 'connection': 'close', 'date': 'Fri, 28 Jul 2017 17:01:20 GMT', 'content-type': 'text/html;charset=utf-8'} 2017-07-28T17:01:20Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.69 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style>
</head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:745)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.69 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.69</h3></body></html>' 2017-07-28T17:01:20Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500 2017-07-28T17:01:20Z DEBUG Waiting for CA to start... 2017-07-28T17:01:21Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-07-28T17:01:21Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 48, in run raise admintool.ScriptError(str(e))
2017-07-28T17:01:21Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s 2017-07-28T17:01:21Z ERROR CA did not start in 300.0s 2017-07-28T17:01:21Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Should I just blindly run ipa-server-upgrade again?
Googling had me look at certificate expirations, they seem to be good.
[root@seattlenfs slapd-BPT-ROCKS]# getcert list | grep expires expires: 2019-05-29 05:54:06 UTC expires: 2019-05-29 05:53:57 UTC expires: 2019-05-29 05:53:16 UTC expires: 2035-07-16 12:51:42 UTC expires: 2019-05-29 05:53:37 UTC expires: 2018-08-15 05:20:24 UTC expires: 2018-08-26 05:01:42 UTC expires: 2018-08-26 05:01:43 UTC
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep ipa- ipa-admintools.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-client.x86_64 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-client-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-python-compat.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server.x86_64 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server-common.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates ipa-server-dns.noarch 4.4.0-14.el7.centos.7 @test-centos7-updates
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep pki- pki-base.noarch 10.3.3-19.el7_3 @updates pki-base-java.noarch 10.3.3-19.el7_3 @updates pki-ca.noarch 10.3.3-19.el7_3 @updates pki-kra.noarch 10.3.3-19.el7_3 @updates pki-server.noarch 10.3.3-19.el7_3 @updates pki-tools.x86_64 10.3.3-19.el7_3 @updates
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep tomcat tomcat.noarch 7.0.69-12.el7_3 @updates tomcat-el-2.2-api.noarch 7.0.69-12.el7_3 @updates tomcat-jsp-2.2-api.noarch 7.0.69-12.el7_3 @updates tomcat-lib.noarch 7.0.69-12.el7_3 @updates tomcat-servlet-3.0-api.noarch 7.0.69-12.el7_3 @updates tomcatjss.noarch 7.1.2-3.el7 @base
[root@seattlenfs slapd-BPT-ROCKS]# yum list | grep java java-1.7.0-openjdk.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.7.0-openjdk-devel.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.7.0-openjdk-headless.x86_64 1:1.7.0.141-2.6.10.1.el7_3 @test-centos7-updates java-1.8.0-openjdk.x86_64 1:1.8.0.141-1.b16.el7_3 @updates java-1.8.0-openjdk-headless.x86_64 1:1.8.0.141-1.b16.el7_3 @updates javamail.noarch 1.4.6-8.el7 @base javapackages-tools.noarch 3.4.1-11.el7 @base javassist.noarch 3.16.1-10.el7 @base nuxwdog-client-java.x86_64 1.0.3-5.el7 @base pki-base-java.noarch 10.3.3-19.el7_3 @updates python-javapackages.noarch 3.4.1-11.el7 @base tzdata-java.noarch 2017a-1.el7 @test-centos7-updates
Any other useful information I can provide?
There is a good page for reporting issues:
https://www.freeipa.org/page/Files_to_be_attached_to_bug_report#Dogtag_CA_fa...
In your case, Dogtag CA did to start so the relevant section is:
ausearch -m AVC > avc.log journalctl -u pki-tomcatd@pki-tomcat.service /var/log/pki/pki-tomcat/ca/debug <---- usually most important
additionally: /var/log/pki/pki-tomcat/ca/selftests.log
only in install: /var/log/pki/pki-ca-spawn.<latest>.log
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
I don't think re-running the upgrade command would help.
rob
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [28/Jul/2017:09:56:24][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: done init id=debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initialized debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: ready to init id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory: init [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init() [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init begins [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init ends [28/Jul/2017:10:08:47][localhost-startStop-1]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:08:47][localhost-startStop-1]: makeConnection: errorIfDown true [28/Jul/2017:10:08:47][localhost-startStop-1]: TCP Keep-Alive: true [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory: init [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init() [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init begins [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init ends [28/Jul/2017:10:13:29][localhost-startStop-2]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:13:29][localhost-startStop-2]: makeConnection: errorIfDown true [28/Jul/2017:10:13:29][localhost-startStop-2]: TCP Keep-Alive: true [28/Jul/2017:10:13:29][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
I don't think re-running the upgrade command would help.
rob
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
HTH, Flo.
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:09:56:24][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: done init id=debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initialized debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: ready to init id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory: init [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init() [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init begins [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init ends [28/Jul/2017:10:08:47][localhost-startStop-1]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:08:47][localhost-startStop-1]: makeConnection: errorIfDown true [28/Jul/2017:10:08:47][localhost-startStop-1]: TCP Keep-Alive: true [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory: init [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init() [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init begins [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init ends [28/Jul/2017:10:13:29][localhost-startStop-2]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:13:29][localhost-startStop-2]: makeConnection: errorIfDown true [28/Jul/2017:10:13:29][localhost-startStop-2]: TCP Keep-Alive: true [28/Jul/2017:10:13:29][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
I don't think re-running the upgrade command would help.
rob
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
ls -a /etc/pki/pki-tomcat/alias/ . .. cert8.db key3.db secmod.db
There are "internal" and "replicationdb" values in /var/lib/pki/pki-tomcat/conf/password.conf but they don't work in response to the ldapsearch prompt above.
Thank you so much for your help!
HTH, Flo.
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:09:56:24][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: done init id=debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initialized debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: ready to init id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory: init [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init() [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init begins [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init ends [28/Jul/2017:10:08:47][localhost-startStop-1]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:08:47][localhost-startStop-1]: makeConnection: errorIfDown true [28/Jul/2017:10:08:47][localhost-startStop-1]: TCP Keep-Alive: true [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory: init [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init() [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init begins [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init ends [28/Jul/2017:10:13:29][localhost-startStop-2]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:13:29][localhost-startStop-2]: makeConnection: errorIfDown true [28/Jul/2017:10:13:29][localhost-startStop-2]: TCP Keep-Alive: true [28/Jul/2017:10:13:29][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
I don't think re-running the upgrade command would help.
rob
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
If the password is not the right one, certutil will prompt you once again for it. This would mean that the password does not allow to access the key from the NSSdb and Dogtag will not be able to use the certificate for authentication.
Flo
ls -a /etc/pki/pki-tomcat/alias/ . .. cert8.db key3.db secmod.db
There are "internal" and "replicationdb" values in /var/lib/pki/pki-tomcat/conf/password.conf but they don't work in response to the ldapsearch prompt above.
Thank you so much for your help!
HTH, Flo.
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:09:56:24][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: done init id=debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initialized debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: ready to init id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory: init [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init() [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init begins [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init ends [28/Jul/2017:10:08:47][localhost-startStop-1]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:08:47][localhost-startStop-1]: makeConnection: errorIfDown true [28/Jul/2017:10:08:47][localhost-startStop-1]: TCP Keep-Alive: true [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory: init [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init() [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init begins [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init ends [28/Jul/2017:10:13:29][localhost-startStop-2]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:13:29][localhost-startStop-2]: makeConnection: errorIfDown true [28/Jul/2017:10:13:29][localhost-startStop-2]: TCP Keep-Alive: true [28/Jul/2017:10:13:29][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
I don't think re-running the upgrade command would help.
rob
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had updates run but had not been restarted. ipactl says pki-tomcatd would not start.
Strangely, the actual service appears to be running:
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
If the password is not the right one, certutil will prompt you once again for it. This would mean that the password does not allow to access the key from the NSSdb and Dogtag will not be able to use the certificate for authentication.
Flo
ls -a /etc/pki/pki-tomcat/alias/ . .. cert8.db key3.db secmod.db
There are "internal" and "replicationdb" values in /var/lib/pki/pki-tomcat/conf/password.conf but they don't work in response to the ldapsearch prompt above.
Thank you so much for your help!
HTH, Flo.
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
atorg.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:09:56:24][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:08:46][localhost-startStop-1]: ============================================ [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: done init id=debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initialized debug [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: CMSEngine: ready to init id=log [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:08:46][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized log [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: done init id=jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initialized jss [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:08:47][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:08:47][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory: init [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init() [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init begins [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapAuthInfo: init ends [28/Jul/2017:10:08:47][localhost-startStop-1]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:08:47][localhost-startStop-1]: makeConnection: errorIfDown true [28/Jul/2017:10:08:47][localhost-startStop-1]: TCP Keep-Alive: true [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
atorg.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)Internal Database Error encountered: Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
atorg.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method) atorg.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
atorg.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)[28/Jul/2017:10:08:47][localhost-startStop-1]: CMSEngine.shutdown() [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [28/Jul/2017:10:13:29][localhost-startStop-2]: ============================================ [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized debug [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized log [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: restart at autoShutdown? false [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: found cert:auditSigningCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: done init id=jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initialized jss [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: initSubsystem id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: CMSEngine: ready to init id=dbs [28/Jul/2017:10:13:29][localhost-startStop-2]: DBSubsystem: init() mEnableSerialMgmt=true [28/Jul/2017:10:13:29][localhost-startStop-2]: Creating LdapBoundConnFactor(DBSubsystem) [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory: init [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapBoundConnFactory:doCloning true [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init() [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init begins [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapAuthInfo: init ends [28/Jul/2017:10:13:29][localhost-startStop-2]: init: before makeConnection errorIfDown is true [28/Jul/2017:10:13:29][localhost-startStop-2]: makeConnection: errorIfDown true [28/Jul/2017:10:13:29][localhost-startStop-2]: TCP Keep-Alive: true [28/Jul/2017:10:13:29][localhost-startStop-2]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [28/Jul/2017:10:13:29][localhost-startStop-2]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
atcom.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654) atcom.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1172) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1078) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:570) at com.netscape.certsrv.apps.CMS.init(CMS.java:188) at com.netscape.certsrv.apps.CMS.start(CMS.java:1621) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:114)
at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) atorg.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
atorg.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
atorg.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1270)
atorg.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1195)
atorg.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1085)
atorg.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5318)
atorg.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5610)
atorg.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
atorg.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
atorg.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
I don't think re-running the upgrade command would help.
rob
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote: > I had an unexpected restart of an IPA server that had apparently had > updates run but had not been restarted. ipactl says pki-tomcatd > would > not start. > > Strangely, the actual service appears to be running: >
dogtag is an application within tomcat so tomcat can run without dogtag running.
We need to see more of the dogtag debug log to see what is going on.
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
Are you running in SELinux enforcing mode?
If so can you run:
# ausearch -m AVC
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
On 07/31/2017 11:34 AM, Rob Crittenden wrote: > Ian Harding via FreeIPA-users wrote: >> I had an unexpected restart of an IPA server that had apparently had >> updates run but had not been restarted. ipactl says pki-tomcatd >> would >> not start. >> >> Strangely, the actual service appears to be running: >> > > dogtag is an application within tomcat so tomcat can run without > dogtag > running. > > We need to see more of the dogtag debug log to see what is going on. >
It looks like an authentication problem...
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting. ---- time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: > > > On 07/31/2017 11:34 AM, Rob Crittenden wrote: >> Ian Harding via FreeIPA-users wrote: >>> I had an unexpected restart of an IPA server that had >>> apparently had >>> updates run but had not been restarted. ipactl says pki-tomcatd >>> would >>> not start. >>> >>> Strangely, the actual service appears to be running: >>> >> >> dogtag is an application within tomcat so tomcat can run without >> dogtag >> running. >> >> We need to see more of the dogtag debug log to see what is going >> on. >> > > It looks like an authentication problem... > > [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake > happened > Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 > Error netscape.ldap.LDAPException: Authentication failed (49) >
Hi,
dogtag stores its internal data in the LDAP server and needs to establish a secure LDAP connection. You can check how this connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for the lines:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=cn=Directory Manager internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.host=vm-... internaldb.ldapconn.port=636 internaldb.ldapconn.secureConn
authtype can be SslClientAuth (authentication with a ssl certificate) or BasicAuth (authentication with a bind DN and password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
You can use this information to manually check the credentials. For instance with sslclientauth:
export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias export LDAPTLS_CERT='subsystemCert cert-pki-ca'
ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
Flo.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: > On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >> >> >> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>> Ian Harding via FreeIPA-users wrote: >>>> I had an unexpected restart of an IPA server that had >>>> apparently had >>>> updates run but had not been restarted. ipactl says pki-tomcatd >>>> would >>>> not start. >>>> >>>> Strangely, the actual service appears to be running: >>>> >>> >>> dogtag is an application within tomcat so tomcat can run without >>> dogtag >>> running. >>> >>> We need to see more of the dogtag debug log to see what is >>> going on. >>> >> >> It looks like an authentication problem... >> >> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake >> happened >> Could not connect to LDAP server host seattlenfs.bpt.rocks port 636 >> Error netscape.ldap.LDAPException: Authentication failed (49) >> > > Hi, > > dogtag stores its internal data in the LDAP server and needs to > establish a secure LDAP connection. You can check how this > connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for > the lines: > > internaldb.ldapauth.authtype=SslClientAuth > internaldb.ldapauth.bindDN=cn=Directory Manager > internaldb.ldapauth.bindPWPrompt=internaldb > internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca > internaldb.ldapconn.host=vm-... > internaldb.ldapconn.port=636 > internaldb.ldapconn.secureConn > > authtype can be SslClientAuth (authentication with a ssl > certificate) or BasicAuth (authentication with a bind DN and > password stored in /var/lib/pki/pki-tomcat/conf/password.conf). > > You can use this information to manually check the credentials. For > instance with sslclientauth: > > export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias > export LDAPTLS_CERT='subsystemCert cert-pki-ca' > > ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL > (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt) >
I found this:
internaldb.ldapauth.authtype=SslClientAuth internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca internaldb.ldapauth.bindPWPrompt=internaldb internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca internaldb.ldapconn.cloneReplicationPort=389 ...
and when I try the ldapsearch I am presented with a prompt to provide a pin/password
Please enter pin, password, or pass phrase for security token 'ldap(0)':
but there is no password file...
Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Thank you again for your time!
Ian
Flo.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote: > On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>> >>> >>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>> Ian Harding via FreeIPA-users wrote: >>>>> I had an unexpected restart of an IPA server that had >>>>> apparently had >>>>> updates run but had not been restarted. ipactl says pki-tomcatd >>>>> would >>>>> not start. >>>>> >>>>> Strangely, the actual service appears to be running: >>>>> >>>> >>>> dogtag is an application within tomcat so tomcat can run without >>>> dogtag >>>> running. >>>> >>>> We need to see more of the dogtag debug log to see what is >>>> going on. >>>> >>> >>> It looks like an authentication problem... >>> >>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake >>> happened >>> Could not connect to LDAP server host seattlenfs.bpt.rocks port >>> 636 >>> Error netscape.ldap.LDAPException: Authentication failed (49) >>> >> >> Hi, >> >> dogtag stores its internal data in the LDAP server and needs to >> establish a secure LDAP connection. You can check how this >> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for >> the lines: >> >> internaldb.ldapauth.authtype=SslClientAuth >> internaldb.ldapauth.bindDN=cn=Directory Manager >> internaldb.ldapauth.bindPWPrompt=internaldb >> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >> internaldb.ldapconn.host=vm-... >> internaldb.ldapconn.port=636 >> internaldb.ldapconn.secureConn >> >> authtype can be SslClientAuth (authentication with a ssl >> certificate) or BasicAuth (authentication with a bind DN and >> password stored in /var/lib/pki/pki-tomcat/conf/password.conf). >> >> You can use this information to manually check the credentials. For >> instance with sslclientauth: >> >> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >> >> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >> (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt) >> > > I found this: > > internaldb.ldapauth.authtype=SslClientAuth > internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca > internaldb.ldapauth.bindPWPrompt=internaldb > internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca > internaldb.ldapconn.cloneReplicationPort=389 > ... > > and when I try the ldapsearch I am presented with a prompt to > provide > a pin/password > > Please enter pin, password, or pass phrase for security token > 'ldap(0)': > > but there is no password file... > Hi,
you are right, in 4.4. there is no pwdfile.txt and the password can be found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag internal=...)
Can you check if the password with the tag internal=... allows to read the keys from the NSS db? certutil -K -d /etc/pki/pki-tomcat/alias (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts: $ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
Flo.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: > On 08/01/2017 03:11 PM, Ian Harding wrote: >> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>> >>>> >>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>> Ian Harding via FreeIPA-users wrote: >>>>>> I had an unexpected restart of an IPA server that had >>>>>> apparently had >>>>>> updates run but had not been restarted. ipactl says >>>>>> pki-tomcatd >>>>>> would >>>>>> not start. >>>>>> >>>>>> Strangely, the actual service appears to be running: >>>>>> >>>>> >>>>> dogtag is an application within tomcat so tomcat can run without >>>>> dogtag >>>>> running. >>>>> >>>>> We need to see more of the dogtag debug log to see what is >>>>> going on. >>>>> >>>> >>>> It looks like an authentication problem... >>>> >>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake >>>> happened >>>> Could not connect to LDAP server host seattlenfs.bpt.rocks >>>> port 636 >>>> Error netscape.ldap.LDAPException: Authentication failed (49) >>>> >>> >>> Hi, >>> >>> dogtag stores its internal data in the LDAP server and needs to >>> establish a secure LDAP connection. You can check how this >>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look >>> for >>> the lines: >>> >>> internaldb.ldapauth.authtype=SslClientAuth >>> internaldb.ldapauth.bindDN=cn=Directory Manager >>> internaldb.ldapauth.bindPWPrompt=internaldb >>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >>> internaldb.ldapconn.host=vm-... >>> internaldb.ldapconn.port=636 >>> internaldb.ldapconn.secureConn >>> >>> authtype can be SslClientAuth (authentication with a ssl >>> certificate) or BasicAuth (authentication with a bind DN and >>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf). >>> >>> You can use this information to manually check the credentials. >>> For >>> instance with sslclientauth: >>> >>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>> >>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>> (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt) >>> >> >> I found this: >> >> internaldb.ldapauth.authtype=SslClientAuth >> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >> internaldb.ldapauth.bindPWPrompt=internaldb >> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >> internaldb.ldapconn.cloneReplicationPort=389 >> ... >> >> and when I try the ldapsearch I am presented with a prompt to >> provide >> a pin/password >> >> Please enter pin, password, or pass phrase for security token >> 'ldap(0)': >> >> but there is no password file... >> > Hi, > > you are right, in 4.4. there is no pwdfile.txt and the password > can be > found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag > internal=...) > > Can you check if the password with the tag internal=... allows to > read > the keys from the NSS db? > certutil -K -d /etc/pki/pki-tomcat/alias > (provide password)
That works...
# certutil -K -d /etc/pki/pki-tomcat/alias certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS Certificate DB:auditSigningCert cert-pki-ca < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert cert-pki-ca < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS Certificate DB:ocspSigningCert cert-pki-ca < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert cert-pki-ca < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS Certificate DB:subsystemCert cert-pki-ca
But this doesn't (with the same password from password.conf)
# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL Please enter pin, password, or pass phrase for security token 'ldap(0)': SASL/EXTERNAL authentication started ldap_sasl_interactive_bind_s: Invalid credentials (49)
That password is getting me somewhere though, since if I put in a nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
Flo.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote: > On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >> On 08/01/2017 03:11 PM, Ian Harding wrote: >>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>>> >>>>> >>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>> I had an unexpected restart of an IPA server that had >>>>>>> apparently had >>>>>>> updates run but had not been restarted. ipactl says >>>>>>> pki-tomcatd >>>>>>> would >>>>>>> not start. >>>>>>> >>>>>>> Strangely, the actual service appears to be running: >>>>>>> >>>>>> >>>>>> dogtag is an application within tomcat so tomcat can run >>>>>> without >>>>>> dogtag >>>>>> running. >>>>>> >>>>>> We need to see more of the dogtag debug log to see what is >>>>>> going on. >>>>>> >>>>> >>>>> It looks like an authentication problem... >>>>> >>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake >>>>> happened >>>>> Could not connect to LDAP server host seattlenfs.bpt.rocks >>>>> port 636 >>>>> Error netscape.ldap.LDAPException: Authentication failed (49) >>>>> >>>> >>>> Hi, >>>> >>>> dogtag stores its internal data in the LDAP server and needs to >>>> establish a secure LDAP connection. You can check how this >>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, >>>> look for >>>> the lines: >>>> >>>> internaldb.ldapauth.authtype=SslClientAuth >>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >>>> internaldb.ldapconn.host=vm-... >>>> internaldb.ldapconn.port=636 >>>> internaldb.ldapconn.secureConn >>>> >>>> authtype can be SslClientAuth (authentication with a ssl >>>> certificate) or BasicAuth (authentication with a bind DN and >>>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf). >>>> >>>> You can use this information to manually check the >>>> credentials. For >>>> instance with sslclientauth: >>>> >>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>> >>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>>> (provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>> >>> >>> I found this: >>> >>> internaldb.ldapauth.authtype=SslClientAuth >>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>> internaldb.ldapauth.bindPWPrompt=internaldb >>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >>> internaldb.ldapconn.cloneReplicationPort=389 >>> ... >>> >>> and when I try the ldapsearch I am presented with a prompt to >>> provide >>> a pin/password >>> >>> Please enter pin, password, or pass phrase for security token >>> 'ldap(0)': >>> >>> but there is no password file... >>> >> Hi, >> >> you are right, in 4.4. there is no pwdfile.txt and the password >> can be >> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag >> internal=...) >> >> Can you check if the password with the tag internal=... allows >> to read >> the keys from the NSS db? >> certutil -K -d /etc/pki/pki-tomcat/alias >> (provide password) > > That works... > > # certutil -K -d /etc/pki/pki-tomcat/alias > certutil: Checking token "NSS Certificate DB" in slot "NSS User > Private > Key and Certificate Services" > Enter Password or Pin for "NSS Certificate DB": > < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) > < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS > Certificate > DB:auditSigningCert cert-pki-ca > < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert > cert-pki-ca > < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS > Certificate > DB:ocspSigningCert cert-pki-ca > < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert > cert-pki-ca > < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS > Certificate > DB:subsystemCert cert-pki-ca > > But this doesn't (with the same password from password.conf) > > # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL > Please enter pin, password, or pass phrase for security token > 'ldap(0)': > SASL/EXTERNAL authentication started > ldap_sasl_interactive_bind_s: Invalid credentials (49) > > That password is getting me somewhere though, since if I put in a > nonsense or incorrect password it just prompts over and over.
Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
Are you running in SELinux enforcing mode?
# getenforce Enforcing
If so can you run:
# ausearch -m AVC
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
I suspect that the subsystem cert was renewed and everything wasn't updated properly. If I'm right this is unrelated to the upgrade it just shone a spotlight on it.
Can you run these commands:
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
and
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
- On server freeipa-sea (=renewal master), the CA component is installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
- On server seattlenfs (where PKI fails to restart), the CA component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
- In the LDAP servers, the cert is stored in the entry uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
Flo.
which seems weird.
The description attribute in the ldapsearch output is of the form 2;#;DN;DN
The # should match the serial number in the first certutil output
The ASCII blob (minus the BEGIN/END headers) should match one of the userCertificate attributes.
rob
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote: > Ian Harding wrote: >> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>>>> >>>>>> >>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>> I had an unexpected restart of an IPA server that had >>>>>>>> apparently had >>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>> pki-tomcatd >>>>>>>> would >>>>>>>> not start. >>>>>>>> >>>>>>>> Strangely, the actual service appears to be running: >>>>>>>> >>>>>>> >>>>>>> dogtag is an application within tomcat so tomcat can run >>>>>>> without >>>>>>> dogtag >>>>>>> running. >>>>>>> >>>>>>> We need to see more of the dogtag debug log to see what is >>>>>>> going on. >>>>>>> >>>>>> >>>>>> It looks like an authentication problem... >>>>>> >>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>> handshake happened >>>>>> Could not connect to LDAP server host seattlenfs.bpt.rocks >>>>>> port 636 >>>>>> Error netscape.ldap.LDAPException: Authentication failed (49) >>>>>> >>>>> >>>>> Hi, >>>>> >>>>> dogtag stores its internal data in the LDAP server and needs to >>>>> establish a secure LDAP connection. You can check how this >>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, >>>>> look for >>>>> the lines: >>>>> >>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>> cert-pki-ca >>>>> internaldb.ldapconn.host=vm-... >>>>> internaldb.ldapconn.port=636 >>>>> internaldb.ldapconn.secureConn >>>>> >>>>> authtype can be SslClientAuth (authentication with a ssl >>>>> certificate) or BasicAuth (authentication with a bind DN and >>>>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf). >>>>> >>>>> You can use this information to manually check the >>>>> credentials. For >>>>> instance with sslclientauth: >>>>> >>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>> >>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>>>> (provide the password from >>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>> >>>> >>>> I found this: >>>> >>>> internaldb.ldapauth.authtype=SslClientAuth >>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >>>> internaldb.ldapconn.cloneReplicationPort=389 >>>> ... >>>> >>>> and when I try the ldapsearch I am presented with a prompt to >>>> provide >>>> a pin/password >>>> >>>> Please enter pin, password, or pass phrase for security token >>>> 'ldap(0)': >>>> >>>> but there is no password file... >>>> >>> Hi, >>> >>> you are right, in 4.4. there is no pwdfile.txt and the >>> password can be >>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag >>> internal=...) >>> >>> Can you check if the password with the tag internal=... allows >>> to read >>> the keys from the NSS db? >>> certutil -K -d /etc/pki/pki-tomcat/alias >>> (provide password) >> >> That works... >> >> # certutil -K -d /etc/pki/pki-tomcat/alias >> certutil: Checking token "NSS Certificate DB" in slot "NSS User >> Private >> Key and Certificate Services" >> Enter Password or Pin for "NSS Certificate DB": >> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >> Certificate >> DB:auditSigningCert cert-pki-ca >> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert >> cert-pki-ca >> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >> Certificate >> DB:ocspSigningCert cert-pki-ca >> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >> cert-pki-ca >> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >> Certificate >> DB:subsystemCert cert-pki-ca >> >> But this doesn't (with the same password from password.conf) >> >> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >> Please enter pin, password, or pass phrase for security token >> 'ldap(0)': >> SASL/EXTERNAL authentication started >> ldap_sasl_interactive_bind_s: Invalid credentials (49) >> >> That password is getting me somewhere though, since if I put in a >> nonsense or incorrect password it just prompts over and over. > > Let's step back a second. You upgraded from what to what?
There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.
# grep ipa /var/log/yum.log Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
# grep pki /var/log/yum.log Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
> > Are you running in SELinux enforcing mode?
# getenforce Enforcing
> > If so can you run: > > # ausearch -m AVC >
# ausearch -m AVC The NOLOG option to log_format is deprecated. Please use the write_logs option. The NOLOG option is overriding the write_logs current setting.
time->Mon Mar 14 10:39:01 2016 type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
> I suspect that the subsystem cert was renewed and everything wasn't > updated properly. If I'm right this is unrelated to the upgrade > it just > shone a spotlight on it. > > Can you run these commands: > > $ ldapsearch -LLL -D 'cn=directory manager' -W -b > uid=pkidbuser,ou=people,o=ipaca userCertificate description >
This returns
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... stuff ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
> and > > # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert > cert-pki-ca' | grep Serial > # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert > cert-pki-ca' -a >
These both return
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services" Enter Password or Pin for "NSS Certificate DB": certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA component
is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC <truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC <truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
Flo.
which seems weird.
> The description attribute in the ldapsearch output is of the > form 2;#;DN;DN > > The # should match the serial number in the first certutil output > > The ASCII blob (minus the BEGIN/END headers) should match one of > the > userCertificate attributes. > > rob >
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote: > On 08/01/2017 12:03 PM, Rob Crittenden wrote: >> Ian Harding wrote: >>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>>>>> >>>>>>> >>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>> I had an unexpected restart of an IPA server that had >>>>>>>>> apparently had >>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>> pki-tomcatd >>>>>>>>> would >>>>>>>>> not start. >>>>>>>>> >>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>> >>>>>>>> >>>>>>>> dogtag is an application within tomcat so tomcat can run >>>>>>>> without >>>>>>>> dogtag >>>>>>>> running. >>>>>>>> >>>>>>>> We need to see more of the dogtag debug log to see what is >>>>>>>> going on. >>>>>>>> >>>>>>> >>>>>>> It looks like an authentication problem... >>>>>>> >>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>> handshake happened >>>>>>> Could not connect to LDAP server host seattlenfs.bpt.rocks >>>>>>> port 636 >>>>>>> Error netscape.ldap.LDAPException: Authentication failed (49) >>>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> dogtag stores its internal data in the LDAP server and needs to >>>>>> establish a secure LDAP connection. You can check how this >>>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, >>>>>> look for >>>>>> the lines: >>>>>> >>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>> cert-pki-ca >>>>>> internaldb.ldapconn.host=vm-... >>>>>> internaldb.ldapconn.port=636 >>>>>> internaldb.ldapconn.secureConn >>>>>> >>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>> certificate) or BasicAuth (authentication with a bind DN and >>>>>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>> >>>>>> You can use this information to manually check the >>>>>> credentials. For >>>>>> instance with sslclientauth: >>>>>> >>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>> >>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>>>>> (provide the password from >>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>> >>>>> >>>>> I found this: >>>>> >>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca >>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>> ... >>>>> >>>>> and when I try the ldapsearch I am presented with a prompt to >>>>> provide >>>>> a pin/password >>>>> >>>>> Please enter pin, password, or pass phrase for security token >>>>> 'ldap(0)': >>>>> >>>>> but there is no password file... >>>>> >>>> Hi, >>>> >>>> you are right, in 4.4. there is no pwdfile.txt and the >>>> password can be >>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag >>>> internal=...) >>>> >>>> Can you check if the password with the tag internal=... allows >>>> to read >>>> the keys from the NSS db? >>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>> (provide password) >>> >>> That works... >>> >>> # certutil -K -d /etc/pki/pki-tomcat/alias >>> certutil: Checking token "NSS Certificate DB" in slot "NSS User >>> Private >>> Key and Certificate Services" >>> Enter Password or Pin for "NSS Certificate DB": >>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>> Certificate >>> DB:auditSigningCert cert-pki-ca >>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert >>> cert-pki-ca >>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>> Certificate >>> DB:ocspSigningCert cert-pki-ca >>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >>> cert-pki-ca >>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>> Certificate >>> DB:subsystemCert cert-pki-ca >>> >>> But this doesn't (with the same password from password.conf) >>> >>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>> Please enter pin, password, or pass phrase for security token >>> 'ldap(0)': >>> SASL/EXTERNAL authentication started >>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>> >>> That password is getting me somewhere though, since if I put in a >>> nonsense or incorrect password it just prompts over and over. >> >> Let's step back a second. You upgraded from what to what? > > There wasn't much of a change... I just assumed someone ran yum > upgrade and didn't restart, then the power outage... it looks > like not much of a version change though. > > # grep ipa /var/log/yum.log > Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:45:32 Installed: > ipa-client-common-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 > Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 > Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch > Jan 08 04:47:04 Installed: > python2-ipalib-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:05 Installed: > python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:05 Updated: > ipa-admintools-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:05 Installed: > ipa-server-common-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:06 Installed: > python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 > Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 > Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 > Jan 08 04:47:27 Updated: > ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:47:36 Installed: > ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch > Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 > Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:42 Updated: > ipa-client-common-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:43 Updated: > ipa-server-common-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:44 Updated: > python2-ipaclient-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:45 Updated: > python2-ipaserver-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 > Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 > Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch > Jul 28 09:55:48 Updated: > ipa-python-compat-4.4.0-14.el7.centos.7.noarch > > # grep pki /var/log/yum.log > Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch > Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 > Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch > Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 > Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch > Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch > Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch > Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch > Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch > Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 > Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch > Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch > Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch > > >> >> Are you running in SELinux enforcing mode? > > # getenforce > Enforcing > >> >> If so can you run: >> >> # ausearch -m AVC >> > > # ausearch -m AVC > The NOLOG option to log_format is deprecated. Please use the > write_logs option. > The NOLOG option is overriding the write_logs current setting. > ---- > time->Mon Mar 14 10:39:01 2016 > type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e > syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 > a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 > gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 > fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" > exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 > key=(null) > type=AVC msg=audit(1457977141.767:17537): avc: denied { read } > for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" > ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 > tcontext=system_u:object_r:unlabeled_t:s0 tclass=file > > >> I suspect that the subsystem cert was renewed and everything wasn't >> updated properly. If I'm right this is unrelated to the upgrade >> it just >> shone a spotlight on it. >> >> Can you run these commands: >> >> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >> uid=pkidbuser,ou=people,o=ipaca userCertificate description >> > > This returns > > $ ldapsearch -LLL -D 'cn=directory manager' -W -b > uid=pkidbuser,ou=people,o=ipaca userCertificate description > Enter LDAP Password: > dn: uid=pkidbuser,ou=people,o=ipaca > userCertificate:: > MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > ... stuff ... > ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= > description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA > Subsystem,O=BPT.ROCKS > >> and >> >> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' | grep Serial >> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' -a >> > > These both return > > # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert > cert-pki-ca' | grep Serial > Enter Password or Pin for "NSS Certificate DB": > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized Object Identifier. > > > # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert > cert-pki-ca' -a > certutil: Checking token "NSS Certificate DB" in slot "NSS User > Private Key and Certificate Services" > Enter Password or Pin for "NSS Certificate DB": > certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: > Unrecognized Object Identifier. > Hi,
you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA component
is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
Flo. > > which seems weird. > >> The description attribute in the ldapsearch output is of the >> form 2;#;DN;DN >> >> The # should match the serial number in the first certutil output >> >> The ASCII blob (minus the BEGIN/END headers) should match one of >> the >> userCertificate attributes. >> >> rob >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: > On 08/02/2017 01:43 AM, Ian Harding wrote: >> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>> Ian Harding wrote: >>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>> I had an unexpected restart of an IPA server that had >>>>>>>>>> apparently had >>>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>>> pki-tomcatd >>>>>>>>>> would >>>>>>>>>> not start. >>>>>>>>>> >>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>> >>>>>>>>> >>>>>>>>> dogtag is an application within tomcat so tomcat can run >>>>>>>>> without >>>>>>>>> dogtag >>>>>>>>> running. >>>>>>>>> >>>>>>>>> We need to see more of the dogtag debug log to see what >>>>>>>>> is going on. >>>>>>>>> >>>>>>>> >>>>>>>> It looks like an authentication problem... >>>>>>>> >>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>> handshake happened >>>>>>>> Could not connect to LDAP server host >>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>> Error netscape.ldap.LDAPException: Authentication failed >>>>>>>> (49) >>>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> dogtag stores its internal data in the LDAP server and >>>>>>> needs to >>>>>>> establish a secure LDAP connection. You can check how this >>>>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, >>>>>>> look for >>>>>>> the lines: >>>>>>> >>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>> cert-pki-ca >>>>>>> internaldb.ldapconn.host=vm-... >>>>>>> internaldb.ldapconn.port=636 >>>>>>> internaldb.ldapconn.secureConn >>>>>>> >>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>> certificate) or BasicAuth (authentication with a bind DN and >>>>>>> password stored in >>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>> >>>>>>> You can use this information to manually check the >>>>>>> credentials. For >>>>>>> instance with sslclientauth: >>>>>>> >>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>> >>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>> EXTERNAL >>>>>>> (provide the password from >>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>> >>>>>> >>>>>> I found this: >>>>>> >>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>> cert-pki-ca >>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>> ... >>>>>> >>>>>> and when I try the ldapsearch I am presented with a prompt >>>>>> to provide >>>>>> a pin/password >>>>>> >>>>>> Please enter pin, password, or pass phrase for security >>>>>> token 'ldap(0)': >>>>>> >>>>>> but there is no password file... >>>>>> >>>>> Hi, >>>>> >>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>> password can be >>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with >>>>> the tag >>>>> internal=...) >>>>> >>>>> Can you check if the password with the tag internal=... >>>>> allows to read >>>>> the keys from the NSS db? >>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>> (provide password) >>>> >>>> That works... >>>> >>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>> User Private >>>> Key and Certificate Services" >>>> Enter Password or Pin for "NSS Certificate DB": >>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>> Certificate >>>> DB:auditSigningCert cert-pki-ca >>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert >>>> cert-pki-ca >>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>> Certificate >>>> DB:ocspSigningCert cert-pki-ca >>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >>>> cert-pki-ca >>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>> Certificate >>>> DB:subsystemCert cert-pki-ca >>>> >>>> But this doesn't (with the same password from password.conf) >>>> >>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL >>>> Please enter pin, password, or pass phrase for security token >>>> 'ldap(0)': >>>> SASL/EXTERNAL authentication started >>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>> >>>> That password is getting me somewhere though, since if I put >>>> in a >>>> nonsense or incorrect password it just prompts over and over. >>> >>> Let's step back a second. You upgraded from what to what? >> >> There wasn't much of a change... I just assumed someone ran yum >> upgrade and didn't restart, then the power outage... it looks >> like not much of a version change though. >> >> # grep ipa /var/log/yum.log >> Jan 08 04:45:32 Installed: >> ipa-common-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:45:32 Installed: >> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >> Jan 08 04:46:07 Updated: >> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch >> Jan 08 04:47:04 Installed: >> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:05 Installed: >> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:05 Updated: >> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:05 Installed: >> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:06 Installed: >> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >> Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >> Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >> Jan 08 04:47:27 Updated: >> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:47:36 Installed: >> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >> Jan 08 04:50:07 Erased: >> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >> Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:42 Updated: >> ipa-client-common-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:43 Updated: >> ipa-server-common-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:43 Updated: >> python2-ipalib-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:44 Updated: >> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:45 Updated: >> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:45 Updated: >> ipa-admintools-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 >> Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 >> Jul 28 09:55:48 Updated: >> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >> Jul 28 09:55:48 Updated: >> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >> >> # grep pki /var/log/yum.log >> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch >> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >> >> >>> >>> Are you running in SELinux enforcing mode? >> >> # getenforce >> Enforcing >> >>> >>> If so can you run: >>> >>> # ausearch -m AVC >>> >> >> # ausearch -m AVC >> The NOLOG option to log_format is deprecated. Please use the >> write_logs option. >> The NOLOG option is overriding the write_logs current setting. >> ---- >> time->Mon Mar 14 10:39:01 2016 >> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e >> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 >> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 >> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 >> egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 >> comm="sshd" exe="/usr/sbin/sshd" >> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >> type=AVC msg=audit(1457977141.767:17537): avc: denied { read } >> for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" >> ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >> >> >>> I suspect that the subsystem cert was renewed and everything >>> wasn't >>> updated properly. If I'm right this is unrelated to the >>> upgrade it just >>> shone a spotlight on it. >>> >>> Can you run these commands: >>> >>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>> >> >> This returns >> >> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >> uid=pkidbuser,ou=people,o=ipaca userCertificate description >> Enter LDAP Password: >> dn: uid=pkidbuser,ou=people,o=ipaca >> userCertificate:: >> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> ... stuff ... >> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >> Subsystem,O=BPT.ROCKS >> >>> and >>> >>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>> cert-pki-ca' | grep Serial >>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>> cert-pki-ca' -a >>> >> >> These both return >> >> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' | grep Serial >> Enter Password or Pin for "NSS Certificate DB": >> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >> Unrecognized Object Identifier. >> >> >> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >> cert-pki-ca' -a >> certutil: Checking token "NSS Certificate DB" in slot "NSS User >> Private Key and Certificate Services" >> Enter Password or Pin for "NSS Certificate DB": >> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >> Unrecognized Object Identifier. >> > Hi, > > you made a typo and requested the keys (-K) instead of the certs > (-L). Please retry with -L and you should be able to see the > certificate serial. >
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a -----BEGIN CERTIFICATE----- MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... sjeiFbUFtoMnfmHLIYpe5QAR -----END CERTIFICATE-----
And this cert is NOT the same.
Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA
component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi and [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
From the output you may only copy/past the ones related to userCertificate
Also to dump the current replication status, send the result of [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" and [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Regards thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
Thank you again for your time!
Ian
> Flo. >> >> which seems weird. >> >>> The description attribute in the ldapsearch output is of the >>> form 2;#;DN;DN >>> >>> The # should match the serial number in the first certutil output >>> >>> The ASCII blob (minus the BEGIN/END headers) should match one >>> of the >>> userCertificate attributes. >>> >>> rob >>> >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: > On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >> On 08/02/2017 01:43 AM, Ian Harding wrote: >>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>> Ian Harding wrote: >>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>> I had an unexpected restart of an IPA server that had >>>>>>>>>>> apparently had >>>>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>>>> pki-tomcatd >>>>>>>>>>> would >>>>>>>>>>> not start. >>>>>>>>>>> >>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> dogtag is an application within tomcat so tomcat can >>>>>>>>>> run without >>>>>>>>>> dogtag >>>>>>>>>> running. >>>>>>>>>> >>>>>>>>>> We need to see more of the dogtag debug log to see what >>>>>>>>>> is going on. >>>>>>>>>> >>>>>>>>> >>>>>>>>> It looks like an authentication problem... >>>>>>>>> >>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>> handshake happened >>>>>>>>> Could not connect to LDAP server host >>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>> Error netscape.ldap.LDAPException: Authentication failed >>>>>>>>> (49) >>>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> dogtag stores its internal data in the LDAP server and >>>>>>>> needs to >>>>>>>> establish a secure LDAP connection. You can check how this >>>>>>>> connection is configured in >>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>> the lines: >>>>>>>> >>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>> cert-pki-ca >>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>> internaldb.ldapconn.port=636 >>>>>>>> internaldb.ldapconn.secureConn >>>>>>>> >>>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>>> certificate) or BasicAuth (authentication with a bind DN and >>>>>>>> password stored in >>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>> >>>>>>>> You can use this information to manually check the >>>>>>>> credentials. For >>>>>>>> instance with sslclientauth: >>>>>>>> >>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>> >>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>> EXTERNAL >>>>>>>> (provide the password from >>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>> >>>>>>> >>>>>>> I found this: >>>>>>> >>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>> cert-pki-ca >>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>> ... >>>>>>> >>>>>>> and when I try the ldapsearch I am presented with a prompt >>>>>>> to provide >>>>>>> a pin/password >>>>>>> >>>>>>> Please enter pin, password, or pass phrase for security >>>>>>> token 'ldap(0)': >>>>>>> >>>>>>> but there is no password file... >>>>>>> >>>>>> Hi, >>>>>> >>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>> password can be >>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with >>>>>> the tag >>>>>> internal=...) >>>>>> >>>>>> Can you check if the password with the tag internal=... >>>>>> allows to read >>>>>> the keys from the NSS db? >>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>> (provide password) >>>>> >>>>> That works... >>>>> >>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>> User Private >>>>> Key and Certificate Services" >>>>> Enter Password or Pin for "NSS Certificate DB": >>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>> Certificate >>>>> DB:auditSigningCert cert-pki-ca >>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert >>>>> cert-pki-ca >>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>> Certificate >>>>> DB:ocspSigningCert cert-pki-ca >>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >>>>> cert-pki-ca >>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>> Certificate >>>>> DB:subsystemCert cert-pki-ca >>>>> >>>>> But this doesn't (with the same password from password.conf) >>>>> >>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>> EXTERNAL >>>>> Please enter pin, password, or pass phrase for security >>>>> token 'ldap(0)': >>>>> SASL/EXTERNAL authentication started >>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>> >>>>> That password is getting me somewhere though, since if I put >>>>> in a >>>>> nonsense or incorrect password it just prompts over and over. >>>> >>>> Let's step back a second. You upgraded from what to what? >>> >>> There wasn't much of a change... I just assumed someone ran >>> yum upgrade and didn't restart, then the power outage... it >>> looks like not much of a version change though. >>> >>> # grep ipa /var/log/yum.log >>> Jan 08 04:45:32 Installed: >>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:45:32 Installed: >>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>> Jan 08 04:46:07 Updated: >>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch >>> Jan 08 04:47:04 Installed: >>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:05 Installed: >>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:05 Updated: >>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:05 Installed: >>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:06 Installed: >>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>> Jan 08 04:47:17 Updated: >>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>> Jan 08 04:47:23 Updated: >>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>> Jan 08 04:47:27 Updated: >>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:47:36 Installed: >>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>> Jan 08 04:50:07 Erased: >>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>> Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:42 Updated: >>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:43 Updated: >>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:43 Updated: >>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:44 Updated: >>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:45 Updated: >>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:45 Updated: >>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 >>> Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 >>> Jul 28 09:55:48 Updated: >>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>> Jul 28 09:55:48 Updated: >>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>> >>> # grep pki /var/log/yum.log >>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch >>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>> >>> >>>> >>>> Are you running in SELinux enforcing mode? >>> >>> # getenforce >>> Enforcing >>> >>>> >>>> If so can you run: >>>> >>>> # ausearch -m AVC >>>> >>> >>> # ausearch -m AVC >>> The NOLOG option to log_format is deprecated. Please use the >>> write_logs option. >>> The NOLOG option is overriding the write_logs current setting. >>> ---- >>> time->Mon Mar 14 10:39:01 2016 >>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e >>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 >>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 >>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 >>> egid=581400001 sgid=0 fsgid=581400001 tty=(none) >>> ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" >>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>> type=AVC msg=audit(1457977141.767:17537): avc: denied { read } >>> for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" >>> ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>> >>> >>>> I suspect that the subsystem cert was renewed and everything >>>> wasn't >>>> updated properly. If I'm right this is unrelated to the >>>> upgrade it just >>>> shone a spotlight on it. >>>> >>>> Can you run these commands: >>>> >>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>> >>> >>> This returns >>> >>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>> Enter LDAP Password: >>> dn: uid=pkidbuser,ou=people,o=ipaca >>> userCertificate:: >>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> ... stuff ... >>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>> Subsystem,O=BPT.ROCKS >>> >>>> and >>>> >>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>> cert-pki-ca' | grep Serial >>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>> cert-pki-ca' -a >>>> >>> >>> These both return >>> >>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>> cert-pki-ca' | grep Serial >>> Enter Password or Pin for "NSS Certificate DB": >>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>> Unrecognized Object Identifier. >>> >>> >>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>> cert-pki-ca' -a >>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>> User Private Key and Certificate Services" >>> Enter Password or Pin for "NSS Certificate DB": >>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>> Unrecognized Object Identifier. >>> >> Hi, >> >> you made a typo and requested the keys (-K) instead of the >> certs (-L). Please retry with -L and you should be able to see >> the certificate serial. >> > > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate > description > Enter LDAP Password: > dn: uid=pkidbuser,ou=people,o=ipaca > userCertificate:: > MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > ... > ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= > description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA > Subsystem,O=BPT.ROCKS > > So the serial is 4? > > [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias > -n 'subsystemCert cert-pki-ca' | grep Serial > Serial Number: 1341718541 (0x4ff9000d) > > Which is NOT 4... > > [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias > -n 'subsystemCert cert-pki-ca' -a > -----BEGIN CERTIFICATE----- > MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > ... > sjeiFbUFtoMnfmHLIYpe5QAR > -----END CERTIFICATE----- > > And this cert is NOT the same. > Hi,
when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.
This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):
$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` $ kinit admin $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
In this example the renewal master is vm-ipaserver.
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA
component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51
and [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86
From the output you may only copy/past the ones related to userCertificate
Also to dump the current replication status, send the result of [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0
and [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
$ sudo /usr/libexec/ipa/certmonger/stop_pkicad $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.
Flo
> Thank you again for your time! > > Ian > >> Flo. >>> >>> which seems weird. >>> >>>> The description attribute in the ldapsearch output is of the >>>> form 2;#;DN;DN >>>> >>>> The # should match the serial number in the first certutil >>>> output >>>> >>>> The ASCII blob (minus the BEGIN/END headers) should match one >>>> of the >>>> userCertificate attributes. >>>> >>>> rob >>>> >>> >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Hi Ian,
Thanks for having gather those data.
# # So pkidbuser entries have a same (old) userCertificate likely generated during install # But only freeipa-sea has a new one created on freeipa-sea around Jun 8th 2017 05:54:16 # This recent certificate is identified by 5938e688000104290000 # [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca ... nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip>
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip>
# # why 5938e688000104290000 value was not propagated to seattlenfs ? # The most recent update (from freeipa-sea) that was replicated to seattlenfs # is 1 year old (57be8043000704290000 - Aug 25th 2016 05:21:07). # In addition seattlenfs received direct update (last one was in Jan 1017) that were not # replicated to freeipa-sea # # The two servers have diverged because they can not replicate to eachother because # they were not correctly initialize. # They have different "replicageneration" (57c291d9000004290000 vs 55c8f3ae000000600000) # # It is looking like freeipa-sea was created one or two years ago and used to initialized seattlenfs. # But later freeipa-sea was recreated. # [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf000004290000 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d4000004290000 57be8043000704290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 00000000
In conclusion: From replication pov, the two instances can not communicate. One solution would be to identify which instance is the good one, you want to keep. and reinit the second one from that reference.
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: > On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>> Ian Harding wrote: >>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>> I had an unexpected restart of an IPA server that had >>>>>>>>>>>> apparently had >>>>>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>> would >>>>>>>>>>>> not start. >>>>>>>>>>>> >>>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> dogtag is an application within tomcat so tomcat can >>>>>>>>>>> run without >>>>>>>>>>> dogtag >>>>>>>>>>> running. >>>>>>>>>>> >>>>>>>>>>> We need to see more of the dogtag debug log to see >>>>>>>>>>> what is going on. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It looks like an authentication problem... >>>>>>>>>> >>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>> handshake happened >>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>> failed (49) >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> dogtag stores its internal data in the LDAP server and >>>>>>>>> needs to >>>>>>>>> establish a secure LDAP connection. You can check how this >>>>>>>>> connection is configured in >>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>> the lines: >>>>>>>>> >>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>> cert-pki-ca >>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>> >>>>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>>>> certificate) or BasicAuth (authentication with a bind DN >>>>>>>>> and >>>>>>>>> password stored in >>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>> >>>>>>>>> You can use this information to manually check the >>>>>>>>> credentials. For >>>>>>>>> instance with sslclientauth: >>>>>>>>> >>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>> >>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>> EXTERNAL >>>>>>>>> (provide the password from >>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>> >>>>>>>> >>>>>>>> I found this: >>>>>>>> >>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>> cert-pki-ca >>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>> ... >>>>>>>> >>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>> prompt to provide >>>>>>>> a pin/password >>>>>>>> >>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>> token 'ldap(0)': >>>>>>>> >>>>>>>> but there is no password file... >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>>> password can be >>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with >>>>>>> the tag >>>>>>> internal=...) >>>>>>> >>>>>>> Can you check if the password with the tag internal=... >>>>>>> allows to read >>>>>>> the keys from the NSS db? >>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>> (provide password) >>>>>> >>>>>> That works... >>>>>> >>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>>> User Private >>>>>> Key and Certificate Services" >>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>> Certificate >>>>>> DB:auditSigningCert cert-pki-ca >>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>> caSigningCert >>>>>> cert-pki-ca >>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>> Certificate >>>>>> DB:ocspSigningCert cert-pki-ca >>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >>>>>> cert-pki-ca >>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>> Certificate >>>>>> DB:subsystemCert cert-pki-ca >>>>>> >>>>>> But this doesn't (with the same password from password.conf) >>>>>> >>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>> EXTERNAL >>>>>> Please enter pin, password, or pass phrase for security >>>>>> token 'ldap(0)': >>>>>> SASL/EXTERNAL authentication started >>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>> >>>>>> That password is getting me somewhere though, since if I >>>>>> put in a >>>>>> nonsense or incorrect password it just prompts over and over. >>>>> >>>>> Let's step back a second. You upgraded from what to what? >>>> >>>> There wasn't much of a change... I just assumed someone ran >>>> yum upgrade and didn't restart, then the power outage... it >>>> looks like not much of a version change though. >>>> >>>> # grep ipa /var/log/yum.log >>>> Jan 08 04:45:32 Installed: >>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:45:32 Installed: >>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>> Jan 08 04:46:07 Updated: >>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch >>>> Jan 08 04:47:04 Installed: >>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:05 Installed: >>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:05 Updated: >>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:05 Installed: >>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:06 Installed: >>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>> Jan 08 04:47:17 Updated: >>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>> Jan 08 04:47:23 Updated: >>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>> Jan 08 04:47:27 Updated: >>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:47:36 Installed: >>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>> Jan 08 04:50:07 Erased: >>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>> Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:42 Updated: >>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:43 Updated: >>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:43 Updated: >>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:44 Updated: >>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:45 Updated: >>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:45 Updated: >>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>> Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>> Jul 28 09:55:48 Updated: >>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>> Jul 28 09:55:48 Updated: >>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>> >>>> # grep pki /var/log/yum.log >>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch >>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>> >>>> >>>>> >>>>> Are you running in SELinux enforcing mode? >>>> >>>> # getenforce >>>> Enforcing >>>> >>>>> >>>>> If so can you run: >>>>> >>>>> # ausearch -m AVC >>>>> >>>> >>>> # ausearch -m AVC >>>> The NOLOG option to log_format is deprecated. Please use the >>>> write_logs option. >>>> The NOLOG option is overriding the write_logs current setting. >>>> ---- >>>> time->Mon Mar 14 10:39:01 2016 >>>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e >>>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 >>>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 >>>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 >>>> egid=581400001 sgid=0 fsgid=581400001 tty=(none) >>>> ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" >>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { read >>>> } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" >>>> ino=116393517 >>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>> >>>> >>>>> I suspect that the subsystem cert was renewed and everything >>>>> wasn't >>>>> updated properly. If I'm right this is unrelated to the >>>>> upgrade it just >>>>> shone a spotlight on it. >>>>> >>>>> Can you run these commands: >>>>> >>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>> >>>> >>>> This returns >>>> >>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>> Enter LDAP Password: >>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>> userCertificate:: >>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>> ... stuff ... >>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>> Subsystem,O=BPT.ROCKS >>>> >>>>> and >>>>> >>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>> cert-pki-ca' | grep Serial >>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>> cert-pki-ca' -a >>>>> >>>> >>>> These both return >>>> >>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>> cert-pki-ca' | grep Serial >>>> Enter Password or Pin for "NSS Certificate DB": >>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>> Unrecognized Object Identifier. >>>> >>>> >>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>> cert-pki-ca' -a >>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>> User Private Key and Certificate Services" >>>> Enter Password or Pin for "NSS Certificate DB": >>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>> Unrecognized Object Identifier. >>>> >>> Hi, >>> >>> you made a typo and requested the keys (-K) instead of the >>> certs (-L). Please retry with -L and you should be able to see >>> the certificate serial. >>> >> >> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >> description >> Enter LDAP Password: >> dn: uid=pkidbuser,ou=people,o=ipaca >> userCertificate:: >> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> ... >> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >> Subsystem,O=BPT.ROCKS >> >> So the serial is 4? >> >> [root@seattlenfs ianh]# certutil -L -d >> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep >> Serial >> Serial Number: 1341718541 (0x4ff9000d) >> >> Which is NOT 4... >> >> [root@seattlenfs ianh]# certutil -L -d >> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >> -----BEGIN CERTIFICATE----- >> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> ... >> sjeiFbUFtoMnfmHLIYpe5QAR >> -----END CERTIFICATE----- >> >> And this cert is NOT the same. >> > Hi, > > when certmonger renews a certificate, it runs pre-save and > post-save commands (which you can see in the output of getcert > list). In your case, the post-save command for subsystemCert > cert-pki-ca probably failed. > > This command (on the renewal master) is responsible for copying > the new cert from the NSS db to the LDAP server. You need first > to check which IPA server is the renewal master (as the serial > numbers are in a completely different range, I assume that you > have multiple IPA servers configured with a CA instance): > > $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` > $ kinit admin > $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b > "cn=masters,cn=ipa,cn=etc,$BASEDN" > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > dn: > cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com > > > In this example the renewal master is vm-ipaserver. >
I get this
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
Which is the other FreeIPA server.
Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA
component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W > -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W > -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W > -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
> If your renewal master is the machine on which the NSS DB > /etc/pki/pki-tomcat/alias contains the newer cert, then you can > manually run the scripts:
If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)
> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad > $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert > cert-pki-ca" >
I ran this, it took a while, then I re-checked and the serial still seems to be 4
$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC ... ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
Here's what was in /var/log/pki/pki-tomcat/ca/debug
[03/Aug/2017:13:52:51][localhost-startStop-1]:
[03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG SUBSYSTEM INITIALIZED ======= [03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================ [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem) [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering! [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
I really appreciate you sticking with me through this. I owe you.
- Ian
> After that, check that the LDAP entry has been updated. If it is > the case, you should be able to restart pki. > > Flo > >> Thank you again for your time! >> >> Ian >> >>> Flo. >>>> >>>> which seems weird. >>>> >>>>> The description attribute in the ldapsearch output is of the >>>>> form 2;#;DN;DN >>>>> >>>>> The # should match the serial number in the first certutil >>>>> output >>>>> >>>>> The ASCII blob (minus the BEGIN/END headers) should match >>>>> one of the >>>>> userCertificate attributes. >>>>> >>>>> rob >>>>> >>>> >>> >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
On 8/9/17 3:05 AM, thierry bordaz wrote:
Hi Ian,
Thanks for having gather those data.
# # So pkidbuser entries have a same (old) userCertificate likely generated during install # But only freeipa-sea has a new one created on freeipa-sea around Jun 8th 2017 05:54:16 # This recent certificate is identified by 5938e688000104290000 # [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca ... nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> # # why 5938e688000104290000 value was not propagated to seattlenfs ? # The most recent update (from freeipa-sea) that was replicated to seattlenfs # is 1 year old (57be8043000704290000 - Aug 25th 2016 05:21:07). # In addition seattlenfs received direct update (last one was in Jan 1017) that were not # replicated to freeipa-sea # # The two servers have diverged because they can not replicate to eachother because # they were not correctly initialize. # They have different "replicageneration" (57c291d9000004290000 vs 55c8f3ae000000600000) # # It is looking like freeipa-sea was created one or two years ago and used to initialized seattlenfs. # But later freeipa-sea was recreated.
That's about right.
# [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf000004290000 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d4000004290000 57be8043000704290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 00000000In conclusion: From replication pov, the two instances can not communicate. One solution would be to identify which instance is the good one, you want to keep. and reinit the second one from that reference.
What exactly does reinit mean? I have run
ipa-replica-manage re-initialize --from freeipa-sea.bpt.rocks
several times over the years when replication has stopped.
Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.
There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error. I don't know if that's related.
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote: > On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: >> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>>> Ian Harding wrote: >>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>>> I had an unexpected restart of an IPA server that >>>>>>>>>>>>> had apparently had >>>>>>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>>> would >>>>>>>>>>>>> not start. >>>>>>>>>>>>> >>>>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> dogtag is an application within tomcat so tomcat can >>>>>>>>>>>> run without >>>>>>>>>>>> dogtag >>>>>>>>>>>> running. >>>>>>>>>>>> >>>>>>>>>>>> We need to see more of the dogtag debug log to see >>>>>>>>>>>> what is going on. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It looks like an authentication problem... >>>>>>>>>>> >>>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>>> handshake happened >>>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>>> failed (49) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> dogtag stores its internal data in the LDAP server and >>>>>>>>>> needs to >>>>>>>>>> establish a secure LDAP connection. You can check how this >>>>>>>>>> connection is configured in >>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>>> the lines: >>>>>>>>>> >>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>> cert-pki-ca >>>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>>> >>>>>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>>>>> certificate) or BasicAuth (authentication with a bind >>>>>>>>>> DN and >>>>>>>>>> password stored in >>>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>>> >>>>>>>>>> You can use this information to manually check the >>>>>>>>>> credentials. For >>>>>>>>>> instance with sslclientauth: >>>>>>>>>> >>>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>>> >>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>>> EXTERNAL >>>>>>>>>> (provide the password from >>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>>> >>>>>>>>> >>>>>>>>> I found this: >>>>>>>>> >>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>> cert-pki-ca >>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>>> ... >>>>>>>>> >>>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>>> prompt to provide >>>>>>>>> a pin/password >>>>>>>>> >>>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>>> token 'ldap(0)': >>>>>>>>> >>>>>>>>> but there is no password file... >>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>>>> password can be >>>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with >>>>>>>> the tag >>>>>>>> internal=...) >>>>>>>> >>>>>>>> Can you check if the password with the tag internal=... >>>>>>>> allows to read >>>>>>>> the keys from the NSS db? >>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>> (provide password) >>>>>>> >>>>>>> That works... >>>>>>> >>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>>>> User Private >>>>>>> Key and Certificate Services" >>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>>> Certificate >>>>>>> DB:auditSigningCert cert-pki-ca >>>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>>> caSigningCert >>>>>>> cert-pki-ca >>>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>>> Certificate >>>>>>> DB:ocspSigningCert cert-pki-ca >>>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert >>>>>>> cert-pki-ca >>>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>>> Certificate >>>>>>> DB:subsystemCert cert-pki-ca >>>>>>> >>>>>>> But this doesn't (with the same password from password.conf) >>>>>>> >>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>> EXTERNAL >>>>>>> Please enter pin, password, or pass phrase for security >>>>>>> token 'ldap(0)': >>>>>>> SASL/EXTERNAL authentication started >>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>>> >>>>>>> That password is getting me somewhere though, since if I >>>>>>> put in a >>>>>>> nonsense or incorrect password it just prompts over and over. >>>>>> >>>>>> Let's step back a second. You upgraded from what to what? >>>>> >>>>> There wasn't much of a change... I just assumed someone ran >>>>> yum upgrade and didn't restart, then the power outage... it >>>>> looks like not much of a version change though. >>>>> >>>>> # grep ipa /var/log/yum.log >>>>> Jan 08 04:45:32 Installed: >>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:45:32 Installed: >>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>> Jan 08 04:46:07 Updated: >>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch >>>>> Jan 08 04:47:04 Installed: >>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:05 Installed: >>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:05 Updated: >>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:05 Installed: >>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:06 Installed: >>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>>> Jan 08 04:47:17 Updated: >>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>>> Jan 08 04:47:23 Updated: >>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>>> Jan 08 04:47:27 Updated: >>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:47:36 Installed: >>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>>> Jan 08 04:50:07 Erased: >>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>>> Jul 28 09:55:41 Updated: >>>>> ipa-common-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:42 Updated: >>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:43 Updated: >>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:43 Updated: >>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:44 Updated: >>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:45 Updated: >>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:45 Updated: >>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:46 Updated: >>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>>> Jul 28 09:55:48 Updated: >>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>>> Jul 28 09:55:48 Updated: >>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>>> Jul 28 09:55:48 Updated: >>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>>> >>>>> # grep pki /var/log/yum.log >>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>>> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch >>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>>> >>>>> >>>>>> >>>>>> Are you running in SELinux enforcing mode? >>>>> >>>>> # getenforce >>>>> Enforcing >>>>> >>>>>> >>>>>> If so can you run: >>>>>> >>>>>> # ausearch -m AVC >>>>>> >>>>> >>>>> # ausearch -m AVC >>>>> The NOLOG option to log_format is deprecated. Please use the >>>>> write_logs option. >>>>> The NOLOG option is overriding the write_logs current setting. >>>>> ---- >>>>> time->Mon Mar 14 10:39:01 2016 >>>>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e >>>>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 >>>>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 >>>>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 >>>>> egid=581400001 sgid=0 fsgid=581400001 tty=(none) >>>>> ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" >>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { >>>>> read } for pid=12965 comm="sshd" name="authorized_keys" >>>>> dev="dm-2" ino=116393517 >>>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>>> >>>>> >>>>>> I suspect that the subsystem cert was renewed and >>>>>> everything wasn't >>>>>> updated properly. If I'm right this is unrelated to the >>>>>> upgrade it just >>>>>> shone a spotlight on it. >>>>>> >>>>>> Can you run these commands: >>>>>> >>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>> >>>>> >>>>> This returns >>>>> >>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>> Enter LDAP Password: >>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>> userCertificate:: >>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>> ... stuff ... >>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>>> Subsystem,O=BPT.ROCKS >>>>> >>>>>> and >>>>>> >>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>>> cert-pki-ca' | grep Serial >>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>> 'subsystemCert cert-pki-ca' -a >>>>>> >>>>> >>>>> These both return >>>>> >>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>> cert-pki-ca' | grep Serial >>>>> Enter Password or Pin for "NSS Certificate DB": >>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>>> Unrecognized Object Identifier. >>>>> >>>>> >>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>> 'subsystemCert cert-pki-ca' -a >>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>> User Private Key and Certificate Services" >>>>> Enter Password or Pin for "NSS Certificate DB": >>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>>> Unrecognized Object Identifier. >>>>> >>>> Hi, >>>> >>>> you made a typo and requested the keys (-K) instead of the >>>> certs (-L). Please retry with -L and you should be able to >>>> see the certificate serial. >>>> >>> >>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >>> description >>> Enter LDAP Password: >>> dn: uid=pkidbuser,ou=people,o=ipaca >>> userCertificate:: >>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> ... >>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>> Subsystem,O=BPT.ROCKS >>> >>> So the serial is 4? >>> >>> [root@seattlenfs ianh]# certutil -L -d >>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>> grep Serial >>> Serial Number: 1341718541 (0x4ff9000d) >>> >>> Which is NOT 4... >>> >>> [root@seattlenfs ianh]# certutil -L -d >>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >>> -----BEGIN CERTIFICATE----- >>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> ... >>> sjeiFbUFtoMnfmHLIYpe5QAR >>> -----END CERTIFICATE----- >>> >>> And this cert is NOT the same. >>> >> Hi, >> >> when certmonger renews a certificate, it runs pre-save and >> post-save commands (which you can see in the output of getcert >> list). In your case, the post-save command for subsystemCert >> cert-pki-ca probably failed. >> >> This command (on the renewal master) is responsible for copying >> the new cert from the NSS db to the LDAP server. You need first >> to check which IPA server is the renewal master (as the serial >> numbers are in a completely different range, I assume that you >> have multiple IPA servers configured with a CA instance): >> >> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-` >> $ kinit admin >> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >> "cn=masters,cn=ipa,cn=etc,$BASEDN" >> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >> dn: >> cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com >> >> >> In this example the renewal master is vm-ipaserver. >> > > I get this > > $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b > "cn=masters,cn=ipa,cn=etc,$BASEDN" > '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn > dn: > cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks > > > Which is the other FreeIPA server. > Hi Ian,
it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is. Sorry.
- On server freeipa-sea (=renewal master), the CA component is
installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA
component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
- In the LDAP servers, the cert is stored in the entry
uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W > -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W > -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?
Thanks, Flo
>> If your renewal master is the machine on which the NSS DB >> /etc/pki/pki-tomcat/alias contains the newer cert, then you can >> manually run the scripts: > > If I'm interpreting this correctly, then it is... so I ran this > on the broken machine (seattlenfs) > >> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad >> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert >> cert-pki-ca" >> > > I ran this, it took a while, then I re-checked and the serial > still seems to be 4 > > $ ldapsearch -LLL -D 'cn=directory manager' -W -b > uid=pkidbuser,ou=people,o=ipaca userCertificate description > Enter LDAP Password: > dn: uid=pkidbuser,ou=people,o=ipaca > userCertificate:: > MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > ... > ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= > description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA > Subsystem,O=BPT.ROCKS > > Here's what was in /var/log/pki/pki-tomcat/ca/debug > > [03/Aug/2017:13:52:51][localhost-startStop-1]: > ============================================ > [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG > SUBSYSTEM INITIALIZED ======= > [03/Aug/2017:13:52:51][localhost-startStop-1]: > ============================================ > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > restart at autoShutdown? false > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > autoShutdown crumb file path? > /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about > to look for cert for auto-shutdown support:auditSigningCert > cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found > cert:auditSigningCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done > init id=debug > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initialized debug > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initSubsystem id=log > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready > to init id=log > [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating > RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) > > [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating > RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) > [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating > RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > restart at autoShutdown? false > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > autoShutdown crumb file path? > /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about > to look for cert for auto-shutdown support:auditSigningCert > cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found > cert:auditSigningCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done > init id=log > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initialized log > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initSubsystem id=jss > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready > to init id=jss > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > restart at autoShutdown? false > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > autoShutdown crumb file path? > /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about > to look for cert for auto-shutdown support:auditSigningCert > cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found > cert:auditSigningCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done > init id=jss > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initialized jss > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: > initSubsystem id=dbs > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready > to init id=dbs > [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: > init() mEnableSerialMgmt=true > [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating > LdapBoundConnFactor(DBSubsystem) > [03/Aug/2017:13:52:51][localhost-startStop-1]: > LdapBoundConnFactory: init > [03/Aug/2017:13:52:51][localhost-startStop-1]: > LdapBoundConnFactory:doCloning true > [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init() > [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: > init begins > [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: > init ends > [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before > makeConnection errorIfDown is true > [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: > errorIfDown true > [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true > [03/Aug/2017:13:52:51][localhost-startStop-1]: > SSLClientCertificateSelectionCB: Setting desired cert nickname > to: subsystemCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: > set client auth cert nickname subsystemCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: > SSLClientCertificatSelectionCB: Entering! > [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: > ocspSigningCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: > subsystemCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: > SSLClientCertificateSelectionCB: desired cert found in list: > subsystemCert cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: > SSLClientCertificateSelectionCB: returning: subsystemCert > cert-pki-ca > [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake > happened > [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown() > > > I really appreciate you sticking with me through this. I owe you. > > - Ian > >> After that, check that the LDAP entry has been updated. If it >> is the case, you should be able to restart pki. >> >> Flo >> >>> Thank you again for your time! >>> >>> Ian >>> >>>> Flo. >>>>> >>>>> which seems weird. >>>>> >>>>>> The description attribute in the ldapsearch output is of >>>>>> the form 2;#;DN;DN >>>>>> >>>>>> The # should match the serial number in the first certutil >>>>>> output >>>>>> >>>>>> The ASCII blob (minus the BEGIN/END headers) should match >>>>>> one of the >>>>>> userCertificate attributes. >>>>>> >>>>>> rob >>>>>> >>>>> >>>> >>> >> >
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
On 08/09/2017 09:30 PM, Ian Harding via FreeIPA-users wrote:
On 8/9/17 3:05 AM, thierry bordaz wrote:
Hi Ian,
Thanks for having gather those data.
# # So pkidbuser entries have a same (old) userCertificate likely generated during install # But only freeipa-sea has a new one created on freeipa-sea around Jun 8th 2017 05:54:16 # This recent certificate is identified by 5938e688000104290000 # [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca ... nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> # # why 5938e688000104290000 value was not propagated to seattlenfs ? # The most recent update (from freeipa-sea) that was replicated to seattlenfs # is 1 year old (57be8043000704290000 - Aug 25th 2016 05:21:07). # In addition seattlenfs received direct update (last one was in Jan 1017) that were not # replicated to freeipa-sea # # The two servers have diverged because they can not replicate to eachother because # they were not correctly initialize. # They have different "replicageneration" (57c291d9000004290000 vs 55c8f3ae000000600000) # # It is looking like freeipa-sea was created one or two years ago and used to initialized seattlenfs. # But later freeipa-sea was recreated.That's about right.
# [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf000004290000 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d4000004290000 57be8043000704290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 00000000In conclusion: From replication pov, the two instances can not communicate. One solution would be to identify which instance is the good one, you want to keep. and reinit the second one from that reference.
What exactly does reinit mean? I have run
ipa-replica-manage re-initialize --from freeipa-sea.bpt.rocks
This triggers the reinit of 'dc=ipadomain,dc=com' suffix but not for o=ipaca suffix. To reinit o=ipaca you may use ipa-cs-replica-manager re-initialize --from freeipa-sea.bpt.rocks. Note that it will overwrite all data of ipaca on seattlenfs with the data from freeipa-sea. This is why it is important to know which server is the valid one.
On freeipa-sea and seattlenfs you may check the ipaca replica agreements status before/after reinit. I would guess they are currently reporting failures between these two replicas.
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
several times over the years when replication has stopped.
Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.
There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error. I don't know if that's related.
Zombie servers have been discussed a lot on this mailing list. You may get rid of them with list-ruv/clean-ruv subcommand. Replication usually manage to work fine even if some zombie entries exist but it is a good practice to clean them.
regards thierry
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote: >> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: >>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>>>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>>>> Ian Harding wrote: >>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>>>> I had an unexpected restart of an IPA server that >>>>>>>>>>>>>> had apparently had >>>>>>>>>>>>>> updates run but had not been restarted. ipactl says >>>>>>>>>>>>>> pki-tomcatd >>>>>>>>>>>>>> would >>>>>>>>>>>>>> not start. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> dogtag is an application within tomcat so tomcat can >>>>>>>>>>>>> run without >>>>>>>>>>>>> dogtag >>>>>>>>>>>>> running. >>>>>>>>>>>>> >>>>>>>>>>>>> We need to see more of the dogtag debug log to see >>>>>>>>>>>>> what is going on. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It looks like an authentication problem... >>>>>>>>>>>> >>>>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>>>> handshake happened >>>>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>>>> failed (49) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> dogtag stores its internal data in the LDAP server and >>>>>>>>>>> needs to >>>>>>>>>>> establish a secure LDAP connection. You can check how >>>>>>>>>>> this >>>>>>>>>>> connection is configured in >>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>>>> the lines: >>>>>>>>>>> >>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>> cert-pki-ca >>>>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>>>> >>>>>>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>>>>>> certificate) or BasicAuth (authentication with a bind >>>>>>>>>>> DN and >>>>>>>>>>> password stored in >>>>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>>>> >>>>>>>>>>> You can use this information to manually check the >>>>>>>>>>> credentials. For >>>>>>>>>>> instance with sslclientauth: >>>>>>>>>>> >>>>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>>>> >>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>>>> EXTERNAL >>>>>>>>>>> (provide the password from >>>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I found this: >>>>>>>>>> >>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>> cert-pki-ca >>>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>>>> prompt to provide >>>>>>>>>> a pin/password >>>>>>>>>> >>>>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>>>> token 'ldap(0)': >>>>>>>>>> >>>>>>>>>> but there is no password file... >>>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>>>>> password can be >>>>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf >>>>>>>>> (with the tag >>>>>>>>> internal=...) >>>>>>>>> >>>>>>>>> Can you check if the password with the tag internal=... >>>>>>>>> allows to read >>>>>>>>> the keys from the NSS db? >>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>> (provide password) >>>>>>>> >>>>>>>> That works... >>>>>>>> >>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>> "NSS User Private >>>>>>>> Key and Certificate Services" >>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>>>> Certificate >>>>>>>> DB:auditSigningCert cert-pki-ca >>>>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>>>> caSigningCert >>>>>>>> cert-pki-ca >>>>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>>>> Certificate >>>>>>>> DB:ocspSigningCert cert-pki-ca >>>>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f >>>>>>>> Server-Cert >>>>>>>> cert-pki-ca >>>>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>>>> Certificate >>>>>>>> DB:subsystemCert cert-pki-ca >>>>>>>> >>>>>>>> But this doesn't (with the same password from password.conf) >>>>>>>> >>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>> EXTERNAL >>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>> token 'ldap(0)': >>>>>>>> SASL/EXTERNAL authentication started >>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>>>> >>>>>>>> That password is getting me somewhere though, since if I >>>>>>>> put in a >>>>>>>> nonsense or incorrect password it just prompts over and >>>>>>>> over. >>>>>>> >>>>>>> Let's step back a second. You upgraded from what to what? >>>>>> >>>>>> There wasn't much of a change... I just assumed someone ran >>>>>> yum upgrade and didn't restart, then the power outage... it >>>>>> looks like not much of a version change though. >>>>>> >>>>>> # grep ipa /var/log/yum.log >>>>>> Jan 08 04:45:32 Installed: >>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:45:32 Installed: >>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>> Jan 08 04:46:07 Updated: >>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>> Jan 08 04:46:08 Installed: >>>>>> python-ipaddress-1.0.16-2.el7.noarch >>>>>> Jan 08 04:47:04 Installed: >>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:05 Installed: >>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:05 Updated: >>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:05 Installed: >>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:06 Installed: >>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>>>> Jan 08 04:47:17 Updated: >>>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>>>> Jan 08 04:47:23 Updated: >>>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>>>> Jan 08 04:47:27 Updated: >>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:47:36 Installed: >>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>>>> Jan 08 04:50:07 Erased: >>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>>>> Jul 28 09:55:41 Updated: >>>>>> ipa-common-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:42 Updated: >>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:43 Updated: >>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:43 Updated: >>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:44 Updated: >>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:45 Updated: >>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:45 Updated: >>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:46 Updated: >>>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>>>> Jul 28 09:55:48 Updated: >>>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>>>> Jul 28 09:55:48 Updated: >>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>>>> Jul 28 09:55:48 Updated: >>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>>>> >>>>>> # grep pki /var/log/yum.log >>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>>>> Jan 08 04:47:19 Installed: >>>>>> pki-base-java-10.3.3-14.el7_3.noarch >>>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>>>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >>>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>>>> >>>>>> >>>>>>> >>>>>>> Are you running in SELinux enforcing mode? >>>>>> >>>>>> # getenforce >>>>>> Enforcing >>>>>> >>>>>>> >>>>>>> If so can you run: >>>>>>> >>>>>>> # ausearch -m AVC >>>>>>> >>>>>> >>>>>> # ausearch -m AVC >>>>>> The NOLOG option to log_format is deprecated. Please use >>>>>> the write_logs option. >>>>>> The NOLOG option is overriding the write_logs current setting. >>>>>> ---- >>>>>> time->Mon Mar 14 10:39:01 2016 >>>>>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e >>>>>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 >>>>>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 >>>>>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 >>>>>> egid=581400001 sgid=0 fsgid=581400001 tty=(none) >>>>>> ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" >>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { >>>>>> read } for pid=12965 comm="sshd" name="authorized_keys" >>>>>> dev="dm-2" ino=116393517 >>>>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>>>> >>>>>> >>>>>>> I suspect that the subsystem cert was renewed and >>>>>>> everything wasn't >>>>>>> updated properly. If I'm right this is unrelated to the >>>>>>> upgrade it just >>>>>>> shone a spotlight on it. >>>>>>> >>>>>>> Can you run these commands: >>>>>>> >>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>> >>>>>> >>>>>> This returns >>>>>> >>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>> Enter LDAP Password: >>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>> userCertificate:: >>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>> ... stuff ... >>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>>>> Subsystem,O=BPT.ROCKS >>>>>> >>>>>>> and >>>>>>> >>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>>>> cert-pki-ca' | grep Serial >>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>> >>>>>> >>>>>> These both return >>>>>> >>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>> 'subsystemCert cert-pki-ca' | grep Serial >>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>>>> Unrecognized Object Identifier. >>>>>> >>>>>> >>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>> 'subsystemCert cert-pki-ca' -a >>>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>>> User Private Key and Certificate Services" >>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: >>>>>> Unrecognized Object Identifier. >>>>>> >>>>> Hi, >>>>> >>>>> you made a typo and requested the keys (-K) instead of the >>>>> certs (-L). Please retry with -L and you should be able to >>>>> see the certificate serial. >>>>> >>>> >>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca >>>> userCertificate description >>>> Enter LDAP Password: >>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>> userCertificate:: >>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>> ... >>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>> Subsystem,O=BPT.ROCKS >>>> >>>> So the serial is 4? >>>> >>>> [root@seattlenfs ianh]# certutil -L -d >>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>>> grep Serial >>>> Serial Number: 1341718541 (0x4ff9000d) >>>> >>>> Which is NOT 4... >>>> >>>> [root@seattlenfs ianh]# certutil -L -d >>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >>>> -----BEGIN CERTIFICATE----- >>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>> ... >>>> sjeiFbUFtoMnfmHLIYpe5QAR >>>> -----END CERTIFICATE----- >>>> >>>> And this cert is NOT the same. >>>> >>> Hi, >>> >>> when certmonger renews a certificate, it runs pre-save and >>> post-save commands (which you can see in the output of getcert >>> list). In your case, the post-save command for subsystemCert >>> cert-pki-ca probably failed. >>> >>> This command (on the renewal master) is responsible for >>> copying the new cert from the NSS db to the LDAP server. You >>> need first to check which IPA server is the renewal master (as >>> the serial numbers are in a completely different range, I >>> assume that you have multiple IPA servers configured with a CA >>> instance): >>> >>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= >>> -f2-` >>> $ kinit admin >>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>> dn: >>> cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com >>> >>> >>> In this example the renewal master is vm-ipaserver. >>> >> >> I get this >> >> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >> "cn=masters,cn=ipa,cn=etc,$BASEDN" >> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >> dn: >> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks >> >> >> Which is the other FreeIPA server. >> > Hi Ian, > > it's getting difficult to follow this thread, so could we step > back and summarize? Yes, it is. Sorry. > > - On server freeipa-sea (=renewal master), the CA component is > installed and 'subsystemCert cert-pki-ca' has Serial number 4 in > the NSSDB /etc/pki/pki-tomcat/alias No?
[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d) > - On server seattlenfs (where PKI fails to restart), the CA > component is installed and 'subsystemCert cert-pki-ca' has > Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias. Yes
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial Serial Number: 1341718541 (0x4ff9000d)
> > - In the LDAP servers, the cert is stored in the entry > uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is > the cert from freeipa-sea). This is where the difference is. There are two certificates on the master and only one on the other. I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.
Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem ,O=BPT.ROCKS
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
<truncated> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO CKS
> If you check the certificates dates, I guess that the most > recent one is on server seattlenfs (because of serial numbers). > Can you confirm this (because our goal is to have the most > recent cert stored in all the NSSDBs and in LDAP)? > > Thanks, > Flo > > >>> If your renewal master is the machine on which the NSS DB >>> /etc/pki/pki-tomcat/alias contains the newer cert, then you >>> can manually run the scripts: >> >> If I'm interpreting this correctly, then it is... so I ran this >> on the broken machine (seattlenfs) >> >>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad >>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert >>> "subsystemCert cert-pki-ca" >>> >> >> I ran this, it took a while, then I re-checked and the serial >> still seems to be 4 >> >> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >> uid=pkidbuser,ou=people,o=ipaca userCertificate description >> Enter LDAP Password: >> dn: uid=pkidbuser,ou=people,o=ipaca >> userCertificate:: >> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> ... >> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >> Subsystem,O=BPT.ROCKS >> >> Here's what was in /var/log/pki/pki-tomcat/ca/debug >> >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> ============================================ >> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG >> SUBSYSTEM INITIALIZED ======= >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> ============================================ >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> restart at autoShutdown? false >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> autoShutdown crumb file path? >> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about >> to look for cert for auto-shutdown support:auditSigningCert >> cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found >> cert:auditSigningCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >> init id=debug >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initialized debug >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initSubsystem id=log >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready >> to init id=log >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) >> >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> restart at autoShutdown? false >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> autoShutdown crumb file path? >> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about >> to look for cert for auto-shutdown support:auditSigningCert >> cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found >> cert:auditSigningCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >> init id=log >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initialized log >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initSubsystem id=jss >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready >> to init id=jss >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> restart at autoShutdown? false >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> autoShutdown crumb file path? >> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about >> to look for cert for auto-shutdown support:auditSigningCert >> cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found >> cert:auditSigningCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >> init id=jss >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initialized jss >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >> initSubsystem id=dbs >> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready >> to init id=dbs >> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: >> init() mEnableSerialMgmt=true >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >> LdapBoundConnFactor(DBSubsystem) >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> LdapBoundConnFactory: init >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> LdapBoundConnFactory:doCloning true >> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >> init() >> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >> init begins >> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >> init ends >> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before >> makeConnection errorIfDown is true >> [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: >> errorIfDown true >> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: >> true >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: Setting desired cert nickname >> to: subsystemCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> LdapJssSSLSocket: set client auth cert nickname subsystemCert >> cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> SSLClientCertificatSelectionCB: Entering! >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: >> ocspSigningCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: >> subsystemCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: desired cert found in list: >> subsystemCert cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> SSLClientCertificateSelectionCB: returning: subsystemCert >> cert-pki-ca >> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake >> happened >> [03/Aug/2017:13:52:51][localhost-startStop-1]: >> CMSEngine.shutdown() >> >> >> I really appreciate you sticking with me through this. I owe you. >> >> - Ian >> >>> After that, check that the LDAP entry has been updated. If it >>> is the case, you should be able to restart pki. >>> >>> Flo >>> >>>> Thank you again for your time! >>>> >>>> Ian >>>> >>>>> Flo. >>>>>> >>>>>> which seems weird. >>>>>> >>>>>>> The description attribute in the ldapsearch output is of >>>>>>> the form 2;#;DN;DN >>>>>>> >>>>>>> The # should match the serial number in the first certutil >>>>>>> output >>>>>>> >>>>>>> The ASCII blob (minus the BEGIN/END headers) should match >>>>>>> one of the >>>>>>> userCertificate attributes. >>>>>>> >>>>>>> rob >>>>>>> >>>>>> >>>>> >>>> >>> >> > _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
On 8/10/17 2:45 AM, thierry bordaz wrote:
On 08/09/2017 09:30 PM, Ian Harding via FreeIPA-users wrote:
On 8/9/17 3:05 AM, thierry bordaz wrote:
Hi Ian,
Thanks for having gather those data.
# # So pkidbuser entries have a same (old) userCertificate likely generated during install # But only freeipa-sea has a new one created on freeipa-sea around Jun 8th 2017 05:54:16 # This recent certificate is identified by 5938e688000104290000 # [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca ... nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR <snip> # # why 5938e688000104290000 value was not propagated to seattlenfs ? # The most recent update (from freeipa-sea) that was replicated to seattlenfs # is 1 year old (57be8043000704290000 - Aug 25th 2016 05:21:07). # In addition seattlenfs received direct update (last one was in Jan 1017) that were not # replicated to freeipa-sea # # The two servers have diverged because they can not replicate to eachother because # they were not correctly initialize. # They have different "replicageneration" (57c291d9000004290000 vs 55c8f3ae000000600000) # # It is looking like freeipa-sea was created one or two years ago and used to initialized seattlenfs. # But later freeipa-sea was recreated.That's about right.
# [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf000004290000 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d4000004290000 57be8043000704290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 00000000In conclusion: From replication pov, the two instances can not communicate. One solution would be to identify which instance is the good one, you want to keep. and reinit the second one from that reference.
What exactly does reinit mean? I have run
ipa-replica-manage re-initialize --from freeipa-sea.bpt.rocks
This triggers the reinit of 'dc=ipadomain,dc=com' suffix but not for o=ipaca suffix. To reinit o=ipaca you may use ipa-cs-replica-manager re-initialize --from freeipa-sea.bpt.rocks. Note that it will overwrite all data of ipaca on seattlenfs with the data from freeipa-sea. This is why it is important to know which server is the valid one.
Hmm.. This looks odd.
[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from freeipa-sea.bpt.rocks Directory Manager password:
ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only initializing the first one returned: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o=ipaca,cn=mapping tree,cn=config Update in progress, 15 seconds elapsed [freeipa-sea.bpt.rocks] reports: Update failed! Status: [32 - LDAP error: No such object]
On freeipa-sea and seattlenfs you may check the ipaca replica agreements status before/after reinit. I would guess they are currently reporting failures between these two replicas.
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica - LDAP error: No such object (connection error)
dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca ,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
several times over the years when replication has stopped.
Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.
There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error. I don't know if that's related.
Zombie servers have been discussed a lot on this mailing list. You may get rid of them with list-ruv/clean-ruv subcommand. Replication usually manage to work fine even if some zombie entries exist but it is a good practice to clean them.
Yeah, I've been through every possible combination of list-ruv/clean-ruv. I have something odd going on in that no replication agreements show up anywhere for freeipa-dal.
regards thierry
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote: > On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote: > >> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote: >>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: >>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>>>>> Ian Harding wrote: >>>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>>>>> I had an unexpected restart of an IPA server that >>>>>>>>>>>>>>> had apparently had >>>>>>>>>>>>>>> updates run but had not been restarted. ipactl >>>>>>>>>>>>>>> says pki-tomcatd >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>> not start. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> dogtag is an application within tomcat so tomcat >>>>>>>>>>>>>> can run without >>>>>>>>>>>>>> dogtag >>>>>>>>>>>>>> running. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We need to see more of the dogtag debug log to see >>>>>>>>>>>>>> what is going on. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> It looks like an authentication problem... >>>>>>>>>>>>> >>>>>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>>>>> handshake happened >>>>>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>>>>> failed (49) >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> dogtag stores its internal data in the LDAP server >>>>>>>>>>>> and needs to >>>>>>>>>>>> establish a secure LDAP connection. You can check how >>>>>>>>>>>> this >>>>>>>>>>>> connection is configured in >>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>>>>> the lines: >>>>>>>>>>>> >>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>>> cert-pki-ca >>>>>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>>>>> >>>>>>>>>>>> authtype can be SslClientAuth (authentication with a ssl >>>>>>>>>>>> certificate) or BasicAuth (authentication with a bind >>>>>>>>>>>> DN and >>>>>>>>>>>> password stored in >>>>>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>>>>> >>>>>>>>>>>> You can use this information to manually check the >>>>>>>>>>>> credentials. For >>>>>>>>>>>> instance with sslclientauth: >>>>>>>>>>>> >>>>>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>>>>> >>>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>>>>> EXTERNAL >>>>>>>>>>>> (provide the password from >>>>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I found this: >>>>>>>>>>> >>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>>>>> >>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>> cert-pki-ca >>>>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>>>>> ... >>>>>>>>>>> >>>>>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>>>>> prompt to provide >>>>>>>>>>> a pin/password >>>>>>>>>>> >>>>>>>>>>> Please enter pin, password, or pass phrase for >>>>>>>>>>> security token 'ldap(0)': >>>>>>>>>>> >>>>>>>>>>> but there is no password file... >>>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>>>>>> password can be >>>>>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf >>>>>>>>>> (with the tag >>>>>>>>>> internal=...) >>>>>>>>>> >>>>>>>>>> Can you check if the password with the tag internal=... >>>>>>>>>> allows to read >>>>>>>>>> the keys from the NSS db? >>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>> (provide password) >>>>>>>>> >>>>>>>>> That works... >>>>>>>>> >>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>>> "NSS User Private >>>>>>>>> Key and Certificate Services" >>>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>>>>> Certificate >>>>>>>>> DB:auditSigningCert cert-pki-ca >>>>>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>>>>> caSigningCert >>>>>>>>> cert-pki-ca >>>>>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>>>>> Certificate >>>>>>>>> DB:ocspSigningCert cert-pki-ca >>>>>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f >>>>>>>>> Server-Cert >>>>>>>>> cert-pki-ca >>>>>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>>>>> Certificate >>>>>>>>> DB:subsystemCert cert-pki-ca >>>>>>>>> >>>>>>>>> But this doesn't (with the same password from >>>>>>>>> password.conf) >>>>>>>>> >>>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>> EXTERNAL >>>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>>> token 'ldap(0)': >>>>>>>>> SASL/EXTERNAL authentication started >>>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>>>>> >>>>>>>>> That password is getting me somewhere though, since if I >>>>>>>>> put in a >>>>>>>>> nonsense or incorrect password it just prompts over and >>>>>>>>> over. >>>>>>>> >>>>>>>> Let's step back a second. You upgraded from what to what? >>>>>>> >>>>>>> There wasn't much of a change... I just assumed someone >>>>>>> ran yum upgrade and didn't restart, then the power >>>>>>> outage... it looks like not much of a version change though. >>>>>>> >>>>>>> # grep ipa /var/log/yum.log >>>>>>> Jan 08 04:45:32 Installed: >>>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:45:32 Installed: >>>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>> Jan 08 04:46:07 Updated: >>>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>> Jan 08 04:46:08 Installed: >>>>>>> python-ipaddress-1.0.16-2.el7.noarch >>>>>>> Jan 08 04:47:04 Installed: >>>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:05 Installed: >>>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:05 Updated: >>>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:05 Installed: >>>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:06 Installed: >>>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>>>>> Jan 08 04:47:17 Updated: >>>>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>> Jan 08 04:47:23 Updated: >>>>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>> Jan 08 04:47:27 Updated: >>>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:47:36 Installed: >>>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>>>>> Jan 08 04:50:07 Erased: >>>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>>>>> Jul 28 09:55:41 Updated: >>>>>>> ipa-common-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:42 Updated: >>>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:43 Updated: >>>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:43 Updated: >>>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:44 Updated: >>>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:45 Updated: >>>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:45 Updated: >>>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:46 Updated: >>>>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>>>>> Jul 28 09:55:48 Updated: >>>>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>>>>> Jul 28 09:55:48 Updated: >>>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>>>>> Jul 28 09:55:48 Updated: >>>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>>>>> >>>>>>> # grep pki /var/log/yum.log >>>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>>>>> Jan 08 04:47:19 Installed: >>>>>>> pki-base-java-10.3.3-14.el7_3.noarch >>>>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>>>>> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch >>>>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Are you running in SELinux enforcing mode? >>>>>>> >>>>>>> # getenforce >>>>>>> Enforcing >>>>>>> >>>>>>>> >>>>>>>> If so can you run: >>>>>>>> >>>>>>>> # ausearch -m AVC >>>>>>>> >>>>>>> >>>>>>> # ausearch -m AVC >>>>>>> The NOLOG option to log_format is deprecated. Please use >>>>>>> the write_logs option. >>>>>>> The NOLOG option is overriding the write_logs current >>>>>>> setting. >>>>>>> ---- >>>>>>> time->Mon Mar 14 10:39:01 2016 >>>>>>> type=SYSCALL msg=audit(1457977141.767:17537): >>>>>>> arch=c000003e syscall=2 success=no exit=-13 >>>>>>> a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 >>>>>>> ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 >>>>>>> euid=581400001 suid=0 fsuid=581400001 egid=581400001 >>>>>>> sgid=0 fsgid=581400001 tty=(none) ses=4294967295 >>>>>>> comm="sshd" exe="/usr/sbin/sshd" >>>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { >>>>>>> read } for pid=12965 comm="sshd" name="authorized_keys" >>>>>>> dev="dm-2" ino=116393517 >>>>>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>>>>> >>>>>>> >>>>>>>> I suspect that the subsystem cert was renewed and >>>>>>>> everything wasn't >>>>>>>> updated properly. If I'm right this is unrelated to the >>>>>>>> upgrade it just >>>>>>>> shone a spotlight on it. >>>>>>>> >>>>>>>> Can you run these commands: >>>>>>>> >>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>>> >>>>>>> >>>>>>> This returns >>>>>>> >>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>> Enter LDAP Password: >>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>>> userCertificate:: >>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>>> ... stuff ... >>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>>> description: 2;4;CN=Certificate >>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS >>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert >>>>>>>> cert-pki-ca' | grep Serial >>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>>> >>>>>>> >>>>>>> These both return >>>>>>> >>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>> 'subsystemCert cert-pki-ca' | grep Serial >>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>> certutil: problem listing keys: >>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>> >>>>>>> >>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>> certutil: Checking token "NSS Certificate DB" in slot "NSS >>>>>>> User Private Key and Certificate Services" >>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>> certutil: problem listing keys: >>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>> >>>>>> Hi, >>>>>> >>>>>> you made a typo and requested the keys (-K) instead of the >>>>>> certs (-L). Please retry with -L and you should be able to >>>>>> see the certificate serial. >>>>>> >>>>> >>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca >>>>> userCertificate description >>>>> Enter LDAP Password: >>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>> userCertificate:: >>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>> ... >>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>>> Subsystem,O=BPT.ROCKS >>>>> >>>>> So the serial is 4? >>>>> >>>>> [root@seattlenfs ianh]# certutil -L -d >>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>>>> grep Serial >>>>> Serial Number: 1341718541 (0x4ff9000d) >>>>> >>>>> Which is NOT 4... >>>>> >>>>> [root@seattlenfs ianh]# certutil -L -d >>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >>>>> -----BEGIN CERTIFICATE----- >>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>> >>>>> ... >>>>> sjeiFbUFtoMnfmHLIYpe5QAR >>>>> -----END CERTIFICATE----- >>>>> >>>>> And this cert is NOT the same. >>>>> >>>> Hi, >>>> >>>> when certmonger renews a certificate, it runs pre-save and >>>> post-save commands (which you can see in the output of >>>> getcert list). In your case, the post-save command for >>>> subsystemCert cert-pki-ca probably failed. >>>> >>>> This command (on the renewal master) is responsible for >>>> copying the new cert from the NSS db to the LDAP server. You >>>> need first to check which IPA server is the renewal master >>>> (as the serial numbers are in a completely different range, I >>>> assume that you have multiple IPA servers configured with a >>>> CA instance): >>>> >>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= >>>> -f2-` >>>> $ kinit admin >>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>>> dn: >>>> cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com >>>> >>>> >>>> In this example the renewal master is vm-ipaserver. >>>> >>> >>> I get this >>> >>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>> dn: >>> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks >>> >>> >>> Which is the other FreeIPA server. >>> >> Hi Ian, >> >> it's getting difficult to follow this thread, so could we step >> back and summarize? > Yes, it is. Sorry. >> >> - On server freeipa-sea (=renewal master), the CA component is >> installed and 'subsystemCert cert-pki-ca' has Serial number 4 >> in the NSSDB /etc/pki/pki-tomcat/alias > No? > > [root@freeipa-sea ianh]# certutil -L -d > /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep > Serial > Serial Number: 1341718541 (0x4ff9000d) >> - On server seattlenfs (where PKI fails to restart), the CA >> component is installed and 'subsystemCert cert-pki-ca' has >> Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias. > Yes > > [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias > -n 'subsystemCert cert-pki-ca' | grep Serial > Serial Number: 1341718541 (0x4ff9000d) > >> >> - In the LDAP servers, the cert is stored in the entry >> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it >> is the cert from freeipa-sea). > This is where the difference is. There are two certificates on > the master and only one on the other. I wonder if this is a > result of the slow motion replication trainwreck I have going on > due to a zombie replica in the ldap database that no longer > exists in the real world and I can't seem to get rid of. > Hi,
the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.
Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?
Flo
[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm...
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry
> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate > description > Enter LDAP Password: > dn: uid=pkidbuser,ou=people,o=ipaca > userCertificate:: > MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > <truncated> > ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= > userCertificate:: > MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK > <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR > description: 2;1341718541;CN=Certificate > Authority,O=BPT.ROCKS;CN=CA Subsystem > ,O=BPT.ROCKS > > > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate > description > Enter LDAP Password: > dn: uid=pkidbuser,ou=people,o=ipaca > userCertificate:: > MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC > <truncated> > ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= > description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA > Subsystem,O=BPT.RO > CKS > >> If you check the certificates dates, I guess that the most >> recent one is on server seattlenfs (because of serial numbers). >> Can you confirm this (because our goal is to have the most >> recent cert stored in all the NSSDBs and in LDAP)? >> >> Thanks, >> Flo >> >> >>>> If your renewal master is the machine on which the NSS DB >>>> /etc/pki/pki-tomcat/alias contains the newer cert, then you >>>> can manually run the scripts: >>> >>> If I'm interpreting this correctly, then it is... so I ran >>> this on the broken machine (seattlenfs) >>> >>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad >>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert >>>> "subsystemCert cert-pki-ca" >>>> >>> >>> I ran this, it took a while, then I re-checked and the serial >>> still seems to be 4 >>> >>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>> Enter LDAP Password: >>> dn: uid=pkidbuser,ou=people,o=ipaca >>> userCertificate:: >>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> ... >>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>> Subsystem,O=BPT.ROCKS >>> >>> Here's what was in /var/log/pki/pki-tomcat/ca/debug >>> >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> ============================================ >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG >>> SUBSYSTEM INITIALIZED ======= >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> ============================================ >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> restart at autoShutdown? false >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> autoShutdown crumb file path? >>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> about to look for cert for auto-shutdown >>> support:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> found cert:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >>> init id=debug >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initialized debug >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initSubsystem id=log >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> ready to init id=log >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) >>> >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> restart at autoShutdown? false >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> autoShutdown crumb file path? >>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> about to look for cert for auto-shutdown >>> support:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> found cert:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >>> init id=log >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initialized log >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initSubsystem id=jss >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> ready to init id=jss >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> restart at autoShutdown? false >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> autoShutdown crumb file path? >>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> about to look for cert for auto-shutdown >>> support:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> found cert:auditSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done >>> init id=jss >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initialized jss >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> initSubsystem id=dbs >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>> ready to init id=dbs >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: >>> init() mEnableSerialMgmt=true >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>> LdapBoundConnFactor(DBSubsystem) >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> LdapBoundConnFactory: init >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> LdapBoundConnFactory:doCloning true >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>> init() >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>> init begins >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>> init ends >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before >>> makeConnection errorIfDown is true >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: >>> errorIfDown true >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: >>> true >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> SSLClientCertificateSelectionCB: Setting desired cert nickname >>> to: subsystemCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> LdapJssSSLSocket: set client auth cert nickname subsystemCert >>> cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> SSLClientCertificatSelectionCB: Entering! >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: >>> ocspSigningCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: >>> subsystemCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> SSLClientCertificateSelectionCB: desired cert found in list: >>> subsystemCert cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> SSLClientCertificateSelectionCB: returning: subsystemCert >>> cert-pki-ca >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake >>> happened >>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>> CMSEngine.shutdown() >>> >>> >>> I really appreciate you sticking with me through this. I owe >>> you. >>> >>> - Ian >>> >>>> After that, check that the LDAP entry has been updated. If it >>>> is the case, you should be able to restart pki. >>>> >>>> Flo >>>> >>>>> Thank you again for your time! >>>>> >>>>> Ian >>>>> >>>>>> Flo. >>>>>>> >>>>>>> which seems weird. >>>>>>> >>>>>>>> The description attribute in the ldapsearch output is of >>>>>>>> the form 2;#;DN;DN >>>>>>>> >>>>>>>> The # should match the serial number in the first >>>>>>>> certutil output >>>>>>>> >>>>>>>> The ASCII blob (minus the BEGIN/END headers) should match >>>>>>>> one of the >>>>>>>> userCertificate attributes. >>>>>>>> >>>>>>>> rob >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
On 8/10/17 11:37 AM, Ian Harding via FreeIPA-users wrote:
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica
- LDAP
error: No such object (connection error)
dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca ,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
So I know I need to ipa-csreplica-manage re-initialize --from freeipa-sea.bpt.rocks on seattlenfs, but also that it fails because of the above.
I think this is the root of the problem where the certificate is not replicated.
Anyone know how I can clean it up? I'm really sorry I've taken up so much of your time. I really appreciate it.
The freeipa-dal problem may or may not be related...
[root@freeipa-sea ianh]# ipa-csreplica-manage list Directory Manager password:
seattlenfs.bpt.rocks: master freeipa-dal.bpt.rocks: CA not configured freeipa-sea.bpt.rocks: master
[root@freeipa-sea ianh]# ipa-csreplica-manage del freeipa-dal.bpt.rocks Directory Manager password:
'freeipa-sea.bpt.rocks' has no replication agreement for 'freeipa-dal.bpt.rocks'
[root@seattlenfs ~]# ipa-csreplica-manage list Directory Manager password:
seattlenfs.bpt.rocks: master freeipa-dal.bpt.rocks: CA not configured freeipa-sea.bpt.rocks: master [root@seattlenfs ~]# ipa-csreplica-manage del freeipa-dal.bpt.rocks Directory Manager password:
'seattlenfs.bpt.rocks' has no replication agreement for 'freeipa-dal.bpt.rocks'
[root@seattlenfs ~]# ipa-replica-manage list-ruv Directory Manager password:
Replica Update Vectors: seattlenfs.bpt.rocks:389: 21 freeipa-sea.bpt.rocks:389: 20 Certificate Server Replica Update Vectors: seattlenfs.bpt.rocks:389: 1290 freeipa-sea.bpt.rocks:389: 1065
Hi Ian
Hmm.. This looks odd.
[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from freeipa-sea.bpt.rocks Directory Manager password:
ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only initializing the first one returned: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o=ipaca,cn=mapping tree,cn=config Update in progress, 15 seconds elapsed [freeipa-sea.bpt.rocks] reports: Update failed! Status: [32 - LDAP error: No such object]
Here there are several replica agreement from freeipa-sea.bpt.rocks-> seattlenfs (for o=ipaca suffix). I am not sure how this can happen. It could be some remaining entries from previous install/uninstall or multiple initialization from different sources.
Looking at the error below, I wonder if there were actually 2 replica agreements freeipa-sea->seattlenfs that are
* cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat * masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat
Would you dump (on all servers freeipa-sea, seattlenfs and freeipa-dal):
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi
best regards thierry
On freeipa-sea and seattlenfs you may check the ipaca replica agreements status before/after reinit. I would guess they are currently reporting failures between these two replicas.
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica
- LDAP
error: No such object (connection error)
dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca ,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
several times over the years when replication has stopped.
Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.
There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error. I don't know if that's related.
Zombie servers have been discussed a lot on this mailing list. You may get rid of them with list-ruv/clean-ruv subcommand. Replication usually manage to work fine even if some zombie entries exist but it is a good practice to clean them.
Yeah, I've been through every possible combination of list-ruv/clean-ruv. I have something odd going on in that no replication agreements show up anywhere for freeipa-dal.
regards thierry
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote: > On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote: >> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote: >> >>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote: >>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: >>>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>>>>>> Ian Harding wrote: >>>>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via >>>>>>>>>>>>> FreeIPA-users wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>>>>>> I had an unexpected restart of an IPA server that >>>>>>>>>>>>>>>> had apparently had >>>>>>>>>>>>>>>> updates run but had not been restarted. ipactl >>>>>>>>>>>>>>>> says pki-tomcatd >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>> not start. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Strangely, the actual service appears to be running: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dogtag is an application within tomcat so tomcat >>>>>>>>>>>>>>> can run without >>>>>>>>>>>>>>> dogtag >>>>>>>>>>>>>>> running. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We need to see more of the dogtag debug log to see >>>>>>>>>>>>>>> what is going on. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> It looks like an authentication problem... >>>>>>>>>>>>>> >>>>>>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>>>>>> handshake happened >>>>>>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>>>>>> failed (49) >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> dogtag stores its internal data in the LDAP server >>>>>>>>>>>>> and needs to >>>>>>>>>>>>> establish a secure LDAP connection. You can check >>>>>>>>>>>>> how this >>>>>>>>>>>>> connection is configured in >>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>>>>>> the lines: >>>>>>>>>>>>> >>>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>>>> cert-pki-ca >>>>>>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>>>>>> >>>>>>>>>>>>> authtype can be SslClientAuth (authentication with a >>>>>>>>>>>>> ssl >>>>>>>>>>>>> certificate) or BasicAuth (authentication with a >>>>>>>>>>>>> bind DN and >>>>>>>>>>>>> password stored in >>>>>>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>>>>>> >>>>>>>>>>>>> You can use this information to manually check the >>>>>>>>>>>>> credentials. For >>>>>>>>>>>>> instance with sslclientauth: >>>>>>>>>>>>> >>>>>>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>>>>>> >>>>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base >>>>>>>>>>>>> -Y EXTERNAL >>>>>>>>>>>>> (provide the password from >>>>>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I found this: >>>>>>>>>>>> >>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>>>>>> >>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>>> cert-pki-ca >>>>>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>>>>>> ... >>>>>>>>>>>> >>>>>>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>>>>>> prompt to provide >>>>>>>>>>>> a pin/password >>>>>>>>>>>> >>>>>>>>>>>> Please enter pin, password, or pass phrase for >>>>>>>>>>>> security token 'ldap(0)': >>>>>>>>>>>> >>>>>>>>>>>> but there is no password file... >>>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> you are right, in 4.4. there is no pwdfile.txt and the >>>>>>>>>>> password can be >>>>>>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf >>>>>>>>>>> (with the tag >>>>>>>>>>> internal=...) >>>>>>>>>>> >>>>>>>>>>> Can you check if the password with the tag >>>>>>>>>>> internal=... allows to read >>>>>>>>>>> the keys from the NSS db? >>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>>> (provide password) >>>>>>>>>> >>>>>>>>>> That works... >>>>>>>>>> >>>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>>>> "NSS User Private >>>>>>>>>> Key and Certificate Services" >>>>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan) >>>>>>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>>>>>> Certificate >>>>>>>>>> DB:auditSigningCert cert-pki-ca >>>>>>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>>>>>> caSigningCert >>>>>>>>>> cert-pki-ca >>>>>>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>>>>>> Certificate >>>>>>>>>> DB:ocspSigningCert cert-pki-ca >>>>>>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f >>>>>>>>>> Server-Cert >>>>>>>>>> cert-pki-ca >>>>>>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>>>>>> Certificate >>>>>>>>>> DB:subsystemCert cert-pki-ca >>>>>>>>>> >>>>>>>>>> But this doesn't (with the same password from >>>>>>>>>> password.conf) >>>>>>>>>> >>>>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y >>>>>>>>>> EXTERNAL >>>>>>>>>> Please enter pin, password, or pass phrase for security >>>>>>>>>> token 'ldap(0)': >>>>>>>>>> SASL/EXTERNAL authentication started >>>>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>>>>>> >>>>>>>>>> That password is getting me somewhere though, since if >>>>>>>>>> I put in a >>>>>>>>>> nonsense or incorrect password it just prompts over and >>>>>>>>>> over. >>>>>>>>> >>>>>>>>> Let's step back a second. You upgraded from what to what? >>>>>>>> >>>>>>>> There wasn't much of a change... I just assumed someone >>>>>>>> ran yum upgrade and didn't restart, then the power >>>>>>>> outage... it looks like not much of a version change though. >>>>>>>> >>>>>>>> # grep ipa /var/log/yum.log >>>>>>>> Jan 08 04:45:32 Installed: >>>>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:45:32 Installed: >>>>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:46:06 Updated: >>>>>>>> libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>>> Jan 08 04:46:07 Updated: >>>>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>>> Jan 08 04:46:08 Installed: >>>>>>>> python-ipaddress-1.0.16-2.el7.noarch >>>>>>>> Jan 08 04:47:04 Installed: >>>>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:05 Installed: >>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:05 Updated: >>>>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:05 Installed: >>>>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:06 Installed: >>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>>>>>> Jan 08 04:47:17 Updated: >>>>>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>>> Jan 08 04:47:23 Updated: >>>>>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>>> Jan 08 04:47:27 Updated: >>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:47:36 Installed: >>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>>>>>> Jan 08 04:50:07 Erased: >>>>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>>>>>> Jul 28 09:55:41 Updated: >>>>>>>> ipa-common-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:42 Updated: >>>>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:43 Updated: >>>>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:43 Updated: >>>>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:44 Updated: >>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:45 Updated: >>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:45 Updated: >>>>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:46 Updated: >>>>>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>>>>>> >>>>>>>> # grep pki /var/log/yum.log >>>>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>>>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>>>>>> Jan 08 04:47:19 Installed: >>>>>>>> pki-base-java-10.3.3-14.el7_3.noarch >>>>>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>>>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>>>>>> Jul 28 10:13:17 Updated: >>>>>>>> pki-base-java-10.3.3-19.el7_3.noarch >>>>>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>>>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Are you running in SELinux enforcing mode? >>>>>>>> >>>>>>>> # getenforce >>>>>>>> Enforcing >>>>>>>> >>>>>>>>> >>>>>>>>> If so can you run: >>>>>>>>> >>>>>>>>> # ausearch -m AVC >>>>>>>>> >>>>>>>> >>>>>>>> # ausearch -m AVC >>>>>>>> The NOLOG option to log_format is deprecated. Please use >>>>>>>> the write_logs option. >>>>>>>> The NOLOG option is overriding the write_logs current >>>>>>>> setting. >>>>>>>> ---- >>>>>>>> time->Mon Mar 14 10:39:01 2016 >>>>>>>> type=SYSCALL msg=audit(1457977141.767:17537): >>>>>>>> arch=c000003e syscall=2 success=no exit=-13 >>>>>>>> a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 >>>>>>>> ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 >>>>>>>> euid=581400001 suid=0 fsuid=581400001 egid=581400001 >>>>>>>> sgid=0 fsgid=581400001 tty=(none) ses=4294967295 >>>>>>>> comm="sshd" exe="/usr/sbin/sshd" >>>>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { >>>>>>>> read } for pid=12965 comm="sshd" name="authorized_keys" >>>>>>>> dev="dm-2" ino=116393517 >>>>>>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>>>>>> >>>>>>>> >>>>>>>>> I suspect that the subsystem cert was renewed and >>>>>>>>> everything wasn't >>>>>>>>> updated properly. If I'm right this is unrelated to the >>>>>>>>> upgrade it just >>>>>>>>> shone a spotlight on it. >>>>>>>>> >>>>>>>>> Can you run these commands: >>>>>>>>> >>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>>>> >>>>>>>> >>>>>>>> This returns >>>>>>>> >>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>>> Enter LDAP Password: >>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>>>> userCertificate:: >>>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>>>> ... stuff ... >>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>>>> description: 2;4;CN=Certificate >>>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS >>>>>>>> >>>>>>>>> and >>>>>>>>> >>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>>>> 'subsystemCert >>>>>>>>> cert-pki-ca' | grep Serial >>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>>>> >>>>>>>> >>>>>>>> These both return >>>>>>>> >>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>>> 'subsystemCert cert-pki-ca' | grep Serial >>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>> certutil: problem listing keys: >>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>>> >>>>>>>> >>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>> "NSS User Private Key and Certificate Services" >>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>> certutil: problem listing keys: >>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> you made a typo and requested the keys (-K) instead of the >>>>>>> certs (-L). Please retry with -L and you should be able to >>>>>>> see the certificate serial. >>>>>>> >>>>>> >>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca >>>>>> userCertificate description >>>>>> Enter LDAP Password: >>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>> userCertificate:: >>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>> ... >>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>>>> Subsystem,O=BPT.ROCKS >>>>>> >>>>>> So the serial is 4? >>>>>> >>>>>> [root@seattlenfs ianh]# certutil -L -d >>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>>>>> grep Serial >>>>>> Serial Number: 1341718541 (0x4ff9000d) >>>>>> >>>>>> Which is NOT 4... >>>>>> >>>>>> [root@seattlenfs ianh]# certutil -L -d >>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >>>>>> -----BEGIN CERTIFICATE----- >>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>> >>>>>> ... >>>>>> sjeiFbUFtoMnfmHLIYpe5QAR >>>>>> -----END CERTIFICATE----- >>>>>> >>>>>> And this cert is NOT the same. >>>>>> >>>>> Hi, >>>>> >>>>> when certmonger renews a certificate, it runs pre-save and >>>>> post-save commands (which you can see in the output of >>>>> getcert list). In your case, the post-save command for >>>>> subsystemCert cert-pki-ca probably failed. >>>>> >>>>> This command (on the renewal master) is responsible for >>>>> copying the new cert from the NSS db to the LDAP server. You >>>>> need first to check which IPA server is the renewal master >>>>> (as the serial numbers are in a completely different range, >>>>> I assume that you have multiple IPA servers configured with >>>>> a CA instance): >>>>> >>>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= >>>>> -f2-` >>>>> $ kinit admin >>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>>>> dn: >>>>> cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com >>>>> >>>>> >>>>> In this example the renewal master is vm-ipaserver. >>>>> >>>> >>>> I get this >>>> >>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>>> dn: >>>> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks >>>> >>>> >>>> Which is the other FreeIPA server. >>>> >>> Hi Ian, >>> >>> it's getting difficult to follow this thread, so could we step >>> back and summarize? >> Yes, it is. Sorry. >>> >>> - On server freeipa-sea (=renewal master), the CA component is >>> installed and 'subsystemCert cert-pki-ca' has Serial number 4 >>> in the NSSDB /etc/pki/pki-tomcat/alias >> No? >> >> [root@freeipa-sea ianh]# certutil -L -d >> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep >> Serial >> Serial Number: 1341718541 (0x4ff9000d) >>> - On server seattlenfs (where PKI fails to restart), the CA >>> component is installed and 'subsystemCert cert-pki-ca' has >>> Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias. >> Yes >> >> [root@seattlenfs ianh]# certutil -L -d >> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep >> Serial >> Serial Number: 1341718541 (0x4ff9000d) >> >>> >>> - In the LDAP servers, the cert is stored in the entry >>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it >>> is the cert from freeipa-sea). >> This is where the difference is. There are two certificates on >> the master and only one on the other. I wonder if this is a >> result of the slow motion replication trainwreck I have going >> on due to a zombie replica in the ldap database that no longer >> exists in the real world and I can't seem to get rid of. >> > Hi, > > the remaining issue is a replication problem. There is a > "Troubleshooting replication" section [1] in IdM Admin guide > that may help you. > > Regarding your zombie replica, which commands did you use to try > to remove it, and what was the output? > > Flo > > [1] > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm... >
Hi,
To debug replication latency or problem we would need several info but starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' > -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0
Regards thierry >> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory >> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >> description >> Enter LDAP Password: >> dn: uid=pkidbuser,ou=people,o=ipaca >> userCertificate:: >> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> <truncated> >> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >> userCertificate:: >> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK >> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR >> description: 2;1341718541;CN=Certificate >> Authority,O=BPT.ROCKS;CN=CA Subsystem >> ,O=BPT.ROCKS >> >> >> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >> description >> Enter LDAP Password: >> dn: uid=pkidbuser,ou=people,o=ipaca >> userCertificate:: >> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >> <truncated> >> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >> Subsystem,O=BPT.RO >> CKS >> >>> If you check the certificates dates, I guess that the most >>> recent one is on server seattlenfs (because of serial >>> numbers). Can you confirm this (because our goal is to have >>> the most recent cert stored in all the NSSDBs and in LDAP)? >>> >>> Thanks, >>> Flo >>> >>> >>>>> If your renewal master is the machine on which the NSS DB >>>>> /etc/pki/pki-tomcat/alias contains the newer cert, then you >>>>> can manually run the scripts: >>>> >>>> If I'm interpreting this correctly, then it is... so I ran >>>> this on the broken machine (seattlenfs) >>>> >>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad >>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert >>>>> "subsystemCert cert-pki-ca" >>>>> >>>> >>>> I ran this, it took a while, then I re-checked and the serial >>>> still seems to be 4 >>>> >>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>> Enter LDAP Password: >>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>> userCertificate:: >>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>> ... >>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>> Subsystem,O=BPT.ROCKS >>>> >>>> Here's what was in /var/log/pki/pki-tomcat/ca/debug >>>> >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> ============================================ >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG >>>> SUBSYSTEM INITIALIZED ======= >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> ============================================ >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> restart at autoShutdown? false >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> autoShutdown crumb file path? >>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> about to look for cert for auto-shutdown >>>> support:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> found cert:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> done init id=debug >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initialized debug >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=log >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> ready to init id=log >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) >>>> >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> restart at autoShutdown? false >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> autoShutdown crumb file path? >>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> about to look for cert for auto-shutdown >>>> support:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> found cert:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> done init id=log >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initialized log >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=jss >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> ready to init id=jss >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> restart at autoShutdown? false >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> autoShutdown crumb file path? >>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> about to look for cert for auto-shutdown >>>> support:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> found cert:auditSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> done init id=jss >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initialized jss >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> initSubsystem id=dbs >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>> ready to init id=dbs >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: >>>> init() mEnableSerialMgmt=true >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>> LdapBoundConnFactor(DBSubsystem) >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> LdapBoundConnFactory: init >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> LdapBoundConnFactory:doCloning true >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>> init() >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>> init begins >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>> init ends >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before >>>> makeConnection errorIfDown is true >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> makeConnection: errorIfDown true >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP >>>> Keep-Alive: true >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> SSLClientCertificateSelectionCB: Setting desired cert >>>> nickname to: subsystemCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> LdapJssSSLSocket: set client auth cert nickname subsystemCert >>>> cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> SSLClientCertificatSelectionCB: Entering! >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate >>>> cert: ocspSigningCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate >>>> cert: subsystemCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> SSLClientCertificateSelectionCB: desired cert found in list: >>>> subsystemCert cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> SSLClientCertificateSelectionCB: returning: subsystemCert >>>> cert-pki-ca >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake >>>> happened >>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>> CMSEngine.shutdown() >>>> >>>> >>>> I really appreciate you sticking with me through this. I owe >>>> you. >>>> >>>> - Ian >>>> >>>>> After that, check that the LDAP entry has been updated. If >>>>> it is the case, you should be able to restart pki. >>>>> >>>>> Flo >>>>> >>>>>> Thank you again for your time! >>>>>> >>>>>> Ian >>>>>> >>>>>>> Flo. >>>>>>>> >>>>>>>> which seems weird. >>>>>>>> >>>>>>>>> The description attribute in the ldapsearch output is of >>>>>>>>> the form 2;#;DN;DN >>>>>>>>> >>>>>>>>> The # should match the serial number in the first >>>>>>>>> certutil output >>>>>>>>> >>>>>>>>> The ASCII blob (minus the BEGIN/END headers) should >>>>>>>>> match one of the >>>>>>>>> userCertificate attributes. >>>>>>>>> >>>>>>>>> rob >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org > To unsubscribe send an email to > freeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
On 8/16/17 5:24 AM, thierry bordaz via FreeIPA-users wrote:
Hi Ian
Hmm.. This looks odd.
[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from freeipa-sea.bpt.rocks Directory Manager password:
ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only initializing the first one returned: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o=ipaca,cn=mapping tree,cn=config Update in progress, 15 seconds elapsed [freeipa-sea.bpt.rocks] reports: Update failed! Status: [32 - LDAP error: No such object]
Here there are several replica agreement from freeipa-sea.bpt.rocks-> seattlenfs (for o=ipaca suffix). I am not sure how this can happen. It could be some remaining entries from previous install/uninstall or multiple initialization from different sources.
If it's possible to do something wrong I am sure I have done it. I have tried the various clean-ruv commands for the apparently existing agreements with nonexistent servers many times but it seems ot have no effect.
Looking at the error below, I wonder if there were actually 2 replica agreements freeipa-sea->seattlenfs that are
- cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat
- masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat
Would you dump (on all servers freeipa-sea, seattlenfs and freeipa-dal):
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi
freeipa-dal no longer exists. Also, since they also appear below... neither do edinburghnfs, fremontnis, bellevuenfs or bpt-nyc1-nfs. The one causing me the most obvious headaches is freeipa-dal.
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: cn: replica nscpentrywsi: createTimestamp: 20160814234939Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: modifyTimestamp: 20170816141634Z nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freei pa-sea.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seat tlenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaId: 1065 nscpentrywsi: nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsState:: KQQAAAAAAADAU5RZAAAAAAAAAAAAAAAAKgAAAAAAAAAAAAAAAAAAAA == nscpentrywsi: nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts, cn=etc,dc=bpt,dc=rocks nscpentrywsi: nsds5replicabinddngroupcheckinterval: 60 nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: numSubordinates: 2 nscpentrywsi: nsds5ReplicaChangeCount: 961 nscpentrywsi: nsds5replicareapactive: 0
dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nscpentrywsi: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=repli ca,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: cn: cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat nscpentrywsi: createTimestamp: 20160814234940Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description: cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: modifyTimestamp: 20170810183437Z nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-free ipa-sea.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindMethod: Simple nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ0da bE1zY2Nod0FQeFlXMkF0Y3RhTA==}AIuTmb+o3/MWgo0mc8r8fw== nscpentrywsi: nsDS5ReplicaHost: seattlenfs.bpt.rocks nscpentrywsi: nsDS5ReplicaPort: 389 nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaTransportInfo: TLS nscpentrywsi: nsds50ruv: {replicageneration} 55c8f3ae000000600000 nscpentrywsi: nsds50ruv: {replica 81 ldap://seattlenfs.bpt.rocks:389} 568ac431 000000510000 57b103ce000500510000 nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b10 3d4000004290000 57b254df000004290000 nscpentrywsi: nsds50ruv: {replica 1095 ldap://freeipa-sea.bpt.rocks:389} 579a9 63c000004470000 57a42389000104470000 nscpentrywsi: nsds50ruv: {replica 96 ldap://freeipa-sea.bpt.rocks:389} 55c8f3b d000000600000 5799a02e000000600000 nscpentrywsi: nsds5ReplicaEnabled: on nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://seattlenfs.bpt.rock s:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1095 ldap://freeipa-sea.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://freeipa-sea.bpt.roc ks:389} 00000000 nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsds5replicationagreement nscpentrywsi: nsds5replicareapactive: 0 nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z nscpentrywsi: nsds5replicaChangesSentSinceStartup: nscpentrywsi: nsds5replicaLastUpdateStatus: Error (32) Problem connecting to r eplica - LDAP error: No such object (connection error) nscpentrywsi: nsds5replicaUpdateInProgress: FALSE nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z
dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nscpentrywsi: dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=repli ca,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: cn: masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat nscpentrywsi: createTimestamp: 20160825052011Z nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: description: masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: modifyTimestamp: 20170706211220Z nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seatt lenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindMethod: Simple nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQnZ6 Z2tGUzErSjlvdEVYRlFJaVQyMA==}yhEAKKElqudqql31/kWsUw== nscpentrywsi: nsDS5ReplicaHost: seattlenfs.bpt.rocks nscpentrywsi: nsDS5ReplicaPort: 389 nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaTransportInfo: TLS nscpentrywsi: nsds50ruv: {replicageneration} 55c8f3ae000000600000 nscpentrywsi: nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be80 4c0000050a0000 587236150000050a0000 nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b10 3d4000004290000 57be8043000704290000 nscpentrywsi: nsds50ruv: {replica 1070 ldap://bellevuenfs.bpt.rocks:389} 57a4f 2700000042e0000 57a4f2700007042e0000 nscpentrywsi: nsds50ruv: {replica 1075 ldap://bpt-nyc1-nfs.bpt.rocks:389} 57a4 7865000004330000 57b111c8000404330000 nscpentrywsi: nsds50ruv: {replica 1085 ldap://fremontnis.bpt.rocks:389} 57a403 e60000043d0000 57a643ea0000043d0000 nscpentrywsi: nsds50ruv: {replica 1090 ldap://freeipa-dal.bpt.rocks:389} 57a2d d35000004420000 57a2dd35000404420000 nscpentrywsi: nsds50ruv: {replica 1195 ldap://edinburghnfs.bpt.rocks:389} 57a4 2390000004ab0000 57a42390000404ab0000 nscpentrywsi: nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.ro cks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1070 ldap://bellevuenfs.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1075 ldap://bpt-nyc1-nfs.bpt. rocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1085 ldap://fremontnis.bpt.ro cks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1090 ldap://freeipa-dal.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1195 ldap://edinburghnfs.bpt. rocks:389} 00000000 nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsds5replicationagreement nscpentrywsi: nsds5replicareapactive: 0 nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z nscpentrywsi: nsds5replicaChangesSentSinceStartup: nscpentrywsi: nsds5replicaLastUpdateStatus: Error (19) Replication error acqui ring replica: Replica has different database generation ID, remote replica ma y need to be initialized (RUV error) nscpentrywsi: nsds5replicaUpdateInProgress: FALSE nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z
##################
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: cn: replica nscpentrywsi: createTimestamp: 20160825052011Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c onfig nscpentrywsi: modifyTimestamp: 20170810182749Z nscpentrywsi: nsDS5Flags: 1 nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seatt lenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaId: 1290 nscpentrywsi: nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaType: 3 nscpentrywsi: nsState:: CgUAAAAAAACkpYxZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA == nscpentrywsi: nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts, cn=etc,dc=bpt,dc=rocks nscpentrywsi: nsds5replicabinddngroupcheckinterval: 60 nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsDS5Replica nscpentrywsi: objectClass: extensibleobject nscpentrywsi: numSubordinates: 1 nscpentrywsi: nsds5ReplicaChangeCount: 3 nscpentrywsi: nsds5replicareapactive: 0
dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca ,cn=mapping tree,cn=config nscpentrywsi: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replic a,cn=o\3Dipaca,cn=mapping tree,cn=config nscpentrywsi: cn: cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat nscpentrywsi: createTimestamp: 20160825052011Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description: cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: modifyTimestamp: 20170810183422Z nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seat tlenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config nscpentrywsi: nsDS5ReplicaBindMethod: Simple nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQzZI d20zcHNMYTZKT3dQVDg4RW9XRA==}PpeZp0WYyNiA7DWqrsJfBw== nscpentrywsi: nsDS5ReplicaHost: freeipa-sea.bpt.rocks nscpentrywsi: nsDS5ReplicaPort: 389 nscpentrywsi: nsDS5ReplicaRoot: o=ipaca nscpentrywsi: nsDS5ReplicaTransportInfo: TLS nscpentrywsi: nsds50ruv: {replicageneration} 57c291d9000004290000 nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f84 0bf000004290000 598ca31f000104290000 nscpentrywsi: nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r ocks:389} 00000000 nscpentrywsi: nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.ro cks:389} 00000000 nscpentrywsi: objectClass: top nscpentrywsi: objectClass: nsds5replicationagreement nscpentrywsi: nsds5ReplicaEnabled: on nscpentrywsi: nsds5replicareapactive: 0 nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z nscpentrywsi: nsds5replicaChangesSentSinceStartup: nscpentrywsi: nsds5replicaLastUpdateStatus: Error (19) Replication error acqui ring replica: Replica has different database generation ID, remote replica ma y need to be initialized (RUV error) nscpentrywsi: nsds5replicaUpdateInProgress: FALSE nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z
best regards thierry
On freeipa-sea and seattlenfs you may check the ipaca replica agreements status before/after reinit. I would guess they are currently reporting failures between these two replicas.
ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica - LDAP error: No such object (connection error)
dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac a,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus Enter LDAP Password: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca ,cn=mapping tree,cn=config nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica: Replica has different database generation ID, remote replica may need to be i nitialized (RUV error)
several times over the years when replication has stopped.
Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.
There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error. I don't know if that's related.
Zombie servers have been discussed a lot on this mailing list. You may get rid of them with list-ruv/clean-ruv subcommand. Replication usually manage to work fine even if some zombie entries exist but it is a good practice to clean them.
Yeah, I've been through every possible combination of list-ruv/clean-ruv. I have something odd going on in that no replication agreements show up anywhere for freeipa-dal.
regards thierry
On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
On 8/7/17 1:44 AM, thierry bordaz wrote:
> > > On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users > wrote: >> On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote: >>> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote: >>> >>>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote: >>>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote: >>>>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote: >>>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote: >>>>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote: >>>>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote: >>>>>>>>>> Ian Harding wrote: >>>>>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote: >>>>>>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote: >>>>>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via >>>>>>>>>>>>>> FreeIPA-users wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote: >>>>>>>>>>>>>>>> Ian Harding via FreeIPA-users wrote: >>>>>>>>>>>>>>>>> I had an unexpected restart of an IPA server >>>>>>>>>>>>>>>>> that had apparently had >>>>>>>>>>>>>>>>> updates run but had not been restarted. ipactl >>>>>>>>>>>>>>>>> says pki-tomcatd >>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>> not start. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Strangely, the actual service appears to be >>>>>>>>>>>>>>>>> running: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dogtag is an application within tomcat so tomcat >>>>>>>>>>>>>>>> can run without >>>>>>>>>>>>>>>> dogtag >>>>>>>>>>>>>>>> running. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We need to see more of the dogtag debug log to >>>>>>>>>>>>>>>> see what is going on. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It looks like an authentication problem... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL >>>>>>>>>>>>>>> handshake happened >>>>>>>>>>>>>>> Could not connect to LDAP server host >>>>>>>>>>>>>>> seattlenfs.bpt.rocks port 636 >>>>>>>>>>>>>>> Error netscape.ldap.LDAPException: Authentication >>>>>>>>>>>>>>> failed (49) >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> dogtag stores its internal data in the LDAP server >>>>>>>>>>>>>> and needs to >>>>>>>>>>>>>> establish a secure LDAP connection. You can check >>>>>>>>>>>>>> how this >>>>>>>>>>>>>> connection is configured in >>>>>>>>>>>>>> /etc/pki/pki-tomcat/ca/CS.cfg, look for >>>>>>>>>>>>>> the lines: >>>>>>>>>>>>>> >>>>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager >>>>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>>>>> cert-pki-ca >>>>>>>>>>>>>> internaldb.ldapconn.host=vm-... >>>>>>>>>>>>>> internaldb.ldapconn.port=636 >>>>>>>>>>>>>> internaldb.ldapconn.secureConn >>>>>>>>>>>>>> >>>>>>>>>>>>>> authtype can be SslClientAuth (authentication with >>>>>>>>>>>>>> a ssl >>>>>>>>>>>>>> certificate) or BasicAuth (authentication with a >>>>>>>>>>>>>> bind DN and >>>>>>>>>>>>>> password stored in >>>>>>>>>>>>>> /var/lib/pki/pki-tomcat/conf/password.conf). >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can use this information to manually check the >>>>>>>>>>>>>> credentials. For >>>>>>>>>>>>>> instance with sslclientauth: >>>>>>>>>>>>>> >>>>>>>>>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias >>>>>>>>>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca' >>>>>>>>>>>>>> >>>>>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base >>>>>>>>>>>>>> -Y EXTERNAL >>>>>>>>>>>>>> (provide the password from >>>>>>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt) >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I found this: >>>>>>>>>>>>> >>>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth >>>>>>>>>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca >>>>>>>>>>>>> >>>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb >>>>>>>>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert >>>>>>>>>>>>> cert-pki-ca >>>>>>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389 >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> and when I try the ldapsearch I am presented with a >>>>>>>>>>>>> prompt to provide >>>>>>>>>>>>> a pin/password >>>>>>>>>>>>> >>>>>>>>>>>>> Please enter pin, password, or pass phrase for >>>>>>>>>>>>> security token 'ldap(0)': >>>>>>>>>>>>> >>>>>>>>>>>>> but there is no password file... >>>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> you are right, in 4.4. there is no pwdfile.txt and >>>>>>>>>>>> the password can be >>>>>>>>>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf >>>>>>>>>>>> (with the tag >>>>>>>>>>>> internal=...) >>>>>>>>>>>> >>>>>>>>>>>> Can you check if the password with the tag >>>>>>>>>>>> internal=... allows to read >>>>>>>>>>>> the keys from the NSS db? >>>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>>>> (provide password) >>>>>>>>>>> >>>>>>>>>>> That works... >>>>>>>>>>> >>>>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias >>>>>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>>>>> "NSS User Private >>>>>>>>>>> Key and Certificate Services" >>>>>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>>>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 >>>>>>>>>>> (orphan) >>>>>>>>>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS >>>>>>>>>>> Certificate >>>>>>>>>>> DB:auditSigningCert cert-pki-ca >>>>>>>>>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e >>>>>>>>>>> caSigningCert >>>>>>>>>>> cert-pki-ca >>>>>>>>>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS >>>>>>>>>>> Certificate >>>>>>>>>>> DB:ocspSigningCert cert-pki-ca >>>>>>>>>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f >>>>>>>>>>> Server-Cert >>>>>>>>>>> cert-pki-ca >>>>>>>>>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS >>>>>>>>>>> Certificate >>>>>>>>>>> DB:subsystemCert cert-pki-ca >>>>>>>>>>> >>>>>>>>>>> But this doesn't (with the same password from >>>>>>>>>>> password.conf) >>>>>>>>>>> >>>>>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base >>>>>>>>>>> -Y EXTERNAL >>>>>>>>>>> Please enter pin, password, or pass phrase for >>>>>>>>>>> security token 'ldap(0)': >>>>>>>>>>> SASL/EXTERNAL authentication started >>>>>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49) >>>>>>>>>>> >>>>>>>>>>> That password is getting me somewhere though, since if >>>>>>>>>>> I put in a >>>>>>>>>>> nonsense or incorrect password it just prompts over >>>>>>>>>>> and over. >>>>>>>>>> >>>>>>>>>> Let's step back a second. You upgraded from what to what? >>>>>>>>> >>>>>>>>> There wasn't much of a change... I just assumed someone >>>>>>>>> ran yum upgrade and didn't restart, then the power >>>>>>>>> outage... it looks like not much of a version change >>>>>>>>> though. >>>>>>>>> >>>>>>>>> # grep ipa /var/log/yum.log >>>>>>>>> Jan 08 04:45:32 Installed: >>>>>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:45:32 Installed: >>>>>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:46:06 Updated: >>>>>>>>> libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>>>> Jan 08 04:46:07 Updated: >>>>>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64 >>>>>>>>> Jan 08 04:46:08 Installed: >>>>>>>>> python-ipaddress-1.0.16-2.el7.noarch >>>>>>>>> Jan 08 04:47:04 Installed: >>>>>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:05 Installed: >>>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:05 Updated: >>>>>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:05 Installed: >>>>>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:06 Installed: >>>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64 >>>>>>>>> Jan 08 04:47:17 Updated: >>>>>>>>> ipa-client-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>>>> Jan 08 04:47:23 Updated: >>>>>>>>> ipa-server-4.4.0-14.el7.centos.1.1.x86_64 >>>>>>>>> Jan 08 04:47:27 Updated: >>>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:47:36 Installed: >>>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch >>>>>>>>> Jan 08 04:50:07 Erased: >>>>>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64 >>>>>>>>> Jul 28 09:55:41 Updated: >>>>>>>>> ipa-common-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:42 Updated: >>>>>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:43 Updated: >>>>>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:43 Updated: >>>>>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:44 Updated: >>>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:45 Updated: >>>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:45 Updated: >>>>>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:46 Updated: >>>>>>>>> ipa-client-4.4.0-14.el7.centos.7.x86_64 >>>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>>> ipa-server-4.4.0-14.el7.centos.7.x86_64 >>>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch >>>>>>>>> Jul 28 09:55:48 Updated: >>>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch >>>>>>>>> >>>>>>>>> # grep pki /var/log/yum.log >>>>>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch >>>>>>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64 >>>>>>>>> Jan 08 04:47:19 Installed: >>>>>>>>> pki-base-java-10.3.3-14.el7_3.noarch >>>>>>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64 >>>>>>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch >>>>>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch >>>>>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch >>>>>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch >>>>>>>>> Jul 28 10:13:17 Updated: >>>>>>>>> pki-base-java-10.3.3-19.el7_3.noarch >>>>>>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64 >>>>>>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch >>>>>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch >>>>>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Are you running in SELinux enforcing mode? >>>>>>>>> >>>>>>>>> # getenforce >>>>>>>>> Enforcing >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If so can you run: >>>>>>>>>> >>>>>>>>>> # ausearch -m AVC >>>>>>>>>> >>>>>>>>> >>>>>>>>> # ausearch -m AVC >>>>>>>>> The NOLOG option to log_format is deprecated. Please use >>>>>>>>> the write_logs option. >>>>>>>>> The NOLOG option is overriding the write_logs current >>>>>>>>> setting. >>>>>>>>> ---- >>>>>>>>> time->Mon Mar 14 10:39:01 2016 >>>>>>>>> type=SYSCALL msg=audit(1457977141.767:17537): >>>>>>>>> arch=c000003e syscall=2 success=no exit=-13 >>>>>>>>> a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 >>>>>>>>> ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 >>>>>>>>> euid=581400001 suid=0 fsuid=581400001 egid=581400001 >>>>>>>>> sgid=0 fsgid=581400001 tty=(none) ses=4294967295 >>>>>>>>> comm="sshd" exe="/usr/sbin/sshd" >>>>>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null) >>>>>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { >>>>>>>>> read } for pid=12965 comm="sshd" name="authorized_keys" >>>>>>>>> dev="dm-2" ino=116393517 >>>>>>>>> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 >>>>>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file >>>>>>>>> >>>>>>>>> >>>>>>>>>> I suspect that the subsystem cert was renewed and >>>>>>>>>> everything wasn't >>>>>>>>>> updated properly. If I'm right this is unrelated to the >>>>>>>>>> upgrade it just >>>>>>>>>> shone a spotlight on it. >>>>>>>>>> >>>>>>>>>> Can you run these commands: >>>>>>>>>> >>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate >>>>>>>>>> description >>>>>>>>>> >>>>>>>>> >>>>>>>>> This returns >>>>>>>>> >>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>>>>>> Enter LDAP Password: >>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>>>>> userCertificate:: >>>>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>>>>> >>>>>>>>> ... stuff ... >>>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>>>>> description: 2;4;CN=Certificate >>>>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS >>>>>>>>> >>>>>>>>>> and >>>>>>>>>> >>>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>>>>> 'subsystemCert >>>>>>>>>> cert-pki-ca' | grep Serial >>>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n >>>>>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>>>>> >>>>>>>>> >>>>>>>>> These both return >>>>>>>>> >>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>>>> 'subsystemCert cert-pki-ca' | grep Serial >>>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>>> certutil: problem listing keys: >>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>>>> >>>>>>>>> >>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n >>>>>>>>> 'subsystemCert cert-pki-ca' -a >>>>>>>>> certutil: Checking token "NSS Certificate DB" in slot >>>>>>>>> "NSS User Private Key and Certificate Services" >>>>>>>>> Enter Password or Pin for "NSS Certificate DB": >>>>>>>>> certutil: problem listing keys: >>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier. >>>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> you made a typo and requested the keys (-K) instead of >>>>>>>> the certs (-L). Please retry with -L and you should be >>>>>>>> able to see the certificate serial. >>>>>>>> >>>>>>> >>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca >>>>>>> userCertificate description >>>>>>> Enter LDAP Password: >>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>>>> userCertificate:: >>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>>> ... >>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>>>> description: 2;4;CN=Certificate >>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS >>>>>>> >>>>>>> So the serial is 4? >>>>>>> >>>>>>> [root@seattlenfs ianh]# certutil -L -d >>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>>>>>> grep Serial >>>>>>> Serial Number: 1341718541 (0x4ff9000d) >>>>>>> >>>>>>> Which is NOT 4... >>>>>>> >>>>>>> [root@seattlenfs ianh]# certutil -L -d >>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a >>>>>>> -----BEGIN CERTIFICATE----- >>>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>>>> >>>>>>> ... >>>>>>> sjeiFbUFtoMnfmHLIYpe5QAR >>>>>>> -----END CERTIFICATE----- >>>>>>> >>>>>>> And this cert is NOT the same. >>>>>>> >>>>>> Hi, >>>>>> >>>>>> when certmonger renews a certificate, it runs pre-save and >>>>>> post-save commands (which you can see in the output of >>>>>> getcert list). In your case, the post-save command for >>>>>> subsystemCert cert-pki-ca probably failed. >>>>>> >>>>>> This command (on the renewal master) is responsible for >>>>>> copying the new cert from the NSS db to the LDAP server. >>>>>> You need first to check which IPA server is the renewal >>>>>> master (as the serial numbers are in a completely different >>>>>> range, I assume that you have multiple IPA servers >>>>>> configured with a CA instance): >>>>>> >>>>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut >>>>>> -d= -f2-` >>>>>> $ kinit admin >>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>>>>> dn: >>>>>> cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com >>>>>> >>>>>> >>>>>> In this example the renewal master is vm-ipaserver. >>>>>> >>>>> >>>>> I get this >>>>> >>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b >>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN" >>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn >>>>> dn: >>>>> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks >>>>> >>>>> >>>>> Which is the other FreeIPA server. >>>>> >>>> Hi Ian, >>>> >>>> it's getting difficult to follow this thread, so could we >>>> step back and summarize? >>> Yes, it is. Sorry. >>>> >>>> - On server freeipa-sea (=renewal master), the CA component >>>> is installed and 'subsystemCert cert-pki-ca' has Serial >>>> number 4 in the NSSDB /etc/pki/pki-tomcat/alias >>> No? >>> >>> [root@freeipa-sea ianh]# certutil -L -d >>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>> grep Serial >>> Serial Number: 1341718541 (0x4ff9000d) >>>> - On server seattlenfs (where PKI fails to restart), the CA >>>> component is installed and 'subsystemCert cert-pki-ca' has >>>> Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias. >>> Yes >>> >>> [root@seattlenfs ianh]# certutil -L -d >>> /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | >>> grep Serial >>> Serial Number: 1341718541 (0x4ff9000d) >>> >>>> >>>> - In the LDAP servers, the cert is stored in the entry >>>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it >>>> is the cert from freeipa-sea). >>> This is where the difference is. There are two certificates >>> on the master and only one on the other. I wonder if this is >>> a result of the slow motion replication trainwreck I have >>> going on due to a zombie replica in the ldap database that no >>> longer exists in the real world and I can't seem to get rid of. >>> >> Hi, >> >> the remaining issue is a replication problem. There is a >> "Troubleshooting replication" section [1] in IdM Admin guide >> that may help you. >> >> Regarding your zombie replica, which commands did you use to >> try to remove it, and what was the output? >> >> Flo >> >> [1] >> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/htm... >> > > Hi, > > To debug replication latency or problem we would need several > info but starting with the results of the following commands > > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005: 75274897 nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042 90004: 20170608055334Z nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290 003: cn=Directory Manager nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: cn: pkidbuser nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: mail: nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI <snip> AR nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: parentid: 2 nscpentrywsi: entryid: 51 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi Enter LDAP Password: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs ystem,O=BPT.ROCKS nscpentrywsi: entryid: 51 nscpentrywsi: parentid: 2 nscpentrywsi: modifyTimestamp: 20150716125146Z nscpentrywsi: createTimestamp: 20150716125146Z nscpentrywsi: modifiersName: cn=directory manager nscpentrywsi: creatorsName: cn=directory manager nscpentrywsi: userPassword:: <snip> nscpentrywsi: userstate: 1 nscpentrywsi: usertype: agentType nscpentrywsi: mail: nscpentrywsi: cn: pkidbuser nscpentrywsi: sn: pkidbuser nscpentrywsi: uid: pkidbuser nscpentrywsi: objectClass: top nscpentrywsi: objectClass: person nscpentrywsi: objectClass: organizationalPerson nscpentrywsi: objectClass: inetOrgPerson nscpentrywsi: objectClass: cmsuser nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074 nscpentrywsi: entryusn: 86 > > From the output you may only copy/past the ones related to > userCertificate > > > Also to dump the current replication status, send the result of > [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc ks-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1065 nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 57c291d9000004290000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000 0 598a1c40000104290000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat tlenfs.bpt.rocks;389;unavailable nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a 1c16 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsds5ReplicaChangeCount: 885 nsds5replicareapactive: 0 > and > [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory > manager' -W -b > "o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" Enter LDAP Password: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock s-pki-tomcat,ou=csusers,cn=config nsDS5ReplicaId: 1290 nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323 nsDS5ReplicaRoot: o=ipaca nsDS5ReplicaType: 3 nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA== nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt, dc=rocks nsds5replicabinddngroupcheckinterval: 60 objectClass: top objectClass: nsDS5Replica objectClass: extensibleobject nsds50ruv: {replicageneration} 55c8f3ae000000600000 nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000 587236150000050a0000 nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000 0 57be8043000704290000 nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei pa-sea.bpt.rocks;389;1065;587236150000050a0000 nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000 000 nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000 0000 nsds5ReplicaChangeCount: 3 nsds5replicareapactive: 0 > > Regards > thierry >>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory >>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >>> description >>> Enter LDAP Password: >>> dn: uid=pkidbuser,ou=people,o=ipaca >>> userCertificate:: >>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> <truncated> >>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>> userCertificate:: >>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK >>> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR >>> description: 2;1341718541;CN=Certificate >>> Authority,O=BPT.ROCKS;CN=CA Subsystem >>> ,O=BPT.ROCKS >>> >>> >>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory >>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate >>> description >>> Enter LDAP Password: >>> dn: uid=pkidbuser,ou=people,o=ipaca >>> userCertificate:: >>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>> <truncated> >>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>> Subsystem,O=BPT.RO >>> CKS >>> >>>> If you check the certificates dates, I guess that the most >>>> recent one is on server seattlenfs (because of serial >>>> numbers). Can you confirm this (because our goal is to have >>>> the most recent cert stored in all the NSSDBs and in LDAP)? >>>> >>>> Thanks, >>>> Flo >>>> >>>> >>>>>> If your renewal master is the machine on which the NSS DB >>>>>> /etc/pki/pki-tomcat/alias contains the newer cert, then you >>>>>> can manually run the scripts: >>>>> >>>>> If I'm interpreting this correctly, then it is... so I ran >>>>> this on the broken machine (seattlenfs) >>>>> >>>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad >>>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert >>>>>> "subsystemCert cert-pki-ca" >>>>>> >>>>> >>>>> I ran this, it took a while, then I re-checked and the >>>>> serial still seems to be 4 >>>>> >>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b >>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description >>>>> Enter LDAP Password: >>>>> dn: uid=pkidbuser,ou=people,o=ipaca >>>>> userCertificate:: >>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC >>>>> ... >>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc= >>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA >>>>> Subsystem,O=BPT.ROCKS >>>>> >>>>> Here's what was in /var/log/pki/pki-tomcat/ca/debug >>>>> >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> ============================================ >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG >>>>> SUBSYSTEM INITIALIZED ======= >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> ============================================ >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> restart at autoShutdown? false >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> autoShutdown crumb file path? >>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> about to look for cert for auto-shutdown >>>>> support:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> found cert:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> done init id=debug >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initialized debug >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initSubsystem id=log >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> ready to init id=log >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit) >>>>> >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system) >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions) >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> restart at autoShutdown? false >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> autoShutdown crumb file path? >>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> about to look for cert for auto-shutdown >>>>> support:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> found cert:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> done init id=log >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initialized log >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initSubsystem id=jss >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> ready to init id=jss >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> restart at autoShutdown? false >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> autoShutdown crumb file path? >>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> about to look for cert for auto-shutdown >>>>> support:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> found cert:auditSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> done init id=jss >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initialized jss >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> initSubsystem id=dbs >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: >>>>> ready to init id=dbs >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: >>>>> init() mEnableSerialMgmt=true >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating >>>>> LdapBoundConnFactor(DBSubsystem) >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> LdapBoundConnFactory: init >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> LdapBoundConnFactory:doCloning true >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>>> init() >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>>> init begins >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: >>>>> init ends >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before >>>>> makeConnection errorIfDown is true >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> makeConnection: errorIfDown true >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP >>>>> Keep-Alive: true >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> SSLClientCertificateSelectionCB: Setting desired cert >>>>> nickname to: subsystemCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> LdapJssSSLSocket: set client auth cert nickname >>>>> subsystemCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> SSLClientCertificatSelectionCB: Entering! >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate >>>>> cert: ocspSigningCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate >>>>> cert: subsystemCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> SSLClientCertificateSelectionCB: desired cert found in list: >>>>> subsystemCert cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> SSLClientCertificateSelectionCB: returning: subsystemCert >>>>> cert-pki-ca >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake >>>>> happened >>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: >>>>> CMSEngine.shutdown() >>>>> >>>>> >>>>> I really appreciate you sticking with me through this. I >>>>> owe you. >>>>> >>>>> - Ian >>>>> >>>>>> After that, check that the LDAP entry has been updated. If >>>>>> it is the case, you should be able to restart pki. >>>>>> >>>>>> Flo >>>>>> >>>>>>> Thank you again for your time! >>>>>>> >>>>>>> Ian >>>>>>> >>>>>>>> Flo. >>>>>>>>> >>>>>>>>> which seems weird. >>>>>>>>> >>>>>>>>>> The description attribute in the ldapsearch output is >>>>>>>>>> of the form 2;#;DN;DN >>>>>>>>>> >>>>>>>>>> The # should match the serial number in the first >>>>>>>>>> certutil output >>>>>>>>>> >>>>>>>>>> The ASCII blob (minus the BEGIN/END headers) should >>>>>>>>>> match one of the >>>>>>>>>> userCertificate attributes. >>>>>>>>>> >>>>>>>>>> rob >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> _______________________________________________ >>> FreeIPA-users mailing list -- >>> freeipa-users@lists.fedorahosted.org >>> To unsubscribe send an email to >>> freeipa-users-leave@lists.fedorahosted.org >> _______________________________________________ >> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org >> To unsubscribe send an email to >> freeipa-users-leave@lists.fedorahosted.org >
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list --freeipa-users@lists.fedorahosted.org To unsubscribe send an email tofreeipa-users-leave@lists.fedorahosted.org
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
freeipa-users@lists.fedorahosted.org
-
Florence Blanc-Renaud -
Ian Harding -
Petr Vobornik -
Rob Crittenden -
thierry bordaz