Failed to start pki-tomcatd Service - javax.ws.rs.ServiceUnavailableException: Subsystem unavailable

[lejeczek]

hi guys,

I fail to troubleshoot this here:

$ ipactl start --ignore-service-failures
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Starting httpd Service
Starting ipa-custodia Service
Starting ntpd Service
Starting pki-tomcatd Service
Failed to start pki-tomcatd Service
Forced start, ignoring pki-tomcatd Service, continuing 
normal operation
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful

logs in /var/log/pki/pki-tomcat:
localhost.2018-03-28.log
...
Mar 28, 2018 11:35:14 AM 
org.apache.catalina.core.StandardHostValve invoke
SEVERE: Exception Processing /ca/admin/ca/getStatus
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
 ?????????????? at 
com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
 ?????????????? at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
 ?????????????? at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
 ?????????????? at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
 ?????????????? at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
 ?????????????? at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
 ?????????????? at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
 ?????????????? at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
 ?????????????? at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
 ?????????????? at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
 ?????????????? at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 ?????????????? at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 ?????????????? at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 ?????????????? at java.lang.Thread.run(Thread.java:748)

in catalina.2018-03-28.log:
...
Mar 28, 2018 11:41:35 AM 
org.apache.catalina.core.ContainerBase backgroundProcess
WARNING: Exception processing realm 
com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
 ?????? at 
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
 ?????? at 
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
 ?????? at 
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
 ?????? at 
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
 ?????? at 
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
 ?????? at 
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
 ?????? at 
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
 ?????? at java.lang.Thread.run(Thread.java:748)

Would you able to conclude anything from those errors? What 
might be a problem?

many thanks, L.

[Florence Blanc-Renaud]

On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> hi guys,
>
> I fail to troubleshoot this here:
>
> $ ipactl start --ignore-service-failures
> Starting Directory Service
> Starting krb5kdc Service
> Starting kadmin Service
> Starting named Service
> Starting httpd Service
> Starting ipa-custodia Service
> Starting ntpd Service
> Starting pki-tomcatd Service
> Failed to start pki-tomcatd Service
> Forced start, ignoring pki-tomcatd Service, continuing normal operation

Hi,

pki-tomcatd may fail to start when the subsystemCert cert-pki-ca did not 
properly get renewed. Please find more information in this blog:
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...

Flo
> Starting ipa-otpd Service
> Starting ipa-dnskeysyncd Service
> ipa: INFO: The ipactl command was successful
>
> logs in /var/log/pki/pki-tomcat:
> localhost.2018-03-28.log
> ...
> Mar 28, 2018 11:35:14 AM org.apache.catalina.core.StandardHostValve invoke
> SEVERE: Exception Processing /ca/admin/ca/getStatus
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> ?????????????? at 
> com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
>
> ?????????????? at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
>
> ?????????????? at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
>
> ?????????????? at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>
> ?????????????? at 
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> ?????????????? at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>
> ?????????????? at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> ?????????????? at 
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
>
> ?????????????? at 
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
>
> ?????????????? at 
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>
> ?????????????? at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> ?????????????? at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> ?????????????? at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>
> ?????????????? at java.lang.Thread.run(Thread.java:748)
>
> in catalina.2018-03-28.log:
> ...
> Mar 28, 2018 11:41:35 AM org.apache.catalina.core.ContainerBase 
> backgroundProcess
> WARNING: Exception processing realm 
> com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> ?????? at 
> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> ?????? at 
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>
> ?????? at 
> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>
> ?????? at 
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>
> ?????? at 
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>
> ?????? at 
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>
> ?????? at 
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>
> ?????? at java.lang.Thread.run(Thread.java:748)
>
> Would you able to conclude anything from those errors? What might be a 
> problem?
>
> many thanks, L.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[lejeczek]

On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > hi guys,
> >
> > I fail to troubleshoot this here:
> >
> > $ ipactl start --ignore-service-failures
> > Starting Directory Service
> > Starting krb5kdc Service
> > Starting kadmin Service
> > Starting named Service
> > Starting httpd Service
> > Starting ipa-custodia Service
> > Starting ntpd Service
> > Starting pki-tomcatd Service
> > Failed to start pki-tomcatd Service
> > Forced start, ignoring pki-tomcatd Service, continuing 
> > normal operation
>
> Hi,
>
> pki-tomcatd may fail to start when the subsystemCert 
> cert-pki-ca did not properly get renewed. Please find more 
> information in this blog:
> https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
>
> Flo
> > Starting ipa-otpd Service
> > Starting ipa-dnskeysyncd Service
> > ipa: INFO: The ipactl command was successful
> >
> > logs in /var/log/pki/pki-tomcat:
> > localhost.2018-03-28.log
> > ...
> > Mar 28, 2018 11:35:14 AM 
> > org.apache.catalina.core.StandardHostValve invoke
> > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > unavailable
> > ?????????????? at 
> > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> >
> > ?????????????? at 
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> >
> > ?????????????? at 
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> >
> > ?????????????? at 
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> >
> > ?????????????? at 
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > ?????????????? at 
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> >
> > ?????????????? at 
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > ?????????????? at 
> > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> >
> > ?????????????? at 
> > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> >
> > ?????????????? at 
> > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> >
> > ?????????????? at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> > ?????????????? at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> > ?????????????? at 
> > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >
> > ?????????????? at java.lang.Thread.run(Thread.java:748)
> >
> > in catalina.2018-03-28.log:
> > ...
> > Mar 28, 2018 11:41:35 AM 
> > org.apache.catalina.core.ContainerBase backgroundProcess
> > WARNING: Exception processing realm 
> > com.netscape.cms.tomcat.ProxyRealm@1e572093 background 
> > process
> > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > unavailable
> > ?????? at 
> > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > ?????? at 
> > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> >
> > ?????? at 
> > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> >
> > ?????? at 
> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> >
> > ?????? at 
> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >
> > ?????? at 
> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> >
> > ?????? at 
> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> >
> > ?????? at java.lang.Thread.run(Thread.java:748)
> >
> > Would you able to conclude anything from those errors? 
> > What might be a problem?
> >
> > many thanks, L.
> > _______________________________________________
> > FreeIPA-users mailing list -- 
> > [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]

I have followed those instructions from the link and it 
seems that both certutil & ldap have the same certificate.
However I also see:

$ sudo journalctl -lf -o cat -u dirsrv@
...
GSSAPI client step 1
GSSAPI client step 1
GSSAPI client step 2
[03/Apr/2018:19:30:53.962565693 +0100] - ERR - 
slapi_ldap_bind - Error: could not send startTLS request: 
error -11 (Connect error)
[03/Apr/2018:19:30:53.965606137 +0100] - ERR - 
slapi_ldap_bind - Error: could not send startTLS request: 
error -11 (Connect error)
GSSAPI client step 1
GSSAPI client step 1
GSSAPI client step 1
..

and in /var/log/pki/pki-tomcat/ca/debug

[03/Apr/2018:19:09:45][localhost-startStop-1]: 
SSLClientCertificatSelectionCB: Entering!
[03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
cert: Server-Cert cert-pki-ca
[03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
cert: caSigningCert cert-pki-ca
[03/Apr/2018:19:09:45][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: returning: null
[03/Apr/2018:19:09:45][localhost-startStop-1]: SSL handshake 
happened
Could not connect to LDAP server host rider.private port 636 
Error netscape.ldap.LDAPException: Authentication failed (48)
         at 
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
         at 
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
         at 
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
         at 
com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
         at 
com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
         at 
com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
         at 
com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
         at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
         at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
         at 
com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
         at 
javax.servlet.GenericServlet.init(GenericServlet.java:158)
         at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at 
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
         at 
org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
         at 
java.security.AccessController.doPrivileged(Native Method)
         at 
javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
         at 
org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
         at 
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
         at 
org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
         at 
org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
         at 
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
         at 
org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
         at 
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
         at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
         at 
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
         at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
         at 
org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
         at 
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
         at 
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
         at 
java.security.AccessController.doPrivileged(Native Method)
         at 
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
         at 
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
         at 
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
         at 
org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
         at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
         at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
..

[Florence Blanc-Renaud]

On 04/03/2018 08:37 PM, lejeczek wrote:
> On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > hi guys,
> > >
> > > I fail to troubleshoot this here:
> > >
> > > $ ipactl start --ignore-service-failures
> > > Starting Directory Service
> > > Starting krb5kdc Service
> > > Starting kadmin Service
> > > Starting named Service
> > > Starting httpd Service
> > > Starting ipa-custodia Service
> > > Starting ntpd Service
> > > Starting pki-tomcatd Service
> > > Failed to start pki-tomcatd Service
> > > Forced start, ignoring pki-tomcatd Service, continuing normal operation
> >
> > Hi,
> >
> > pki-tomcatd may fail to start when the subsystemCert cert-pki-ca did 
> > not properly get renewed. Please find more information in this blog:
> > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> >
> > Flo
> > > Starting ipa-otpd Service
> > > Starting ipa-dnskeysyncd Service
> > > ipa: INFO: The ipactl command was successful
> > >
> > > logs in /var/log/pki/pki-tomcat:
> > > localhost.2018-03-28.log
> > > ...
> > > Mar 28, 2018 11:35:14 AM org.apache.catalina.core.StandardHostValve 
> > > invoke
> > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > ?????????????? at 
> > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > >
> > > ?????????????? at 
> > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > >
> > > ?????????????? at 
> > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> > >
> > > ?????????????? at 
> > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > >
> > > ?????????????? at 
> > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > >
> > > ?????????????? at 
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > > ?????????????? at 
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >
> > > ?????????????? at 
> > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > >
> > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > >
> > > in catalina.2018-03-28.log:
> > > ...
> > > Mar 28, 2018 11:41:35 AM org.apache.catalina.core.ContainerBase 
> > > backgroundProcess
> > > WARNING: Exception processing realm 
> > > com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
> > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > ?????? at 
> > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > >
> > > ?????? at 
> > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > >
> > > ?????? at java.lang.Thread.run(Thread.java:748)
> > >
> > > Would you able to conclude anything from those errors? What might be 
> > > a problem?
> > >
> > > many thanks, L.
> > > _______________________________________________
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to 
> > > [e-mail]
>
> I have followed those instructions from the link and it seems that both 
> certutil & ldap have the same certificate.
> However I also see:
>
> $ sudo journalctl -lf -o cat -u dirsrv@
> ...
> GSSAPI client step 1
> GSSAPI client step 1
> GSSAPI client step 2
> [03/Apr/2018:19:30:53.962565693 +0100] - ERR - slapi_ldap_bind - Error: 
> could not send startTLS request: error -11 (Connect error)
> [03/Apr/2018:19:30:53.965606137 +0100] - ERR - slapi_ldap_bind - Error: 
> could not send startTLS request: error -11 (Connect error)
> GSSAPI client step 1
> GSSAPI client step 1
> GSSAPI client step 1
> ..
>
> and in /var/log/pki/pki-tomcat/ca/debug
>
> [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> SSLClientCertificatSelectionCB: Entering!
> [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> Server-Cert cert-pki-ca
> [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> caSigningCert cert-pki-ca
> [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> SSLClientCertificateSelectionCB: returning: null

Hi,
it looks like the subsystemCert is not picked to authenticate to the 
LDAP server. Can you check if the content of 
/etc/pki/pki-tomcat/ca/CS.cfg is also consistent: this file contains an 
entry for ca.subsystem.cert=MII.. that should match the cert 
'subsystemCert cert-pki-ca' stored in /etc/pki/pki-tomcat/alias/ and in 
LDAP.

Flo
> [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL handshake happened
> Could not connect to LDAP server host rider.private port 636 Error 
> netscape.ldap.LDAPException: Authentication failed (48)
>          at 
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
>
> at 
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
>
> at 
> com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
>
> at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
>          at 
> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
>          at 
> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
>          at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
>          at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
>          at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
>          at 
> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
>
> at javax.servlet.GenericServlet.init(GenericServlet.java:158)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>          at 
> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
>          at 
> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>          at 
> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
>          at 
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
>
> at 
> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
>
> at 
> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
>
> at 
> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
>
> at 
> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
>          at 
> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
>
> at 
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
>
> at 
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>          at 
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
>
> at 
> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
>          at 
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
>
> at 
> org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
>
> at java.security.AccessController.doPrivileged(Native Method)
>          at 
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
>          at 
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>          at 
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
>
> at 
> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
>
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ..

[lejeczek]

On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> On 04/03/2018 08:37 PM, lejeczek wrote:
> > On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > > hi guys,
> > > >
> > > > I fail to troubleshoot this here:
> > > >
> > > > $ ipactl start --ignore-service-failures
> > > > Starting Directory Service
> > > > Starting krb5kdc Service
> > > > Starting kadmin Service
> > > > Starting named Service
> > > > Starting httpd Service
> > > > Starting ipa-custodia Service
> > > > Starting ntpd Service
> > > > Starting pki-tomcatd Service
> > > > Failed to start pki-tomcatd Service
> > > > Forced start, ignoring pki-tomcatd Service, continuing 
> > > > normal operation
> > >
> > > Hi,
> > >
> > > pki-tomcatd may fail to start when the subsystemCert 
> > > cert-pki-ca did not properly get renewed. Please find 
> > > more information in this blog:
> > > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > >
> > > Flo
> > > > Starting ipa-otpd Service
> > > > Starting ipa-dnskeysyncd Service
> > > > ipa: INFO: The ipactl command was successful
> > > >
> > > > logs in /var/log/pki/pki-tomcat:
> > > > localhost.2018-03-28.log
> > > > ...
> > > > Mar 28, 2018 11:35:14 AM 
> > > > org.apache.catalina.core.StandardHostValve invoke
> > > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > unavailable
> > > > ?????????????? at 
> > > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > > >
> > > > ?????????????? at 
> > > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> > > >
> > > > ?????????????? at 
> > > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > > >
> > > > ?????????????? at 
> > > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > >
> > > > ?????????????? at 
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > > > ?????????????? at 
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > >
> > > > ?????????????? at 
> > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > >
> > > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > > >
> > > > in catalina.2018-03-28.log:
> > > > ...
> > > > Mar 28, 2018 11:41:35 AM 
> > > > org.apache.catalina.core.ContainerBase backgroundProcess
> > > > WARNING: Exception processing realm 
> > > > com.netscape.cms.tomcat.ProxyRealm@1e572093 background 
> > > > process
> > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > unavailable
> > > > ?????? at 
> > > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > > >
> > > > ?????? at java.lang.Thread.run(Thread.java:748)
> > > >
> > > > Would you able to conclude anything from those errors? 
> > > > What might be a problem?
> > > >
> > > > many thanks, L.
> > > > _______________________________________________
> > > > FreeIPA-users mailing list -- 
> > > > [e-mail]
> > > > To unsubscribe send an email to 
> > > > [e-mail]
> >
> > I have followed those instructions from the link and it 
> > seems that both certutil & ldap have the same certificate.
> > However I also see:
> >
> > $ sudo journalctl -lf -o cat -u dirsrv@
> > ...
> > GSSAPI client step 1
> > GSSAPI client step 1
> > GSSAPI client step 2
> > [03/Apr/2018:19:30:53.962565693 +0100] - ERR - 
> > slapi_ldap_bind - Error: could not send startTLS request: 
> > error -11 (Connect error)
> > [03/Apr/2018:19:30:53.965606137 +0100] - ERR - 
> > slapi_ldap_bind - Error: could not send startTLS request: 
> > error -11 (Connect error)
> > GSSAPI client step 1
> > GSSAPI client step 1
> > GSSAPI client step 1
> > ..
> >
> > and in /var/log/pki/pki-tomcat/ca/debug
> >
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > SSLClientCertificatSelectionCB: Entering!
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
> > cert: Server-Cert cert-pki-ca
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
> > cert: caSigningCert cert-pki-ca
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > SSLClientCertificateSelectionCB: returning: null
>
> Hi,
> it looks like the subsystemCert is not picked to 
> authenticate to the LDAP server. Can you check if the 
> content of /etc/pki/pki-tomcat/ca/CS.cfg is also 
> consistent: this file contains an entry for 
> ca.subsystem.cert=MII.. that should match the cert 
> 'subsystemCert cert-pki-ca' stored in 
> /etc/pki/pki-tomcat/alias/ and in LDAP.
>
> Flo

hi
all three places:

$ /etc/pki/pki-tomcat/ca/CS.cfg
$ certutil -L -d /etc/pki/pki-tomcat/alias/ -n 
'subsystemCert cert-pki-ca' -f 
/etc/pki/pki-tomcat/alias/pwdfile.txt -a
$ ldapsearch -D 'cn=directory manager' -W -b 
uid=pkidbuser,ou=people,o=ipaca userCertificate description 
seeAlso

show identical certificate. More from debug log:
...
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
============================================
[04/Apr/2018:10:20:54][localhost-startStop-1]: =====  DEBUG 
SUBSYSTEM INITIALIZED   =======
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
============================================
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
restart at autoShutdown? false
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
autoShutdown crumb file path? 
/var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
about to look for cert for auto-shutdown 
support:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
found cert:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
done init id=debug
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initialized debug
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initSubsystem id=log
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
ready to init id=log
[04/Apr/2018:10:20:54][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[04/Apr/2018:10:20:54][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[04/Apr/2018:10:20:54][localhost-startStop-1]: Creating 
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
restart at autoShutdown? false
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
autoShutdown crumb file path? 
/var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
about to look for cert for auto-shutdown 
support:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
found cert:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
done init id=log
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initialized log
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initSubsystem id=jss
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
ready to init id=jss
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
initializing JSS subsystem
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
enabled: true
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
NSS database: /var/lib/pki/pki-tomcat/alias/
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
initializing CryptoManager
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
initializing SSL
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
random:
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
- algorithm: pkcs11prng
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
- provider: Mozilla-JSS
[04/Apr/2018:10:20:54][localhost-startStop-1]: JssSubsystem: 
initialization complete
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
restart at autoShutdown? false
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
autoShutdown crumb file path? 
/var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
about to look for cert for auto-shutdown 
support:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
found cert:auditSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
done init id=jss
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initialized jss
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
initSubsystem id=dbs
[04/Apr/2018:10:20:54][localhost-startStop-1]: CMSEngine: 
ready to init id=dbs
[04/Apr/2018:10:20:54][localhost-startStop-1]: DBSubsystem: 
init() mEnableSerialMgmt=true
[04/Apr/2018:10:20:54][localhost-startStop-1]: Creating 
LdapBoundConnFactor(DBSubsystem)
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
LdapBoundConnFactory: init
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
LdapBoundConnFactory:doCloning true
[04/Apr/2018:10:20:54][localhost-startStop-1]: LdapAuthInfo: 
init()
[04/Apr/2018:10:20:54][localhost-startStop-1]: LdapAuthInfo: 
init begins
[04/Apr/2018:10:20:54][localhost-startStop-1]: LdapAuthInfo: 
init ends
[04/Apr/2018:10:20:54][localhost-startStop-1]: init: before 
makeConnection errorIfDown is true
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
makeConnection: errorIfDown true
[04/Apr/2018:10:20:54][localhost-startStop-1]: TCP 
Keep-Alive: true
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: Setting desired cert 
nickname to: subsystemCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
LdapJssSSLSocket: set client auth cert nickname 
subsystemCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
SSLClientCertificatSelectionCB: Entering!
[04/Apr/2018:10:20:54][localhost-startStop-1]: Candidate 
cert: Server-Cert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: Candidate 
cert: caSigningCert cert-pki-ca
[04/Apr/2018:10:20:54][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: returning: null
[04/Apr/2018:10:20:54][localhost-startStop-1]: SSL handshake 
happened
Could not connect to LDAP server host ...

What's the way out of this? It's two masters IPA setup.
Is there a way to stop/start only pki without doing so to 
the rest of the services?
I cannot just remove PKI from the IPA, can I?

many thanks, L.
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL 
> > handshake happened
> > Could not connect to LDAP server host rider.private port 
> > 636 Error netscape.ldap.LDAPException: Authentication 
> > failed (48)
> >          at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> >
> > at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> >
> > at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> >
> > at 
> > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> >          at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
> >          at 
> > com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
> >          at 
> > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
> >
> > at 
> > javax.servlet.GenericServlet.init(GenericServlet.java:158)
> >          at 
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >          at 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >          at 
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> >
> > at 
> > java.security.AccessController.doPrivileged(Native Method)
> >          at 
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> >          at 
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
> >
> > at 
> > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
> >
> > at 
> > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
> >
> > at 
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> >
> > at 
> > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> >
> > at 
> > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> >          at 
> > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> >
> > at 
> > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> >
> > at 
> > java.security.AccessController.doPrivileged(Native Method)
> >          at 
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> >
> > at 
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> >
> > at 
> > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> >
> > at 
> > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> >
> > at 
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >
> > at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > ..

[lejeczek]

On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> On 04/03/2018 08:37 PM, lejeczek wrote:
> > On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > > hi guys,
> > > >
> > > > I fail to troubleshoot this here:
> > > >
> > > > $ ipactl start --ignore-service-failures
> > > > Starting Directory Service
> > > > Starting krb5kdc Service
> > > > Starting kadmin Service
> > > > Starting named Service
> > > > Starting httpd Service
> > > > Starting ipa-custodia Service
> > > > Starting ntpd Service
> > > > Starting pki-tomcatd Service
> > > > Failed to start pki-tomcatd Service
> > > > Forced start, ignoring pki-tomcatd Service, continuing 
> > > > normal operation
> > >
> > > Hi,
> > >
> > > pki-tomcatd may fail to start when the subsystemCert 
> > > cert-pki-ca did not properly get renewed. Please find 
> > > more information in this blog:
> > > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > >
> > > Flo
> > > > Starting ipa-otpd Service
> > > > Starting ipa-dnskeysyncd Service
> > > > ipa: INFO: The ipactl command was successful
> > > >
> > > > logs in /var/log/pki/pki-tomcat:
> > > > localhost.2018-03-28.log
> > > > ...
> > > > Mar 28, 2018 11:35:14 AM 
> > > > org.apache.catalina.core.StandardHostValve invoke
> > > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > unavailable
> > > > ?????????????? at 
> > > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > >
> > > > ?????????????? at 
> > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > > >
> > > > ?????????????? at 
> > > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> > > >
> > > > ?????????????? at 
> > > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > > >
> > > > ?????????????? at 
> > > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > >
> > > > ?????????????? at 
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >
> > > > ?????????????? at 
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > >
> > > > ?????????????? at 
> > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > >
> > > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > > >
> > > > in catalina.2018-03-28.log:
> > > > ...
> > > > Mar 28, 2018 11:41:35 AM 
> > > > org.apache.catalina.core.ContainerBase backgroundProcess
> > > > WARNING: Exception processing realm 
> > > > com.netscape.cms.tomcat.ProxyRealm@1e572093 background 
> > > > process
> > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > unavailable
> > > > ?????? at 
> > > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > >
> > > > ?????? at 
> > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > > >
> > > > ?????? at java.lang.Thread.run(Thread.java:748)
> > > >
> > > > Would you able to conclude anything from those errors? 
> > > > What might be a problem?
> > > >
> > > > many thanks, L.
> > > > _______________________________________________
> > > > FreeIPA-users mailing list -- 
> > > > [e-mail]
> > > > To unsubscribe send an email to 
> > > > [e-mail]
> >
> > I have followed those instructions from the link and it 
> > seems that both certutil & ldap have the same certificate.
> > However I also see:
> >
> > $ sudo journalctl -lf -o cat -u dirsrv@
> > ...
> > GSSAPI client step 1
> > GSSAPI client step 1
> > GSSAPI client step 2
> > [03/Apr/2018:19:30:53.962565693 +0100] - ERR - 
> > slapi_ldap_bind - Error: could not send startTLS request: 
> > error -11 (Connect error)
> > [03/Apr/2018:19:30:53.965606137 +0100] - ERR - 
> > slapi_ldap_bind - Error: could not send startTLS request: 
> > error -11 (Connect error)
> > GSSAPI client step 1
> > GSSAPI client step 1
> > GSSAPI client step 1
> > ..
> >
> > and in /var/log/pki/pki-tomcat/ca/debug
> >
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > SSLClientCertificatSelectionCB: Entering!
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
> > cert: Server-Cert cert-pki-ca
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate 
> > cert: caSigningCert cert-pki-ca
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > SSLClientCertificateSelectionCB: returning: null
>
> Hi,
> it looks like the subsystemCert is not picked to 
> authenticate to the LDAP server. Can you check if the 
> content of /etc/pki/pki-tomcat/ca/CS.cfg is also 
> consistent: this file contains an entry for 
> ca.subsystem.cert=MII.. that should match the cert 
> 'subsystemCert cert-pki-ca' stored in 
> /etc/pki/pki-tomcat/alias/ and in LDAP.
>
> Flo
> > [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL 
> > handshake happened
> > Could not connect to LDAP server host rider.private port 
> > 636 Error netscape.ldap.LDAPException: Authentication 
> > failed (48)
> >          at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> >
> > at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> >
> > at 
> > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> >
> > at 
> > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
> >
> > at 
> > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> >          at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
> >          at 
> > com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
> >          at 
> > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
> >
> > at 
> > javax.servlet.GenericServlet.init(GenericServlet.java:158)
> >          at 
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >          at 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at 
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.lang.reflect.Method.invoke(Method.java:498)
> >          at 
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> >
> > at 
> > java.security.AccessController.doPrivileged(Native Method)
> >          at 
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> >          at 
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> >
> > at 
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
> >
> > at 
> > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
> >
> > at 
> > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
> >
> > at 
> > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
> >
> > at 
> > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> >
> > at 
> > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> >
> > at 
> > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> >          at 
> > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> >
> > at 
> > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> >
> > at 
> > java.security.AccessController.doPrivileged(Native Method)
> >          at 
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> >
> > at 
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> >
> > at 
> > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> >
> > at 
> > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> >
> > at 
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >
> > at 
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > ..

could this have some causation in expired certs?

$ getcert list | grep -E "Request ID|status|certificate|expires"
Number of certificates and requests being tracked: 9.
Request ID '20170920090053':
     status: MONITORING
     certificate: 
type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
     expires: 2018-09-20 09:00:53 UTC
     certificate template/profile: KDCs_PKINIT_Certs
Request ID '20171221120303':
     status: CA_WORKING
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     expires: 2018-03-27 14:07:51 UTC
Request ID '20171221120304':
     status: CA_WORKING
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     expires: 2018-03-27 14:07:50 UTC
Request ID '20171221120305':
     status: CA_WORKING
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
cert-pki-ca',token='NSS Certificate DB'
     expires: 2018-03-27 14:07:51 UTC
Request ID '20171221120306':
     status: MONITORING
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
cert-pki-ca',token='NSS Certificate DB'
     expires: 2036-04-06 14:07:49 UTC
Request ID '20171221120307':
     status: CA_WORKING
     certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
     expires: 2018-03-27 14:08:18 UTC
Request ID '20171221120308':
     status: MONITORING
     certificate: 
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
cert-pki-ca',token='NSS Certificate DB'
     expires: 2019-07-25 16:13:23 UTC
Request ID '20171221120309':
     status: MONITORING
     certificate: 
type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-CCNR-CEB-PRIVATE-CAM-AC-UK',nickname='Server-Cert',token='NSS 
Certificate DB'
     expires: 2019-04-09 12:11:52 UTC
Request ID '20171221120310':
     status: MONITORING
     certificate: 
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
Certificate DB'
     expires: 2019-04-09 12:11:54 UTC

[Florence Blanc-Renaud]

On 04/04/2018 12:37 PM, lejeczek via FreeIPA-users wrote:
> On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> > On 04/03/2018 08:37 PM, lejeczek wrote:
> > > On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > > > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > > > hi guys,
> > > > >
> > > > > I fail to troubleshoot this here:
> > > > >
> > > > > $ ipactl start --ignore-service-failures
> > > > > Starting Directory Service
> > > > > Starting krb5kdc Service
> > > > > Starting kadmin Service
> > > > > Starting named Service
> > > > > Starting httpd Service
> > > > > Starting ipa-custodia Service
> > > > > Starting ntpd Service
> > > > > Starting pki-tomcatd Service
> > > > > Failed to start pki-tomcatd Service
> > > > > Forced start, ignoring pki-tomcatd Service, continuing normal 
> > > > > operation
> > > >
> > > > Hi,
> > > >
> > > > pki-tomcatd may fail to start when the subsystemCert cert-pki-ca did 
> > > > not properly get renewed. Please find more information in this blog:
> > > > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > > >
> > > > Flo
> > > > > Starting ipa-otpd Service
> > > > > Starting ipa-dnskeysyncd Service
> > > > > ipa: INFO: The ipactl command was successful
> > > > >
> > > > > logs in /var/log/pki/pki-tomcat:
> > > > > localhost.2018-03-28.log
> > > > > ...
> > > > > Mar 28, 2018 11:35:14 AM org.apache.catalina.core.StandardHostValve 
> > > > > invoke
> > > > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > > > ?????????????? at 
> > > > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > > >
> > > > > ?????????????? at 
> > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > >
> > > > > ?????????????? at 
> > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > >
> > > > > ?????????????? at 
> > > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > > >
> > > > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > > > >
> > > > > in catalina.2018-03-28.log:
> > > > > ...
> > > > > Mar 28, 2018 11:41:35 AM org.apache.catalina.core.ContainerBase 
> > > > > backgroundProcess
> > > > > WARNING: Exception processing realm 
> > > > > com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
> > > > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > > > ?????? at 
> > > > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > > >
> > > > > ?????? at 
> > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > > > >
> > > > > ?????? at java.lang.Thread.run(Thread.java:748)
> > > > >
> > > > > Would you able to conclude anything from those errors? What might 
> > > > > be a problem?
> > > > >
> > > > > many thanks, L.
> > > > > _______________________________________________
> > > > > FreeIPA-users mailing list -- [e-mail]
> > > > > To unsubscribe send an email to 
> > > > > [e-mail]
> > >
> > > I have followed those instructions from the link and it seems that 
> > > both certutil & ldap have the same certificate.
> > > However I also see:
> > >
> > > $ sudo journalctl -lf -o cat -u dirsrv@
> > > ...
> > > GSSAPI client step 1
> > > GSSAPI client step 1
> > > GSSAPI client step 2
> > > [03/Apr/2018:19:30:53.962565693 +0100] - ERR - slapi_ldap_bind - 
> > > Error: could not send startTLS request: error -11 (Connect error)
> > > [03/Apr/2018:19:30:53.965606137 +0100] - ERR - slapi_ldap_bind - 
> > > Error: could not send startTLS request: error -11 (Connect error)
> > > GSSAPI client step 1
> > > GSSAPI client step 1
> > > GSSAPI client step 1
> > > ..
> > >
> > > and in /var/log/pki/pki-tomcat/ca/debug
> > >
> > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > SSLClientCertificatSelectionCB: Entering!
> > > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> > > Server-Cert cert-pki-ca
> > > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> > > caSigningCert cert-pki-ca
> > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > SSLClientCertificateSelectionCB: returning: null
> >
> > Hi,
> > it looks like the subsystemCert is not picked to authenticate to the 
> > LDAP server. Can you check if the content of 
> > /etc/pki/pki-tomcat/ca/CS.cfg is also consistent: this file contains 
> > an entry for ca.subsystem.cert=MII.. that should match the cert 
> > 'subsystemCert cert-pki-ca' stored in /etc/pki/pki-tomcat/alias/ and 
> > in LDAP.
> >
> > Flo
> > > [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL handshake happened
> > > Could not connect to LDAP server host rider.private port 636 Error 
> > > netscape.ldap.LDAPException: Authentication failed (48)
> > >          at 
> > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> > >
> > > at 
> > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> > >
> > > at 
> > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> > >
> > > at 
> > > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> > >          at 
> > > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
> > >          at 
> > > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
> > >          at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> > >          at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
> > >          at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
> > >          at 
> > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
> > >
> > > at javax.servlet.GenericServlet.init(GenericServlet.java:158)
> > >          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >          at 
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >
> > > at 
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > >          at 
> > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> > >          at 
> > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> > >          at java.security.AccessController.doPrivileged(Native Method)
> > >          at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > >          at 
> > > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> > >          at 
> > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> > >
> > > at 
> > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> > >
> > > at 
> > > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
> > >
> > > at 
> > > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
> > >
> > > at 
> > > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
> > >          at 
> > > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
> > >
> > > at 
> > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
> > >
> > > at 
> > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> > >          at 
> > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> > >
> > > at 
> > > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> > >
> > > at 
> > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> > >
> > > at 
> > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> > >
> > > at java.security.AccessController.doPrivileged(Native Method)
> > >          at 
> > > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> > >          at 
> > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> > >          at 
> > > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> > >
> > > at 
> > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> > >
> > > at 
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > >          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > ..
>
> could this have some causation in expired certs?

Hi,

CA_WORKING means that certmonger's helper is trying to download the 
certificate from LDAP, but does not find new certs.

In topologies with multiple servers, only one server is the renewal 
master. When one of auditSigningCert cert-pki-ca, ocspSigningCert 
cert-pki-ca, subsystemCert cert-pki-ca or caSigningCert cert-pki-ca 
expires, the renewal master is the one that actually handles the 
renewal, and the other masters simply download the new certs from LDAP.

You need to check which server is your renewal master (ipa config-show | 
grep 'IPA CA renewal master'), then make sure that the certs were 
properly renewed on this master (check consistency between 
/etc/pki/pki-tomcat/alias, the certs in 
cn=certificates,cn=ipa,cn=etc,$BASEDN, and the content in 
/etc/pki/pki-tomcat/ca/CS.cfg).

Then check that replication is working between the renewal master and 
the other masters. If the replication is broken, the certs will not be 
copied on the other masters and the download will not detect new 
certificates.

HTH,
Flo
> $ getcert list | grep -E "Request ID|status|certificate|expires"
> Number of certificates and requests being tracked: 9.
> Request ID '20170920090053':
>      status: MONITORING
>      certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>      expires: 2018-09-20 09:00:53 UTC
>      certificate template/profile: KDCs_PKINIT_Certs
> Request ID '20171221120303':
>      status: CA_WORKING
>      certificate: 
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
> cert-pki-ca',token='NSS Certificate DB'
>      expires: 2018-03-27 14:07:51 UTC
> Request ID '20171221120304':
>      status: CA_WORKING
>      certificate: 
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS 
> Certificate DB'
>      expires: 2018-03-27 14:07:50 UTC
> Request ID '20171221120305':
>      status: CA_WORKING
>      certificate: 
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
> cert-pki-ca',token='NSS Certificate DB'
>      expires: 2018-03-27 14:07:51 UTC
> Request ID '20171221120306':
>      status: MONITORING
>      certificate: 
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
> cert-pki-ca',token='NSS Certificate DB'
>      expires: 2036-04-06 14:07:49 UTC
> Request ID '20171221120307':
>      status: CA_WORKING
>      certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>      expires: 2018-03-27 14:08:18 UTC
> Request ID '20171221120308':
>      status: MONITORING
>      certificate: 
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
> cert-pki-ca',token='NSS Certificate DB'
>      expires: 2019-07-25 16:13:23 UTC
> Request ID '20171221120309':
>      status: MONITORING
>      certificate: 
> type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-CCNR-CEB-PRIVATE-CAM-AC-UK',nickname='Server-Cert',token='NSS 
> Certificate DB'
>      expires: 2019-04-09 12:11:52 UTC
> Request ID '20171221120310':
>      status: MONITORING
>      certificate: 
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
> Certificate DB'
>      expires: 2019-04-09 12:11:54 UTC
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[lejeczek]

On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> On 04/04/2018 12:37 PM, lejeczek via FreeIPA-users wrote:
> > On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> > > On 04/03/2018 08:37 PM, lejeczek wrote:
> > > > On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > > > > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > > > > hi guys,
> > > > > >
> > > > > > I fail to troubleshoot this here:
> > > > > >
> > > > > > $ ipactl start --ignore-service-failures
> > > > > > Starting Directory Service
> > > > > > Starting krb5kdc Service
> > > > > > Starting kadmin Service
> > > > > > Starting named Service
> > > > > > Starting httpd Service
> > > > > > Starting ipa-custodia Service
> > > > > > Starting ntpd Service
> > > > > > Starting pki-tomcatd Service
> > > > > > Failed to start pki-tomcatd Service
> > > > > > Forced start, ignoring pki-tomcatd Service, 
> > > > > > continuing normal operation
> > > > >
> > > > > Hi,
> > > > >
> > > > > pki-tomcatd may fail to start when the subsystemCert 
> > > > > cert-pki-ca did not properly get renewed. Please find 
> > > > > more information in this blog:
> > > > > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > > > >
> > > > > Flo
> > > > > > Starting ipa-otpd Service
> > > > > > Starting ipa-dnskeysyncd Service
> > > > > > ipa: INFO: The ipactl command was successful
> > > > > >
> > > > > > logs in /var/log/pki/pki-tomcat:
> > > > > > localhost.2018-03-28.log
> > > > > > ...
> > > > > > Mar 28, 2018 11:35:14 AM 
> > > > > > org.apache.catalina.core.StandardHostValve invoke
> > > > > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > > > unavailable
> > > > > > ?????????????? at 
> > > > > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > >
> > > > > > ?????????????? at 
> > > > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> > > > > >
> > > > > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > > > > >
> > > > > > in catalina.2018-03-28.log:
> > > > > > ...
> > > > > > Mar 28, 2018 11:41:35 AM 
> > > > > > org.apache.catalina.core.ContainerBase backgroundProcess
> > > > > > WARNING: Exception processing realm 
> > > > > > com.netscape.cms.tomcat.ProxyRealm@1e572093 
> > > > > > background process
> > > > > > javax.ws.rs.ServiceUnavailableException: Subsystem 
> > > > > > unavailable
> > > > > > ?????? at 
> > > > > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
> > > > > >
> > > > > > ?????? at 
> > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
> > > > > >
> > > > > > ?????? at java.lang.Thread.run(Thread.java:748)
> > > > > >
> > > > > > Would you able to conclude anything from those 
> > > > > > errors? What might be a problem?
> > > > > >
> > > > > > many thanks, L.
> > > > > > _______________________________________________
> > > > > > FreeIPA-users mailing list -- 
> > > > > > [e-mail]
> > > > > > To unsubscribe send an email to 
> > > > > > [e-mail]
> > > >
> > > > I have followed those instructions from the link and it 
> > > > seems that both certutil & ldap have the same certificate.
> > > > However I also see:
> > > >
> > > > $ sudo journalctl -lf -o cat -u dirsrv@
> > > > ...
> > > > GSSAPI client step 1
> > > > GSSAPI client step 1
> > > > GSSAPI client step 2
> > > > [03/Apr/2018:19:30:53.962565693 +0100] - ERR - 
> > > > slapi_ldap_bind - Error: could not send startTLS 
> > > > request: error -11 (Connect error)
> > > > [03/Apr/2018:19:30:53.965606137 +0100] - ERR - 
> > > > slapi_ldap_bind - Error: could not send startTLS 
> > > > request: error -11 (Connect error)
> > > > GSSAPI client step 1
> > > > GSSAPI client step 1
> > > > GSSAPI client step 1
> > > > ..
> > > >
> > > > and in /var/log/pki/pki-tomcat/ca/debug
> > > >
> > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > SSLClientCertificatSelectionCB: Entering!
> > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > Candidate cert: Server-Cert cert-pki-ca
> > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > Candidate cert: caSigningCert cert-pki-ca
> > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > SSLClientCertificateSelectionCB: returning: null
> > >
> > > Hi,
> > > it looks like the subsystemCert is not picked to 
> > > authenticate to the LDAP server. Can you check if the 
> > > content of /etc/pki/pki-tomcat/ca/CS.cfg is also 
> > > consistent: this file contains an entry for 
> > > ca.subsystem.cert=MII.. that should match the cert 
> > > 'subsystemCert cert-pki-ca' stored in 
> > > /etc/pki/pki-tomcat/alias/ and in LDAP.
> > >
> > > Flo
> > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL 
> > > > handshake happened
> > > > Could not connect to LDAP server host rider.private 
> > > > port 636 Error netscape.ldap.LDAPException: 
> > > > Authentication failed (48)
> > > >          at 
> > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> > > >
> > > > at 
> > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> > > >
> > > > at 
> > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> > > >
> > > > at 
> > > > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> > > >
> > > > at 
> > > > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
> > > >
> > > > at 
> > > > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
> > > >
> > > > at 
> > > > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> > > >
> > > > at 
> > > > com.netscape.certsrv.apps.CMS.init(CMS.java:189)
> > > >          at 
> > > > com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
> > > >          at 
> > > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
> > > >
> > > > at 
> > > > javax.servlet.GenericServlet.init(GenericServlet.java:158)
> > > >          at 
> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> > > > Method)
> > > >          at 
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > >
> > > > at 
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >
> > > > at 
> > > > java.lang.reflect.Method.invoke(Method.java:498)
> > > >          at 
> > > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> > > >
> > > > at 
> > > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> > > >
> > > > at 
> > > > java.security.AccessController.doPrivileged(Native Method)
> > > >          at 
> > > > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > > >
> > > > at 
> > > > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> > > >
> > > > at 
> > > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> > > >
> > > > at 
> > > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
> > > >
> > > > at 
> > > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> > > >
> > > > at 
> > > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> > > >
> > > > at 
> > > > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> > > >
> > > > at 
> > > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> > > >
> > > > at 
> > > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> > > >
> > > > at 
> > > > java.security.AccessController.doPrivileged(Native Method)
> > > >          at 
> > > > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> > > >
> > > > at 
> > > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> > > >
> > > > at 
> > > > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> > > >
> > > > at 
> > > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> > > >
> > > > at 
> > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > >
> > > > at 
> > > > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > ..
> >
> > could this have some causation in expired certs?
>
> Hi,
>
> CA_WORKING means that certmonger's helper is trying to 
> download the certificate from LDAP, but does not find new 
> certs.
>
> In topologies with multiple servers, only one server is 
> the renewal master. When one of auditSigningCert 
> cert-pki-ca, ocspSigningCert cert-pki-ca, subsystemCert 
> cert-pki-ca or caSigningCert cert-pki-ca expires, the 
> renewal master is the one that actually handles the 
> renewal, and the other masters simply download the new 
> certs from LDAP.
>
> You need to check which server is your renewal master (ipa 
> config-show | grep 'IPA CA renewal master'), then make 
> sure that the certs were properly renewed on this master 
> (check consistency between /etc/pki/pki-tomcat/alias, the 
> certs in cn=certificates,cn=ipa,cn=etc,$BASEDN, and the 
> content in /etc/pki/pki-tomcat/ca/CS.cfg).
>
> Then check that replication is working between the renewal 
> master and the other masters. If the replication is 
> broken, the certs will not be copied on the other masters 
> and the download will not detect new certificates.
>
> HTH,
> Flo

I had no renewal master at all! Was it gone with a replica a 
removed from IPA? If/when CA renewal master is removed from 
IPA(if it's allowed in the first place?) does another server 
assume this role automatically?

Now I've set up a renewal server as per the doc but still 
cannot start that new CA renewal server.
How to renew those expired certs? Maybe this has to be done 
manually somehow prior, in order to start it all?
> > $ getcert list | grep -E "Request 
> > ID|status|certificate|expires"
> > Number of certificates and requests being tracked: 9.
> > Request ID '20170920090053':
> >      status: MONITORING
> >      certificate: 
> > type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
> >      expires: 2018-09-20 09:00:53 UTC
> >      certificate template/profile: KDCs_PKINIT_Certs
> > Request ID '20171221120303':
> >      status: CA_WORKING
> >      certificate: 
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
> > cert-pki-ca',token='NSS Certificate DB'
> >      expires: 2018-03-27 14:07:51 UTC
> > Request ID '20171221120304':
> >      status: CA_WORKING
> >      certificate: 
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
> > cert-pki-ca',token='NSS Certificate DB'
> >      expires: 2018-03-27 14:07:50 UTC
> > Request ID '20171221120305':
> >      status: CA_WORKING
> >      certificate: 
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
> > cert-pki-ca',token='NSS Certificate DB'
> >      expires: 2018-03-27 14:07:51 UTC
> > Request ID '20171221120306':
> >      status: MONITORING
> >      certificate: 
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
> > cert-pki-ca',token='NSS Certificate DB'
> >      expires: 2036-04-06 14:07:49 UTC
> > Request ID '20171221120307':
> >      status: CA_WORKING
> >      certificate: 
> > type=FILE,location='/var/lib/ipa/ra-agent.pem'
> >      expires: 2018-03-27 14:08:18 UTC
> > Request ID '20171221120308':
> >      status: MONITORING
> >      certificate: 
> > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
> > cert-pki-ca',token='NSS Certificate DB'
> >      expires: 2019-07-25 16:13:23 UTC
> > Request ID '20171221120309':
> >      status: MONITORING
> >      certificate: 
> > type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-CCNR-CEB-PRIVATE-CAM-AC-UK',nickname='Server-Cert',token='NSS 
> > Certificate DB'
> >      expires: 2019-04-09 12:11:52 UTC
> > Request ID '20171221120310':
> >      status: MONITORING
> >      certificate: 
> > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
> > Certificate DB'
> >      expires: 2019-04-09 12:11:54 UTC
> > _______________________________________________
> > FreeIPA-users mailing list -- 
> > [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]

[Florence Blanc-Renaud]

On 04/04/2018 02:49 PM, lejeczek via FreeIPA-users wrote:
> On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> > On 04/04/2018 12:37 PM, lejeczek via FreeIPA-users wrote:
> > > On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> > > > On 04/03/2018 08:37 PM, lejeczek wrote:
> > > > > On 29/03/18 12:43, Florence Blanc-Renaud wrote:
> > > > > > On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
> > > > > > > hi guys,
> > > > > > >
> > > > > > > I fail to troubleshoot this here:
> > > > > > >
> > > > > > > $ ipactl start --ignore-service-failures
> > > > > > > Starting Directory Service
> > > > > > > Starting krb5kdc Service
> > > > > > > Starting kadmin Service
> > > > > > > Starting named Service
> > > > > > > Starting httpd Service
> > > > > > > Starting ipa-custodia Service
> > > > > > > Starting ntpd Service
> > > > > > > Starting pki-tomcatd Service
> > > > > > > Failed to start pki-tomcatd Service
> > > > > > > Forced start, ignoring pki-tomcatd Service, continuing normal 
> > > > > > > operation
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > pki-tomcatd may fail to start when the subsystemCert cert-pki-ca 
> > > > > > did not properly get renewed. Please find more information in this 
> > > > > > blog:
> > > > > > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcat...
> > > > > >
> > > > > > Flo
> > > > > >
> > > > > > > Starting ipa-otpd Service
> > > > > > > Starting ipa-dnskeysyncd Service
> > > > > > > ipa: INFO: The ipactl command was successful
> > > > > > >
> > > > > > > logs in /var/log/pki/pki-tomcat:
> > > > > > > localhost.2018-03-28.log
> > > > > > > ...
> > > > > > > Mar 28, 2018 11:35:14 AM 
> > > > > > > org.apache.catalina.core.StandardHostValve invoke
> > > > > > > SEVERE: Exception Processing /ca/admin/ca/getStatus
> > > > > > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > > > > > ?????????????? at 
> > > > > > > com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
> > > > > > >
> > > > > > > ?????????????? at 
> > > > > > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 
> > > > > > >
> > > > > > > ?????????????? at java.lang.Thread.run(Thread.java:748)
> > > > > > >
> > > > > > >
> > > > > > > in catalina.2018-03-28.log:
> > > > > > > ...
> > > > > > > Mar 28, 2018 11:41:35 AM org.apache.catalina.core.ContainerBase 
> > > > > > > backgroundProcess
> > > > > > > WARNING: Exception processing realm 
> > > > > > > com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
> > > > > > > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
> > > > > > > ?????? at 
> > > > > > > com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552) 
> > > > > > >
> > > > > > > ?????? at 
> > > > > > > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520) 
> > > > > > >
> > > > > > > ?????? at java.lang.Thread.run(Thread.java:748)
> > > > > > >
> > > > > > >
> > > > > > > Would you able to conclude anything from those errors? What might 
> > > > > > > be a problem?
> > > > > > >
> > > > > > > many thanks, L.
> > > > > > > _______________________________________________
> > > > > > > FreeIPA-users mailing list -- [e-mail]
> > > > > > > To unsubscribe send an email to 
> > > > > > > [e-mail]
> > > > >
> > > > > I have followed those instructions from the link and it seems that 
> > > > > both certutil & ldap have the same certificate.
> > > > > However I also see:
> > > > >
> > > > > $ sudo journalctl -lf -o cat -u dirsrv@
> > > > > ...
> > > > > GSSAPI client step 1
> > > > > GSSAPI client step 1
> > > > > GSSAPI client step 2
> > > > > [03/Apr/2018:19:30:53.962565693 +0100] - ERR - slapi_ldap_bind - 
> > > > > Error: could not send startTLS request: error -11 (Connect error)
> > > > > [03/Apr/2018:19:30:53.965606137 +0100] - ERR - slapi_ldap_bind - 
> > > > > Error: could not send startTLS request: error -11 (Connect error)
> > > > > GSSAPI client step 1
> > > > > GSSAPI client step 1
> > > > > GSSAPI client step 1
> > > > > ..
> > > > >
> > > > > and in /var/log/pki/pki-tomcat/ca/debug
> > > > >
> > > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > > SSLClientCertificatSelectionCB: Entering!
> > > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> > > > > Server-Cert cert-pki-ca
> > > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert: 
> > > > > caSigningCert cert-pki-ca
> > > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: 
> > > > > SSLClientCertificateSelectionCB: returning: null
> > > >
> > > > Hi,
> > > > it looks like the subsystemCert is not picked to authenticate to the 
> > > > LDAP server. Can you check if the content of 
> > > > /etc/pki/pki-tomcat/ca/CS.cfg is also consistent: this file contains 
> > > > an entry for ca.subsystem.cert=MII.. that should match the cert 
> > > > 'subsystemCert cert-pki-ca' stored in /etc/pki/pki-tomcat/alias/ and 
> > > > in LDAP.
> > > >
> > > > Flo
> > > > > [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL handshake happened
> > > > > Could not connect to LDAP server host rider.private port 636 Error 
> > > > > netscape.ldap.LDAPException: Authentication failed (48)
> > > > >          at 
> > > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
> > > > >
> > > > > at 
> > > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
> > > > >
> > > > > at 
> > > > > com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
> > > > >
> > > > > at 
> > > > > com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
> > > > >          at 
> > > > > com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
> > > > >          at 
> > > > > com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
> > > > >
> > > > > at 
> > > > > com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
> > > > >          at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
> > > > >          at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
> > > > >          at 
> > > > > com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
> > > > >
> > > > > at javax.servlet.GenericServlet.init(GenericServlet.java:158)
> > > > >          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
> > > > > Method)
> > > > >          at 
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > >
> > > > > at 
> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > >
> > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > >          at 
> > > > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
> > > > >          at 
> > > > > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
> > > > >          at java.security.AccessController.doPrivileged(Native Method)
> > > > >          at 
> > > > > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > > > >          at 
> > > > > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> > > > >          at 
> > > > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
> > > > >
> > > > > at java.security.AccessController.doPrivileged(Native Method)
> > > > >          at 
> > > > > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> > > > >          at 
> > > > > org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
> > > > >
> > > > > at 
> > > > > org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
> > > > >
> > > > > at 
> > > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > >
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > ..
> > >
> > > could this have some causation in expired certs?
> >
> > Hi,
> >
> > CA_WORKING means that certmonger's helper is trying to download the 
> > certificate from LDAP, but does not find new certs.
> >
> > In topologies with multiple servers, only one server is the renewal 
> > master. When one of auditSigningCert cert-pki-ca, ocspSigningCert 
> > cert-pki-ca, subsystemCert cert-pki-ca or caSigningCert cert-pki-ca 
> > expires, the renewal master is the one that actually handles the 
> > renewal, and the other masters simply download the new certs from LDAP.
> >
> > You need to check which server is your renewal master (ipa config-show 
> > | grep 'IPA CA renewal master'), then make sure that the certs were 
> > properly renewed on this master (check consistency between 
> > /etc/pki/pki-tomcat/alias, the certs in 
> > cn=certificates,cn=ipa,cn=etc,$BASEDN, and the content in 
> > /etc/pki/pki-tomcat/ca/CS.cfg).
> >
> > Then check that replication is working between the renewal master and 
> > the other masters. If the replication is broken, the certs will not be 
> > copied on the other masters and the download will not detect new 
> > certificates.
> >
> > HTH,
> > Flo
>
> I had no renewal master at all! Was it gone with a replica a removed 
> from IPA? If/when CA renewal master is removed from IPA(if it's allowed 
> in the first place?) does another server assume this role automatically?

Hi,

when the renewal master is removed (for instance through 
ipa-server-install --uninstall), FreeIPA should pick another host where 
CA instance is configured and set this host as renewal master.

Flo
> Now I've set up a renewal server as per the doc but still cannot start 
> that new CA renewal server > How to renew those expired certs? Maybe this has to be done manually
> somehow prior, in order to start it all?
> > > $ getcert list | grep -E "Request ID|status|certificate|expires"
> > > Number of certificates and requests being tracked: 9.
> > > Request ID '20170920090053':
> > >      status: MONITORING
> > >      certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
> > >      expires: 2018-09-20 09:00:53 UTC
> > >      certificate template/profile: KDCs_PKINIT_Certs
> > > Request ID '20171221120303':
> > >      status: CA_WORKING
> > >      certificate: 
> > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert 
> > > cert-pki-ca',token='NSS Certificate DB'
> > >      expires: 2018-03-27 14:07:51 UTC
> > > Request ID '20171221120304':
> > >      status: CA_WORKING
> > >      certificate: 
> > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert 
> > > cert-pki-ca',token='NSS Certificate DB'
> > >      expires: 2018-03-27 14:07:50 UTC
> > > Request ID '20171221120305':
> > >      status: CA_WORKING
> > >      certificate: 
> > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert 
> > > cert-pki-ca',token='NSS Certificate DB'
> > >      expires: 2018-03-27 14:07:51 UTC
> > > Request ID '20171221120306':
> > >      status: MONITORING
> > >      certificate: 
> > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert 
> > > cert-pki-ca',token='NSS Certificate DB'
> > >      expires: 2036-04-06 14:07:49 UTC
> > > Request ID '20171221120307':
> > >      status: CA_WORKING
> > >      certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
> > >      expires: 2018-03-27 14:08:18 UTC
> > > Request ID '20171221120308':
> > >      status: MONITORING
> > >      certificate: 
> > > type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert 
> > > cert-pki-ca',token='NSS Certificate DB'
> > >      expires: 2019-07-25 16:13:23 UTC
> > > Request ID '20171221120309':
> > >      status: MONITORING
> > >      certificate: 
> > > type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-CCNR-CEB-PRIVATE-CAM-AC-UK',nickname='Server-Cert',token='NSS 
> > > Certificate DB'
> > >      expires: 2019-04-09 12:11:52 UTC
> > > Request ID '20171221120310':
> > >      status: MONITORING
> > >      certificate: 
> > > type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS 
> > > Certificate DB'
> > >      expires: 2019-04-09 12:11:54 UTC
> > > _______________________________________________
> > > FreeIPA-users mailing list -- [e-mail]
> > > To unsubscribe send an email to 
> > > [e-mail]
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[lejeczek]

On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> Hi,
>
> CA_WORKING means that certmonger's helper is trying to 
> download the certificate from LDAP, but does not find new 
> certs.
>
> In topologies with multiple servers, only one server is 
> the renewal master. When one of auditSigningCert 
> cert-pki-ca, ocspSigningCert cert-pki-ca, subsystemCert 
> cert-pki-ca or caSigningCert cert-pki-ca expires, the 
> renewal master is the one that actually handles the 
> renewal, and the other masters simply download the new 
> certs from LDAP.
>
> You need to check which server is your renewal master (ipa 
> config-show | grep 'IPA CA renewal master'), then make 
> sure that the certs were properly renewed on this master 
> (check consistency between /etc/pki/pki-tomcat/alias, the 
> certs in cn=certificates,cn=ipa,cn=etc,$BASEDN, and the 
> content in /etc/pki/pki-tomcat/ca/CS.cfg).
>
> Then check that replication is working between the renewal 
> master and the other masters. If the replication is 
> broken, the certs will not be copied on the other masters 
> and the download will not detect new certificates.
>
> HTH,

I also see differences here in case it mattes(and then what 
to do about it):
on rider:

Replica Update Vectors:
     rider.private:389: 71
     whale.private:389: 91
Certificate Server Replica Update Vectors:
     rider.private:389: 1075
     whale.private:389: 1170

on whale:

Replica Update Vectors:
     whale.private:389: 91
     rider.private:389: 71
Certificate Server Replica Update Vectors:
     whale.private:389: 1170

Also on whale host I see:
..
[04/Apr/2018:14:19:28.872403514 +0100] - WARN - 
NSMMReplicationPlugin - repl5_inc_run - 
agmt="cn=cloneAgreement1-whale.private-pki-tomcat" 
(rider:389): The remote replica has a different database 
generation ID than the local database.  You may have to 
reinitialize the remote replica, or the local replica.

even though on rider I did:

$ ipa-replica-manage re-initialize --from

[Florence Blanc-Renaud]

On 04/04/2018 03:21 PM, lejeczek via FreeIPA-users wrote:
> On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> > Hi,
> >
> > CA_WORKING means that certmonger's helper is trying to download the 
> > certificate from LDAP, but does not find new certs.
> >
> > In topologies with multiple servers, only one server is the renewal 
> > master. When one of auditSigningCert cert-pki-ca, ocspSigningCert 
> > cert-pki-ca, subsystemCert cert-pki-ca or caSigningCert cert-pki-ca 
> > expires, the renewal master is the one that actually handles the 
> > renewal, and the other masters simply download the new certs from LDAP.
> >
> > You need to check which server is your renewal master (ipa config-show 
> > | grep 'IPA CA renewal master'), then make sure that the certs were 
> > properly renewed on this master (check consistency between 
> > /etc/pki/pki-tomcat/alias, the certs in 
> > cn=certificates,cn=ipa,cn=etc,$BASEDN, and the content in 
> > /etc/pki/pki-tomcat/ca/CS.cfg).
> >
> > Then check that replication is working between the renewal master and 
> > the other masters. If the replication is broken, the certs will not be 
> > copied on the other masters and the download will not detect new 
> > certificates.
> >
> > HTH,
>
> I also see differences here in case it mattes(and then what to do about 
> it):
> on rider:
>
> Replica Update Vectors:
>      rider.private:389: 71
>      whale.private:389: 91
> Certificate Server Replica Update Vectors:
>      rider.private:389: 1075
>      whale.private:389: 1170
>
> on whale:
>
> Replica Update Vectors:
>      whale.private:389: 91
>      rider.private:389: 71
> Certificate Server Replica Update Vectors:
>      whale.private:389: 1170
>
> Also on whale host I see:
> ..
> [04/Apr/2018:14:19:28.872403514 +0100] - WARN - NSMMReplicationPlugin - 
> repl5_inc_run - agmt="cn=cloneAgreement1-whale.private-pki-tomcat" 
> (rider:389): The remote replica has a different database generation ID 
> than the local database.  You may have to reinitialize the remote 
> replica, or the local replica.
>
> even though on rider I did:
>
> $ ipa-replica-manage re-initialize --from

The replication handles 2 different suffixes, one for IdM data (below 
dc=domain,dc=com), and one for CA data (below o=ipaca). In your case, 
the replication of CA data is broken and the right command to fix that 
is ipa-csreplica-manage re-initialize instead of ipa-replica-manage 
re-initialize.

Flo
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[lejeczek]

On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> You need to check which server is your renewal master (ipa 
> config-show | grep 'IPA CA renewal master'), then make 
> sure that the certs were properly renewed on this master 
> (check consistency between /etc/pki/pki-tomcat/alias, the 
> certs in cn=certificates,cn=ipa,cn=etc,$BASEDN, and the 
> content in /etc/pki/pki-tomcat/ca/CS.cfg).

I have only one cert, a ipaCertSubject: CN=Certificate 
Authority,O=PRIVATE.CCNR.CEB.PRIVATE.CAM.AC.UK
which seems to correspond with:
$ certutil -L -d /etc/pki/pki-tomcat/alias -f 
/tmp/pwdfile.txt -n 'caSigningCert cert-pki-ca'

which is also in /etc/pki/pki-tomcat/ca/CS.cfg, and that is: 
ca.signing.cert which is different from ca.subsystem.cert
But I'd imagine that's expected(?)

New CA master renewing server still fails:
...
[04/Apr/2018:15:14:44][localhost-startStop-1]: 
SSLClientCertificatSelectionCB: Entering!
[04/Apr/2018:15:14:44][localhost-startStop-1]: Candidate 
cert: Server-Cert cert-pki-ca
[04/Apr/2018:15:14:44][localhost-startStop-1]: Candidate 
cert: caSigningCert cert-pki-ca
[04/Apr/2018:15:14:44][localhost-startStop-1]: 
SSLClientCertificateSelectionCB: returning: null
[04/Apr/2018:15:14:44][localhost-startStop-1]: SSL handshake 
happened
Could not connect to LDAP server host whale port 636 Error 
netscape.ldap.LDAPException: Authentication failed (48)
...

It seems that these certs are as they should be. How can 
troubleshoot it further? Can logs verbosity be upped?
Many thanks, L.

[Florence Blanc-Renaud]

On 04/04/2018 04:16 PM, lejeczek via FreeIPA-users wrote:
> On 04/04/18 12:43, Florence Blanc-Renaud wrote:
> > You need to check which server is your renewal master (ipa config-show 
> > | grep 'IPA CA renewal master'), then make sure that the certs were 
> > properly renewed on this master (check consistency between 
> > /etc/pki/pki-tomcat/alias, the certs in 
> > cn=certificates,cn=ipa,cn=etc,$BASEDN, and the content in 
> > /etc/pki/pki-tomcat/ca/CS.cfg).
>
> I have only one cert, a ipaCertSubject: CN=Certificate 
> Authority,O=PRIVATE.CCNR.CEB.PRIVATE.CAM.AC.UK
> which seems to correspond with:
> $ certutil -L -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 
> 'caSigningCert cert-pki-ca'

The renewed certificates (if any) can be found in LDAP below 
cn=ca_renewal,cn=ipa,cn=etc,$BASEDN. If your replication got broken at 
one point, you need to check on different masters.
> which is also in /etc/pki/pki-tomcat/ca/CS.cfg, and that is: 
> ca.signing.cert which is different from ca.subsystem.cert
> But I'd imagine that's expected(?)
>
> New CA master renewing server still fails:
> ...
> [04/Apr/2018:15:14:44][localhost-startStop-1]: 
> SSLClientCertificatSelectionCB: Entering!
> [04/Apr/2018:15:14:44][localhost-startStop-1]: Candidate cert: 
> Server-Cert cert-pki-ca
> [04/Apr/2018:15:14:44][localhost-startStop-1]: Candidate cert: 
> caSigningCert cert-pki-ca
> [04/Apr/2018:15:14:44][localhost-startStop-1]: 
> SSLClientCertificateSelectionCB: returning: null
> [04/Apr/2018:15:14:44][localhost-startStop-1]: SSL handshake happened
> Could not connect to LDAP server host whale port 636 Error 
> netscape.ldap.LDAPException: Authentication failed (48)
> ...
>
> It seems that these certs are as they should be. How can troubleshoot it 
> further? Can logs verbosity be upped?

You can add verbosity by creating this file:
$ cat /etc/ipa/server.conf
[global]
debug=True

then restart ipa stack. This will add information in httpd's logs.

HTH,
Flo
> Many thanks, L.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]