From thofmann at fedoraproject.org Wed Jul 24 12:12:59 2019 Content-Type: multipart/mixed; boundary="===============1907082459813153612==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 12:12:29 +0000 Message-ID: <20190724121229.12270.6237@mailman01.phx2.fedoraproject.org> --===============1907082459813153612== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi all, I'm trying to set up a replica on CentOS 7, the master is on CentOS 6. Even= tually, I want to retire the CentOS 6 host. I'm following this migration gu= ide: https://www.freeipa.org/page/Howto/Migration#Migrating_existing_FreeIP= A_deployment However, running `ipa-replica-install --setup-ca ./replica-info-replica.fqd= n.gpg` always gets stuck and eventually fails when setting up pki-tomcatd: Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/28]: configuring certificate server instance [2/28]: exporting Dogtag certificate store pin [3/28]: stopping certificate server instance to update CS.cfg [4/28]: backing up CS.cfg [5/28]: disabling nonces [6/28]: set up CRL publishing [7/28]: enable PKIX certificate path discovery and validation [8/28]: starting certificate server instance [9/28]: configure certmonger for renewals [10/28]: importing RA certificate from PKCS #12 file [11/28]: setting audit signing renewal to 2 years [12/28]: restarting certificate server [13/28]: authorizing RA to modify profiles [14/28]: authorizing RA to manage lightweight CAs [15/28]: Ensure lightweight CAs container exists [16/28]: Ensuring backward compatibility [17/28]: configure certificate renewals [18/28]: configure Server-Cert certificate renewal [19/28]: Configure HTTP to proxy connections [20/28]: restarting certificate server [21/28]: updating IPA configuration [22/28]: enabling CA instance [23/28]: exposing CA instance on LDAP [24/28]: migrating certificate profiles to LDAP [25/28]: importing IPA certificate profiles [26/28]: adding default CA ACL [27/28]: adding 'ipa' CA entry [28/28]: configuring certmonger renewal for lightweight CAs Done configuring certificate server (pki-tomcatd). Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. ipapython.admintool: ERROR CA did not start in 300.0s ipapython.admintool: ERROR The ipa-replica-install command failed. See /= var/log/ipareplica-install.log for more information Looking at `ipareplica-install.log`: 2019-07-24T11:14:21Z DEBUG stderr=3D 2019-07-24T11:14:21Z DEBUG wait_for_open_ports: localhost [8080, 8443] time= out 300 2019-07-24T11:14:21Z DEBUG waiting for port: 8080 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on ::1 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on 127.0.0.1 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8080 2019-07-24T11:14:25Z DEBUG waiting for port: 8443 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8443 2019-07-24T11:14:25Z DEBUG Start of pki-tomcatd(a)pki-tomcat.service comple= te 2019-07-24T11:14:25Z DEBUG Waiting until the CA is running 2019-07-24T11:14:25Z DEBUG request POST http://replica.fqdn:8080/ca/admin/c= a/getStatus 2019-07-24T11:14:25Z DEBUG request body '' 2019-07-24T11:14:44Z DEBUG response status 500 2019-07-24T11:14:44Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=3Dutf-8 Content-Language: en Content-Length: 2208 Date: Wed, 24 Jul 2019 11:14:44 GMT Connection: close 2019-07-24T11:14:44Z DEBUG response body 'Apache Tomcat/= 7.0.76 - Error report

HTTP Status 500 - Sub= system unavailable


type Ex= ception report

message Subsystem unavailable

d= escription The server encountered an internal error that prevented i= t from fulfilling this requ est.

exception

javax.ws.rs.ServiceUnavailableExcepti=
on: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurit=
yConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Auth=
enticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.val=
ves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catali=
na.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.cata=
lina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.=
coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:=
1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proce=
ss(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$Soc=
ketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolEx=
ecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.Thre=
adPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat=
.util.threads.TaskThrea
 d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.=
java:748)\n

note The full stack trace of the root cau= se is available in the Apache Tomcat/7.0.76 logs.


Apache Tomcat/7.0.76

' 2019-07-24T11:14:44Z DEBUG The CA status is: check interrupted due to error= : Retrieving CA status failed with status 500 2019-07-24T11:14:44Z DEBUG Waiting for CA to start... 2019-07-24T11:14:45Z DEBUG request POST http://replica.fqdn:8080/ca/admin/c= a/getStatus 2019-07-24T11:14:45Z DEBUG request body '' 2019-07-24T11:14:45Z DEBUG response status 500 2019-07-24T11:14:45Z DEBUG response headers Server: Apache-Coyote/1.1 Content-Type: text/html;charset=3Dutf-8 Content-Language: en Content-Length: 2208 Date: Wed, 24 Jul 2019 11:14:45 GMT Connection: close Looking into the log of pki-tomcatd, I see the following: Internal Database Error encountered: Could not connect to LDAP server host = replica.fqdn port 636 Error netscape.ldap.LDAPException: Authentication fai= led (48) [...] WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm(a)6a= e79124 background process javax.ws.rs.ServiceUnavailableException: Subsystem unavailable at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.j= ava:1356) at org.apache.catalina.core.StandardContext.backgroundProcess(StandardConte= xt.java:5958) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.proc= essChildren(ContainerBase.java:1542) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.proc= essChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.proc= essChildren(ContainerBase.java:1552) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(= ContainerBase.java:1520) at java.lang.Thread.run(Thread.java:748) I checked that the pki-tomcatd uses the right certificates, following this = guide: https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomca= td-fails-to-start/ Everything looked fine, i.e., tomcat uses the correct certificate and can a= lso read the private key. Interestingly, during the setup of the replica, the setup is stuck for quit= e some time (~30 minutes) in the step " [1/28]: configuring certificate se= rver instance". In the ns-slapd log, I can see a lot of the following: INFO - import_monitor_threads - import ipaca: Processed 40105 entries -- av= erage rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% I'm surprised by the number of entries. I had set up the same host as a rep= lica in a previous try, but needed to remove it due to another error. May t= hose be left-overs from the previous replica instance? I didn't see this ha= ppening on the first attempt. Before redoing the setup, I removed the host = from the replica set with `ipa-replica-manage del --force`, from the csrepl= ica with `ipa-csreplica-manage del --force`, and also deleted the host entr= y itself with `ipa host-del`. I also uninstalled the freeipa server on the = replica host. I'm also wondering about the `Authentication failed (48)`, as 48 indicates = LDAP_INAPPROPRIATE_AUTH. I'm not sure how to debug this. Any help is appreciated! Kind regards, Till --===============1907082459813153612==-- From fcami at redhat.com Wed Jul 24 12:32:40 2019 Content-Type: multipart/mixed; boundary="===============6259351590160670130==" MIME-Version: 1.0 From: =?utf-8?q?Fran=C3=A7ois_Cami_=3Cfcami_at_redhat=2Ecom=3E?= To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 14:32:03 +0200 Message-ID: In-Reply-To: 20190724121229.12270.6237@mailman01.phx2.fedoraproject.org --===============6259351590160670130== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, On Wed, Jul 24, 2019 at 2:13 PM Till Hofmann via FreeIPA-users wrote: > > Hi all, > > I'm trying to set up a replica on CentOS 7, the master is on CentOS 6. Ev= entually, I want to retire the CentOS 6 host. I'm following this migration = guide: https://www.freeipa.org/page/Howto/Migration#Migrating_existing_Free= IPA_deployment > > However, running `ipa-replica-install --setup-ca ./replica-info-replica.f= qdn.gpg` always gets stuck and eventually fails when setting up pki-tomcatd: > Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes > [1/28]: configuring certificate server instance > [2/28]: exporting Dogtag certificate store pin > [3/28]: stopping certificate server instance to update CS.cfg > [4/28]: backing up CS.cfg > [5/28]: disabling nonces > [6/28]: set up CRL publishing > [7/28]: enable PKIX certificate path discovery and validation > [8/28]: starting certificate server instance > [9/28]: configure certmonger for renewals > [10/28]: importing RA certificate from PKCS #12 file > [11/28]: setting audit signing renewal to 2 years > [12/28]: restarting certificate server > [13/28]: authorizing RA to modify profiles > [14/28]: authorizing RA to manage lightweight CAs > [15/28]: Ensure lightweight CAs container exists > [16/28]: Ensuring backward compatibility > [17/28]: configure certificate renewals > [18/28]: configure Server-Cert certificate renewal > [19/28]: Configure HTTP to proxy connections > [20/28]: restarting certificate server > [21/28]: updating IPA configuration > [22/28]: enabling CA instance > [23/28]: exposing CA instance on LDAP > [24/28]: migrating certificate profiles to LDAP > [25/28]: importing IPA certificate profiles > [26/28]: adding default CA ACL > [27/28]: adding 'ipa' CA entry > [28/28]: configuring certmonger renewal for lightweight CAs > Done configuring certificate server (pki-tomcatd). > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > ipapython.admintool: ERROR CA did not start in 300.0s > ipapython.admintool: ERROR The ipa-replica-install command failed. See= /var/log/ipareplica-install.log for more information > > Looking at `ipareplica-install.log`: > 2019-07-24T11:14:21Z DEBUG stderr=3D > 2019-07-24T11:14:21Z DEBUG wait_for_open_ports: localhost [8080, 8443] ti= meout 300 > 2019-07-24T11:14:21Z DEBUG waiting for port: 8080 > 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on ::1 > 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on 127.0.0.1 > 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8080 > 2019-07-24T11:14:25Z DEBUG waiting for port: 8443 > 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8443 > 2019-07-24T11:14:25Z DEBUG Start of pki-tomcatd(a)pki-tomcat.service comp= lete > 2019-07-24T11:14:25Z DEBUG Waiting until the CA is running > 2019-07-24T11:14:25Z DEBUG request POST http://replica.fqdn:8080/ca/admin= /ca/getStatus > 2019-07-24T11:14:25Z DEBUG request body '' > 2019-07-24T11:14:44Z DEBUG response status 500 > 2019-07-24T11:14:44Z DEBUG response headers Server: Apache-Coyote/1.1 > Content-Type: text/html;charset=3Dutf-8 > Content-Language: en > Content-Length: 2208 > Date: Wed, 24 Jul 2019 11:14:44 GMT > Connection: close > > 2019-07-24T11:14:44Z DEBUG response body 'Apache Tomca= t/7.0.76 - Error report

HTTP Status 500 - S= ubsystem unavailable


type = Exception report

message Subsystem unavailable

description The server encountered an internal error that prevented= it from fulfilling this requ > est.

exception

javax.ws.rs.ServiceUnavailableExcep=
tion: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecur=
ityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Au=
thenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v=
alves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.cata=
lina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.ca=
talina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apach=
e.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.jav=
a:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.pro=
cess(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$S=
ocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPool=
Executor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.Th=
readPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomc=
at.util.threads.TaskThrea
>  d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Threa=
d.java:748)\n

note The full stack trace of the root c= ause is available in the Apache Tomcat/7.0.76 logs.


Apache Tomcat/7.0.76

' > 2019-07-24T11:14:44Z DEBUG The CA status is: check interrupted due to err= or: Retrieving CA status failed with status 500 > 2019-07-24T11:14:44Z DEBUG Waiting for CA to start... > 2019-07-24T11:14:45Z DEBUG request POST http://replica.fqdn:8080/ca/admin= /ca/getStatus > 2019-07-24T11:14:45Z DEBUG request body '' > 2019-07-24T11:14:45Z DEBUG response status 500 > 2019-07-24T11:14:45Z DEBUG response headers Server: Apache-Coyote/1.1 > Content-Type: text/html;charset=3Dutf-8 > Content-Language: en > Content-Length: 2208 > Date: Wed, 24 Jul 2019 11:14:45 GMT > Connection: close > > Looking into the log of pki-tomcatd, I see the following: > Internal Database Error encountered: Could not connect to LDAP server hos= t replica.fqdn port 636 Error netscape.ldap.LDAPException: Authentication f= ailed (48) > [...] > WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm(a)= 6ae79124 background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:1= 37) > at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase= .java:1356) > at org.apache.catalina.core.StandardContext.backgroundProcess(StandardCon= text.java:5958) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1542) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1552) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1552) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.ru= n(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > > I checked that the pki-tomcatd uses the right certificates, following thi= s guide: > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom= catd-fails-to-start/ > Everything looked fine, i.e., tomcat uses the correct certificate and can= also read the private key. > > Interestingly, during the setup of the replica, the setup is stuck for qu= ite some time (~30 minutes) in the step " [1/28]: configuring certificate = server instance". In the ns-slapd log, I can see a lot of the following: > INFO - import_monitor_threads - import ipaca: Processed 40105 entries -- = average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% > I'm surprised by the number of entries. I had set up the same host as a r= eplica in a previous try, but needed to remove it due to another error. May= those be left-overs from the previous replica instance? I didn't see this = happening on the first attempt. Before redoing the setup, I removed the hos= t from the replica set with `ipa-replica-manage del --force`, from the csre= plica with `ipa-csreplica-manage del --force`, and also deleted the host en= try itself with `ipa host-del`. I also uninstalled the freeipa server on th= e replica host. Could you count the actual number of requests records in the o=3Dipaca suffix and examine them? Cheers Fran=C3=A7ois > I'm also wondering about the `Authentication failed (48)`, as 48 indicate= s LDAP_INAPPROPRIATE_AUTH. > > I'm not sure how to debug this. Any help is appreciated! > > Kind regards, > Till > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org --===============6259351590160670130==-- From flo at redhat.com Wed Jul 24 12:59:53 2019 Content-Type: multipart/mixed; boundary="===============3365864260157982345==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 14:59:18 +0200 Message-ID: In-Reply-To: 20190724121229.12270.6237@mailman01.phx2.fedoraproject.org --===============3365864260157982345== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 7/24/19 2:12 PM, Till Hofmann via FreeIPA-users wrote: > Hi all, > = > I'm trying to set up a replica on CentOS 7, the master is on CentOS 6. Ev= entually, I want to retire the CentOS 6 host. I'm following this migration = guide: https://www.freeipa.org/page/Howto/Migration#Migrating_existing_Free= IPA_deployment > = > However, running `ipa-replica-install --setup-ca ./replica-info-replica.f= qdn.gpg` always gets stuck and eventually fails when setting up pki-tomcatd: > Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes > [1/28]: configuring certificate server instance > [2/28]: exporting Dogtag certificate store pin > [3/28]: stopping certificate server instance to update CS.cfg > [4/28]: backing up CS.cfg > [5/28]: disabling nonces > [6/28]: set up CRL publishing > [7/28]: enable PKIX certificate path discovery and validation > [8/28]: starting certificate server instance > [9/28]: configure certmonger for renewals > [10/28]: importing RA certificate from PKCS #12 file > [11/28]: setting audit signing renewal to 2 years > [12/28]: restarting certificate server > [13/28]: authorizing RA to modify profiles > [14/28]: authorizing RA to manage lightweight CAs > [15/28]: Ensure lightweight CAs container exists > [16/28]: Ensuring backward compatibility > [17/28]: configure certificate renewals > [18/28]: configure Server-Cert certificate renewal > [19/28]: Configure HTTP to proxy connections > [20/28]: restarting certificate server > [21/28]: updating IPA configuration > [22/28]: enabling CA instance > [23/28]: exposing CA instance on LDAP > [24/28]: migrating certificate profiles to LDAP > [25/28]: importing IPA certificate profiles > [26/28]: adding default CA ACL > [27/28]: adding 'ipa' CA entry > [28/28]: configuring certmonger renewal for lightweight CAs > Done configuring certificate server (pki-tomcatd). > Your system may be partly configured. > Run /usr/sbin/ipa-server-install --uninstall to clean up. > ipapython.admintool: ERROR CA did not start in 300.0s > ipapython.admintool: ERROR The ipa-replica-install command failed. See= /var/log/ipareplica-install.log for more information > = > Looking at `ipareplica-install.log`: > 2019-07-24T11:14:21Z DEBUG stderr=3D > 2019-07-24T11:14:21Z DEBUG wait_for_open_ports: localhost [8080, 8443] ti= meout 300 > 2019-07-24T11:14:21Z DEBUG waiting for port: 8080 > 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on ::1 > 2019-07-24T11:14:21Z DEBUG Failed to connect to port 8080 tcp on 127.0.0.1 > 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8080 > 2019-07-24T11:14:25Z DEBUG waiting for port: 8443 > 2019-07-24T11:14:25Z DEBUG SUCCESS: port: 8443 > 2019-07-24T11:14:25Z DEBUG Start of pki-tomcatd(a)pki-tomcat.service comp= lete > 2019-07-24T11:14:25Z DEBUG Waiting until the CA is running > 2019-07-24T11:14:25Z DEBUG request POST http://replica.fqdn:8080/ca/admin= /ca/getStatus > 2019-07-24T11:14:25Z DEBUG request body '' > 2019-07-24T11:14:44Z DEBUG response status 500 > 2019-07-24T11:14:44Z DEBUG response headers Server: Apache-Coyote/1.1 > Content-Type: text/html;charset=3Dutf-8 > Content-Language: en > Content-Length: 2208 > Date: Wed, 24 Jul 2019 11:14:44 GMT > Connection: close > = > 2019-07-24T11:14:44Z DEBUG response body 'Apache Tomca= t/7.0.76 - Error report

HTTP Status 500 - S= ubsystem unavailable


type = Exception report

message Subsystem unavailable

description The server encountered an internal error that prevented= it from fulfilling this requ > est.

exception

javax.ws.rs.ServiceUnavailableExce=
ption: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecu=
rityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.A=
uthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.=
valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.cat=
alina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.c=
atalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apac=
he.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.ja=
va:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.pr=
ocess(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$=
SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoo=
lExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.T=
hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tom=
cat.util.threads.TaskThrea
>   d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre=
ad.java:748)\n

note The full stack trace of the root = cause is available in the Apache Tomcat/7.0.76 logs.


Apache Tomcat/7.0.76

' > 2019-07-24T11:14:44Z DEBUG The CA status is: check interrupted due to err= or: Retrieving CA status failed with status 500 > 2019-07-24T11:14:44Z DEBUG Waiting for CA to start... > 2019-07-24T11:14:45Z DEBUG request POST http://replica.fqdn:8080/ca/admin= /ca/getStatus > 2019-07-24T11:14:45Z DEBUG request body '' > 2019-07-24T11:14:45Z DEBUG response status 500 > 2019-07-24T11:14:45Z DEBUG response headers Server: Apache-Coyote/1.1 > Content-Type: text/html;charset=3Dutf-8 > Content-Language: en > Content-Length: 2208 > Date: Wed, 24 Jul 2019 11:14:45 GMT > Connection: close > = > Looking into the log of pki-tomcatd, I see the following: > Internal Database Error encountered: Could not connect to LDAP server hos= t replica.fqdn port 636 Error netscape.ldap.LDAPException: Authentication f= ailed (48) Hi, a few things to check on the replica: - is the ldap server running and listening on port 636? - is there any SSL error in the log of pki-tomcatd? I recall an issue = 6577 [1] with a topology initially deployed as CA-less, then CA got = installed but the admin forgot to run ipa-certupdate on the nodes. As a = result, the CA cert was not put in all the relevant databases and = replica files did not contain the CA cert. [1] https://pagure.io/freeipa/issue/6577 flo > [...] > WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm(a)= 6ae79124 background process > javax.ws.rs.ServiceUnavailableException: Subsystem unavailable > at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:1= 37) > at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase= .java:1356) > at org.apache.catalina.core.StandardContext.backgroundProcess(StandardCon= text.java:5958) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1542) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1552) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr= ocessChildren(ContainerBase.java:1552) > at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.ru= n(ContainerBase.java:1520) > at java.lang.Thread.run(Thread.java:748) > = > I checked that the pki-tomcatd uses the right certificates, following thi= s guide: > https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom= catd-fails-to-start/ > Everything looked fine, i.e., tomcat uses the correct certificate and can= also read the private key. > = > Interestingly, during the setup of the replica, the setup is stuck for qu= ite some time (~30 minutes) in the step " [1/28]: configuring certificate = server instance". In the ns-slapd log, I can see a lot of the following: > INFO - import_monitor_threads - import ipaca: Processed 40105 entries -- = average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% > I'm surprised by the number of entries. I had set up the same host as a r= eplica in a previous try, but needed to remove it due to another error. May= those be left-overs from the previous replica instance? I didn't see this = happening on the first attempt. Before redoing the setup, I removed the hos= t from the replica set with `ipa-replica-manage del --force`, from the csre= plica with `ipa-csreplica-manage del --force`, and also deleted the host en= try itself with `ipa host-del`. I also uninstalled the freeipa server on th= e replica host. > = > I'm also wondering about the `Authentication failed (48)`, as 48 indicate= s LDAP_INAPPROPRIATE_AUTH. > = > I'm not sure how to debug this. Any help is appreciated! > = > Kind regards, > Till > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org > = --===============3365864260157982345==-- From thofmann at fedoraproject.org Wed Jul 24 14:04:12 2019 Content-Type: multipart/mixed; boundary="===============8818805164479705873==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 16:03:48 +0200 Message-ID: In-Reply-To: CAGdKLmX=Y_cL=eK0rR=jYeq-uP2NSRm9-6e45gM=Oqbwu81s=Q@mail.gmail.com --===============8818805164479705873== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Fran=C3=A7ois, Thanks for the reply! On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: >> >> Interestingly, during the setup of the replica, the setup is stuck for q= uite some time (~30 minutes) in the step " [1/28]: configuring certificate= server instance". In the ns-slapd log, I can see a lot of the following: >> INFO - import_monitor_threads - import ipaca: Processed 40105 entries --= average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% >> I'm surprised by the number of entries. I had set up the same host as a = replica in a previous try, but needed to remove it due to another error. Ma= y those be left-overs from the previous replica instance? I didn't see this= happening on the first attempt. Before redoing the setup, I removed the ho= st from the replica set with `ipa-replica-manage del --force`, from the csr= eplica with `ipa-csreplica-manage del --force`, and also deleted the host e= ntry itself with `ipa host-del`. I also uninstalled the freeipa server on t= he replica host. > = > Could you count the actual number of requests records in the o=3Dipaca > suffix and examine them? I'm not exactly sure what you mean (I don't have much experience with LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (but I guess that's not what you meant). On the replica, ns-slapd processed 267358 entries before finishing. Kind regards, Till --===============8818805164479705873==-- From thofmann at fedoraproject.org Wed Jul 24 14:32:10 2019 Content-Type: multipart/mixed; boundary="===============6139165044010631622==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 16:31:40 +0200 Message-ID: <838b1b09-a373-324c-f7d0-3f2b4c92a86f@fedoraproject.org> In-Reply-To: ec97f29a-0f82-e584-4c42-ed1fdca8d8be@redhat.com --===============6139165044010631622== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Florence, Thanks for the pointers! On 7/24/19 2:59 PM, Florence Blanc-Renaud wrote: > = > Hi, > = > a few things to check on the replica: > - is the ldap server running and listening on port 636? Yes, the server is running and listening to port 636. I can also query the server, but only after running `export LDAPTLS_CACERT=3D/etc/ipa/ca.crt`. If I don't set the CACERT, then the connection fails with ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) > - is there any SSL error in the log of pki-tomcatd? I recall an issue > 6577 [1] with a topology initially deployed as CA-less, then CA got > installed but the admin forgot to run ipa-certupdate on the nodes. As a > result, the CA cert was not put in all the relevant databases and > replica files did not contain the CA cert. Do you mean other than this one: Internal Database Error encountered: Could not connect to LDAP server host replica.fqdn port 636 Error netscape.ldap.LDAPException: Authentication failed (48) I can also see the following error, but that's before pki-tomcatd is restarted: testLDAPConnection: The specified user cn=3DReplication Manager masterAgreement1-replica.fqdn.de-pki-tomcat,cn=3Dconfig does not exist CMSEngine: init(): password test execution failed for replicationdbwith NO_SUCH_USER. This may not be a latest instance. Ignoring .. The error in #6577 looks quite similar, but the scenario is different. I can't just call ipa-certupdate because there is no configured ipa client on the replica. I'm also wondering if I could just set up the replica without the CA, shut down the old master and then use ipa-ca-install to set up the CA on the new server. Would that work? Kind regards, Till > = > [1] https://pagure.io/freeipa/issue/6577 > = > flo >=20 --===============6139165044010631622==-- From thofmann at fedoraproject.org Wed Jul 24 15:49:09 2019 Content-Type: multipart/mixed; boundary="===============2968910065502240754==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 17:48:24 +0200 Message-ID: In-Reply-To: a86b12d4-6cfd-10ea-1103-95fbc19cb1a1@fedoraproject.org --===============2968910065502240754== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 7/24/19 4:03 PM, Till Hofmann wrote: > Hi Fran=C3=A7ois, > = > Thanks for the reply! > = > On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: > = >>> >>> Interestingly, during the setup of the replica, the setup is stuck for = quite some time (~30 minutes) in the step " [1/28]: configuring certificat= e server instance". In the ns-slapd log, I can see a lot of the following: >>> INFO - import_monitor_threads - import ipaca: Processed 40105 entries -= - average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% >>> I'm surprised by the number of entries. I had set up the same host as a= replica in a previous try, but needed to remove it due to another error. M= ay those be left-overs from the previous replica instance? I didn't see thi= s happening on the first attempt. Before redoing the setup, I removed the h= ost from the replica set with `ipa-replica-manage del --force`, from the cs= replica with `ipa-csreplica-manage del --force`, and also deleted the host = entry itself with `ipa host-del`. I also uninstalled the freeipa server on = the replica host. >> >> Could you count the actual number of requests records in the o=3Dipaca >> suffix and examine them? > = > = > I'm not exactly sure what you mean (I don't have much experience with > LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (but I > guess that's not what you meant). On the replica, ns-slapd processed > 267358 entries before finishing. OK, I was looking in the wrong place. The number of request records in LDAP is 268721. I'm not sure what exactly I should be looking for, but I don't see anything unusual. I'm currently looking into the ldap auth config of tomcat, I noticed that it looks quite different compared to the master instance. Kind regards, Till --===============2968910065502240754==-- From fcami at redhat.com Wed Jul 24 15:53:20 2019 Content-Type: multipart/mixed; boundary="===============1715706093827213924==" MIME-Version: 1.0 From: =?utf-8?q?Fran=C3=A7ois_Cami_=3Cfcami_at_redhat=2Ecom=3E?= To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 17:52:40 +0200 Message-ID: In-Reply-To: c7a3a126-294f-4044-b4a0-7e3f94d26397@fedoraproject.org --===============1715706093827213924== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, Jul 24, 2019 at 5:48 PM Till Hofmann = wrote: > > > > On 7/24/19 4:03 PM, Till Hofmann wrote: > > Hi Fran=C3=A7ois, > > > > Thanks for the reply! > > > > On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: > > > >>> > >>> Interestingly, during the setup of the replica, the setup is stuck fo= r quite some time (~30 minutes) in the step " [1/28]: configuring certific= ate server instance". In the ns-slapd log, I can see a lot of the following: > >>> INFO - import_monitor_threads - import ipaca: Processed 40105 entries= -- average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% > >>> I'm surprised by the number of entries. I had set up the same host as= a replica in a previous try, but needed to remove it due to another error.= May those be left-overs from the previous replica instance? I didn't see t= his happening on the first attempt. Before redoing the setup, I removed the= host from the replica set with `ipa-replica-manage del --force`, from the = csreplica with `ipa-csreplica-manage del --force`, and also deleted the hos= t entry itself with `ipa host-del`. I also uninstalled the freeipa server o= n the replica host. > >> > >> Could you count the actual number of requests records in the o=3Dipaca > >> suffix and examine them? > > > > > > I'm not exactly sure what you mean (I don't have much experience with > > LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (but I > > guess that's not what you meant). On the replica, ns-slapd processed > > 267358 entries before finishing. > > OK, I was looking in the wrong place. The number of request records in > LDAP is 268721. I'm not sure what exactly I should be looking for, but I > don't see anything unusual. I could be wrong but at 114 entries processed per second, 268721 would need 37 mins to complete and the timeout is at 5 mins (the 300 seconds above). Let me investigate a bit more and I'll get back to you. Cheers Fran=C3=A7ois > I'm currently looking into the ldap auth config of tomcat, I noticed > that it looks quite different compared to the master instance. > > Kind regards, > Till --===============1715706093827213924==-- From fcami at redhat.com Wed Jul 24 16:04:44 2019 Content-Type: multipart/mixed; boundary="===============5007028754785990017==" MIME-Version: 1.0 From: =?utf-8?q?Fran=C3=A7ois_Cami_=3Cfcami_at_redhat=2Ecom=3E?= To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 18:03:43 +0200 Message-ID: In-Reply-To: CAGdKLmWFcnxDPteTKFaX9g73X5APS0sSMdVfYXSu6o+78dngeQ@mail.gmail.com --===============5007028754785990017== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, Jul 24, 2019 at 5:52 PM Fran=C3=A7ois Cami wro= te: > > On Wed, Jul 24, 2019 at 5:48 PM Till Hofmann wrote: > > > > > > > > On 7/24/19 4:03 PM, Till Hofmann wrote: > > > Hi Fran=C3=A7ois, > > > > > > Thanks for the reply! > > > > > > On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: > > > > > >>> > > >>> Interestingly, during the setup of the replica, the setup is stuck = for quite some time (~30 minutes) in the step " [1/28]: configuring certif= icate server instance". In the ns-slapd log, I can see a lot of the followi= ng: > > >>> INFO - import_monitor_threads - import ipaca: Processed 40105 entri= es -- average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% > > >>> I'm surprised by the number of entries. I had set up the same host = as a replica in a previous try, but needed to remove it due to another erro= r. May those be left-overs from the previous replica instance? I didn't see= this happening on the first attempt. Before redoing the setup, I removed t= he host from the replica set with `ipa-replica-manage del --force`, from th= e csreplica with `ipa-csreplica-manage del --force`, and also deleted the h= ost entry itself with `ipa host-del`. I also uninstalled the freeipa server= on the replica host. > > >> > > >> Could you count the actual number of requests records in the o=3Dipa= ca > > >> suffix and examine them? > > > > > > > > > I'm not exactly sure what you mean (I don't have much experience with > > > LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (but= I > > > guess that's not what you meant). On the replica, ns-slapd processed > > > 267358 entries before finishing. > > > > OK, I was looking in the wrong place. The number of request records in > > LDAP is 268721. I'm not sure what exactly I should be looking for, but I > > don't see anything unusual. > > I could be wrong but at 114 entries processed per second, 268721 would > need 37 mins to complete and the timeout is at 5 mins (the 300 seconds > above). > > Let me investigate a bit more and I'll get back to you. So it looks like you are hitting an issue Rob is already working on: https://github.com/freeipa/freeipa/pull/3374 but the fix is WIP. In the meantime, can you edit ipalib/constants.py in your system? Around line 175 there should be: ('replication_wait_timeout', 300), It should be safe to replace 300 with 3600 at replica install time. If install succeeds then wait at least 1h then shut down services: # ipactl stop revert the above change, and restart IPA: # ipactl start Please let us know if this helped. Fran=C3=A7ois > Cheers > Fran=C3=A7ois > > > I'm currently looking into the ldap auth config of tomcat, I noticed > > that it looks quite different compared to the master instance. > > > > Kind regards, > > Till --===============5007028754785990017==-- From thofmann at fedoraproject.org Wed Jul 24 19:31:06 2019 Content-Type: multipart/mixed; boundary="===============4741864732279172261==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Wed, 24 Jul 2019 21:30:20 +0200 Message-ID: <96c9db35-bbe5-70f2-f89f-ddc8fe821ac1@fedoraproject.org> In-Reply-To: CAGdKLmVf-X+_8bTY-XfV9VabO+hgi997myemnSXHWZd6V9FXsw@mail.gmail.com --===============4741864732279172261== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 7/24/19 6:03 PM, Fran=C3=A7ois Cami wrote: > On Wed, Jul 24, 2019 at 5:52 PM Fran=C3=A7ois Cami w= rote: >> >> On Wed, Jul 24, 2019 at 5:48 PM Till Hofmann wrote: >>> >>> >>> >>> On 7/24/19 4:03 PM, Till Hofmann wrote: >>>> Hi Fran=C3=A7ois, >>>> >>>> Thanks for the reply! >>>> >>>> On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: >>>> >>>>>> >>>>>> Interestingly, during the setup of the replica, the setup is stuck f= or quite some time (~30 minutes) in the step " [1/28]: configuring certifi= cate server instance". In the ns-slapd log, I can see a lot of the followin= g: >>>>>> INFO - import_monitor_threads - import ipaca: Processed 40105 entrie= s -- average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% >>>>>> I'm surprised by the number of entries. I had set up the same host a= s a replica in a previous try, but needed to remove it due to another error= . May those be left-overs from the previous replica instance? I didn't see = this happening on the first attempt. Before redoing the setup, I removed th= e host from the replica set with `ipa-replica-manage del --force`, from the= csreplica with `ipa-csreplica-manage del --force`, and also deleted the ho= st entry itself with `ipa host-del`. I also uninstalled the freeipa server = on the replica host. >>>>> >>>>> Could you count the actual number of requests records in the o=3Dipaca >>>>> suffix and examine them? >>>> >>>> >>>> I'm not exactly sure what you mean (I don't have much experience with >>>> LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (but I >>>> guess that's not what you meant). On the replica, ns-slapd processed >>>> 267358 entries before finishing. >>> >>> OK, I was looking in the wrong place. The number of request records in >>> LDAP is 268721. I'm not sure what exactly I should be looking for, but I >>> don't see anything unusual. >> >> I could be wrong but at 114 entries processed per second, 268721 would >> need 37 mins to complete and the timeout is at 5 mins (the 300 seconds >> above). >> >> Let me investigate a bit more and I'll get back to you. > = > So it looks like you are hitting an issue Rob is already working on: > https://github.com/freeipa/freeipa/pull/3374 > but the fix is WIP. > = > In the meantime, can you edit ipalib/constants.py in your system? > Around line 175 there should be: > ('replication_wait_timeout', 300), > = > It should be safe to replace 300 with 3600 at replica install time. > = > If install succeeds then wait at least 1h then shut down services: > # ipactl stop > revert the above change, and restart IPA: > # ipactl start > = > Please let us know if this helped. > = Thanks for the suggestion. Unfortunately, that didn't help. The timeout timer only starts after the last step (28/28), but the syncing already happens before that, so the timeout doesn't affect that. I see the same errors as before, just for a longer time. I could get one step further by modifying the tomcat config so it wouldn't use the certificate, similar to what they did here: https://www.redhat.com/archives/freeipa-users/2017-January/msg00216.html However, that's just a workaround. Also, I immediately get another error: testLDAPConnection: The specified user cn=3DReplication Manager masterAgreement1-replica.fqdn-pki-tomcat,cn=3Dconfig does not exist CMSEngine: init(): password test execution failed for replicationdbwith NO_SUCH_USER. This may not be a latest instance. Ignoring .. Kind regards, Till --===============4741864732279172261==-- From arpittolani at gmail.com Fri Jul 26 07:18:47 2019 Content-Type: multipart/mixed; boundary="===============2332721283956479462==" MIME-Version: 1.0 From: Arpit Tolani To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Fri, 26 Jul 2019 12:48:08 +0530 Message-ID: In-Reply-To: 96c9db35-bbe5-70f2-f89f-ddc8fe821ac1@fedoraproject.org --===============2332721283956479462== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I added Replication timeout in /usr/share/dirsrv/data/template-dse.ldif on replica before ipa-replica-install which took care of time consumed for large data getting replicated. https://access.redhat.com/documentation/en-us/red_hat_directory_server/10/h= tml/administration_guide/setting-replication-timeout-periods You need to setup timeout as per this link. On Thu, Jul 25, 2019 at 1:37 AM Till Hofmann via FreeIPA-users < freeipa-users(a)lists.fedorahosted.org> wrote: > > > On 7/24/19 6:03 PM, Fran=C3=A7ois Cami wrote: > > On Wed, Jul 24, 2019 at 5:52 PM Fran=C3=A7ois Cami = wrote: > >> > >> On Wed, Jul 24, 2019 at 5:48 PM Till Hofmann < > thofmann(a)fedoraproject.org> wrote: > >>> > >>> > >>> > >>> On 7/24/19 4:03 PM, Till Hofmann wrote: > >>>> Hi Fran=C3=A7ois, > >>>> > >>>> Thanks for the reply! > >>>> > >>>> On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: > >>>> > >>>>>> > >>>>>> Interestingly, during the setup of the replica, the setup is stuck > for quite some time (~30 minutes) in the step " [1/28]: configuring > certificate server instance". In the ns-slapd log, I can see a lot of the > following: > >>>>>> INFO - import_monitor_threads - import ipaca: Processed 40105 > entries -- average rate 123.8/sec, recent rate 114.0/sec, hit ratio 100% > >>>>>> I'm surprised by the number of entries. I had set up the same host > as a replica in a previous try, but needed to remove it due to another > error. May those be left-overs from the previous replica instance? I didn= 't > see this happening on the first attempt. Before redoing the setup, I > removed the host from the replica set with `ipa-replica-manage del > --force`, from the csreplica with `ipa-csreplica-manage del --force`, and > also deleted the host entry itself with `ipa host-del`. I also uninstalled > the freeipa server on the replica host. > >>>>> > >>>>> Could you count the actual number of requests records in the o=3Dip= aca > >>>>> suffix and examine them? > >>>> > >>>> > >>>> I'm not exactly sure what you mean (I don't have much experience with > >>>> LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results (bu= t I > >>>> guess that's not what you meant). On the replica, ns-slapd processed > >>>> 267358 entries before finishing. > >>> > >>> OK, I was looking in the wrong place. The number of request records in > >>> LDAP is 268721. I'm not sure what exactly I should be looking for, but > I > >>> don't see anything unusual. > >> > >> I could be wrong but at 114 entries processed per second, 268721 would > >> need 37 mins to complete and the timeout is at 5 mins (the 300 seconds > >> above). > >> > >> Let me investigate a bit more and I'll get back to you. > > > > So it looks like you are hitting an issue Rob is already working on: > > https://github.com/freeipa/freeipa/pull/3374 > > but the fix is WIP. > > > > In the meantime, can you edit ipalib/constants.py in your system? > > Around line 175 there should be: > > ('replication_wait_timeout', 300), > > > > It should be safe to replace 300 with 3600 at replica install time. > > > > If install succeeds then wait at least 1h then shut down services: > > # ipactl stop > > revert the above change, and restart IPA: > > # ipactl start > > > > Please let us know if this helped. > > > > Thanks for the suggestion. Unfortunately, that didn't help. The timeout > timer only starts after the last step (28/28), but the syncing already > happens before that, so the timeout doesn't affect that. I see the same > errors as before, just for a longer time. > > I could get one step further by modifying the tomcat config so it > wouldn't use the certificate, similar to what they did here: > https://www.redhat.com/archives/freeipa-users/2017-January/msg00216.html > > However, that's just a workaround. Also, I immediately get another error: > testLDAPConnection: The specified user cn=3DReplication Manager > masterAgreement1-replica.fqdn-pki-tomcat,cn=3Dconfig does not exist > CMSEngine: init(): password test execution failed for replicationdbwith > NO_SUCH_USER. This may not be a latest instance. Ignoring .. > > > Kind regards, > Till > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedora= hosted.org > -- = Thanks & Regards Arpit Tolani --===============2332721283956479462== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdj5JIGFkZGVkIFJlcGxpY2F0aW9uIHRpbWVvdXQgaW4gL3Vzci9z aGFyZS9kaXJzcnYvZGF0YS90ZW1wbGF0ZS08c3BhbiBjbGFzcz0iZ21haWwtaWwiPmRzZTwvc3Bh bj4uPHNwYW4gY2xhc3M9ImdtYWlsLWlsIj5sZGlmIG9uIHJlcGxpY2EgYmVmb3JlIGlwYS1yZXBs aWNhLWluc3RhbGwgd2hpY2ggdG9vayBjYXJlIG9mIHRpbWUgY29uc3VtZWQgZm9yIGxhcmdlIGRh dGEgZ2V0dGluZyByZXBsaWNhdGVkLjwvc3Bhbj48YnI+PC9kaXY+PGRpdj48c3BhbiBjbGFzcz0i Z21haWwtaWwiPjxicj48L3NwYW4+PC9kaXY+PGRpdj48YSBocmVmPSJodHRwczovL2FjY2Vzcy5y ZWRoYXQuY29tL2RvY3VtZW50YXRpb24vZW4tdXMvcmVkX2hhdF9kaXJlY3Rvcnlfc2VydmVyLzEw L2h0bWwvYWRtaW5pc3RyYXRpb25fZ3VpZGUvc2V0dGluZy1yZXBsaWNhdGlvbi10aW1lb3V0LXBl cmlvZHMiPmh0dHBzOi8vYWNjZXNzLnJlZGhhdC5jb20vZG9jdW1lbnRhdGlvbi9lbi11cy9yZWRf aGF0X2RpcmVjdG9yeV9zZXJ2ZXIvMTAvaHRtbC9hZG1pbmlzdHJhdGlvbl9ndWlkZS9zZXR0aW5n LXJlcGxpY2F0aW9uLXRpbWVvdXQtcGVyaW9kczwvYT7CoFlvdSBuZWVkIHRvIHNldHVwIHRpbWVv dXQgYXMgcGVyIHRoaXMgbGluay7CoDxicj48L2Rpdj48L2Rpdj48YnI+PGRpdiBjbGFzcz0iZ21h aWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBUaHUsIEp1bCAy NSwgMjAxOSBhdCAxOjM3IEFNIFRpbGwgSG9mbWFubiB2aWEgRnJlZUlQQS11c2VycyAmbHQ7PGEg aHJlZj0ibWFpbHRvOmZyZWVpcGEtdXNlcnNAbGlzdHMuZmVkb3JhaG9zdGVkLm9yZyI+ZnJlZWlw YS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3JnPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2Pjxi bG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAw LjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6 MWV4Ij48YnI+Cjxicj4KT24gNy8yNC8xOSA2OjAzIFBNLCBGcmFuw6dvaXMgQ2FtaSB3cm90ZTo8 YnI+CiZndDsgT24gV2VkLCBKdWwgMjQsIDIwMTkgYXQgNTo1MiBQTSBGcmFuw6dvaXMgQ2FtaSAm bHQ7PGEgaHJlZj0ibWFpbHRvOmZjYW1pQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5mY2Ft aUByZWRoYXQuY29tPC9hPiZndDsgd3JvdGU6PGJyPgomZ3Q7Jmd0Ozxicj4KJmd0OyZndDsgT24g V2VkLCBKdWwgMjQsIDIwMTkgYXQgNTo0OCBQTSBUaWxsIEhvZm1hbm4gJmx0OzxhIGhyZWY9Im1h aWx0bzp0aG9mbWFubkBmZWRvcmFwcm9qZWN0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPnRob2ZtYW5u QGZlZG9yYXByb2plY3Qub3JnPC9hPiZndDsgd3JvdGU6PGJyPgomZ3Q7Jmd0OyZndDs8YnI+CiZn dDsmZ3Q7Jmd0Ozxicj4KJmd0OyZndDsmZ3Q7PGJyPgomZ3Q7Jmd0OyZndDsgT24gNy8yNC8xOSA0 OjAzIFBNLCBUaWxsIEhvZm1hbm4gd3JvdGU6PGJyPgomZ3Q7Jmd0OyZndDsmZ3Q7IEhpIEZyYW7D p29pcyw8YnI+CiZndDsmZ3Q7Jmd0OyZndDs8YnI+CiZndDsmZ3Q7Jmd0OyZndDsgVGhhbmtzIGZv ciB0aGUgcmVwbHkhPGJyPgomZ3Q7Jmd0OyZndDsmZ3Q7PGJyPgomZ3Q7Jmd0OyZndDsmZ3Q7IE9u IDcvMjQvMTkgMjozMiBQTSwgRnJhbsOnb2lzIENhbWkgd3JvdGU6PGJyPgomZ3Q7Jmd0OyZndDsm Z3Q7PGJyPgomZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyZndDs8YnI+CiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7 Jmd0OyBJbnRlcmVzdGluZ2x5LCBkdXJpbmcgdGhlIHNldHVwIG9mIHRoZSByZXBsaWNhLCB0aGUg c2V0dXAgaXMgc3R1Y2sgZm9yIHF1aXRlIHNvbWUgdGltZSAofjMwIG1pbnV0ZXMpIGluIHRoZSBz dGVwICZxdW90O8KgIFsxLzI4XTogY29uZmlndXJpbmcgY2VydGlmaWNhdGUgc2VydmVyIGluc3Rh bmNlJnF1b3Q7LiBJbiB0aGUgbnMtc2xhcGQgbG9nLCBJIGNhbiBzZWUgYSBsb3Qgb2YgdGhlIGZv bGxvd2luZzo8YnI+CiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBJTkZPIC0gaW1wb3J0X21vbml0 b3JfdGhyZWFkcyAtIGltcG9ydCBpcGFjYTogUHJvY2Vzc2VkIDQwMTA1IGVudHJpZXMgLS0gYXZl cmFnZSByYXRlIDEyMy44L3NlYywgcmVjZW50IHJhdGUgMTE0LjAvc2VjLCBoaXQgcmF0aW8gMTAw JTxicj4KJmd0OyZndDsmZ3Q7Jmd0OyZndDsmZ3Q7IEkmIzM5O20gc3VycHJpc2VkIGJ5IHRoZSBu dW1iZXIgb2YgZW50cmllcy4gSSBoYWQgc2V0IHVwIHRoZSBzYW1lIGhvc3QgYXMgYSByZXBsaWNh IGluIGEgcHJldmlvdXMgdHJ5LCBidXQgbmVlZGVkIHRvIHJlbW92ZSBpdCBkdWUgdG8gYW5vdGhl ciBlcnJvci4gTWF5IHRob3NlIGJlIGxlZnQtb3ZlcnMgZnJvbSB0aGUgcHJldmlvdXMgcmVwbGlj YSBpbnN0YW5jZT8gSSBkaWRuJiMzOTt0IHNlZSB0aGlzIGhhcHBlbmluZyBvbiB0aGUgZmlyc3Qg YXR0ZW1wdC4gQmVmb3JlIHJlZG9pbmcgdGhlIHNldHVwLCBJIHJlbW92ZWQgdGhlIGhvc3QgZnJv bSB0aGUgcmVwbGljYSBzZXQgd2l0aCBgaXBhLXJlcGxpY2EtbWFuYWdlIGRlbCAtLWZvcmNlYCwg ZnJvbSB0aGUgY3NyZXBsaWNhIHdpdGggYGlwYS1jc3JlcGxpY2EtbWFuYWdlIGRlbCAtLWZvcmNl YCwgYW5kIGFsc28gZGVsZXRlZCB0aGUgaG9zdCBlbnRyeSBpdHNlbGYgd2l0aCBgaXBhIGhvc3Qt ZGVsYC4gSSBhbHNvIHVuaW5zdGFsbGVkIHRoZSBmcmVlaXBhIHNlcnZlciBvbiB0aGUgcmVwbGlj YSBob3N0Ljxicj4KJmd0OyZndDsmZ3Q7Jmd0OyZndDs8YnI+CiZndDsmZ3Q7Jmd0OyZndDsmZ3Q7 IENvdWxkIHlvdSBjb3VudCB0aGUgYWN0dWFsIG51bWJlciBvZiByZXF1ZXN0cyByZWNvcmRzIGlu IHRoZSBvPWlwYWNhPGJyPgomZ3Q7Jmd0OyZndDsmZ3Q7Jmd0OyBzdWZmaXggYW5kIGV4YW1pbmUg dGhlbT88YnI+CiZndDsmZ3Q7Jmd0OyZndDs8YnI+CiZndDsmZ3Q7Jmd0OyZndDs8YnI+CiZndDsm Z3Q7Jmd0OyZndDsgSSYjMzk7bSBub3QgZXhhY3RseSBzdXJlIHdoYXQgeW91IG1lYW4gKEkgZG9u JiMzOTt0IGhhdmUgbXVjaCBleHBlcmllbmNlIHdpdGg8YnI+CiZndDsmZ3Q7Jmd0OyZndDsgTERB UCkuIFNlYXJjaGluZyBmb3IgJnF1b3Q7KG9iamVjdGNsYXNzPWlwYWNhKikmcXVvdDsgZ2l2ZXMg bWUgMiByZXN1bHRzIChidXQgSTxicj4KJmd0OyZndDsmZ3Q7Jmd0OyBndWVzcyB0aGF0JiMzOTtz IG5vdCB3aGF0IHlvdSBtZWFudCkuIE9uIHRoZSByZXBsaWNhLCBucy1zbGFwZCBwcm9jZXNzZWQ8 YnI+CiZndDsmZ3Q7Jmd0OyZndDsgMjY3MzU4IGVudHJpZXMgYmVmb3JlIGZpbmlzaGluZy48YnI+ CiZndDsmZ3Q7Jmd0Ozxicj4KJmd0OyZndDsmZ3Q7IE9LLCBJIHdhcyBsb29raW5nIGluIHRoZSB3 cm9uZyBwbGFjZS4gVGhlIG51bWJlciBvZiByZXF1ZXN0IHJlY29yZHMgaW48YnI+CiZndDsmZ3Q7 Jmd0OyBMREFQIGlzIDI2ODcyMS4gSSYjMzk7bSBub3Qgc3VyZSB3aGF0IGV4YWN0bHkgSSBzaG91 bGQgYmUgbG9va2luZyBmb3IsIGJ1dCBJPGJyPgomZ3Q7Jmd0OyZndDsgZG9uJiMzOTt0IHNlZSBh bnl0aGluZyB1bnVzdWFsLjxicj4KJmd0OyZndDs8YnI+CiZndDsmZ3Q7IEkgY291bGQgYmUgd3Jv bmcgYnV0IGF0IDExNCBlbnRyaWVzIHByb2Nlc3NlZCBwZXIgc2Vjb25kLCAyNjg3MjEgd291bGQ8 YnI+CiZndDsmZ3Q7IG5lZWQgMzcgbWlucyB0byBjb21wbGV0ZSBhbmQgdGhlIHRpbWVvdXQgaXMg YXQgNSBtaW5zICh0aGUgMzAwIHNlY29uZHM8YnI+CiZndDsmZ3Q7IGFib3ZlKS48YnI+CiZndDsm Z3Q7PGJyPgomZ3Q7Jmd0OyBMZXQgbWUgaW52ZXN0aWdhdGUgYSBiaXQgbW9yZSBhbmQgSSYjMzk7 bGwgZ2V0IGJhY2sgdG8geW91Ljxicj4KJmd0OyA8YnI+CiZndDsgU28gaXQgbG9va3MgbGlrZSB5 b3UgYXJlIGhpdHRpbmcgYW4gaXNzdWUgUm9iIGlzIGFscmVhZHkgd29ya2luZyBvbjo8YnI+CiZn dDsgPGEgaHJlZj0iaHR0cHM6Ly9naXRodWIuY29tL2ZyZWVpcGEvZnJlZWlwYS9wdWxsLzMzNzQi IHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vZ2l0aHViLmNvbS9mcmVl aXBhL2ZyZWVpcGEvcHVsbC8zMzc0PC9hPjxicj4KJmd0OyBidXQgdGhlIGZpeCBpcyBXSVAuPGJy PgomZ3Q7IDxicj4KJmd0OyBJbiB0aGUgbWVhbnRpbWUsIGNhbiB5b3UgZWRpdCBpcGFsaWIvY29u c3RhbnRzLnB5IGluIHlvdXIgc3lzdGVtPzxicj4KJmd0OyBBcm91bmQgbGluZSAxNzUgdGhlcmUg c2hvdWxkIGJlOjxicj4KJmd0OyAoJiMzOTtyZXBsaWNhdGlvbl93YWl0X3RpbWVvdXQmIzM5Oywg MzAwKSw8YnI+CiZndDsgPGJyPgomZ3Q7IEl0IHNob3VsZCBiZSBzYWZlIHRvIHJlcGxhY2UgMzAw IHdpdGggMzYwMCBhdCByZXBsaWNhIGluc3RhbGwgdGltZS48YnI+CiZndDsgPGJyPgomZ3Q7IElm IGluc3RhbGwgc3VjY2VlZHMgdGhlbiB3YWl0IGF0IGxlYXN0IDFoIHRoZW4gc2h1dCBkb3duIHNl cnZpY2VzOjxicj4KJmd0OyAjIGlwYWN0bCBzdG9wPGJyPgomZ3Q7IHJldmVydCB0aGUgYWJvdmUg Y2hhbmdlLCBhbmQgcmVzdGFydCBJUEE6PGJyPgomZ3Q7ICMgaXBhY3RsIHN0YXJ0PGJyPgomZ3Q7 IDxicj4KJmd0OyBQbGVhc2UgbGV0IHVzIGtub3cgaWYgdGhpcyBoZWxwZWQuPGJyPgomZ3Q7IDxi cj4KPGJyPgpUaGFua3MgZm9yIHRoZSBzdWdnZXN0aW9uLiBVbmZvcnR1bmF0ZWx5LCB0aGF0IGRp ZG4mIzM5O3QgaGVscC4gVGhlIHRpbWVvdXQ8YnI+CnRpbWVyIG9ubHkgc3RhcnRzIGFmdGVyIHRo ZSBsYXN0IHN0ZXAgKDI4LzI4KSwgYnV0IHRoZSBzeW5jaW5nIGFscmVhZHk8YnI+CmhhcHBlbnMg YmVmb3JlIHRoYXQsIHNvIHRoZSB0aW1lb3V0IGRvZXNuJiMzOTt0IGFmZmVjdCB0aGF0LiBJIHNl ZSB0aGUgc2FtZTxicj4KZXJyb3JzIGFzIGJlZm9yZSwganVzdCBmb3IgYSBsb25nZXIgdGltZS48 YnI+Cjxicj4KSSBjb3VsZCBnZXQgb25lIHN0ZXAgZnVydGhlciBieSBtb2RpZnlpbmcgdGhlIHRv bWNhdCBjb25maWcgc28gaXQ8YnI+CndvdWxkbiYjMzk7dCB1c2UgdGhlIGNlcnRpZmljYXRlLCBz aW1pbGFyIHRvIHdoYXQgdGhleSBkaWQgaGVyZTo8YnI+CjxhIGhyZWY9Imh0dHBzOi8vd3d3LnJl ZGhhdC5jb20vYXJjaGl2ZXMvZnJlZWlwYS11c2Vycy8yMDE3LUphbnVhcnkvbXNnMDAyMTYuaHRt bCIgcmVsPSJub3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly93d3cucmVkaGF0LmNv bS9hcmNoaXZlcy9mcmVlaXBhLXVzZXJzLzIwMTctSmFudWFyeS9tc2cwMDIxNi5odG1sPC9hPjxi cj4KPGJyPgpIb3dldmVyLCB0aGF0JiMzOTtzIGp1c3QgYSB3b3JrYXJvdW5kLiBBbHNvLCBJIGlt bWVkaWF0ZWx5IGdldCBhbm90aGVyIGVycm9yOjxicj4KdGVzdExEQVBDb25uZWN0aW9uOiBUaGUg c3BlY2lmaWVkIHVzZXIgY249UmVwbGljYXRpb24gTWFuYWdlcjxicj4KbWFzdGVyQWdyZWVtZW50 MS1yZXBsaWNhLmZxZG4tcGtpLXRvbWNhdCxjbj1jb25maWcgZG9lcyBub3QgZXhpc3Q8YnI+CkNN U0VuZ2luZTogaW5pdCgpOiBwYXNzd29yZCB0ZXN0IGV4ZWN1dGlvbiBmYWlsZWQgZm9yIHJlcGxp Y2F0aW9uZGJ3aXRoPGJyPgpOT19TVUNIX1VTRVIuwqAgVGhpcyBtYXkgbm90IGJlIGEgbGF0ZXN0 IGluc3RhbmNlLsKgIElnbm9yaW5nIC4uPGJyPgo8YnI+Cjxicj4KS2luZCByZWdhcmRzLDxicj4K VGlsbDxicj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188 YnI+CkZyZWVJUEEtdXNlcnMgbWFpbGluZyBsaXN0IC0tIDxhIGhyZWY9Im1haWx0bzpmcmVlaXBh LXVzZXJzQGxpc3RzLmZlZG9yYWhvc3RlZC5vcmciIHRhcmdldD0iX2JsYW5rIj5mcmVlaXBhLXVz ZXJzQGxpc3RzLmZlZG9yYWhvc3RlZC5vcmc8L2E+PGJyPgpUbyB1bnN1YnNjcmliZSBzZW5kIGFu IGVtYWlsIHRvIDxhIGhyZWY9Im1haWx0bzpmcmVlaXBhLXVzZXJzLWxlYXZlQGxpc3RzLmZlZG9y YWhvc3RlZC5vcmciIHRhcmdldD0iX2JsYW5rIj5mcmVlaXBhLXVzZXJzLWxlYXZlQGxpc3RzLmZl ZG9yYWhvc3RlZC5vcmc8L2E+PGJyPgpGZWRvcmEgQ29kZSBvZiBDb25kdWN0OiA8YSBocmVmPSJo dHRwczovL2RvY3MuZmVkb3JhcHJvamVjdC5vcmcvZW4tVVMvcHJvamVjdC9jb2RlLW9mLWNvbmR1 Y3QvIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwczovL2RvY3MuZmVkb3Jh cHJvamVjdC5vcmcvZW4tVVMvcHJvamVjdC9jb2RlLW9mLWNvbmR1Y3QvPC9hPjxicj4KTGlzdCBH dWlkZWxpbmVzOiA8YSBocmVmPSJodHRwczovL2ZlZG9yYXByb2plY3Qub3JnL3dpa2kvTWFpbGlu Z19saXN0X2d1aWRlbGluZXMiIHJlbD0ibm9yZWZlcnJlciIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBz Oi8vZmVkb3JhcHJvamVjdC5vcmcvd2lraS9NYWlsaW5nX2xpc3RfZ3VpZGVsaW5lczwvYT48YnI+ Ckxpc3QgQXJjaGl2ZXM6IDxhIGhyZWY9Imh0dHBzOi8vbGlzdHMuZmVkb3JhaG9zdGVkLm9yZy9h cmNoaXZlcy9saXN0L2ZyZWVpcGEtdXNlcnNAbGlzdHMuZmVkb3JhaG9zdGVkLm9yZyIgcmVsPSJu b3JlZmVycmVyIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly9saXN0cy5mZWRvcmFob3N0ZWQub3Jn L2FyY2hpdmVzL2xpc3QvZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3JnPC9hPjxi cj4KPC9ibG9ja3F1b3RlPjwvZGl2PjxiciBjbGVhcj0iYWxsIj48ZGl2Pjxicj48L2Rpdj4tLSA8 YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+ ClRoYW5rcyAmYW1wOyBSZWdhcmRzPGJyPkFycGl0IFRvbGFuaTxicj48YnI+PC9kaXY+PC9kaXY+ Cg== --===============2332721283956479462==-- From thofmann at fedoraproject.org Fri Jul 26 08:34:10 2019 Content-Type: multipart/mixed; boundary="===============2989589499081818087==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Fri, 26 Jul 2019 10:33:43 +0200 Message-ID: In-Reply-To: CAD3MydBei3O190vdUKbDkV74Gu5SBH_zeJYc9XN_qBUD1QVo-g@mail.gmail.com --===============2989589499081818087== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Arpit, On 7/26/19 9:18 AM, Arpit Tolani wrote: > I added Replication timeout in /usr/share/dirsrv/data/template-dse.ldif > on replica before ipa-replica-install which took care of time consumed > for large data getting replicated. > = > https://access.redhat.com/documentation/en-us/red_hat_directory_server/10= /html/administration_guide/setting-replication-timeout-periods=C2=A0You > need to setup timeout as per this link. = Thanks for the hint, but I really don't think a timeout is the issue here. After I did what Fran=C3=A7ois suggested, the replication went through (as before) and it then waited for 60 (instead of 5) minutes for pki-tomcatd to start, which repeatedly fails to authenticate with the following error: netscape.ldap.LDAPException: Authentication failed (48) There's is nothing else going on, so increasing more timeouts will just increase the time until it fails. To see if this is related to some leftover config flying around for the replica host, I set up a second replica host, which now fails much earlier during setup with the following error: NSMMReplicationPlugin - acquire_replica - agmt=3D"cn=3Dmaster.fqdn" (master:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. I tried this multiple times, always resulting in the same error. Kind regards, Till > = > On Thu, Jul 25, 2019 at 1:37 AM Till Hofmann via FreeIPA-users > > wrote: > = > = > = > On 7/24/19 6:03 PM, Fran=C3=A7ois Cami wrote: > > On Wed, Jul 24, 2019 at 5:52 PM Fran=C3=A7ois Cami > wrote: > >> > >> On Wed, Jul 24, 2019 at 5:48 PM Till Hofmann > > = wrote: > >>> > >>> > >>> > >>> On 7/24/19 4:03 PM, Till Hofmann wrote: > >>>> Hi Fran=C3=A7ois, > >>>> > >>>> Thanks for the reply! > >>>> > >>>> On 7/24/19 2:32 PM, Fran=C3=A7ois Cami wrote: > >>>> > >>>>>> > >>>>>> Interestingly, during the setup of the replica, the setup is > stuck for quite some time (~30 minutes) in the step "=C2=A0 [1/28]: > configuring certificate server instance". In the ns-slapd log, I can > see a lot of the following: > >>>>>> INFO - import_monitor_threads - import ipaca: Processed 40105 > entries -- average rate 123.8/sec, recent rate 114.0/sec, hit ratio 1= 00% > >>>>>> I'm surprised by the number of entries. I had set up the same > host as a replica in a previous try, but needed to remove it due to > another error. May those be left-overs from the previous replica > instance? I didn't see this happening on the first attempt. Before > redoing the setup, I removed the host from the replica set with > `ipa-replica-manage del --force`, from the csreplica with > `ipa-csreplica-manage del --force`, and also deleted the host entry > itself with `ipa host-del`. I also uninstalled the freeipa server on > the replica host. > >>>>> > >>>>> Could you count the actual number of requests records in the > o=3Dipaca > >>>>> suffix and examine them? > >>>> > >>>> > >>>> I'm not exactly sure what you mean (I don't have much > experience with > >>>> LDAP). Searching for "(objectclass=3Dipaca*)" gives me 2 results > (but I > >>>> guess that's not what you meant). On the replica, ns-slapd > processed > >>>> 267358 entries before finishing. > >>> > >>> OK, I was looking in the wrong place. The number of request > records in > >>> LDAP is 268721. I'm not sure what exactly I should be looking > for, but I > >>> don't see anything unusual. > >> > >> I could be wrong but at 114 entries processed per second, 268721 > would > >> need 37 mins to complete and the timeout is at 5 mins (the 300 > seconds > >> above). > >> > >> Let me investigate a bit more and I'll get back to you. > > > > So it looks like you are hitting an issue Rob is already working on: > > https://github.com/freeipa/freeipa/pull/3374 > > but the fix is WIP. > > > > In the meantime, can you edit ipalib/constants.py in your system? > > Around line 175 there should be: > > ('replication_wait_timeout', 300), > > > > It should be safe to replace 300 with 3600 at replica install time. > > > > If install succeeds then wait at least 1h then shut down services: > > # ipactl stop > > revert the above change, and restart IPA: > > # ipactl start > > > > Please let us know if this helped. > > > = > Thanks for the suggestion. Unfortunately, that didn't help. The timeo= ut > timer only starts after the last step (28/28), but the syncing already > happens before that, so the timeout doesn't affect that. I see the sa= me > errors as before, just for a longer time. > = > I could get one step further by modifying the tomcat config so it > wouldn't use the certificate, similar to what they did here: > https://www.redhat.com/archives/freeipa-users/2017-January/msg00216.h= tml > = > However, that's just a workaround. Also, I immediately get another > error: > testLDAPConnection: The specified user cn=3DReplication Manager > masterAgreement1-replica.fqdn-pki-tomcat,cn=3Dconfig does not exist > CMSEngine: init(): password test execution failed for replicationdbwi= th > NO_SUCH_USER.=C2=A0 This may not be a latest instance.=C2=A0 Ignoring= .. > = > = > Kind regards, > Till > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > > To unsubscribe send an email to > freeipa-users-leave(a)lists.fedorahosted.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelin= es > List Archives: > https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fe= dorahosted.org > = > = > = > -- = > Thanks & Regards > Arpit Tolani >=20 --===============2989589499081818087==-- From thofmann at fedoraproject.org Fri Jul 26 12:55:26 2019 Content-Type: multipart/mixed; boundary="===============8322397115984478557==" MIME-Version: 1.0 From: Till Hofmann To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Fri, 26 Jul 2019 12:55:05 +0000 Message-ID: <20190726125505.9625.26589@mailman01.phx2.fedoraproject.org> In-Reply-To: 20190724121229.12270.6237@mailman01.phx2.fedoraproject.org --===============8322397115984478557== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi all, I managed to work around the issue by: 1. Setting up the replica without the CA (i.e., `ipa-replica-install` witho= ut `--setup-ca`) 2. Set up the CA with `ipa-ca-install`. This also failed at some point (bec= ause it could not contact the old master on port 8443), but it seemed to do= "enough" so I could actually ignore the missing steps. I turned off the original master, verified that I could still log in on the= clients and also tested certificate renewal with `ipa-cacert-manage renew`= , which was successful. I don't know what the missing steps were, I hope this won't bite me in the = long run. Do you have any suggestions what else I could test to verify that= the CA is also working properly? Thanks for all the help! Kind regards, Till --===============8322397115984478557==-- From rcritten at redhat.com Fri Jul 26 13:06:46 2019 Content-Type: multipart/mixed; boundary="===============6650838527606875400==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: ipa-replica-install fails to start pki-tomcatd Date: Fri, 26 Jul 2019 09:06:19 -0400 Message-ID: <84e2d5a0-9f08-e676-0f42-adc5967116e7@redhat.com> In-Reply-To: 20190726125505.9625.26589@mailman01.phx2.fedoraproject.org --===============6650838527606875400== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Till Hofmann via FreeIPA-users wrote: > Hi all, > = > I managed to work around the issue by: > 1. Setting up the replica without the CA (i.e., `ipa-replica-install` wit= hout `--setup-ca`) > 2. Set up the CA with `ipa-ca-install`. This also failed at some point (b= ecause it could not contact the old master on port 8443), but it seemed to = do "enough" so I could actually ignore the missing steps. > = > I turned off the original master, verified that I could still log in on t= he clients and also tested certificate renewal with `ipa-cacert-manage rene= w`, which was successful. This is not a sufficient test. This command only renews the CA certificate. > I don't know what the missing steps were, I hope this won't bite me in th= e long run. Do you have any suggestions what else I could test to verify th= at the CA is also working properly? If you want to test that the CA works then issue a new certificate, revoke it, ensure it lands in the CRL at next generation. You also need to set the certificate renewal master and CRL generator. --===============6650838527606875400==--