Thanks Rob,
This is the output of ldap-ca-master
# matches for CA REST API
<LocationMatch
"^/ca/rest/account/login|^/ca/rest/account/logout|^/ca/rest/installer/installToken|^/ca/rest/securityDomain/domainInfo|^/ca/rest/securityDomain/installToken|^/ca/rest/profiles|^/ca/rest/authorities|^/ca/rest/admin/kraconnector/remove">
NSSOptions +StdEnvVars +ExportCertData +StrictRequire +OptRenegotiate
NSSVerifyClient optional
ProxyPassMatch ajp://localhost:8009
ProxyPassReverse ajp://localhost:8009
</LocationMatch>
/var/log/httpd/access_log
10.32.1.60 - host/ldap-b-3.example.com(a)EXAMPLE.COM
[16/Sep/2019:12:01:17 -0400] "POST /ipa/xml HTTP/1.1" 200 316
10.31.1.24 - - [16/Sep/2019:12:01:32 -0400] "GET
https://ldap-ca-master.example.com:443/ca/rest/account/login HTTP/1.1"
200 218
10.31.1.24 - - [16/Sep/2019:12:01:32 -0400] "GET
https://ldap-ca-master.example.com:443/ca/rest/authorities/42a9fffc-199d-...
HTTP/1.1" 200 905
10.31.1.24 - - [16/Sep/2019:12:01:32 -0400] "GET
https://ldap-ca-master.example.com:443/ca/rest/account/logout
HTTP/1.1" 204 -
10.31.1.24 - - [16/Sep/2019:12:01:32 -0400] "POST
https://ldap-ca-master.example.com:443/ca/rest/certrequests?issuer-id=42a...
HTTP/1.1" 404 218
10.32.1.60 - - [16/Sep/2019:12:01:32 -0400] "POST /ipa/xml HTTP/1.1" 401 1474
10.32.1.60 - host/ldap-b-3.example.com(a)EXAMPLE.COM
[16/Sep/2019:12:01:32 -0400] "POST /ipa/xml HTTP/1.1" 200 316
10.31.1.24 - - [16/Sep/2019:12:01:47 -0400] "GET
https://ldap-ca-master.example.com:443/ca/rest/account/login HTTP/1.1"
200 218
10.31.1.24 - - [16/Sep/2019:12:01:47 -0400] "GET
https://ldap-ca-master.example.com:443/ca/rest/authorities/42a9fffc-199d-...
HTTP/1.1" 200 905
[root@ldap-ca-master conf.d]# ipa-replica-manage list -v `hostname`
Directory Manager password:
ldap-b-1.example.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (0) Replica acquired successfully:
Incremental update succeeded
last update ended: 2019-09-17 22:13:04+00:00
[root@ldap-b-1 conf.d]# ipa-replica-manage list -v `hostname`
Directory Manager password:
ldap-ca-master.example.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (18) Replication error acquiring replica:
Incremental update transient error. Backing off, will retry update
later. (transient error)
last update ended: 1970-01-01 00:00:00+00:00
ldap-b-2.example.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (15) Replication error acquiring replica:
Changelog database error was encountered (changelog error)
last update ended: 1970-01-01 00:00:00+00:00
ldap-b-3.example.com: replica
last init status: 0 Total update succeeded
last init ended: 2019-09-16 15:56:54+00:00
last update status: Error (3) Replication error acquiring replica:
Unable to acquire replica: permission denied. The bind dn does not
have permission to supply replication updates to the replica. Will
retry later. (permission denied)
last update ended: 2019-09-16 15:56:55+00:00
[root@ldap-b-2 ~]# ipa-replica-manage list -v `hostname`
Directory Manager password:
ldap-b-1.example.com: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: Error (0) Replica acquired successfully:
Incremental update succeeded
last update ended: 2019-09-17 22:32:26+00:00
ldap-b-3.example.com i am trying to add in cluster throwing error for
CA_REJECT.
Let me know if you need more data or log?
On Tue, Sep 17, 2019 at 1:55 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
>
> Satish Patel via FreeIPA-users wrote:
> > Folks,
> >
> > Stay with me while i explain my issue because its little complex, We
> > had 2 working ldap running in datacenter-A for many months and life
> > was good.
> >
> > Last year company decided to shutdown datacenter-A and migrate
> > everything from there to new datacenter-B.
> >
> > This is what i did for migration, I have created two new LDAP server
> > in Datacenter-B and run create replica from Datacenter-A ( but my bad
> > luck we forgot to do --setup-ca option which create CA replica) In
> > short we have no CA running in new datacenter-B
> >
> > Fun part start now. so finally few months back we shutdown
> > datacenter-A and archived all data (LDAP was running in VMware so we
> > archived vmdk), after 8 month we found our LDAP server running under
> > load so we decided to create more replica and we found we have no CA
> > master so we can't create replica. Damn it.
> >
> > We dig into datacenter-A archived and start ldap-ca-master start on
> > new IP address we gave it same DNS name so it won't create any issue,
> > when i start ldap-ca-master it started throwing error that some certs
> > expired blah..blah.. so finally i renew them and this LDAP looks good
> > now CA is also running.
> >
> > Hostname:
> >
> > ldap-ca-master (This is old datacenter LDAP with CA, awakened few days ago)
> > ldap-b-1 (new datacenter LDAP without CA)
> > ldap-b-2 (new datacenter LDAP without CA)
> >
> > Now i am trying to create new ldap-b-3 in new datacenter using
> > ldap-b-1 as my master to create new replica and somehow i am getting
> > following error
> >
> >
> > RuntimeError: Certificate issuance failed (CA_REJECTED: Server at
> >
https://ldap-b-1.example.com/ipa/xml failed request, will retry: 4035
> > (RPC failed at server. Request failed with status 404: Non-2xx
> > response from CA REST API: 404. ).)
> > Installation failed. Rolling back changes.
> > Unenrolling client from IPA server
> > Unenrolling host failed: RPC failed at server. invalid 'hostname': An
> > IPA master host cannot be deleted or disabled
> >
> > Question:
> >
> > 1. My all other ldap running 4.5.x but new replica is on 4.6 not sure
> > that is the issue here or not?
> >
> > 2. I can see ldap-ca-master node isn't fully sync with ldap-b-1 and
> > ldap-b-2 because i brought that machine in life after 8 month (do you
> > think i should do force sync ldap-ca-master to sync with ldap-b-1 ?)
> >
> > 3. Should i use ldap-ca-master to create replica or i can pick any
> > node to create replica?
> >
> > What are the options i have here to troubleshoot this issue?
>
> Look in /etc/httpd/conf.d/ipa-pki-proxy.conf for a section like:
>
> <LocationMatch "^/ca/rest/account/login|...
>
> Show us the full contents.
>
> See what URL is being requested in /var/log/httpd/access_log
>
> ipa-replica-manage list -v `hostname` on all the masters will show you
> the current status.
>
> rob