ipa-replica-install fails during initial replication

[Markus Rexhepi-Lindberg]

The replication step fails while installing a new ipa replica server.

Some facts:

* Both servers running version 4.9.12.
* Both servers running RHEL 8.9
* Master located in Sweden and replica located in USA.
* Actual domain has been substituted with "example.com".

Some logs:

= replica=

replica# ipa-replica-install --verbose --setup-dns --forwarder 10.0.2.200 --forwarder 10.0.2.201 --forwarder 10.0.2.202 --setup-ca
...
Created connection context.ldap2_140175491229624
Fetching nsDS5ReplicaId from master [attempt 1/5]
retrieving schema for SchemaCache url=ldap://se-rhidm02x.se.example.com:389 conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7f7d2304e278>
Successfully updated nsDS5ReplicaId.
Add or update replica config cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
Added replica config cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
update_entry modlist [(0, 'nsDS5ReplicaBindDN', [b'cn=ldap/[e-mail],cn=config'])]
Add or update replica config cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
No update to cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config necessary
Waiting up to 300 seconds for replication (ldap://se-rhidm02x.se.example.com:389) cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config (objectclass=*)
Entry found [LDAPEntry(ipapython.dn.DN('cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config'), {'objectClass': [b'nsds5replicationagreement', b'top']
, 'cn': [b'meTousidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicaHost': [b'usidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicaPort': [b'389'], 'nsds5replicaTimeout': [b'120'], 'nsDS5ReplicaRoot': [b'dc=lnx,d
c=example,dc=com'], 'description': [b'me to usidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicatedAttributeList': [b'(objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfaile
dauth krbloginfailedcount passwordgraceusertime'], 'nsDS5ReplicaTransportInfo': [b'LDAP'], 'nsDS5ReplicaBindMethod': [b'SASL/GSSAPI'], 'nsds5ReplicaStripAttrs': [b'modifiersName modifyTimestamp internalMo
difiersName internalModifyTimestamp'], 'nsDS5ReplicatedAttributeListTotal': [b'(objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount passwordgraceusertime'], 'nsds
5replicareapactive': [b'0'], 'nsds5replicaLastUpdateStart': [b'19700101000000Z'], 'nsds5replicaLastUpdateEnd': [b'19700101000000Z'], 'nsds5replicaChangesSentSinceStartup': [b''], 'nsds5replicaLastUpdateSt
atus': [b'Error (-2) Problem connecting to replica - LDAP error: Local error (connection error)'], 'nsds5replicaLastUpdateStatusJSON': [b'{"state": "red", "ldap_rc": "-2", "ldap_rc_text": "Local error", "
repl_rc": "16", "repl_rc_text": "connection error", "date": "2024-02-15T14:35:36Z", "message": "Error (-2) Problem connecting to replica - LDAP error: Local error (connection error)"}'], 'nsds5replicaUpda
teInProgress': [b'FALSE'], 'nsds5replicaLastInitStart': [b'19700101000000Z'], 'nsds5replicaLastInitEnd': [b'19700101000000Z']})]
Waiting up to 300 seconds for replication (ldapi://%2Frun%2Fslapd-LNX-EXAMPLE-COM.socket) cn=meTose-rhidm02x.se.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config (objectclass=*)
Entry found [LDAPEntry(ipapython.dn.DN('cn=meTose-rhidm02x.se.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config'), {'objectClass': [b'nsds5replicationagreement', b'top'], 'cn': [
b'meTose-rhidm02x.se.example.com'], 'nsDS5ReplicaHost': [b'se-rhidm02x.se.example.com'], 'nsDS5ReplicaPort': [b'389'], 'nsds5replicaTimeout': [b'120'], 'nsDS5ReplicaRoot': [b'dc=lnx,dc=example,dc=com'], 'descripti
on': [b'me to se-rhidm02x.se.example.com'], 'nsDS5ReplicatedAttributeList': [b'(objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount passwordgr
aceusertime'], 'nsDS5ReplicaTransportInfo': [b'LDAP'], 'nsDS5ReplicaBindMethod': [b'SASL/GSSAPI'], 'nsds5ReplicaStripAttrs': [b'modifiersName modifyTimestamp internalModifiersName internalModifyTimestamp'
], 'nsDS5ReplicatedAttributeListTotal': [b'(objectclass=*) $ EXCLUDE entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount passwordgraceusertime'], 'nsds5replicareapactive': [b'0'], 'nsds5r
eplicaLastUpdateStart': [b'19700101000000Z'], 'nsds5replicaLastUpdateEnd': [b'19700101000000Z'], 'nsds5replicaChangesSentSinceStartup': [b''], 'nsds5replicaLastUpdateStatus': [b'Error (0) No replication s
essions started since server startup'], 'nsds5replicaLastUpdateStatusJSON': [b'{"state": "green", "ldap_rc": "0", "ldap_rc_text": "success", "repl_rc": "0", "repl_rc_text": "replica acquired", "date": "20
24-02-15T14:35:28Z", "message": "Error (0) No replication sessions started since server startup"}'], 'nsds5replicaUpdateInProgress': [b'FALSE'], 'nsds5replicaLastInitStart': [b'19700101000000Z'], 'nsds5re
plicaLastInitEnd': [b'19700101000000Z']})]
Starting replication, please wait until this has completed.
Update in progress, 15 seconds elapsed
[ldap://se-rhidm02x.se.example.com:389] reports: Update failed! Status: [Error (-2) - LDAP error: Local error - no response received]

replica# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/errors
...
[15/Feb/2024:09:35:58.128874085 -0500] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTose-rhidm02x.se.example.com" (se-rhidm02x:389): The remote replica has a different database generation ID than the local database.  You may have to reinitialize the remote replica, or the local replica.
...

replica# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/access
...
[15/Feb/2024:09:35:28.821998361 -0500] conn=6 fd=119 slot=119 connection from 10.0.13.145 to 192.168.224.21
[15/Feb/2024:09:35:28.827100928 -0500] conn=6 op=0 UNBIND
[15/Feb/2024:09:35:28.827120206 -0500] conn=6 op=0 fd=119 closed error - U1
...

= master =

master# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/access
...
[15/Feb/2024:15:35:44.803292478 +0100] conn=37567 op=31 SRCH base="cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config" scope=0 filter="(objectC
lass=*)" attrs="nsds5BeginReplicaRefresh nsds5replicaLastInitStart cn nsds5replicaLastInitStatusJSON nsds5replicaLastInitEnd nsds5replicaUpdateInProgress nsds5replicaLastInitStatus"
[15/Feb/2024:15:35:44.803737834 +0100] conn=37567 op=31 RESULT err=0 tag=101 nentries=1 wtime=0.000219465 optime=0.000451462 etime=0.000669200
[15/Feb/2024:15:35:45.170456864 +0100] conn=37383 op=16 UNBIND
[15/Feb/2024:15:35:45.170486056 +0100] conn=37383 op=16 fd=273 closed error - U1
...

master# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/errors
...
[15/Feb/2024:15:35:37.160764934 +0100] - WARN - NSMMReplicationPlugin - repl5_tot_run - Unable to acquire replica for total update, error: -2, retrying in 1 seconds.
[15/Feb/2024:15:35:38.274695202 +0100] - WARN - NSMMReplicationPlugin - repl5_tot_run - Unable to acquire replica for total update, error: -2, retrying in 2 seconds.
[15/Feb/2024:15:35:40.388281036 +0100] - WARN - NSMMReplicationPlugin - repl5_tot_run - Unable to acquire replica for total update, error: -2, retrying in 3 seconds.
[15/Feb/2024:15:35:43.503252882 +0100] - WARN - NSMMReplicationPlugin - repl5_tot_run - Unable to acquire replica for total update, error: -2, retrying in 4 seconds.
[15/Feb/2024:15:35:47.618537566 +0100] - WARN - NSMMReplicationPlugin - repl5_tot_run - Unable to acquire replica for total update, error: -2, retrying in 5 seconds.
...

[Florence Blanc-Renaud]

Hi,

On Thu, Feb 15, 2024 at 3:50 PM Markus Rexhepi-Lindberg via FreeIPA-users <
[e-mail]> wrote:
> The replication step fails while installing a new ipa replica server.
>
> Some facts:
> Both servers running version 4.9.12.
> Both servers running RHEL 8.9
> Master located in Sweden and replica located in USA.
> Actual domain has been substituted with "example.com".
>
> Some logs:
>
> = replica=
>
> replica# ipa-replica-install --verbose --setup-dns --forwarder 10.0.2.200
> --forwarder 10.0.2.201 --forwarder 10.0.2.202 --setup-ca
> ...
> Created connection context.ldap2_140175491229624
> Fetching nsDS5ReplicaId from master [attempt 1/5]
> retrieving schema for SchemaCache url=ldap://
> se-rhidm02x.se.example.com:389 conn=<ldap.ldapobject.SimpleLDAPObject
> object at 0x7f7d2304e278>
> Successfully updated nsDS5ReplicaId.
> Add or update replica config
> cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
> Added replica config
> cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
> update_entry modlist [(0, 'nsDS5ReplicaBindDN', [b'cn=ldap/
> [e-mail],cn=config'])]
> Add or update replica config
> cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config
> No update to cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping
> tree,cn=config necessary

The replica creates an entry on the master
cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping tree,cn=config that
should contain nsDS5ReplicaId, nsDS5ReplicaBindDN: cn=replication
manager,cn=config and nsDS5ReplicaBindDNGroup: cn=replication
managers,cn=sysaccounts,cn=etc,dc=ipa,dc=test

Since there is a message "No update to ... necessary, it looks like the
master already knows about this replica, maybe it is not the first time you
try to add it?
> Waiting up to 300 seconds for replication (ldap://
> se-rhidm02x.se.example.com:389) cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping
> tree,cn=config (objectclass=*)
> Entry found [LDAPEntry(ipapython.dn.DN('cn=
> meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping
> tree,cn=config'), {'objectClass': [b'nsds5replicationagreement', b'top']
> , 'cn': [b'meTousidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicaHost':
> [b'usidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicaPort': [b'389'],
> 'nsds5replicaTimeout': [b'120'], 'nsDS5ReplicaRoot': [b'dc=lnx,d
> c=example,dc=com'], 'description': [b'me to
> usidc1-rhidm01x.idc1.us.example.com'], 'nsDS5ReplicatedAttributeList':
> [b'(objectclass=*) $ EXCLUDE memberof idnssoaserial entryusn
> krblastsuccessfulauth krblastfaile
> dauth krbloginfailedcount passwordgraceusertime'],
> 'nsDS5ReplicaTransportInfo': [b'LDAP'], 'nsDS5ReplicaBindMethod':
> [b'SASL/GSSAPI'], 'nsds5ReplicaStripAttrs': [b'modifiersName
> modifyTimestamp internalMo
> difiersName internalModifyTimestamp'],
> 'nsDS5ReplicatedAttributeListTotal': [b'(objectclass=*) $ EXCLUDE entryusn
> krblastsuccessfulauth krblastfailedauth krbloginfailedcount
> passwordgraceusertime'], 'nsds
> 5replicareapactive': [b'0'], 'nsds5replicaLastUpdateStart':
> [b'19700101000000Z'], 'nsds5replicaLastUpdateEnd': [b'19700101000000Z'],
> 'nsds5replicaChangesSentSinceStartup': [b''], 'nsds5replicaLastUpdateSt
> atus': [b'Error (-2) Problem connecting to replica - LDAP error: Local
> error (connection error)'], 'nsds5replicaLastUpdateStatusJSON':
> [b'{"state": "red", "ldap_rc": "-2", "ldap_rc_text": "Local error", "
> repl_rc": "16", "repl_rc_text": "connection error", "date":
> "2024-02-15T14:35:36Z", "message": "Error (-2) Problem connecting to
> replica - LDAP error: Local error (connection error)"}'], 'nsds5replicaUpda
> teInProgress': [b'FALSE'], 'nsds5replicaLastInitStart':
> [b'19700101000000Z'], 'nsds5replicaLastInitEnd': [b'19700101000000Z']})]

The replica starts replication by setting nsds5BeginReplicaRefresh=start on
the master (entry cn=meToreplica,...) and reads the entry to check the
replication status.
This message *"Error (-2) Problem connecting to replica - LDAP error: Local
error (connection error)"* indicates an issue when the master tries to
communicate with the replica. You may find corresponding logs in the
master's 389ds error log and maybe a failed connection in the replica's
389ds access log.

Waiting up to 300 seconds for replication
> (ldapi://%2Frun%2Fslapd-LNX-EXAMPLE-COM.socket) cn=
> meTose-rhidm02x.se.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping
> tree,cn=config (objectclass=*)
> Entry found [LDAPEntry(ipapython.dn.DN('cn=meTose-rhidm02x.se.example.com,cn=replica,cn=dc=lnx,dc=example,dc=com,cn=mapping
> tree,cn=config'), {'objectClass': [b'nsds5replicationagreement', b'top'],
> 'cn': [
> b'meTose-rhidm02x.se.example.com'], 'nsDS5ReplicaHost': [b'
> se-rhidm02x.se.example.com'], 'nsDS5ReplicaPort': [b'389'],
> 'nsds5replicaTimeout': [b'120'], 'nsDS5ReplicaRoot':
> [b'dc=lnx,dc=example,dc=com'], 'descripti
> on': [b'me to se-rhidm02x.se.example.com'],
> 'nsDS5ReplicatedAttributeList': [b'(objectclass=*) $ EXCLUDE memberof
> idnssoaserial entryusn krblastsuccessfulauth krblastfailedauth
> krbloginfailedcount passwordgr
> aceusertime'], 'nsDS5ReplicaTransportInfo': [b'LDAP'],
> 'nsDS5ReplicaBindMethod': [b'SASL/GSSAPI'], 'nsds5ReplicaStripAttrs':
> [b'modifiersName modifyTimestamp internalModifiersName
> internalModifyTimestamp'
> ], 'nsDS5ReplicatedAttributeListTotal': [b'(objectclass=*) $ EXCLUDE
> entryusn krblastsuccessfulauth krblastfailedauth krbloginfailedcount
> passwordgraceusertime'], 'nsds5replicareapactive': [b'0'], 'nsds5r
> eplicaLastUpdateStart': [b'19700101000000Z'], 'nsds5replicaLastUpdateEnd':
> [b'19700101000000Z'], 'nsds5replicaChangesSentSinceStartup': [b''],
> 'nsds5replicaLastUpdateStatus': [b'Error (0) No replication s
> essions started since server startup'],
> 'nsds5replicaLastUpdateStatusJSON': [b'{"state": "green", "ldap_rc": "0",
> "ldap_rc_text": "success", "repl_rc": "0", "repl_rc_text": "replica
> acquired", "date": "20
> 24-02-15T14:35:28Z", "message": "Error (0) No replication sessions started
> since server startup"}'], 'nsds5replicaUpdateInProgress': [b'FALSE'],
> 'nsds5replicaLastInitStart': [b'19700101000000Z'], 'nsds5re
> plicaLastInitEnd': [b'19700101000000Z']})]
> Starting replication, please wait until this has completed.
> Update in progress, 15 seconds elapsed
> [ldap://se-rhidm02x.se.example.com:389] reports: Update failed! Status:
> [Error (-2) - LDAP error: Local error - no response received]
>
> replica# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/errors
> ...
> [15/Feb/2024:09:35:58.128874085 -0500] - WARN - NSMMReplicationPlugin -
> repl5_inc_run - agmt="cn=meTose-rhidm02x.se.example.com"
> (se-rhidm02x:389): The remote replica has a different database generation
> ID than the local database.  You may have to reinitialize the remote
> replica, or the local replica.
> ...
>
> replica# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/access
> ...
> [15/Feb/2024:09:35:28.821998361 -0500] conn=6 fd=119 slot=119 connection
> from 10.0.13.145 to 192.168.224.21
> [15/Feb/2024:09:35:28.827100928 -0500] conn=6 op=0 UNBIND
> [15/Feb/2024:09:35:28.827120206 -0500] conn=6 op=0 fd=119 closed error - U1
> ...
>
> = master =
>
> master# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/access
> ...
> [15/Feb/2024:15:35:44.803292478 +0100] conn=37567 op=31 SRCH base="cn=
> meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config" scope=0 filter="(objectC
> lass=*)" attrs="nsds5BeginReplicaRefresh nsds5replicaLastInitStart cn
> nsds5replicaLastInitStatusJSON nsds5replicaLastInitEnd
> nsds5replicaUpdateInProgress nsds5replicaLastInitStatus"
> [15/Feb/2024:15:35:44.803737834 +0100] conn=37567 op=31 RESULT err=0
> tag=101 nentries=1 wtime=0.000219465 optime=0.000451462 etime=0.000669200
> [15/Feb/2024:15:35:45.170456864 +0100] conn=37383 op=16 UNBIND
> [15/Feb/2024:15:35:45.170486056 +0100] conn=37383 op=16 fd=273 closed
> error - U1
> ...
>
> master# cat /var/log/dirsrv/slapd-LNX-EXAMPLE-COM/errors
> ...
> [15/Feb/2024:15:35:37.160764934 +0100] - WARN - NSMMReplicationPlugin -
> repl5_tot_run - Unable to acquire replica for total update, error: -2,
> retrying in 1 seconds.
> [15/Feb/2024:15:35:38.274695202 +0100] - WARN - NSMMReplicationPlugin -
> repl5_tot_run - Unable to acquire replica for total update, error: -2,
> retrying in 2 seconds.
> [15/Feb/2024:15:35:40.388281036 +0100] - WARN - NSMMReplicationPlugin -
> repl5_tot_run - Unable to acquire replica for total update, error: -2,
> retrying in 3 seconds.
> [15/Feb/2024:15:35:43.503252882 +0100] - WARN - NSMMReplicationPlugin -
> repl5_tot_run - Unable to acquire replica for total update, error: -2,
> retrying in 4 seconds.
> [15/Feb/2024:15:35:47.618537566 +0100] - WARN - NSMMReplicationPlugin -
> repl5_tot_run - Unable to acquire replica for total update, error: -2,
> retrying in 5 seconds.
> ...

Is there any log right before that one which would indicate a bind issue
from master to replica?

You can re-try a clean install:
- on the replica, ipa-server-install --uninstall -U; kdestroy -A
- on the master, ipa server-del <replica fqdn>
- on the replica, ipa-replica-install <your options>

flo
> --
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Markus Rexhepi-Lindberg]

Hi Florence,

Thanks for looking into this I appreciate it very much!

```
master# ldapsearch -xLLL -o ldif-wrap=no -D "cn=directory manager" -W -s sub -b cn=config objectclass=nsds5replicationagreement dn
Enter LDAP Password:
dn: cn=meTose-rhidm03x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=se-rhidm02x.se.example.com-to-se-rhidm01x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=se-rhidm02x.se.example.com-to-se-rhidm04x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config

dn: cn=caTose-rhidm03x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config

dn: cn=se-rhidm02x.se.example.com-to-se-rhidm01x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config

dn: cn=se-rhidm02x.se.example.com-to-se-rhidm04x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
```

On the master "meTousidc1-rhidm01x.idc1.us.example.com" is there after
running ipa-replica install <...> from the replica. This has been found
after all my install attempts and I have been removing that entry using.

```
master# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
changetype: delete
EOF
```

I tried a clean install as per your suggestion but it fails in the same
way. Worth to note that `ipa server-del <replica fqdn>` was not
possible since I could not find the replica using `ipa server-find`.
Maybe that indicates an issue?

When running the `ipa-replica-install <...>` command I get the
following error and warning.

```
Could not resolve hostname se-rhidm03x.se.example.com using DNS. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
Continue? [no]: yes
...
WARNING: 2 existing users or groups do not have a SID identifier assigned.
Installer can run a task to have ipa-sidgen Directory Server plugin generate
the SID identifier for all these users. Please note, in case of a high
number of users and groups, the operation might lead to high replication
traffic and performance degradation. Refer to ipa-adtrust-install(1) man page
for details.

Do you want to run the ipa-sidgen task? [no]: no
```

What I do to install the replica is first manually installing it as a
client, adding it to the ipaservers hostgroup and then running the
`ipa-replica-install <...>` command.

```
replica# ipa-client-install --domain lnx.example.com --force-join --mkhomedir --no-ntp --principal idmsrvjoin --realm LNX.EXAMPLE.COM

master# ipa hostgroup-add-member ipaservers --hosts usidc1-rhidm01x.idc1.us.example.com

replica# ipa-replica-install --verbose --setup-dns --forwarder 10.0.2.200 --forwarder 10.0.2.201 --forwarder 10.0.2.202 --setup-ca
```

I tried sending an e-mail with the following files in a tar ball, but
it seems to not have been accepted due to the its large size. I have
published them on my own website instead, hope that works.

master ds389 access: https://www.rexhepi-lindberg.com/iparepl/master/access
master ds389 errors: https://www.rexhepi-lindberg.com/iparepl/master/errors
replica ds389 access: https://www.rexhepi-lindberg.com/iparepl/replica/access
replica ds389 errors: https://www.rexhepi-lindberg.com/iparepl/replica/errors
replica-install.log: https://www.rexhepi-lindberg.com/iparepl/replica/ipareplica-install.log

--
Markus

[Florence Blanc-Renaud]

Hi Markus,

On Mon, Feb 19, 2024 at 9:07 AM Markus Rexhepi-Lindberg via FreeIPA-users <
[e-mail]> wrote:
> Hi Florence,
>
> Thanks for looking into this I appreciate it very much!
> master# ldapsearch -xLLL -o ldif-wrap=no -D "cn=directory manager" -W -s
> sub -b cn=config objectclass=nsds5replicationagreement dn
> Enter LDAP Password:
> dn: cn=meTose-rhidm03x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config
>
> dn: cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config
>
> dn: cn=se-rhidm02x.se.example.com-to-se-rhidm01x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config
>
> dn: cn=se-rhidm02x.se.example.com-to-se-rhidm04x.se.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config
>
> dn: cn=caTose-rhidm03x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping
> tree,cn=config
>
> dn: cn=se-rhidm02x.se.example.com-to-se-rhidm01x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping
> tree,cn=config
>
> dn: cn=se-rhidm02x.se.example.com-to-se-rhidm04x.se.example.com,cn=replica,cn=o\3Dipaca,cn=mapping
> tree,cn=config
>
> On the master "meTousidc1-rhidm01x.idc1.us.example.com" is there after
> running ipa-replica install <...> from the replica. This has been found
> after all my install attempts and I have been removing that entry using.
> master# ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
> tree,cn=config
> changetype: delete
> EOF
>
> I tried a clean install as per your suggestion but it fails in the same
> way. Worth to note that `ipa server-del <replica fqdn>` was not
> possible since I could not find the replica using `ipa server-find`.
> Maybe that indicates an issue?
>
> When running the `ipa-replica-install <...>` command I get the
> following error and warning.
> Could not resolve hostname se-rhidm03x.se.example.com using DNS. Clients
> may not function properly. Please check your DNS setup. (Note that this
> check queries IPA DNS directly and ignores /etc/hosts.)
> Continue? [no]: yes
> ...
> WARNING: 2 existing users or groups do not have a SID identifier assigned.
> Installer can run a task to have ipa-sidgen Directory Server plugin
> generate
> the SID identifier for all these users. Please note, in case of a high
> number of users and groups, the operation might lead to high replication
> traffic and performance degradation. Refer to ipa-adtrust-install(1) man
> page
> for details.
>
> Do you want to run the ipa-sidgen task? [no]: no
>
> What I do to install the replica is first manually installing it as a
> client, adding it to the ipaservers hostgroup and then running the
> `ipa-replica-install <...>` command.
> replica# ipa-client-install --domain lnx.example.com --force-join
> --mkhomedir --no-ntp --principal idmsrvjoin --realm LNX.EXAMPLE.COM
>
> master# ipa hostgroup-add-member ipaservers --hosts
> usidc1-rhidm01x.idc1.us.example.com
>
> replica# ipa-replica-install --verbose --setup-dns --forwarder 10.0.2.200
> --forwarder 10.0.2.201 --forwarder 10.0.2.202 --setup-ca
>
> I tried sending an e-mail with the following files in a tar ball, but
> it seems to not have been accepted due to the its large size. I have
> published them on my own website instead, hope that works.
>
> master ds389 access:
> https://www.rexhepi-lindberg.com/iparepl/master/access
> master ds389 errors:
> https://www.rexhepi-lindberg.com/iparepl/master/errors
> replica ds389 access:
> https://www.rexhepi-lindberg.com/iparepl/replica/access
> replica ds389 errors:
> https://www.rexhepi-lindberg.com/iparepl/replica/errors
> replica-install.log
> https://www.rexhepi-lindberg.com/iparepl/replica/errorsreplica-install.log:
> https://www.rexhepi-lindberg.com/iparepl/replica/ipareplica-install.log

Can you check your DNS configuration? From the logs on the master (
se-rhidm03x.se.example.com), it seems the master is listening on IP address
10.0.13.*145*, but the replica is resolving the master as 10.0.13.*146*.

Maybe you have a conflicting entry in /etc/hosts on the replica.
flo

--
> Markus
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Markus Rexhepi-Lindberg]

Hi Florence,

From what I can see it is setup correctly on both the master(s) and replica.

I got the following during `ipa-replica-install`:
```
Search DNS server se-rhidm01x.se.example.com (['10.0.13.139', '10.0.13.139', '10.0.13.139']) for se-rhidm03x.se.example.com
Could not resolve hostname se-rhidm03x.se.example.com using DNS. Clients may not function properly. Please check your DNS setup. (Note that this check queries IPA DNS directly and ignores /etc/hosts.)
Continue? [no]:
```

Which I solved by enabling rescursion in the named config on the masters. `ipa-replica-install` now outputs this instead:
```
...
raw: domainlevel_get(version='2.251')
domainlevel_get(version='2.251')
raw: hostgroup_find(None, cn='ipaservers', version='2.251', host=['usidc1-rhidm01x.idc1.us.example.com'])
hostgroup_find(None, cn='ipaservers', all=False, raw=False, version='2.251', no_members=True, pkey_only=False, host=('usidc1-rhidm01x.idc1.us.example.com',))
Lookup failed: Preferred host usidc1-rhidm01x.idc1.us.example.com does not provide DNS.
Check forward/reverse DNS resolution
Search DNS server se-rhidm04x.se.example.com (['10.0.11.190', '10.0.11.190', '10.0.11.190']) for se-rhidm03x.se.example.com
Check reverse address 10.0.13.146 (se-rhidm03x.se.example.com)
Address 10.0.13.146 resolves to: se-rhidm03x.se.example.com..
Search DNS server se-rhidm04x.se.example.com (['10.0.11.190', '10.0.11.190', '10.0.11.190']) for usidc1-rhidm01x.idc1.us.example.com
Check reverse address 192.168.224.21 (usidc1-rhidm01x.idc1.us.example.com)
Address 192.168.224.21 resolves to: usidc1-rhidm01x.idc1.us.example.com..
Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
raw: dns_is_enabled(version='2.251')
dns_is_enabled(version='2.251')
Name usidc1-rhidm01x.idc1.us.example.com resolved to {UnsafeIPAddress('192.168.224.21')}
Searching for an interface of IP address: 192.168.224.21
Testing local IP address: 127.0.0.1/255.0.0.0 (interface: lo)
Testing local IP address: 192.168.224.21/255.255.255.128 (interface: eth0)
IP address 192.168.224.21 belongs to a private range, using forward policy only
Checking DNS forwarders, please wait ...
Checking DNS server: 10.0.2.200
DNS server 10.0.2.200 does not support DNSSEC: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.

DNS server 10.0.2.200: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.
Checking DNS server: 10.0.2.201
DNS server 10.0.2.201 does not support DNSSEC: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.

DNS server 10.0.2.201: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.
Checking DNS server: 10.0.2.202
DNS server 10.0.2.202 does not support DNSSEC: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.

DNS server 10.0.2.202: answer to query '. SOA' is missing DNSSEC signatures (no RRSIG data)
Please fix forwarder configuration to enable DNSSEC support.
WARNING: DNSSEC validation will be disabled
will use DNS forwarders: [CheckedIPAddressLoopback('10.0.2.200'), CheckedIPAddressLoopback('10.0.2.201'), CheckedIPAddressLoopback('10.0.2.202')]
...
```

--
Markus

[Florence Blanc-Renaud]

Hi,

On Fri, Feb 23, 2024 at 12:38 PM Markus Rexhepi-Lindberg via FreeIPA-users <
[e-mail]> wrote:
> Hi Florence,
>
> From what I can see it is setup correctly on both the master(s) and
> replica.

I now understand the confusion: the logs provided in
master ds389 access: https://www.rexhepi-lindberg.com/iparepl/master/access
master ds389 errors: https://www.rexhepi-lindberg.com/iparepl/master/errors
are taken on the server *se-rhidm02x*.se.example.com, but the replica was
using se-rhidm03x.se.example.com as source of data.
You can identify which master is used in the ipareplica-install.log file.
Can you then share the logs from the corresponding master?
> I got the following during `ipa-replica-install`:
> Search DNS server se-rhidm01x.se.example.com (['10.0.13.139',
> '10.0.13.139', '10.0.13.139']) for se-rhidm03x.se.example.com
> Could not resolve hostname se-rhidm03x.se.example.com using DNS. Clients
> may not function properly. Please check your DNS setup. (Note that this
> check queries IPA DNS directly and ignores /etc/hosts.)
> Continue? [no]:
>
> Which I solved by enabling rescursion in the named config on the masters.
> `ipa-replica-install` now outputs this instead:
> ...
> raw: domainlevel_get(version='2.251')
> domainlevel_get(version='2.251')
> raw: hostgroup_find(None, cn='ipaservers', version='2.251', host=['
> usidc1-rhidm01x.idc1.us.example.com'])
> hostgroup_find(None, cn='ipaservers', all=False, raw=False,
> version='2.251', no_members=True, pkey_only=False, host=('
> usidc1-rhidm01x.idc1.us.example.com',))
> Lookup failed: Preferred host usidc1-rhidm01x.idc1.us.example.com does
> not provide DNS.
> Check forward/reverse DNS resolution
> Search DNS server se-rhidm04x.se.example.com (['10.0.11.190',
> '10.0.11.190', '10.0.11.190']) for se-rhidm03x.se.example.com
> Check reverse address 10.0.13.146 (se-rhidm03x.se.example.com)
> Address 10.0.13.146 resolves to: se-rhidm03x.se.example.com..
> Search DNS server se-rhidm04x.se.example.com (['10.0.11.190',
> '10.0.11.190', '10.0.11.190']) for usidc1-rhidm01x.idc1.us.example.com
> Check reverse address 192.168.224.21 (usidc1-rhidm01x.idc1.us.example.com)
> Address 192.168.224.21 resolves to: usidc1-rhidm01x.idc1.us.example.com..
> Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index'
> raw: dns_is_enabled(version='2.251')
> dns_is_enabled(version='2.251')
> Name usidc1-rhidm01x.idc1.us.example.com resolved to
> {UnsafeIPAddress('192.168.224.21')}
> Searching for an interface of IP address: 192.168.224.21
> Testing local IP address: 127.0.0.1/255.0.0.0 (interface: lo)
> Testing local IP address: 192.168.224.21/255.255.255.128 (interface: eth0)
> IP address 192.168.224.21 belongs to a private range, using forward policy
> only
> Checking DNS forwarders, please wait ...
> Checking DNS server: 10.0.2.200
> DNS server 10.0.2.200 does not support DNSSEC: answer to query '. SOA' is
> missing DNSSEC signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
>
> DNS server 10.0.2.200: answer to query '. SOA' is missing DNSSEC
> signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
> Checking DNS server: 10.0.2.201
> DNS server 10.0.2.201 does not support DNSSEC: answer to query '. SOA' is
> missing DNSSEC signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
>
> DNS server 10.0.2.201: answer to query '. SOA' is missing DNSSEC
> signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
> Checking DNS server: 10.0.2.202
> DNS server 10.0.2.202 does not support DNSSEC: answer to query '. SOA' is
> missing DNSSEC signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
>
> DNS server 10.0.2.202: answer to query '. SOA' is missing DNSSEC
> signatures (no RRSIG data)
> Please fix forwarder configuration to enable DNSSEC support.
> WARNING: DNSSEC validation will be disabled
> will use DNS forwarders: [CheckedIPAddressLoopback('10.0.2.200'),
> CheckedIPAddressLoopback('10.0.2.201'),
> CheckedIPAddressLoopback('10.0.2.202')]

The above message is only a warning and should not prevent the
installation. If your DNS servers don't support DNSSEC, you can also
provide the option --no-dnssec-validation.

flo

...
> --
> Markus
> --
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Markus Rexhepi-Lindberg]

Hi,

You are right, sorry for the confusion. I have performed a new `ipa-replica-install` and you can find the logs for the master and replica in these links:

master ds389 access: https://www.rexhepi-lindberg.com/iparepl/20230223/se-rhidm03_access
master ds389 errors: https://www.rexhepi-lindberg.com/iparepl/20230223/se-rhidm03_errors
replica ds389 access: https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_access
replica ds389 errors: https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_errors
replica-install.log: https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_ipareplica...

master = se-rhidm03x
replica = usidc1-rhidm01x

Thanks for clarifying the DNSSEC warnings.

--
Markus

[Florence Blanc-Renaud]

Hi,

On Fri, Feb 23, 2024 at 2:49 PM Markus Rexhepi-Lindberg via FreeIPA-users <
[e-mail]> wrote:
> Hi,
>
> You are right, sorry for the confusion. I have performed a new
> `ipa-replica-install` and you can find the logs for the master and replica
> in these links:
>
> master ds389 access:
> https://www.rexhepi-lindberg.com/iparepl/20230223/se-rhidm03_access
> master ds389 errors:
> https://www.rexhepi-lindberg.com/iparepl/20230223/se-rhidm03_errors
> replica ds389 access:
> https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_access
> replica ds389 errors:
> https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_errors
> replica-install.log
> https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_errorsreplica-install.log:
>
> https://www.rexhepi-lindberg.com/iparepl/20230223/usidc1-rhidm01x_ipareplica...
>
> master = se-rhidm03x
> replica = usidc1-rhidm01x

The replication is enabled by setting nsds5BeginReplicaRefresh=start on the
master (entry

cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config

). We can see this operation on the master logs:

[23/Feb/2024:12:00:57.569011330 +0100] conn=167303 op=21 MOD
dn="cn=meTousidc1-rhidm01x.idc1.us.example.com,cn=replica,cn=dc\3Dlnx\2Cdc\3Dexample\2Cdc\3Dcom,cn=mapping
tree,cn=config"

After that, the master tries to establish the connection to the replica but
this fails. The logs on the replica show:

[23/Feb/2024:06:00:58.077083917 -0500] conn=6 fd=119 slot=119
connection from 10.0.13.146 to 192.168.224.21
[23/Feb/2024:06:00:58.083111562 -0500] conn=6 op=0 UNBIND
[23/Feb/2024:06:00:58.083129212 -0500] conn=6 op=0 fd=119 closed error - U1

This connection should be established from the master using the kerberos
ticket for the principal
ldap/[e-mail] obtained
from /etc/dirsrv/ds.keytab, thanks to a mapping defined on the replica
in cn=Peer
Master,cn=mapping,cn=sasl,cn=config:

objectclass: top, nsSaslMapping
nsSaslMapRegexString: '^[^:@]+$'
nsSaslMapBaseDNTemplate: cn=config
nsSaslMapFilterTemplate: '(cn=&@IPA.TEST)'
nsSaslMapPriority: 1

This allows to map ldap/[e-mail] to the
entry cn=ldap/[e-mail],cn=config

You can try to reproduce the error with (from the master):
kinit -kt /etc/dirsrv/ds.keytab ldap/`hostname`
ldapsearch -Y GSSAPI -H ldap://usidc1-rhidm01x.idc1.us.example.com -b "" -s
base

The output may help understand why the connection is immediately closed
instead of trying the GSSAPI bind operation.

flo

Thanks for clarifying the DNSSEC warnings.
> --
> Markus
> --
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Markus Rexhepi-Lindberg]

Hi,

Thank you Fabian, your suggested commands lead me to the answer and a solution!

```
[root@se-rhidm01x ~]# ldapsearch -Y GSSAPI -H ldap://usidc1-rhidm01x.idc1.us.example.com -b "" -s base
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[e-mail] not found in Kerberos database)
```

We are using lnx.example.com as the IPA domain and not example.com since example.com is already an AD domain. We did not define a Kerberos realm for the idc1.us.example.com domain so Kerberos defaulted to example.com and translated it into the EXAMPLE.COM Kerberos realm which does exist and is reachable but it is the incorrect one for the IPA hosts since they are using the LNX.EXAMPLE.COM Kerberos realm.

This could be seen by using Kerberos commands on the masters.

```
[root@se-rhidm01x ~]# ldapsearch -Y GSSAPI -H ldap://usidc1-rhidm01x.idc1.us.example.com -b "" -s base
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
        additional info: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server krbtgt/[e-mail] not found in Kerberos database)
```

Here krbtgt/[e-mail] should be krbtgt/[e-mail].

To fix this I added the following to the /etc/krb5.conf file on all the masters:
```
...
[domain_realm]
...
 .idc1.us.example.com = LNX.EXAMPLE.COM
 idc1.us.example.com = LNX.EXAMPLE.COM
...
```

Ultimately we should probably define it as such to "catch all":
```
...
[domain_realm]
.example.com = LNX.EXAMPLE.COM
example.com = LNX.EXAMPLE.COM
...
```

--
Markus