Hi. I'm having problems adding a replica. I think it's related to id ranges not being set correctly on the existing server.
Some context that isn't strictly related to the error but I think is relevant. Initially there used to be 2 servers, ipa1 and ipa2, both CentOS 7, fully up to date. Some time last year replication fell and I tried to fix it but was unable to. Since the error was in the ipa2-to-ipa1 direction I tried to reinit ipa1 but that failed and left ipa1 in a deconfigured, non-working state (I could give you more details but I don't think it's necessary at this time). I thus uninstalled ipa1 and went to try to salvage the still working ipa2 server. After making both a backup of the VM and a full ipa-backup, I went through the process of adding a new replica, which I decided to install on Rocky Linux 8 in order to start the upgrade process at the same time (with a goal to get to Rocky 9 later).
First I hit the "SASL encrypted packet length exceeds maximum allowed limit" error which I solved by increasing nsslapd-maxsasliosize and nsslapd-sasl-max-buffer-size on ipa2 and setting those values in ipa-replica-install --dirsrv-config-file.
After that I hit this "Failed to add fallback group." error. I found two existing threads on this mailing list, one from 2017 and one from just a couple days ago, and a Red Hat KB page that I can't view. [1]
I understand the fix may be to modify/set ID ranges but I'm not exactly sure how so I'm asking for your help.
Below are logs of the error and current state of range settings.
### ipa-replica-install error on ipa3 full log is here: https://0x0.st/oFDg.txt
Configuring SID generation [1/7]: creating samba domain object [2/7]: adding admin(group) SIDs [3/7]: adding RID bases [4/7]: updating Kerberos config 'dns_lookup_kdc' already set to 'true', nothing to do. [5/7]: activating sidgen task [6/7]: restarting Directory Server to take MS PAC and LDAP plugins changes into account [7/7]: adding fallback group Failed to load default-smb-group.ldif: CalledProcessError(Command ['/usr/bin/ldapmodify', '-v', '-f', '/tmp/tmpbg9tdvpw', '-H', 'ld api://%2Frun%2Fslapd-ABAK-SI.socket', '-Y', 'EXTERNAL'] returned non-zero exit status 1: 'ldap_initialize( ldapi://%2Frun%2Fslapd-A BAK-SI.socket/??base )\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=aut h\nSASL SSF: 0\nldap_add: Operations error (1)\n\tadditional info: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.\n') Failed to add fallback group.
### range settings [jernej@ipa2 ~]$ sudo ipa-replica-manage dnarange-show ipa2.abak.si: No range set ipa3.abak.si: No range set [jernej@ipa2 ~]$ ipa idrange-find ---------------- 2 ranges matched ---------------- Range name: ABAK.SI_id_range First Posix ID of the range: 792600000 Number of IDs in the range: 200000 First RID of the corresponding RID range: 1000 First RID of the secondary RID range: 100000000 Range type: local domain range
Range name: ABAK.SI_subid_range First Posix ID of the range: 2147483648 Number of IDs in the range: 2147352576 First RID of the corresponding RID range: 2147283648 Domain SID of the trusted domain: S-1-5-21-738065-838566-3187085368 Range type: Active Directory domain range ---------------------------- Number of entries returned 2 ---------------------------- [jernej@ipa2 ~]$ sudo ipa-replica-manage list ipa3.abak.si: master ipa2.abak.si: master [jernej@ipa2 ~]$ sudo ipa-replica-manage dnanextrange-show Directory Manager password:
ipa2.abak.si: No on-deck range set ipa3.abak.si: No on-deck range set [jernej@ipa2 ~]$ ldapsearch -x -D 'cn=Directory Manager' -W -b 'cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn =config' Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# Posix IDs, Distributed Numeric Assignment Plugin, plugins, config dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config cn: Posix IDs dnaExcludeScope: cn=provisioning,dc=abak,dc=si dnaFilter: (|(objectClass=posixAccount)(objectClass=posixGroup)(objectClass=ip aIDobject)) dnaMagicRegen: -1 dnaMaxValue: 1100 dnaNextValue: 1101 dnaScope: dc=abak,dc=si dnaSharedCfgDN: cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=abak,dc=si dnaThreshold: 500 dnaType: uidNumber dnaType: gidNumber objectClass: top objectClass: extensibleObject
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
Jernej Jakob via FreeIPA-users wrote:
Hi. I'm having problems adding a replica. I think it's related to id ranges not being set correctly on the existing server.
Some context that isn't strictly related to the error but I think is relevant. Initially there used to be 2 servers, ipa1 and ipa2, both CentOS 7, fully up to date. Some time last year replication fell and I tried to fix it but was unable to. Since the error was in the ipa2-to-ipa1 direction I tried to reinit ipa1 but that failed and left ipa1 in a deconfigured, non-working state (I could give you more details but I don't think it's necessary at this time). I thus uninstalled ipa1 and went to try to salvage the still working ipa2 server. After making both a backup of the VM and a full ipa-backup, I went through the process of adding a new replica, which I decided to install on Rocky Linux 8 in order to start the upgrade process at the same time (with a goal to get to Rocky 9 later).
First I hit the "SASL encrypted packet length exceeds maximum allowed limit" error which I solved by increasing nsslapd-maxsasliosize and nsslapd-sasl-max-buffer-size on ipa2 and setting those values in ipa-replica-install --dirsrv-config-file.
After that I hit this "Failed to add fallback group." error. I found two existing threads on this mailing list, one from 2017 and one from just a couple days ago, and a Red Hat KB page that I can't view. [1]
I understand the fix may be to modify/set ID ranges but I'm not exactly sure how so I'm asking for your help.
Below are logs of the error and current state of range settings.
### ipa-replica-install error on ipa3 full log is here: https://0x0.st/oFDg.txt
Configuring SID generation [1/7]: creating samba domain object [2/7]: adding admin(group) SIDs [3/7]: adding RID bases [4/7]: updating Kerberos config 'dns_lookup_kdc' already set to 'true', nothing to do. [5/7]: activating sidgen task [6/7]: restarting Directory Server to take MS PAC and LDAP plugins changes into account [7/7]: adding fallback group Failed to load default-smb-group.ldif: CalledProcessError(Command ['/usr/bin/ldapmodify', '-v', '-f', '/tmp/tmpbg9tdvpw', '-H', 'ld api://%2Frun%2Fslapd-ABAK-SI.socket', '-Y', 'EXTERNAL'] returned non-zero exit status 1: 'ldap_initialize( ldapi://%2Frun%2Fslapd-A BAK-SI.socket/??base )\nSASL/EXTERNAL authentication started\nSASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=aut h\nSASL SSF: 0\nldap_add: Operations error (1)\n\tadditional info: Allocation of a new value for range cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config failed! Unable to proceed.\n') Failed to add fallback group.
### range settings [jernej@ipa2 ~]$ sudo ipa-replica-manage dnarange-show ipa2.abak.si: No range set ipa3.abak.si: No range set [jernej@ipa2 ~]$ ipa idrange-find
2 ranges matched
Range name: ABAK.SI_id_range First Posix ID of the range: 792600000 Number of IDs in the range: 200000 First RID of the corresponding RID range: 1000 First RID of the secondary RID range: 100000000 Range type: local domain range
Range name: ABAK.SI_subid_range First Posix ID of the range: 2147483648 Number of IDs in the range: 2147352576 First RID of the corresponding RID range: 2147283648 Domain SID of the trusted domain: S-1-5-21-738065-838566-3187085368 Range type: Active Directory domain range
Number of entries returned 2
[jernej@ipa2 ~]$ sudo ipa-replica-manage list ipa3.abak.si: master ipa2.abak.si: master [jernej@ipa2 ~]$ sudo ipa-replica-manage dnanextrange-show Directory Manager password:
ipa2.abak.si: No on-deck range set ipa3.abak.si: No on-deck range set [jernej@ipa2 ~]$ ldapsearch -x -D 'cn=Directory Manager' -W -b 'cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn =config' Enter LDAP Password: # extended LDIF # # LDAPv3 # base <cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config> with scope subtree # filter: (objectclass=*) # requesting: ALL #
# Posix IDs, Distributed Numeric Assignment Plugin, plugins, config dn: cn=Posix IDs,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config cn: Posix IDs dnaExcludeScope: cn=provisioning,dc=abak,dc=si dnaFilter: (|(objectClass=posixAccount)(objectClass=posixGroup)(objectClass=ip aIDobject)) dnaMagicRegen: -1 dnaMaxValue: 1100 dnaNextValue: 1101 dnaScope: dc=abak,dc=si dnaSharedCfgDN: cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=abak,dc=si dnaThreshold: 500 dnaType: uidNumber dnaType: gidNumber objectClass: top objectClass: extensibleObject
# search result search: 2 result: 0 Success
# numResponses: 2 # numEntries: 1
Since the ipa3 installation failed I'd start by uninstalling the remnants.
You can use ipa-replica-manage dnarange-set on ipa2 to set the range to cover your entire range. I'd encourage you to find the highest value already used just to over a bunch of overlap searching in the DNA plugin.
You can get a rough estimate of the last issued value with a search like:
ldapsearch -LLL -Q -Y GSSAPI -b cn=accounts,dc=example,dc=test uidnumber gidnumber | cut -d: -f2 | sort -un
If you want to test it before trying another replica install create a test user or group and it should get an uid/gid.
On the next replica install it should give the new server half the remaining range.
rob
On Wed, 1 Feb 2023 10:00:56 -0500 Rob Crittenden rcritten@redhat.com wrote:
Since the ipa3 installation failed I'd start by uninstalling the remnants.
You can use ipa-replica-manage dnarange-set on ipa2 to set the range to cover your entire range. I'd encourage you to find the highest value already used just to over a bunch of overlap searching in the DNA plugin.
You can get a rough estimate of the last issued value with a search like:
ldapsearch -LLL -Q -Y GSSAPI -b cn=accounts,dc=example,dc=test uidnumber gidnumber | cut -d: -f2 | sort -un
If you want to test it before trying another replica install create a test user or group and it should get an uid/gid.
On the next replica install it should give the new server half the remaining range.
rob
Thanks. I managed to install the replica successfully.
The ldapsearch command showed there were two ranges of used ID's. 792600000-792600036 and 792700504-792700509. (I think the first was assigned to ipa1 - the now uninstalled replica, the second to ipa2) So I chose 792600040-792700499 as dnarange for ipa2. I also set 792700510-792799999 as dnanextrange for ipa2.
Then I could add the new replica with no problem. It chose 792750501-792799999 as the range for the new replica, taken from dnanextrange for ipa2. I don't think that will be a problem as I'm very unlikely to ever need more ID's.
I also had a problem when uninstalling the failed replica from the last attempt that ended at this "Failed to add fallback group." error. I had done this a couple times before (due to other errors) and always used the procedure: - 'ipa-server-install --uninstall' - on ipa2: 'ipa-replica-manage clean-dangling-ruv' (as there were always leftover RUVs that the uninstall didn't delete) - checked there wasn't a leftover topology or server - then re-ran ipa-client-install and ipa-replica-install.
This time the 'clean-dangling-ruv' step did not complete. It removed the RUV for 'domain' but could not delete the 'ca' RUV. Unfortunately the slapd error log got rotated which deleted the error in question but I know it was "Unable to acquire replica: error: duplicate replica ID detected" from my search history. I could not find any relevant info on this ruv cleanup error. I tried cancelling and resubmitting the cleanup but it never succeeded. So I restored the server to a snapshot I made a couple days ago before I started trying to add a new replica. After this I was able to install the replica successfully.
I also registered on the Red Hat Customer Portal which allowed me to view the knowledgebase docs. They were helpful in pointing me to relevant docs pages. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Jernej Jakob wrote:
On Wed, 1 Feb 2023 10:00:56 -0500 Rob Crittenden rcritten@redhat.com wrote:
Since the ipa3 installation failed I'd start by uninstalling the remnants.
You can use ipa-replica-manage dnarange-set on ipa2 to set the range to cover your entire range. I'd encourage you to find the highest value already used just to over a bunch of overlap searching in the DNA plugin.
You can get a rough estimate of the last issued value with a search like:
ldapsearch -LLL -Q -Y GSSAPI -b cn=accounts,dc=example,dc=test uidnumber gidnumber | cut -d: -f2 | sort -un
If you want to test it before trying another replica install create a test user or group and it should get an uid/gid.
On the next replica install it should give the new server half the remaining range.
rob
Thanks. I managed to install the replica successfully.
The ldapsearch command showed there were two ranges of used ID's. 792600000-792600036 and 792700504-792700509. (I think the first was assigned to ipa1 - the now uninstalled replica, the second to ipa2) So I chose 792600040-792700499 as dnarange for ipa2. I also set 792700510-792799999 as dnanextrange for ipa2.
Then I could add the new replica with no problem. It chose 792750501-792799999 as the range for the new replica, taken from dnanextrange for ipa2. I don't think that will be a problem as I'm very unlikely to ever need more ID's.
I also had a problem when uninstalling the failed replica from the last attempt that ended at this "Failed to add fallback group." error. I had done this a couple times before (due to other errors) and always used the procedure:
- 'ipa-server-install --uninstall'
- on ipa2: 'ipa-replica-manage clean-dangling-ruv' (as there were always leftover RUVs that the uninstall didn't delete)
- checked there wasn't a leftover topology or server
- then re-ran ipa-client-install and ipa-replica-install.
This time the 'clean-dangling-ruv' step did not complete. It removed the RUV for 'domain' but could not delete the 'ca' RUV. Unfortunately the slapd error log got rotated which deleted the error in question but I know it was "Unable to acquire replica: error: duplicate replica ID detected" from my search history. I could not find any relevant info on this ruv cleanup error. I tried cancelling and resubmitting the cleanup but it never succeeded. So I restored the server to a snapshot I made a couple days ago before I started trying to add a new replica. After this I was able to install the replica successfully.
I also registered on the Red Hat Customer Portal which allowed me to view the knowledgebase docs. They were helpful in pointing me to relevant docs pages. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
Glad you got it working.
For future knowledge, there are also list-ruv and clean-ruv options to ipa-replica-manage to help ferret out invalid RUVs individually.
rob
freeipa-users@lists.fedorahosted.org
-
Jernej Jakob -
Rob Crittenden