sidgen_task fails with SID conflict

[Basile Pinsard]

Hi, 
I had an existing instance of freeipa that went broken so badly (pki-tomcat unrecoverable) that the only option was spinning up a new one and `ipa migrate-ds` from the broken one. 
The new instance was set to reuse the same id-range as the previous one, so all is good for the users in that range.
The older instance has a number of user that were imported from an even older LDAP with IDs out of the IPA range.

So after import, I quickly figured out that I need to create a small (1000) `legacy`  range that covers these, most of these legacy users were then able to login.
Here are the id-ranges after the legacy was added.

# ipa idrange-find
----------------
3 ranges matched
----------------
  Range name: DOMAIN_id_range
  First Posix ID of the range: 944200000
  Number of IDs in the range: 200000
  First RID of the corresponding RID range: 1000
  First RID of the secondary RID range: 100000000
  Range type: local domain range

Range name: DOMAIN_id_range_legacy
  First Posix ID of the range: 1000
  Number of IDs in the range: 1000
  First RID of the corresponding RID range: 302000
  First RID of the secondary RID range: 200000000
  Range type: local domain range

Range name: DOMAIN_subid_range
  First Posix ID of the range: 2147483648
  Number of IDs in the range: 2147352576
  First RID of the corresponding RID range: 2147283648
  Domain SID of the trusted domain: S-1-5-21-738065-838566-3977953474
  Range type: Active Directory domain range
----------------------------
Number of entries returned 3
----------------------------

However, a number of the legacy users still cannot login. 
I tried starting the sidgen task, and from the logs it seems that there is a conflict with the ranges I chose for the legacy id range.

```
[18/Jul/2024:16:24:12.358313104 +0000] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 194]: Sidgen task starts ...
[18/Jul/2024:16:24:12.598768115 +0000] - ERR - rid_to_sid_with_check - [file ipa_sidgen_common.c, line 384]: SID [S-1-5-21-3076474616-2786889582-2859700629-302272] is already used.
[18/Jul/2024:16:24:12.637972455 +0000] - ERR - rid_to_sid_with_check - [file ipa_sidgen_common.c, line 384]: SID [S-1-5-21-3076474616-2786889582-2859700629-200000272] is already used.
[18/Jul/2024:16:24:12.696381619 +0000] - ERR - find_sid_for_id - [file ipa_sidgen_common.c, line 432]: Secondary SID is used as well.
[18/Jul/2024:16:24:12.746590836 +0000] - ERR - find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 532]: Cannot convert Posix ID [1272] into an unused SID.
[18/Jul/2024:16:24:12.796710604 +0000] - ERR - do_work - [file ipa_sidgen_task.c, line 154]: Cannot add SID to existing entry.
[18/Jul/2024:16:24:12.854320074 +0000] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 199]: Sidgen task finished [19].
```

I cannot figure out what is my error and the documentation is quite scarce on how to choose first-rids, except saying that ranges shouldn't overlap, which I thought I made care of when creating the legacy range.
Maybe I am too dumb to understand where is the overlap I created, or why I do get conflict.

Thanks for your help and expertise!

[Florence Blanc-Renaud]

Hi,

your range configuration looks good to me.

The first range covers posix ids from 944'200'000 to 944'400'000 and RIDs
from 1'000 to 201'000  / secondary RIDs from 100'000'000 to 100'200'000.
The legacy range covers posix ids from 1000 to 2000 and RIDs from 302'000
to 303'000 / secondary RIDs from 200'000'000 to 200'001'000. There is no
overlap in the posix ids, and there is no overlap in the RIDs.

You can check if you already have an entry with the specified SID (replace
dc=ipa,dc=test with your own base DN):
ldapsearch -LLL -o ldif-wrap=no -D cn=directory\ manager -W -b
dc=ipa,dc=test
"(ipaNTSecurityIdentifier=S-1-5-21-3076474616-2786889582-2859700629-302272)"
dn

And check if you have entries with the same uidnumber/gidnumber:
ldapsearch -LLL -o ldif-wrap=no -D cn=directory\ manager -w Secret123 -b
dc=ipa,dc=test "(|(uidnumber=1272)(gidnumber=1272))" dn

It's ok to have a user and a group with the same value, when the group is
the private group for the user, but IIRC the migration creates regular
groups instead of private groups.

flo

On Thu, Jul 18, 2024 at 7:38 PM Basile Pinsard via FreeIPA-users <
[e-mail]> wrote:
> Hi,
> I had an existing instance of freeipa that went broken so badly
> (pki-tomcat unrecoverable) that the only option was spinning up a new one
> and `ipa migrate-ds` from the broken one.
> The new instance was set to reuse the same id-range as the previous one,
> so all is good for the users in that range.
> The older instance has a number of user that were imported from an even
> older LDAP with IDs out of the IPA range.
>
> So after import, I quickly figured out that I need to create a small
> (1000) `legacy`  range that covers these, most of these legacy users were
> then able to login.
> Here are the id-ranges after the legacy was added.
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 944200000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
>
> Range name: DOMAIN_id_range_legacy
>   First Posix ID of the range: 1000
>   Number of IDs in the range: 1000
>   First RID of the corresponding RID range: 302000
>   First RID of the secondary RID range: 200000000
>   Range type: local domain range
>
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-3977953474
>   Range type: Active Directory domain range
> Number of entries returned 3
>
> However, a number of the legacy users still cannot login.
> I tried starting the sidgen task, and from the logs it seems that there is
> a conflict with the ranges I chose for the legacy id range.
> [18/Jul/2024:16:24:12.358313104 +0000] - ERR - sidgen_task_thread - [file
> ipa_sidgen_task.c, line 194]: Sidgen task starts ...
> [18/Jul/2024:16:24:12.598768115 +0000] - ERR - rid_to_sid_with_check -
> [file ipa_sidgen_common.c, line 384]: SID
> [S-1-5-21-3076474616-2786889582-2859700629-302272] is already used.
> [18/Jul/2024:16:24:12.637972455 +0000] - ERR - rid_to_sid_with_check -
> [file ipa_sidgen_common.c, line 384]: SID
> [S-1-5-21-3076474616-2786889582-2859700629-200000272] is already used.
> [18/Jul/2024:16:24:12.696381619 +0000] - ERR - find_sid_for_id - [file
> ipa_sidgen_common.c, line 432]: Secondary SID is used as well.
> [18/Jul/2024:16:24:12.746590836 +0000] - ERR - find_sid_for_ldap_entry -
> [file ipa_sidgen_common.c, line 532]: Cannot convert Posix ID [1272] into
> an unused SID.
> [18/Jul/2024:16:24:12.796710604 +0000] - ERR - do_work - [file
> ipa_sidgen_task.c, line 154]: Cannot add SID to existing entry.
> [18/Jul/2024:16:24:12.854320074 +0000] - ERR - sidgen_task_thread - [file
> ipa_sidgen_task.c, line 199]: Sidgen task finished [19].
>
> I cannot figure out what is my error and the documentation is quite scarce
> on how to choose first-rids, except saying that ranges shouldn't overlap,
> which I thought I made care of when creating the legacy range.
> Maybe I am too dumb to understand where is the overlap I created, or why I
> do get conflict.
> Thanks for your help and expertise!
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[jan wagner]

Hey!

Have similar issue.

Is it ok, to change the RIDs for DOMAIN_id_range_legacy? Even if they previously started (I assume) with
> First RID of the corresponding RID range: 1000
>  First RID of the secondary RID range: 100000000

which is the default (I assume)? Does this affect some later process, or is it exclusively used just for SID generation?
Or is it ok as long as you dont use the ranges to create any new users?

Thanks!

[Florence Blanc-Renaud]

Hi,

On Fri, Aug 16, 2024 at 9:39 AM jan wagner via FreeIPA-users <
[e-mail]> wrote:
> Hey!
>
> Have similar issue.

Can you give more details? Similar how? What is not working?
> Is it ok, to change the RIDs for DOMAIN_id_range_legacy? Even if they
> previously started (I assume) with
> > First RID of the corresponding RID range: 1000
> >  First RID of the secondary RID range: 100000000

If your user entries already contain a ipaNtSecurityIdentifier but you
change the Base RID and Secondary Base RID, the mapping algorithm will not
evaluate ipaNtSecurityIdentifier the same way for a given posix ID and
vice-versa and the user entry content will be inconsistent. Why do you plan
to modify the RIDs?

which is the default (I assume)? Does this affect some later process, or is
> it exclusively used just for SID generation?
> Or is it ok as long as you dont use the ranges to create any new users?
> Thanks!
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Jan Wagner]

Thanks for the quick reply!
I need to migrate a old ipa instance/domain (lets name it ipa_old.example.com) to an new IPA instance/domain (lets name it ipa_new.example.com)

Let's assume, that ipa_old.example.com has these ranges:
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 576000000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-3977953474
>   Range type: Active Directory domain range

And that ipa_new.example.com these ranges:
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 768200000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
>   Range type: Active Directory domain range

I want to use ipa migrate-ds for the migration, however it will copy over ipaNtSecurityIdentifier entries with the ipa_old.example.com's Domain SID.
I should then add the range from ipa_old.example.com to the ipa_new.example.com, however I would not be able to do it as is, because RIDs would conflict.

Is it ok, to set ranges as follows for ipa_new.example.com, if there are yet no users created (other then 'admin') in ipa_new.example.com and if I change ipaNtSecurityIdentifier and U/GID of admin to match the new range?
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 768200000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 201000
>   First RID of the secondary RID range: 100001000
>   Range type: local domain range
> Range name: DOMAIN_id_range_legacy
>   First Posix ID of the range: 576000000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
>   Range type: Active Directory domain range

If I did it this way instead:
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 768200000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
> Range name: DOMAIN_id_range_legacy
>   First Posix ID of the range: 576000000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 201000
>   First RID of the secondary RID range: 100001000
>   Range type: local domain range
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
>   Range type: Active Directory domain range

Then ipaNtSecurityIdentifiers would be indeed inconsistent. What would it cause? Would IPA generate already existing ipaNtSecurityIdentifiers for new users and cause conflicts?

Would it be better, if I used these ranges (the ones that were created on installation) instead:
> # ipa idrange-find
> 3 ranges matched
>
> Range name: DOMAIN_id_range
>   First Posix ID of the range: 768200000
>   Number of IDs in the range: 200000
>   First RID of the corresponding RID range: 1000
>   First RID of the secondary RID range: 100000000
>   Range type: local domain range
> Range name: DOMAIN_subid_range
>   First Posix ID of the range: 2147483648
>   Number of IDs in the range: 2147352576
>   First RID of the corresponding RID range: 2147283648
>   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
>   Range type: Active Directory domain range

and programmatically created all the users from ipa_old.example.com in ipa_new.example.com with different IDs (IPA would allocate some from DOMAIN_id_range), and then used ID Views in ipa_new.example.com, to map them to their posix user IDs from ipa_old.example.com (so file permissions on hosts previously enrolled in ipa_old.example.com would remain functional/semantically the same)?

Thanks!

[Florence Blanc-Renaud]

Hi,

On Fri, Aug 16, 2024 at 2:38 PM Jan Wagner via FreeIPA-users <
[e-mail]> wrote:
> Thanks for the quick reply!
> I need to migrate a old ipa instance/domain (lets name it
> ipa_old.example.com) to an new IPA instance/domain (lets name it
> ipa_new.example.com)
>
> Let's assume, that ipa_old.example.com has these ranges:
> > # ipa idrange-find
> > 3 ranges matched
> >
> > Range name: DOMAIN_id_range
> >   First Posix ID of the range: 576000000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 1000
> >   First RID of the secondary RID range: 100000000
> >   Range type: local domain range
> >   Range name: DOMAIN_subid_range
> >
> > First Posix ID of the range: 2147483648
> >   Number of IDs in the range: 2147352576
> >   First RID of the corresponding RID range: 2147283648
> >   Domain SID of the trusted domain: S-1-5-21-738065-838566-3977953474
> >   Range type: Active Directory domain range
>
> And that ipa_new.example.com these ranges:
> > # ipa idrange-find
> > 3 ranges matched
> >
> > Range name: DOMAIN_id_range
> >   First Posix ID of the range: 768200000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 1000
> >   First RID of the secondary RID range: 100000000
> >   Range type: local domain range
> >   Range name: DOMAIN_subid_range
> >
> > First Posix ID of the range: 2147483648
> >   Number of IDs in the range: 2147352576
> >   First RID of the corresponding RID range: 2147283648
> >   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
> >   Range type: Active Directory domain range
>
> I want to use ipa migrate-ds for the migration, however it will copy over
> ipaNtSecurityIdentifier entries with the ipa_old.example.com's Domain SID.
> I should then add the range from ipa_old.example.com to the
> ipa_new.example.com, however I would not be able to do it as is, because
> RIDs would conflict.
>
> If you are using migrate-ds, you will have to remove the

ipantsecuritydentifier from the migrated entries because they contain a
domain SID from the old installation, and it will not be the same as the
new installation domain SID. You can read
https://lists.fedorahosted.org/archives/list/[e-mail]...
for more explanations.

Is it ok, to set ranges as follows for ipa_new.example.com, if there are
> yet no users created (other then 'admin') in ipa_new.example.com and if I
> change ipaNtSecurityIdentifier and U/GID of admin to match the new range?

Note that if you are starting the new server installation from scratch you
can provide some arguments directly to ipa-server-install instead of
modifying the values later on:
ipa-server-install [...] --rid-base=RID_BASE
--secondary-rid-base=SECONDARY_RID_BASE --idstart=IDSTART
> > # ipa idrange-find
> > 3 ranges matched
> >
> > Range name: DOMAIN_id_range
> >   First Posix ID of the range: 768200000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 201000
> >   First RID of the secondary RID range: 100001000
> >   Range type: local domain range
>
> This range will use

- POSIX ids [768'200'000;768'400'000[
- primary RIDS [201'000;401'000[
- secondary RIDS [*100'001'000;100'201'000*[
> Range name: DOMAIN_id_range_legacy
> > First Posix ID of the range: 576000000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 1000
> >   First RID of the secondary RID range: 100000000
> >   Range type: local domain range

This range will use
- POSIX ids [576'000'000;576'200'000[
- primary RIDS [1'000;201'000[
- secondary RIDs [*100'000'000;100'200'000*[

The secondary RIDs are overlapping. For DOMAIN_id_range you could use
instead 100'200'000.
flo
> >   Range name: DOMAIN_subid_range
> >
> > First Posix ID of the range: 2147483648
> >   Number of IDs in the range: 2147352576
> >   First RID of the corresponding RID range: 2147283648
> >   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
> >   Range type: Active Directory domain range
>
> If I did it this way instead:
> > # ipa idrange-find
> > 3 ranges matched
> >
> > Range name: DOMAIN_id_range
> >   First Posix ID of the range: 768200000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 1000
> >   First RID of the secondary RID range: 100000000
> >   Range type: local domain range
> > Range name: DOMAIN_id_range_legacy
> >   First Posix ID of the range: 576000000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 201000
> >   First RID of the secondary RID range: 100001000
> >   Range type: local domain range
> >   Range name: DOMAIN_subid_range
> >
> > First Posix ID of the range: 2147483648
> >   Number of IDs in the range: 2147352576
> >   First RID of the corresponding RID range: 2147283648
> >   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
> >   Range type: Active Directory domain range
>
> Then ipaNtSecurityIdentifiers would be indeed inconsistent. What would it
> cause? Would IPA generate already existing ipaNtSecurityIdentifiers for new
> users and cause conflicts?
>
> Would it be better, if I used these ranges (the ones that were created on
> installation) instead:
> > # ipa idrange-find
> > 3 ranges matched
> >
> > Range name: DOMAIN_id_range
> >   First Posix ID of the range: 768200000
> >   Number of IDs in the range: 200000
> >   First RID of the corresponding RID range: 1000
> >   First RID of the secondary RID range: 100000000
> >   Range type: local domain range
> >   Range name: DOMAIN_subid_range
> >
> > First Posix ID of the range: 2147483648
> >   Number of IDs in the range: 2147352576
> >   First RID of the corresponding RID range: 2147283648
> >   Domain SID of the trusted domain: S-1-5-21-738065-838566-418768009
> >   Range type: Active Directory domain range
>
> and programmatically created all the users from ipa_old.example.com in
> ipa_new.example.com with different IDs (IPA would allocate some from
> DOMAIN_id_range), and then used ID Views in ipa_new.example.com, to map
> them to their posix user IDs from ipa_old.example.com (so file
> permissions on hosts previously enrolled in ipa_old.example.com would
> remain functional/semantically the same)?
> Thanks!
>
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue