Everything getting lowercased migrating between FreeIPA instances

[Mitchell Smith]

Hi List,

I am trying to migrate an old FreeIPA 4.3.1 server running on Ubuntu
16.04 to a new FreeIPA 4.5.4 server running on Centos 7.

I am doing the migration via the "ipa migrate-ds" command, the command
is running successfully and the users get migrated, even the custom
schema attributes come over which is nice, but everything is getting
converted to lowercase, even things like object classes, which is
causing some issues for things talking to LDAP and expecting specific
values.

A very simplistic example without going in to our custom schema is

ObjectClass: posixAccount

and after the migration

ObjectClass: posixaccount

I have tweaked /usr/lib/python2.7/site-packages/ipaserver/plugins/migration.py
as follows to try and work around this however the migration is still
lowercasing everything.

If anyone could please suggest where else in the code I should start
digging where the migration might be getting normalized into
lowercase, I would really appreciate any feedback.

--- migration.orig	2018-11-22 00:50:07.335290536 +0000
+++ migration.py	2018-11-22 00:51:40.938290536 +0000
@@ -284,7 +284,7 @@
                     continue

                 api.log.debug('converting DN value %s for %s in %s' %
(value, attr, dn))
-                rdnval = remote_entry[primary_key][0].lower()
+                rdnval = remote_entry[primary_key][0]
                 entry_attrs[attr][ind] = DN((primary_key, rdnval),
container, api.env.basedn)

     return dn
@@ -697,7 +697,7 @@
         for name in names:
             if options[name]:
                 options[name] = tuple(
-                    v.lower() for v in options[name]
+                    v for v in options[name]
                 )
             else:
                 options[name] = tuple()
@@ -801,9 +801,9 @@
                     # In case if pkey attribute is in the migrated object DN
                     # and the original LDAP is multivalued, make sure that
                     # we pick the correct value (the unique one stored in DN)
-                    pkey = ava.value.lower()
+                    pkey = ava.value
                 else:
-                    pkey = entry_attrs[ldap_obj.primary_key.name][0].lower()
+                    pkey = entry_attrs[ldap_obj.primary_key.name][0]

                 if pkey in exclude:
                     continue
@@ -813,10 +813,10 @@
                     set(
                         config.get(
                             ldap_obj.object_class_config, ldap_obj.object_class
-                        ) + [o.lower() for o in entry_attrs['objectclass']]
+                        ) + [o for o in entry_attrs['objectclass']]
                     )
                 )
-                entry_attrs[ldap_obj.primary_key.name][0] =
entry_attrs[ldap_obj.primary_key.name][0].lower()
+                entry_attrs[ldap_obj.primary_key.name][0] =
entry_attrs[ldap_obj.primary_key.name][0]

                 callback = self.migrate_objects[ldap_obj_name]['pre_callback']
                 if callable(callback):

Thanks for any suggestions.

Cheers

[Rob Crittenden]

Mitchell Smith via FreeIPA-users wrote:
>  Hi List,
>  
>  I am trying to migrate an old FreeIPA 4.3.1 server running on Ubuntu
>  16.04 to a new FreeIPA 4.5.4 server running on Centos 7.
>  
>  I am doing the migration via the "ipa migrate-ds" command, the command
>  is running successfully and the users get migrated, even the custom
>  schema attributes come over which is nice, but everything is getting
>  converted to lowercase, even things like object classes, which is
>  causing some issues for things talking to LDAP and expecting specific
>  values.
>  
>  A very simplistic example without going in to our custom schema is
>  
>  ObjectClass: posixAccount
>  
>  and after the migration
>  
>  ObjectClass: posixaccount
>  
>  I have tweaked /usr/lib/python2.7/site-packages/ipaserver/plugins/migration.py
>  as follows to try and work around this however the migration is still
>  lowercasing everything.
>  
>  If anyone could please suggest where else in the code I should start
>  digging where the migration might be getting normalized into
>  lowercase, I would really appreciate any feedback. 
Sorry, I don't know what might be doing this. It could be in the ldap2
module itself.

But, relying on case in an objectclass is not a good idea. The RFC isn't
very specific about the matching criteria but 389-ds has it implemented
as case insensitive.

I guess another question would be why not create a 4.5.4 replica based
on your current master and then eventually retire the old one?

rob


rob

>  
>  --- migration.orig	2018-11-22 00:50:07.335290536 +0000
>  +++ migration.py	2018-11-22 00:51:40.938290536 +0000
>  @@ -284,7 +284,7 @@
>                       continue
>  
>                   api.log.debug('converting DN value %s for %s in %s' %
>  (value, attr, dn))
>  -                rdnval = remote_entry[primary_key][0].lower()
>  +                rdnval = remote_entry[primary_key][0]
>                   entry_attrs[attr][ind] = DN((primary_key, rdnval),
>  container, api.env.basedn)
>  
>       return dn
>  @@ -697,7 +697,7 @@
>           for name in names:
>               if options[name]:
>                   options[name] = tuple(
>  -                    v.lower() for v in options[name]
>  +                    v for v in options[name]
>                   )
>               else:
>                   options[name] = tuple()
>  @@ -801,9 +801,9 @@
>                       # In case if pkey attribute is in the migrated object DN
>                       # and the original LDAP is multivalued, make sure that
>                       # we pick the correct value (the unique one stored in DN)
>  -                    pkey = ava.value.lower()
>  +                    pkey = ava.value
>                   else:
>  -                    pkey = entry_attrs[ldap_obj.primary_key.name][0].lower()
>  +                    pkey = entry_attrs[ldap_obj.primary_key.name][0]
>  
>                   if pkey in exclude:
>                       continue
>  @@ -813,10 +813,10 @@
>                       set(
>                           config.get(
>                               ldap_obj.object_class_config, ldap_obj.object_class
>  -                        ) + [o.lower() for o in entry_attrs['objectclass']]
>  +                        ) + [o for o in entry_attrs['objectclass']]
>                       )
>                   )
>  -                entry_attrs[ldap_obj.primary_key.name][0] =
>  entry_attrs[ldap_obj.primary_key.name][0].lower()
>  +                entry_attrs[ldap_obj.primary_key.name][0] =
>  entry_attrs[ldap_obj.primary_key.name][0]
>  
>                   callback = self.migrate_objects[ldap_obj_name]['pre_callback']
>                   if callable(callback):
>  
>  Thanks for any suggestions.
>  
>  Cheers
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>

[Mitchell Smith]

Hi Rob,

Yes my initial thought was to add the new instance as a replica and then eventually retire
the old one.

I ran in to some problems with the ‘ipa-replica-install’ command though.

I was able to join as a client no problem, but when I went to run ‘ipa-replica-install’ it
failed while configuring the directory server component.

  [25/42]: restarting directory server
  [26/42]: creating DS keytab
  [27/42]: ignore time skew for initial replication
  [28/42]: setting up initial replication
  [error] DatabaseError: Server is unwilling to perform: modification of attribute
nsds5replicareleasetimeout is not allowed in replica entry
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

I thought this might have something to do with differences between 4.3.1 and 4.5.4 but I
wasn’t entirely sure.

If there is a work around for this issue, it would be a significantly easier transition to
the new FreeIPA instance.

Cheers,

Mitch


>  On Nov 28, 2018, at 4:45 AM, Rob Crittenden
> <rcritten(a)redhat.com&gt; wrote:
>  
>  Mitchell Smith via FreeIPA-users wrote:
> > Hi List,
> > 
> > I am trying to migrate an old FreeIPA 4.3.1 server running on Ubuntu
> > 16.04 to a new FreeIPA 4.5.4 server running on Centos 7.
> > 
> > I am doing the migration via the "ipa migrate-ds" command, the command
> > is running successfully and the users get migrated, even the custom
> > schema attributes come over which is nice, but everything is getting
> > converted to lowercase, even things like object classes, which is
> > causing some issues for things talking to LDAP and expecting specific
> > values.
> > 
> > A very simplistic example without going in to our custom schema is
> > 
> > ObjectClass: posixAccount
> > 
> > and after the migration
> > 
> > ObjectClass: posixaccount
> > 
> > I have tweaked /usr/lib/python2.7/site-packages/ipaserver/plugins/migration.py
> > as follows to try and work around this however the migration is still
> > lowercasing everything.
> > 
> > If anyone could please suggest where else in the code I should start
> > digging where the migration might be getting normalized into
> > lowercase, I would really appreciate any feedback.
>  
>  Sorry, I don't know what might be doing this. It could be in the ldap2
>  module itself.
>  
>  But, relying on case in an objectclass is not a good idea. The RFC isn't
>  very specific about the matching criteria but 389-ds has it implemented
>  as case insensitive.
>  
>  I guess another question would be why not create a 4.5.4 replica based
>  on your current master and then eventually retire the old one?
>  
>  rob
>  
>  
>  rob
>  
> > 
> > --- migration.orig	2018-11-22 00:50:07.335290536 +0000
> > +++ migration.py	2018-11-22 00:51:40.938290536 +0000
> > @@ -284,7 +284,7 @@
> >                     continue
> > 
> >                 api.log.debug('converting DN value %s for %s in %s' %
> > (value, attr, dn))
> > -                rdnval = remote_entry[primary_key][0].lower()
> > +                rdnval = remote_entry[primary_key][0]
> >                 entry_attrs[attr][ind] = DN((primary_key, rdnval),
> > container, api.env.basedn)
> > 
> >     return dn
> > @@ -697,7 +697,7 @@
> >         for name in names:
> >             if options[name]:
> >                 options[name] = tuple(
> > -                    v.lower() for v in options[name]
> > +                    v for v in options[name]
> >                 )
> >             else:
> >                 options[name] = tuple()
> > @@ -801,9 +801,9 @@
> >                     # In case if pkey attribute is in the migrated object DN
> >                     # and the original LDAP is multivalued, make sure that
> >                     # we pick the correct value (the unique one stored in DN)
> > -                    pkey = ava.value.lower()
> > +                    pkey = ava.value
> >                 else:
> > -                    pkey = entry_attrs[ldap_obj.primary_key.name][0].lower()
> > +                    pkey = entry_attrs[ldap_obj.primary_key.name][0]
> > 
> >                 if pkey in exclude:
> >                     continue
> > @@ -813,10 +813,10 @@
> >                     set(
> >                         config.get(
> >                             ldap_obj.object_class_config, ldap_obj.object_class
> > -                        ) + [o.lower() for o in entry_attrs['objectclass']]
> > +                        ) + [o for o in entry_attrs['objectclass']]
> >                     )
> >                 )
> > -                entry_attrs[ldap_obj.primary_key.name][0] =
> > entry_attrs[ldap_obj.primary_key.name][0].lower()
> > +                entry_attrs[ldap_obj.primary_key.name][0] =
> > entry_attrs[ldap_obj.primary_key.name][0]
> > 
> >                 callback =
> self.migrate_objects[ldap_obj_name]['pre_callback']
> >                 if callable(callback):
> > 
> > Thanks for any suggestions.
> > 
> > Cheers
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> <mailto:[e-mail]>
> > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> <mailto:[e-mail]>
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> <https://getfedora.org/code-of-conduct.html>
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> <https://fedoraproject.org/wiki/Mailing_list_guidelines>
> > List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> <https://lists.fedorahosted.org/archives/list/[e-mail]...