[389-users] 389-console LDIF export question

[Michael Lang]

On 06/27/2013 08:52 PM, Rich Megginson wrote:
> On 06/27/2013 12:32 PM, Michael Lang wrote:
>> On 06/27/2013 06:13 PM, Rich Megginson wrote:
>>> On 06/27/2013 04:57 AM, Michael Lang wrote:
>>>> Dear all,
>>>>
>>>> I would like to clarify how the procedure to "export -> LDIF" 
>>>> through the 389-console GUI is done (or how if reproduce should be 
>>>> done).
>>>> As far as I've understood the exports on the server itself is done 
>>>> through creating a appropriate entry in cn=export,cn=task,cn=config
>>>> For "remote" machines using the GUI this isn't valid as the 
>>>> nsFilename attribute requires a local write able location. What 
>>>> I've seen through looking at the
>>>> LDAP protocol queries done during an export-to-ldif on console 
>>>> machine task is a query to the schema listing all attributeTypes 
>>>> which are then sent as attribute-retrieve-list in the ldap search. 
>>>> At least that's how I was able to reproduce the same LDIF from 
>>>> GUI-Console export and manually. Is this correct and should this be 
>>>> done in that way ? (I've noticed also the "Warning: If you don't 
>>>> have permissions" statement when using the LDIF export in the 
>>>> Console which more-or-less ack's my approach)
>>>
>>> Right.  There is no way to do a database export to LDIF to a remote 
>>> file on a remote machine.
>>>
>>> Are you trying to figure out a way to generate an LDIF file using 
>>> ldapsearch that looks exactly like a database export LDIF file? Why?
>>
>> reason for that, I might not have "file based" access on the server 
>> itself but need to do backup's of the database which are not "setup 
>> specific" (with setup specific I mean without the right database 
>> configurations I might not be able to import the content).
>>
>> in this special case why I started to look into the ldif create, I 
>> had database backups but ran into a situation where my directory 
>> console was showing content but ldapsearch (and other 
>> tools/applications) where not getting any content at all. It was a 
>> strange situation and restoring the database trough restore didn't 
>> help where droping the databases and importing and LDIF was working 
>> (I still had another server running with the content). As mentioned 
>> above for the machines I have under complete access this shouldn't be 
>> the issue, but I also have other machines where I might be forced to 
>> do this through the LDAP protocol.
>
> That particular case sounds like you were missing the "aci" attribute 
> which is an operational attribute.
>
> In general, if you want "everything"(1), you can do a search like this:
>
> ldapsearch -b dc=your,dc=suffix 
> '(|(objectclass=*)(objectclass=ldapsubentry))' \* $OPERATIONAL_ATTRS
>
> where $OPERATIONAL_ATTRS is a list of operational attributes you can 
> get from the schema.  You'll definitely want "aci" in that list.
>
> If you know python, python-ldap has a nice schema parser.
>
> (1) This still won't get you everything - specifically, you won't get 
> deleted entries and other replication metadata - If you think you need 
> that information, add (objectclass=nsTombstone) to the search filter 
> above.


Rich,

thanks for you explanation, the 389-console GUI itself doesn't export 
nsTombstone objects and the purpose is to be able to restore the data 
not the replication setting for the database the suffice resides on (at 
least that's what I would understood from the attributes in the entry 
returned by the search)

again, thanks for the python-schema hint, much easier then doing it 
manually, this is how I would scripted reproduce the 389-console LDIF 
Database Export without the GUI)

<source lang="python">
#!/usr/bin/env python
import ldap
import ldap.schema
import logging
import sys
import ldif

logger  = logging.getLogger('LDIFexport')
logger.addHandler(logging.StreamHandler(sys.stderr))
logger.setLevel(logging.INFO)

ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_NEVER)

class LDIFDumper(object):
     def __init__(self, uri=None):
         self.uri                = uri
         self.dn, self.schema    = ldap.schema.urlfetch(self.uri)
         self.objclass           = set()
         self.attributes         = set()
         self._objects           = []
         self.__fetch_objectClasses__()
         self.__fetch_attributes__()
     def __fetch_objectClasses__(self):
         for obj in self.schema.tree(ldap.schema.ObjectClass).keys():
             obj = self.schema.get_obj(ldap.schema.ObjectClass, obj)
             if obj == None:
                 continue
             self.objclass.update(obj.names)
     def __fetch_attributes__(self):
         for attr in self.schema.tree(ldap.schema.AttributeType).keys():
             attr = self.schema.get_obj(ldap.schema.AttributeType, attr)
             if attr == None:
                 continue
             self.attributes.update(attr.names)
     def db2ldif(self, basedn=None, binddn=None, bindpw=None, 
fileName=None):
         try:
             if not isinstance(fileName, ldif.LDIFWriter):
                 if not isinstance(fileName, file):
                     fileName    = open(fileName, 'w')
                 ldifFile    = ldif.LDIFWriter(fileName)
         except IOError, e:
             return (False, u'unable to create LDIFFile %s' % fileName.name)
         try:
             self.srv    = ldap.initialize(self.uri)
             self.srv.start_tls_s()
         except ldap.CONNECT_ERROR:
             logger.info(u'unable to initialize start_tls, continue 
without encryption')
         except ldap.SERVER_DOWN:
             return (False, u'cannot connect to %s SERVER_DOWN' % self.uri)
         try:
             if not binddn == None:
                 if self.srv.simple_bind_s(binddn, bindpw) != (97, []):
                     return (False, u'couldn\'t authenticate as %s 
against %s' % (binddn, self.srv._uri))
         except ldap.INVALID_CREDENTIALS:
             return (False, u'couldn\'t authenticate as %s against %s' % 
(binddn, self.srv._uri))
         try:
             # for additional data change filterstr to
             # 
filterstr='(|(|(objectClass=*)(objectClass=ldapsubentry))(objectClass=nsTombstone))' 
by Rich Megginson <rmeggins at redhat.com>
             for dn, attrs in self.srv.search_s(basedn, 
ldap.SCOPE_SUBTREE, attrlist=list(self.attributes)):
                 self._objects.append((dn, attrs))
                 ldifFile.unparse(dn, attrs)
         except ldap.ADMINLIMIT_EXCEEDED:
             return (False, u'couldn\'t fetch all records as %s need 
more privileges or increase limits on the server' % binddn)
         except ldap.SIZELIMIT_EXCEEDED:
             return (False, u'couldn\'t fetch all records as %s need 
more privileges or increase limits on the server' % binddn)
         self.srv.unbind_s()
         return (True, True)

if __name__ == '__main__':
     import getpass
     lfd = LDIFDumper(uri='ldap://localhost/')
     lfd.db2ldif(basedn='dc=ctbto,dc=org', binddn='cn=Directory 
Manager', bindpw=getpass.getpass(), fileName='/tmp/dc=ctbto,dc=org.ldif')

</source>

head from /tmp/dc=ctbto,dc=org.ldif

dn: dc=ctbto,dc=org
aci: (targetattr != "userPassword") (version 3.0; acl "Anonymous access"; al
  low (read, search, compare)userdn = "ldap:///anyone";)
aci: (targetattr != "nsroledn||aci")(version 3.0; acl "Allow self entry modi
  fication except for nsroledn and aci attributes"; allow (write)userdn 
="lda
  p:///self";)
aci: (targetattr = "*")(version 3.0; acl "Configuration Adminstrator"; allow
   (all) userdn = 
"ldap:///uid=admin,ou=Administrators,ou=TopologyManagement,
  o=NetscapeRoot";)
aci: (targetattr ="*")(version 3.0;acl "Configuration Administrators Group";
  allow (all) (groupdn = "ldap:///cn=Configuration Administrators, 
ou=Groups,
   ou=TopologyManagement, o=NetscapeRoot");)
aci: (targetattr = "*")(version 3.0; acl "SIE Group"; allow (all)groupdn = "
  ldap:///dc=ctbto,dc=org";)
createTimestamp: 20130626125103Z
creatorsName: cn=directory manager
domaincomponent: ctbto
entrydn: dc=ctbto,dc=org
entryid: 1
hasSubordinates: FALSE
modifiersName: cn=directory manager
modifyTimestamp: 20130626125103Z
nsUniqueId: f13af0a6-de5e11e2-ab7e96d1-4e21dbc8
numSubordinates: 0
objectClass: top
objectClass: dcobject
subschemaSubentry: cn=schema


this should be sufficient in the sense of being able to restore the 
content of the Directory independent of the "original database 
distribution". Comparing the object attributes from the manually fetch 
one and the GUI fetched on there are on both objects 19 attributes. Of 
course it doesn't provide any configuration (cn=config) or replication 
settings but that's not the main goal of this exercise.

thanks for you support

regards
mIke

>>
>> thanks for your response
>>
>> regards
>> mIke
>>
>>
>>>
>>>>
>>>> thanks for any hint.
>>>> regards
>>>>
>>>> Mike
>>>>
>>>> -- 
>>>> 389 users mailing list
>>>> 389-users at lists.fedoraproject.org
>>>> https://admin.fedoraproject.org/mailman/listinfo/389-users
>>>
>>
>