[lib389] Deref control advice needed

[William Brown]

Hi,

In relation to ticket 47757, I have started work on a deref control for Noriko.
The idea is to get it working in lib389, then get it upstreamed into pyldap.

At this point it's all done, except that the actual request control doesn't
appear to work. Could one of the lib389 / ldap python experts cast their eye
over this and let me know where I've gone wrong?

For one, I don't think that the DerferenceControl decodeControlValue function is
ever called, as I'm never seeing the encodedControlValue printed in my logs.

Second, the results I get from result3 are:

[('cn=testgroup,dc=example,dc=com', {'objectClass': ['top', 'extensibleobject'],
'uniqueMember': ['uid=test,dc=example,dc=com'], 'cn': ['testgroup']})]
[]

Which again, doesn't seem correct, as there should be a result from the control.

Additionally, any tips on how to make the code nicer would be appreciated.

I've attached the complete patch, with a unit test to trigger the search, but
the request control looks like:

"""
     controlValue ::= SEQUENCE OF derefSpec DerefSpec

DerefSpec ::= SEQUENCE {
         derefAttr       attributeDescription,    ; with DN syntax
         attributes      AttributeList }

AttributeList ::= SEQUENCE OF attr AttributeDescription
    """
    class AttributeList(univ.SequenceOf):
        componentType = AttributeDescription()

class DerefSpec(univ.Sequence):
        componentType = namedtype.NamedTypes(
            namedtype.NamedType('derefAttr', AttributeDescription()),
            namedtype.NamedType('attributes', AttributeList()),
        )

class ControlValue(univ.SequenceOf):
        componentType = DerefSpec()

class DereferenceControl(LDAPControl):
        """
        Dereference Control
        """

def __init__(self, criticality, deref):
            LDAPControl.__init__(self, CONTROL_DEREF, criticality)
            self.deref = deref

def encodeControlValue(self):
            # How does -E ask for many values?
            derefAttr, attributes = self.deref.split(':')
            attributes = attributes.split(',')
            al = AttributeList()
            i = 0
            while len(attributes) > 0:
                al.setComponentByPosition(i, attributes.pop())
                i += 1
            ds = DerefSpec()
            ds.setComponentByName('derefAttr', derefAttr)
            ds.setComponentByName('attributes', al)
            cv = ControlValue()
            cv.setComponentByPosition(0, ds)
            print(cv.prettyPrint())
            return encoder.encode(cv)

def decodeControlValue(self,encodedControlValue):
            print(encodedControlValue)

[William Brown]

> In relation to ticket 47757, I have started work on a deref control for
> Noriko.
> The idea is to get it working in lib389, then get it upstreamed into pyldap.
>
> At this point it's all done, except that the actual request control doesn't
> appear to work. Could one of the lib389 / ldap python experts cast their eye
> over this and let me know where I've gone wrong?

I have improved this, but am having issues with the asn1spec for ber decoding.

I have attached the updated patch, but specifically the issue is in _controls.py

I would appreciate if anyone could take a look at this, and let me know if there
is something I have missed.

"""
 controlValue ::= SEQUENCE OF derefRes DerefRes

DerefRes ::= SEQUENCE {
     derefAttr       AttributeDescription,
     derefVal        LDAPDN,
     attrVals        [0] PartialAttributeList OPTIONAL }

PartialAttributeList ::= SEQUENCE OF
                partialAttribute PartialAttribute
"""

class DerefRes(univ.Sequence):
    componentType = namedtype.NamedTypes(
        namedtype.NamedType('derefAttr', AttributeDescription()),
        namedtype.NamedType('derefVal', LDAPDN()),
        namedtype.OptionalNamedType('attrVals', PartialAttributeList()),
    )

class DerefResultControlValue(univ.SequenceOf):
    componentType = DerefRes()

....

def decodeControlValue(self,encodedControlValue):
        self.entry = {}
        #decodedValue,_ =
decoder.decode(encodedControlValue,asn1Spec=DerefResultControlValue())
        # Gets the error: TagSet(Tag(tagClass=0, tagFormat=32, tagId=16),
Tag(tagClass=128, tagFormat=32, tagId=0)) not in asn1Spec:
{TagSet(Tag(tagClass=0, tagFormat=32, tagId=16)): PartialAttributeList()}/{}
        decodedValue,_ = decoder.decode(encodedControlValue)
        print(decodedValue.prettyPrint())
        # Pretty print yields
        #Sequence:  <-- Sequence of
        # <no-name>=Sequence:  <-- derefRes
        #  <no-name>=uniqueMember <-- derefAttr
        #  <no-name>=uid=test,dc=example,dc=com <-- derefVal
        #  <no-name>=Sequence: <-- attrVals
        #   <no-name>=uid
        #   <no-name>=Set:
        #    <no-name>=test
        # For now, while asn1spec is sad, we'll just rely on it being well
formed
        # However, this isn't good, as without the asn1spec, we seem to actually
be dropping values ....
        for result in decodedValue:
            derefAttr, derefVal, _ = result
            self.entry[str(derefAttr)] = str(derefVal)

[Rich Megginson]

On 08/26/2015 03:28 AM, William Brown wrote:
> > In relation to ticket 47757, I have started work on a deref control for
> > Noriko.
> > The idea is to get it working in lib389, then get it upstreamed into pyldap.
> >
> > At this point it's all done, except that the actual request control doesn't
> > appear to work. Could one of the lib389 / ldap python experts cast their eye
> > over this and let me know where I've gone wrong?
>
> I have improved this, but am having issues with the asn1spec for ber decoding.
>
> I have attached the updated patch, but specifically the issue is in _controls.py
>
> I would appreciate if anyone could take a look at this, and let me know if there
> is something I have missed.

Not sure, but here is some code I did without using pyasn:
https://github.com/richm/scripts/blob/master/derefctrl.py
This is quite old by now, and is probably bit rotted with respect to 
python-ldap and python3.
> """
>   controlValue ::= SEQUENCE OF derefRes DerefRes
>
> DerefRes ::= SEQUENCE {
>       derefAttr       AttributeDescription,
>       derefVal        LDAPDN,
>       attrVals        [0] PartialAttributeList OPTIONAL }
>
> PartialAttributeList ::= SEQUENCE OF
>                  partialAttribute PartialAttribute
> """
>
> class DerefRes(univ.Sequence):
>      componentType = namedtype.NamedTypes(
>          namedtype.NamedType('derefAttr', AttributeDescription()),
>          namedtype.NamedType('derefVal', LDAPDN()),
>          namedtype.OptionalNamedType('attrVals', PartialAttributeList()),
>      )
>
> class DerefResultControlValue(univ.SequenceOf):
>      componentType = DerefRes()
>
> ....
>  def decodeControlValue(self,encodedControlValue):
>      self.entry = {}
>      #decodedValue,_ =
>
> decoder.decode(encodedControlValue,asn1Spec=DerefResultControlValue())
>          # Gets the error: TagSet(Tag(tagClass=0, tagFormat=32, tagId=16),
> Tag(tagClass=128, tagFormat=32, tagId=0)) not in asn1Spec:
> {TagSet(Tag(tagClass=0, tagFormat=32, tagId=16)): PartialAttributeList()}/{}
>          decodedValue,_ = decoder.decode(encodedControlValue)
>          print(decodedValue.prettyPrint())
>          # Pretty print yields
>          #Sequence:  <-- Sequence of
>          # <no-name>=Sequence:  <-- derefRes
>          #  <no-name>=uniqueMember <-- derefAttr
>          #  <no-name>=uid=test,dc=example,dc=com <-- derefVal
>          #  <no-name>=Sequence: <-- attrVals
>          #   <no-name>=uid
>          #   <no-name>=Set:
>          #    <no-name>=test
>          # For now, while asn1spec is sad, we'll just rely on it being well
> formed
>          # However, this isn't good, as without the asn1spec, we seem to actually
> be dropping values ....
>          for result in decodedValue:
>              derefAttr, derefVal, _ = result
>              self.entry[str(derefAttr)] = str(derefVal)
>
> --
> 389-devel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-devel

[William Brown]

> > I would appreciate if anyone could take a look at this, and let me
> > know if there
> > is something I have missed.
>
> Not sure, but here is some code I did without using pyasn:
> https://github.com/richm/scripts/blob/master/derefctrl.py
> This is quite old by now, and is probably bit rotted with respect to 
> python-ldap and python3.

Yep. It appears to do exactly the same as my control, just that I'm
using pyasn. Your structures look really similar to what I have
defined, so that is a good start on my part.

Thanks for the interesting read, very interesting to see how you do the
hand encoding.

Sincerely,

[thierry bordaz]

On 08/27/2015 02:31 AM, Rich Megginson wrote:
> On 08/26/2015 03:28 AM, William Brown wrote:
> > > In relation to ticket 47757, I have started work on a deref control for
> > > Noriko.
> > > The idea is to get it working in lib389, then get it upstreamed into pyldap.
> > >
> > > At this point it's all done, except that the actual request control doesn't
> > > appear to work. Could one of the lib389 / ldap python experts cast their eye
> > > over this and let me know where I've gone wrong?
> >
> > I have improved this, but am having issues with the asn1spec for ber decoding.
> >
> > I have attached the updated patch, but specifically the issue is in _controls.py
> >
> > I would appreciate if anyone could take a look at this, and let me know if there
> > is something I have missed.
>
> Not sure, but here is some code I did without using pyasn:
> https://github.com/richm/scripts/blob/master/derefctrl.py
> This is quite old by now, and is probably bit rotted with respect to 
> python-ldap and python3.

Old !! but it worked like a charm for me. I just had to do this modif 
because of change in python-ldap IIRC

diff derefctrl.py /tmp/derefctrl_orig.py
    0a1
     >
    151,152c152
    <         self.criticality,self.derefspeclist,self.entry =
    criticality,derefspeclist or [],None
    <
    #LDAPControl.__init__(self,DerefCtrl.controlType,criticality,derefspeclist)
    ---
     >
    LDAPControl.__init__(self,DerefCtrl.controlType,criticality,derefspeclist)
    154c154
    <     def encodeControlValue(self):
    ---
     >     def encodeControlValue(self,value):
    156c156
    <         for (derefattr,attrs) in self.derefspeclist:
    ---
     >         for (derefattr,attrs) in value:
> > """
> >   controlValue ::= SEQUENCE OF derefRes DerefRes
> >
> > DerefRes ::= SEQUENCE {
> >       derefAttr       AttributeDescription,
> >       derefVal        LDAPDN,
> >       attrVals        [0] PartialAttributeList OPTIONAL }
> >
> > PartialAttributeList ::= SEQUENCE OF
> >                  partialAttribute PartialAttribute
> > """
> >
> > class DerefRes(univ.Sequence):
> >      componentType = namedtype.NamedTypes(
> >          namedtype.NamedType('derefAttr', AttributeDescription()),
> >          namedtype.NamedType('derefVal', LDAPDN()),
> >          namedtype.OptionalNamedType('attrVals', PartialAttributeList()),
> >      )
> >
> > class DerefResultControlValue(univ.SequenceOf):
> >      componentType = DerefRes()
> >
> > ....
> >  def decodeControlValue(self,encodedControlValue):
> >      self.entry = {}
> >      #decodedValue,_ =
> >
> > decoder.decode(encodedControlValue,asn1Spec=DerefResultControlValue())
> >          # Gets the error: TagSet(Tag(tagClass=0, tagFormat=32, tagId=16),
> > Tag(tagClass=128, tagFormat=32, tagId=0)) not in asn1Spec:
> > {TagSet(Tag(tagClass=0, tagFormat=32, tagId=16)): PartialAttributeList()}/{}
> >          decodedValue,_ = decoder.decode(encodedControlValue)
> >          print(decodedValue.prettyPrint())
> >          # Pretty print yields
> >          #Sequence:  <-- Sequence of
> >          # <no-name>=Sequence:  <-- derefRes
> >          #  <no-name>=uniqueMember <-- derefAttr
> >          #  <no-name>=uid=test,dc=example,dc=com <-- derefVal
> >          #  <no-name>=Sequence: <-- attrVals
> >          #   <no-name>=uid
> >          #   <no-name>=Set:
> >          #    <no-name>=test
> >          # For now, while asn1spec is sad, we'll just rely on it being well
> > formed
> >          # However, this isn't good, as without the asn1spec, we seem to actually
> > be dropping values ....
> >          for result in decodedValue:
> >              derefAttr, derefVal, _ = result
> >              self.entry[str(derefAttr)] = str(derefVal)
> >
> > --
> > 389-devel mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-devel
>
> --
> 389-devel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-devel

[Rich Megginson]

On 09/02/2015 10:35 AM, thierry bordaz wrote:
> On 08/27/2015 02:31 AM, Rich Megginson wrote:
> > On 08/26/2015 03:28 AM, William Brown wrote:
> > > > In relation to ticket 47757, I have started work on a deref control for
> > > > Noriko.
> > > > The idea is to get it working in lib389, then get it upstreamed into pyldap.
> > > >
> > > > At this point it's all done, except that the actual request control doesn't
> > > > appear to work. Could one of the lib389 / ldap python experts cast their eye
> > > > over this and let me know where I've gone wrong?
> > >
> > > I have improved this, but am having issues with the asn1spec for ber decoding.
> > >
> > > I have attached the updated patch, but specifically the issue is in _controls.py
> > >
> > > I would appreciate if anyone could take a look at this, and let me know if there
> > > is something I have missed.
> >
> > Not sure, but here is some code I did without using pyasn:
> > https://github.com/richm/scripts/blob/master/derefctrl.py
> > This is quite old by now, and is probably bit rotted with respect to 
> > python-ldap and python3.
>
> Old !! but it worked like a charm for me. I just had to do this modif 
> because of change in python-ldap IIRC

OK.  But I would rather use William's version which is based on pyasn1 - 
it hurts my brain to hand code BER . . .
> diff derefctrl.py /tmp/derefctrl_orig.py
> 0a1
> >
> 151,152c152
> <         self.criticality,self.derefspeclist,self.entry =
> criticality,derefspeclist or [],None
> <
> #LDAPControl.__init__(self,DerefCtrl.controlType,criticality,derefspeclist)
> ---
> >
> LDAPControl.__init__(self,DerefCtrl.controlType,criticality,derefspeclist)
> 154c154
> <     def encodeControlValue(self):
> ---
> >     def encodeControlValue(self,value):
> 156c156
> <         for (derefattr,attrs) in self.derefspeclist:
> ---
> >         for (derefattr,attrs) in value:
>
>
> > > """
> > >   controlValue ::= SEQUENCE OF derefRes DerefRes
> > >
> > > DerefRes ::= SEQUENCE {
> > >       derefAttr       AttributeDescription,
> > >       derefVal        LDAPDN,
> > >       attrVals        [0] PartialAttributeList OPTIONAL }
> > >
> > > PartialAttributeList ::= SEQUENCE OF
> > >                  partialAttribute PartialAttribute
> > > """
> > >
> > > class DerefRes(univ.Sequence):
> > >      componentType = namedtype.NamedTypes(
> > >          namedtype.NamedType('derefAttr', AttributeDescription()),
> > >          namedtype.NamedType('derefVal', LDAPDN()),
> > >          namedtype.OptionalNamedType('attrVals', PartialAttributeList()),
> > >      )
> > >
> > > class DerefResultControlValue(univ.SequenceOf):
> > >      componentType = DerefRes()
> > >
> > > ....
> > >  def decodeControlValue(self,encodedControlValue):
> > >      self.entry = {}
> > >      #decodedValue,_ =
> > >
> > > decoder.decode(encodedControlValue,asn1Spec=DerefResultControlValue())
> > >          # Gets the error: TagSet(Tag(tagClass=0, tagFormat=32, tagId=16),
> > > Tag(tagClass=128, tagFormat=32, tagId=0)) not in asn1Spec:
> > > {TagSet(Tag(tagClass=0, tagFormat=32, tagId=16)): PartialAttributeList()}/{}
> > >          decodedValue,_ = decoder.decode(encodedControlValue)
> > >          print(decodedValue.prettyPrint())
> > >          # Pretty print yields
> > >          #Sequence:  <-- Sequence of
> > >          # <no-name>=Sequence:  <-- derefRes
> > >          #  <no-name>=uniqueMember <-- derefAttr
> > >          #  <no-name>=uid=test,dc=example,dc=com <-- derefVal
> > >          #  <no-name>=Sequence: <-- attrVals
> > >          #   <no-name>=uid
> > >          #   <no-name>=Set:
> > >          #    <no-name>=test
> > >          # For now, while asn1spec is sad, we'll just rely on it being well
> > > formed
> > >          # However, this isn't good, as without the asn1spec, we seem to actually
> > > be dropping values ....
> > >          for result in decodedValue:
> > >              derefAttr, derefVal, _ = result
> > >              self.entry[str(derefAttr)] = str(derefVal)
> > >
> > > --
> > > 389-devel mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/389-devel
> >
> > --
> > 389-devel mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-devel

[William Brown]

> > Old !! but it worked like a charm for me. I just had to do this modif 
> > because of change in python-ldap IIRC
>
> OK.  But I would rather use William's version which is based on pyasn1 - 
> it hurts my brain to hand code BER . . .

https://fedorahosted.org/389/ticket/48262

Please take a look and review.