Some bind DNs sporadically can't search users

[Morgan Jones]

We're pulling our hair out over this issue and wondering if it rings a bell for anyone or perhaps there's a bug fix in a  later version of 389 that might resolve it.  I've looked and not found anything but it's also not the easiest issue to search for.  We are on CentOS DS now but can move to 389 of course but hesitate to put forth the effort if we aren't reasonably sure it will resolve our issue.

Certain bind DNs (so far only service accounts) are unable to search for users.  For instance:

[20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 BIND dn="uid=copier,ou=employees,dc=domain,dc=org" method=128 version=3
[20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=copier,ou=employees,dc=domain,dc=org"
[20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 SRCH base="ou=employees,dc=domain,dc=org" scope=2 filter="(uid=user1)" attrs="cn mail uid"
[20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 RESULT err=0 tag=101 nentries=0 etime=0

I did a search of the directory at the same time on the same replica as myself and other users with the same access level and was able to search uid=user1.

It sometimes only lasts a few minutes though in a few instances it was much longer.  In one case we resolved it by adding a service account with a different name but the same access level and moved the application to that account.  In another case where hundreds of copiers are configured with the binddn and thus changing the binddn wasn't practical we were able to resolve it by restoring the masters from a db2ldif and re-initializing the consumers. It's not isolated to once consumer--we've seen it on at least 2: one partial, one full.

Our environment is CentOS DS on CentOS 5.  We have ~35,000 entries in the directory, most of them users.  We only have 9 acis, most of which provide access via groups.  We have 2 multi-masters, 2 full consumers and until recently 2 partial replicas.  We converted the partial replicas to full replicas recently as we know there are/were issues with partial replication and wanted to rule that out.  We're at the latest version of CentOS DS:

[morgan@ldap01 ~]$ rpm -qa|grep centos-ds
centos-ds-admin-8.2.1-1.el5.centos
centos-ds-console-8.2.0-4.el5.centos
centos-ds-base-8.2.8-2.el5.centos
centos-ds-8.2.0-2.el5.centos
[morgan@ldap01 ~]$

This issue is killing us as it effectively denies access to a service for all users that use it.  This could be the VPN for instance which many users depend on all day for their work.

We are considering many options, primarily just moving to 389 on CentOS 6.  That's a fairly big task for a site of our size so we'd like some confidence that we won't see a repeat of this issue.  There's talk of considering other products which no one wants to do but we're all edgy as this has caused a fair amount of downtime and about all I can do at the moment is monitor for it and re-initialize when it happens which doesn't inspire confidence.

thank you,

-morgan

[Ludwig Krispenz]

Hi,

so you say that a search with a specific bind user sometimes succeeds 
and sometimes doesn't ?
If so, could you run this withe aci logging enabled ?
Are groups involved in the acis and do these groups during these runs ?
Could you post your acis ?

Ludwig

On 03/03/2014 04:56 PM, Morgan Jones wrote:
> We're pulling our hair out over this issue and wondering if it rings a bell for anyone or perhaps there's a bug fix in a  later version of 389 that might resolve it.  I've looked and not found anything but it's also not the easiest issue to search for.  We are on CentOS DS now but can move to 389 of course but hesitate to put forth the effort if we aren't reasonably sure it will resolve our issue.
>
> Certain bind DNs (so far only service accounts) are unable to search for users.  For instance:
>
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 BIND dn="uid=copier,ou=employees,dc=domain,dc=org" method=128 version=3
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=copier,ou=employees,dc=domain,dc=org"
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 SRCH base="ou=employees,dc=domain,dc=org" scope=2 filter="(uid=user1)" attrs="cn mail uid"
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 RESULT err=0 tag=101 nentries=0 etime=0
>
> I did a search of the directory at the same time on the same replica as myself and other users with the same access level and was able to search uid=user1.
>
> It sometimes only lasts a few minutes though in a few instances it was much longer.  In one case we resolved it by adding a service account with a different name but the same access level and moved the application to that account.  In another case where hundreds of copiers are configured with the binddn and thus changing the binddn wasn't practical we were able to resolve it by restoring the masters from a db2ldif and re-initializing the consumers. It's not isolated to once consumer--we've seen it on at least 2: one partial, one full.
>
> Our environment is CentOS DS on CentOS 5.  We have ~35,000 entries in the directory, most of them users.  We only have 9 acis, most of which provide access via groups.  We have 2 multi-masters, 2 full consumers and until recently 2 partial replicas.  We converted the partial replicas to full replicas recently as we know there are/were issues with partial replication and wanted to rule that out.  We're at the latest version of CentOS DS:
>
> [morgan@ldap01 ~]$ rpm -qa|grep centos-ds
> centos-ds-admin-8.2.1-1.el5.centos
> centos-ds-console-8.2.0-4.el5.centos
> centos-ds-base-8.2.8-2.el5.centos
> centos-ds-8.2.0-2.el5.centos
> [morgan@ldap01 ~]$
>
> This issue is killing us as it effectively denies access to a service for all users that use it.  This could be the VPN for instance which many users depend on all day for their work.
>
> We are considering many options, primarily just moving to 389 on CentOS 6.  That's a fairly big task for a site of our size so we'd like some confidence that we won't see a repeat of this issue.  There's talk of considering other products which no one wants to do but we're all edgy as this has caused a fair amount of downtime and about all I can do at the moment is monitor for it and re-initialize when it happens which doesn't inspire confidence.
>
> thank you,
> -morgan
>
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Morgan Jones]

On Mar 3, 2014, at 11:07 AM, Ludwig Krispenz  wrote:
> Hi,
>
> so you say that a search with a specific bind user sometimes succeeds and sometimes doesn't ?

Correct.
> If so, could you run this withe aci logging enabled ?

Sure though we are still unable to repeat the problem reliably so haven't caught it happening with aci debugging turned on.  I'm going to work on various scenarios to do so in our dev environment.
> Are groups involved in the acis and do these groups during these runs ?

Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.  I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.
> Could you post your acis ?

Probably.  I'm working on permission to do so.

thanks,

-morgan

[Ludwig Krispenz]

On 03/03/2014 08:08 PM, Morgan Jones wrote:
> On Mar 3, 2014, at 11:07 AM, Ludwig Krispenz  wrote:
> > Hi,
> >
> > so you say that a search with a specific bind user sometimes succeeds and sometimes doesn't ?
>
> Correct.
> > If so, could you run this withe aci logging enabled ?
>
> Sure though we are still unable to repeat the problem reliably so haven't caught it happening with aci debugging turned on.  I'm going to work on various scenarios to do so in our dev environment.
> > Are groups involved in the acis and do these groups during these runs ?
>
> Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.

you can't, it was incomplete. I wanted to know if these groups are 
modified during the runs when you see the failure.
> I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.
> > Could you post your acis ?
>
> Probably.  I'm working on permission to do so.
>
> thanks,
>
> -morgan
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Morgan Jones]

On Mar 4, 2014, at 3:20 AM, Ludwig Krispenz  wrote:
> > > Are groups involved in the acis and do these groups during these runs ?
> >
> > Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.
>
> you can't, it was incomplete. I wanted to know if these groups are modified during the runs when you see the failure.
> > I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.

No, groups were not modified.  They are relatively small as we're still migrating to this environment--maybe 10-15 DNs per group and they're only modified when we add/remove privileged accounts which isn't very often.
> > > Could you post your acis ?
> >
> > Probably.  I'm working on permission to do so.

The compromise I came to with my management and security team is to obfuscate the ACIs such that the attribute counts and structure are intact but the names are changed.  Is the below useful?

# Employee LDAP Access Control
#
dn: dc=domain,dc=org
changetype: modify
replace: aci
#
aci: (target = "ldap:///ou=employees,dc=domain,dc=org")
 (targetattr = "userpassword")
 (version 3.0; acl "limited user self write";
 allow (write) userdn = "ldap:///self";)
#
aci: (target = "ldap:///dc=domain,dc=org" )
 (targetfilter = "(|(objectclass=orgAssociate)(objectclass=orgEmployee)(objectclass=domain)
 (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
 (targetattr = "attr1 || attr2 || ... || attr40")
 (version 3.0; acl "general access, replaces anonymous access"; 
 allow (read, search, compare) 
 (userdn = "ldap:///self") or 
 (groupdn = "ldap:///cn=orgGroup1,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=orgGroup3,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or 
 (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
 ;)
#
aci: (target = "ldap:///dc=domain,dc=org" )
 (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
 (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
 (targetattr = "attr1 || attr2 || ... || attr40 ")
 (version 3.0; acl "general access, replaces anonymous access"; 
 allow (read, search, compare) 
 (userdn = "ldap:///self") or 
 (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or 
 (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
 ;)
#
aci: (target = "ldap:///dc=domain,dc=org")
 (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
 (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=orgServiceAccount)(objectclass=orgOrgAccount))")
 (targetattr = "attr1 || attr2 || ... || attr40")
 (version 3.0; acl "general access plus service and organizational accounts"; 
 allow (read, search, compare) 
 (userdn = "ldap:///self") or 
 (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
 (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or 
 (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
 ;)
#
aci: (target = "ldap:///dc=domain,dc=org")(targetattr = "attr1 ||
 attr2 || ... || attr30")
 (version 3.0; acl "limited read access to non-public attributes for delegated admins"; 
 allow (read, search, compare)
 (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or 
 (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
 ;)
#
aci: (target = "ldap:///dc=domain,dc=org")
 (targetattr = "attr1 || attr2 || ... || attr28")
 (version 3.0; acl "limited write access for delegated admins"; 
 allow (write) groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org";)
#
aci: (target = "ldap:///dc=domain,dc=org")
 (targetattr = "*")(version 3.0; acl "full access for delegated admins"; 
 allow (all) groupdn = "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org";)
#
aci: (target = "ldap:///dc=domain,dc=org")
 (targetfilter="(memberof=cn=orgGroup6,ou=Groups,dc=domain,dc=org)")
 (targetattr="userpassword")
 (version 3.0; acl "deny non-admin user write access to admin users' passwords";
 deny (all) groupdn != "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org"
 ;)
#
aci: (target = "ldap:///dc=domain,dc=org")
 (targetattr = "attr1 || attr2 || ... || attr19")
 (version 3.0; acl "access to posixaccount attributes for proxyagent";
 allow (read,search,compare) userdn = "ldap:///uid=binddn1,ou=svc_accts,dc=domain,dc=org";)

thanks,

-morgan

[Ludwig Krispenz]

On 03/04/2014 11:10 PM, Morgan Jones wrote:
> On Mar 4, 2014, at 3:20 AM, Ludwig Krispenz  wrote:
> > > > Are groups involved in the acis and do these groups during these runs ?
> > >
> > > Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.
> >
> > you can't, it was incomplete. I wanted to know if these groups are modified during the runs when you see the failure.
> > > I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.
>
> No, groups were not modified.  They are relatively small as we're still migrating to this environment--maybe 10-15 DNs per group and they're only modified when we add/remove privileged accounts which isn't very often.
> > > > Could you post your acis ?
> > >
> > > Probably.  I'm working on permission to do so.
>
> The compromise I came to with my management and security team is to obfuscate the ACIs such that the attribute counts and structure are intact but the names are changed.  Is the below useful?

yes, but II can't see anything wrong with the acis.

One more question. Do the searches always match only one entry or one 
they should see and some they shouldn't ?
> # Employee LDAP Access Control
> #
> dn: dc=domain,dc=org
> changetype: modify
> replace: aci
> #
> aci: (target = "ldap:///ou=employees,dc=domain,dc=org")
>   (targetattr = "userpassword")
>   (version 3.0; acl "limited user self write";
>   allow (write) userdn = "ldap:///self";)
> #
> aci: (target = "ldap:///dc=domain,dc=org" )
>   (targetfilter = "(|(objectclass=orgAssociate)(objectclass=orgEmployee)(objectclass=domain)
>   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
>   (targetattr = "attr1 || attr2 || ... || attr40")
>   (version 3.0; acl "general access, replaces anonymous access";
>   allow (read, search, compare)
>   (userdn = "ldap:///self") or
>   (groupdn = "ldap:///cn=orgGroup1,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup3,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
>   ;)
> #
> aci: (target = "ldap:///dc=domain,dc=org" )
>   (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
>   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
>   (targetattr = "attr1 || attr2 || ... || attr40 ")
>   (version 3.0; acl "general access, replaces anonymous access";
>   allow (read, search, compare)
>   (userdn = "ldap:///self") or
>   (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
>   ;)
> #
> aci: (target = "ldap:///dc=domain,dc=org")
>   (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
>   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=orgServiceAccount)(objectclass=orgOrgAccount))")
>   (targetattr = "attr1 || attr2 || ... || attr40")
>   (version 3.0; acl "general access plus service and organizational accounts";
>   allow (read, search, compare)
>   (userdn = "ldap:///self") or
>   (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
>   ;)
> #
> aci: (target = "ldap:///dc=domain,dc=org")(targetattr = "attr1 ||
>   attr2 || ... || attr30")
>   (version 3.0; acl "limited read access to non-public attributes for delegated admins";
>   allow (read, search, compare)
>   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
>   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
>   ;)
> #
> aci: (target = "ldap:///dc=domain,dc=org")
>   (targetattr = "attr1 || attr2 || ... || attr28")
>   (version 3.0; acl "limited write access for delegated admins";
>   allow (write) groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org";)
> #
> aci: (target = "ldap:///dc=domain,dc=org")
>   (targetattr = "*")(version 3.0; acl "full access for delegated admins";
>   allow (all) groupdn = "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org";)
> #
> aci: (target = "ldap:///dc=domain,dc=org")
>   (targetfilter="(memberof=cn=orgGroup6,ou=Groups,dc=domain,dc=org)")
>   (targetattr="userpassword")
>   (version 3.0; acl "deny non-admin user write access to admin users' passwords";
>   deny (all) groupdn != "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org"
>   ;)
> #
> aci: (target = "ldap:///dc=domain,dc=org")
>   (targetattr = "attr1 || attr2 || ... || attr19")
>   (version 3.0; acl "access to posixaccount attributes for proxyagent";
>   allow (read,search,compare) userdn = "ldap:///uid=binddn1,ou=svc_accts,dc=domain,dc=org";)
>
> thanks,
>
> -morgan
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Morgan Jones]

On Mar 6, 2014, at 11:32 AM, Ludwig Krispenz  wrote:
> On 03/04/2014 11:10 PM, Morgan Jones wrote:
> > On Mar 4, 2014, at 3:20 AM, Ludwig Krispenz  wrote:
> > > > > Are groups involved in the acis and do these groups during these runs ?
> > > >
> > > > Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.
> > >
> > > you can't, it was incomplete. I wanted to know if these groups are modified during the runs when you see the failure.
> > > > I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.
> >
> > No, groups were not modified.  They are relatively small as we're still migrating to this environment--maybe 10-15 DNs per group and they're only modified when we add/remove privileged accounts which isn't very often.
> > > > > Could you post your acis ?
> > > >
> > > > Probably.  I'm working on permission to do so.
> >
> > The compromise I came to with my management and security team is to obfuscate the ACIs such that the attribute counts and structure are intact but the names are changed.  Is the below useful?
>
> yes, but II can't see anything wrong with the acis.

Thanks for your input on the ACIs.
> One more question. Do the searches always match only one entry or one they should see and some they shouldn't ?

In every case where we've seen this problem it's a search for one entry (uid=username) that the bind dn is able to see.

Thanks for your input, we're working on repeating it reliably in 389.
> > # Employee LDAP Access Control
> > #
> > dn: dc=domain,dc=org
> > changetype: modify
> > replace: aci
> > #
> > aci: (target = "ldap:///ou=employees,dc=domain,dc=org")
> >  (targetattr = "userpassword")
> >  (version 3.0; acl "limited user self write";
> >  allow (write) userdn = "ldap:///self";)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org" )
> >  (targetfilter = "(|(objectclass=orgAssociate)(objectclass=orgEmployee)(objectclass=domain)
> >  (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
> >  (targetattr = "attr1 || attr2 || ... || attr40")
> >  (version 3.0; acl "general access, replaces anonymous access";
> >  allow (read, search, compare)
> >  (userdn = "ldap:///self") or
> >  (groupdn = "ldap:///cn=orgGroup1,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup3,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> >  ;)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org" )
> >  (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
> >  (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
> >  (targetattr = "attr1 || attr2 || ... || attr40 ")
> >  (version 3.0; acl "general access, replaces anonymous access";
> >  allow (read, search, compare)
> >  (userdn = "ldap:///self") or
> >  (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> >  ;)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")
> >  (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
> >  (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=orgServiceAccount)(objectclass=orgOrgAccount))")
> >  (targetattr = "attr1 || attr2 || ... || attr40")
> >  (version 3.0; acl "general access plus service and organizational accounts";
> >  allow (read, search, compare)
> >  (userdn = "ldap:///self") or
> >  (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> >  ;)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")(targetattr = "attr1 ||
> >  attr2 || ... || attr30")
> >  (version 3.0; acl "limited read access to non-public attributes for delegated admins";
> >  allow (read, search, compare)
> >  (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> >  (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> >  ;)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")
> >  (targetattr = "attr1 || attr2 || ... || attr28")
> >  (version 3.0; acl "limited write access for delegated admins";
> >  allow (write) groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org";)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")
> >  (targetattr = "*")(version 3.0; acl "full access for delegated admins";
> >  allow (all) groupdn = "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org";)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")
> >  (targetfilter="(memberof=cn=orgGroup6,ou=Groups,dc=domain,dc=org)")
> >  (targetattr="userpassword")
> >  (version 3.0; acl "deny non-admin user write access to admin users' passwords";
> >  deny (all) groupdn != "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org"
> >  ;)
> > #
> > aci: (target = "ldap:///dc=domain,dc=org")
> >  (targetattr = "attr1 || attr2 || ... || attr19")
> >  (version 3.0; acl "access to posixaccount attributes for proxyagent";
> >  allow (read,search,compare) userdn = "ldap:///uid=binddn1,ou=svc_accts,dc=domain,dc=org";)
> >
> > thanks,
> >
> > -morgan
> >
> > --
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Ludwig Krispenz]

On 03/06/2014 05:42 PM, Morgan Jones wrote:
> On Mar 6, 2014, at 11:32 AM, Ludwig Krispenz  wrote:
> > On 03/04/2014 11:10 PM, Morgan Jones wrote:
> > > On Mar 4, 2014, at 3:20 AM, Ludwig Krispenz  wrote:
> > > > > > Are groups involved in the acis and do these groups during these runs ?
> > > > >
> > > > > Yes, most of our ACIs use groups to determine access.  I'm not sure I understand the second part of your question though.
> > > >
> > > > you can't, it was incomplete. I wanted to know if these groups are modified during the runs when you see the failure.
> > > > > I do suspect this has something to do with access control though as it's behaving exactly like the user is denied by the ACIs.
> > >
> > > No, groups were not modified.  They are relatively small as we're still migrating to this environment--maybe 10-15 DNs per group and they're only modified when we add/remove privileged accounts which isn't very often.
> > > > > > Could you post your acis ?
> > > > >
> > > > > Probably.  I'm working on permission to do so.
> > >
> > > The compromise I came to with my management and security team is to obfuscate the ACIs such that the attribute counts and structure are intact but the names are changed.  Is the below useful?
> >
> > yes, but II can't see anything wrong with the acis.
>
> Thanks for your input on the ACIs.
> > One more question. Do the searches always match only one entry or one they should see and some they shouldn't ?
>
> In every case where we've seen this problem it's a search for one entry (uid=username) that the bind dn is able to see.

what i was thinking of is a scenario where there is a cn=user1 in two 
subtrees, the bound user should only see one. I remember a case where 
the deny for the one entry was cached and the other entry was not returned
> Thanks for your input, we're working on repeating it reliably in 389.

That would be great
> > > # Employee LDAP Access Control
> > > #
> > > dn: dc=domain,dc=org
> > > changetype: modify
> > > replace: aci
> > > #
> > > aci: (target = "ldap:///ou=employees,dc=domain,dc=org")
> > >   (targetattr = "userpassword")
> > >   (version 3.0; acl "limited user self write";
> > >   allow (write) userdn = "ldap:///self";)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org" )
> > >   (targetfilter = "(|(objectclass=orgAssociate)(objectclass=orgEmployee)(objectclass=domain)
> > >   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
> > >   (targetattr = "attr1 || attr2 || ... || attr40")
> > >   (version 3.0; acl "general access, replaces anonymous access";
> > >   allow (read, search, compare)
> > >   (userdn = "ldap:///self") or
> > >   (groupdn = "ldap:///cn=orgGroup1,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup3,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> > >   ;)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org" )
> > >   (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
> > >   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=groupofuniquenames))")
> > >   (targetattr = "attr1 || attr2 || ... || attr40 ")
> > >   (version 3.0; acl "general access, replaces anonymous access";
> > >   allow (read, search, compare)
> > >   (userdn = "ldap:///self") or
> > >   (groupdn = "ldap:///cn=orgGroup2,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> > >   ;)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")
> > >   (targetfilter = "(|(objectclass=orgExternalEmployee)(objectclass=domain)
> > >   (objectclass=organizationalunit)(objectclass=groupofnames)(objectclass=orgServiceAccount)(objectclass=orgOrgAccount))")
> > >   (targetattr = "attr1 || attr2 || ... || attr40")
> > >   (version 3.0; acl "general access plus service and organizational accounts";
> > >   allow (read, search, compare)
> > >   (userdn = "ldap:///self") or
> > >   (groupdn = "ldap:///cn=OrgGroup3,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> > >   ;)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")(targetattr = "attr1 ||
> > >   attr2 || ... || attr30")
> > >   (version 3.0; acl "limited read access to non-public attributes for delegated admins";
> > >   allow (read, search, compare)
> > >   (groupdn = "ldap:///cn=orgGroup4,ou=groups,dc=domain,dc=org") or
> > >   (groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org")
> > >   ;)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")
> > >   (targetattr = "attr1 || attr2 || ... || attr28")
> > >   (version 3.0; acl "limited write access for delegated admins";
> > >   allow (write) groupdn = "ldap:///cn=orgGroup5,ou=groups,dc=domain,dc=org";)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")
> > >   (targetattr = "*")(version 3.0; acl "full access for delegated admins";
> > >   allow (all) groupdn = "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org";)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")
> > >   (targetfilter="(memberof=cn=orgGroup6,ou=Groups,dc=domain,dc=org)")
> > >   (targetattr="userpassword")
> > >   (version 3.0; acl "deny non-admin user write access to admin users' passwords";
> > >   deny (all) groupdn != "ldap:///cn=orgGroup6,ou=groups,dc=domain,dc=org"
> > >   ;)
> > > #
> > > aci: (target = "ldap:///dc=domain,dc=org")
> > >   (targetattr = "attr1 || attr2 || ... || attr19")
> > >   (version 3.0; acl "access to posixaccount attributes for proxyagent";
> > >   allow (read,search,compare) userdn = "ldap:///uid=binddn1,ou=svc_accts,dc=domain,dc=org";)
> > >
> > > thanks,
> > >
> > > -morgan
> > >
> > > --
> > > 389 users mailing list
> > > [e-mail]
> > > https://admin.fedoraproject.org/mailman/listinfo/389-users
> >
> > --
> > 389 users mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/389-users
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Morgan Jones]

On Mar 6, 2014, at 11:51 AM, Ludwig Krispenz  wrote:
> > > One more question. Do the searches always match only one entry or one they should see and some they shouldn't ?
> >
> > In every case where we've seen this problem it's a search for one entry (uid=username) that the bind dn is able to see.
>
> what i was thinking of is a scenario where there is a cn=user1 in two subtrees, the bound user should only see one. I remember a case where the deny for the one entry was cached and the other entry was not returned

Oh, interesting.  That is not the case for us though.
> > Thanks for your input, we're working on repeating it reliably in 389.
>
> That would be great

I'll see what I can do.

thanks,

-morgan

[Rich Megginson]

On 03/03/2014 08:56 AM, Morgan Jones wrote:
> We're pulling our hair out over this issue and wondering if it rings a bell for anyone or perhaps there's a bug fix in a  later version of 389 that might resolve it.  I've looked and not found anything but it's also not the easiest issue to search for.  We are on CentOS DS now but can move to 389 of course but hesitate to put forth the effort if we aren't reasonably sure it will resolve our issue.
>
> Certain bind DNs (so far only service accounts) are unable to search for users.  For instance:
>
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 BIND dn="uid=copier,ou=employees,dc=domain,dc=org" method=128 version=3
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=1 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=copier,ou=employees,dc=domain,dc=org"
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 SRCH base="ou=employees,dc=domain,dc=org" scope=2 filter="(uid=user1)" attrs="cn mail uid"
> [20/Feb/2014:10:18:05 -0500] conn=15600927 op=2 RESULT err=0 tag=101 nentries=0 etime=0
>
> I did a search of the directory at the same time on the same replica as myself and other users with the same access level and was able to search uid=user1.
>
> It sometimes only lasts a few minutes though in a few instances it was much longer.  In one case we resolved it by adding a service account with a different name but the same access level and moved the application to that account.  In another case where hundreds of copiers are configured with the binddn and thus changing the binddn wasn't practical we were able to resolve it by restoring the masters from a db2ldif and re-initializing the consumers. It's not isolated to once consumer--we've seen it on at least 2: one partial, one full.
>
> Our environment is CentOS DS on CentOS 5.  We have ~35,000 entries in the directory, most of them users.  We only have 9 acis, most of which provide access via groups.  We have 2 multi-masters, 2 full consumers and until recently 2 partial replicas.  We converted the partial replicas to full replicas recently as we know there are/were issues with partial replication and wanted to rule that out.  We're at the latest version of CentOS DS:
>
> [morgan@ldap01 ~]$ rpm -qa|grep centos-ds
> centos-ds-admin-8.2.1-1.el5.centos
> centos-ds-console-8.2.0-4.el5.centos
> centos-ds-base-8.2.8-2.el5.centos
> centos-ds-8.2.0-2.el5.centos
> [morgan@ldap01 ~]$
>
> This issue is killing us as it effectively denies access to a service for all users that use it.  This could be the VPN for instance which many users depend on all day for their work.
>
> We are considering many options, primarily just moving to 389 on CentOS 6.  That's a fairly big task for a site of our size so we'd like some confidence that we won't see a repeat of this issue.  There's talk of considering other products which no one wants to do but we're all edgy as this has caused a fair amount of downtime and about all I can do at the moment is monitor for it and re-initialize when it happens which doesn't inspire confidence.

I've never seen this problem, so I can't be sure that upgrading will 
solve it.  However, it is almost impossible for us to support centos-ds 
8.2, so we are unlikely to be able to debug 8.2 and provide a patch for 8.2.

If you can reproduce the problem with 389-ds-base 1.2.11.x, it will be 
much easier for us to debug and fix.
> thank you,
> -morgan
>
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users

[Morgan Jones]

On Mar 3, 2014, at 11:24 AM, Rich Megginson  wrote:
> On 03/03/2014 08:56 AM, Morgan Jones wrote:
> > We're pulling our hair out over this issue and wondering if it rings a bell for anyone or perhaps there's a bug fix in a  later version of 389 that might resolve it.  I've looked and not found anything but it's also not the easiest issue to search for.  We are on CentOS DS now but can move to 389 of course but hesitate to put forth the effort if we aren't reasonably sure it will resolve our issue.

...
> > We are considering many options, primarily just moving to 389 on CentOS 6.  That's a fairly big task for a site of our size so we'd like some confidence that we won't see a repeat of this issue.  There's talk of considering other products which no one wants to do but we're all edgy as this has caused a fair amount of downtime and about all I can do at the moment is monitor for it and re-initialize when it happens which doesn't inspire confidence.
>
> I've never seen this problem, so I can't be sure that upgrading will solve it.  However, it is almost impossible for us to support centos-ds 8.2, so we are unlikely to be able to debug 8.2 and provide a patch for 8.2.

This is understandable.  Now that I have a better idea of how the community works I would not implement on CentOS-DS again.
> If you can reproduce the problem with 389-ds-base 1.2.11.x, it will be much easier for us to debug and fix.

We are still unable to repeat it reliably but are working to do so in our dev environment.  I'll be in touch when/if we are able to do so in 389 1.2.11.x.

thanks,

-morgan

[Rich Megginson]

On 03/03/2014 12:05 PM, Morgan Jones wrote:
> On Mar 3, 2014, at 11:24 AM, Rich Megginson  wrote:
> > On 03/03/2014 08:56 AM, Morgan Jones wrote:
> > > We're pulling our hair out over this issue and wondering if it rings a bell for anyone or perhaps there's a bug fix in a  later version of 389 that might resolve it.  I've looked and not found anything but it's also not the easiest issue to search for.  We are on CentOS DS now but can move to 389 of course but hesitate to put forth the effort if we aren't reasonably sure it will resolve our issue.
>
> ...
> > > We are considering many options, primarily just moving to 389 on CentOS 6.  That's a fairly big task for a site of our size so we'd like some confidence that we won't see a repeat of this issue.  There's talk of considering other products which no one wants to do but we're all edgy as this has caused a fair amount of downtime and about all I can do at the moment is monitor for it and re-initialize when it happens which doesn't inspire confidence.
> >
> > I've never seen this problem, so I can't be sure that upgrading will solve it.  However, it is almost impossible for us to support centos-ds 8.2, so we are unlikely to be able to debug 8.2 and provide a patch for 8.2.
>
> This is understandable.  Now that I have a better idea of how the community works I would not implement on CentOS-DS again.

CentOS is downstream of RHEL.  RHEL is downstream of 389.  Even if a 389 
developer is able to reproduce on CentOS, we have to first get the patch 
into upstream 389, then into RHEL, then there has to be a RHEL release, 
then someone in CentOS has to pick up that build and respin for CentOS.
> > If you can reproduce the problem with 389-ds-base 1.2.11.x, it will be much easier for us to debug and fix.
>
> We are still unable to repeat it reliably but are working to do so in our dev environment.  I'll be in touch when/if we are able to do so in 389 1.2.11.x.
>
> thanks,
>
> -morgan
>
> --
> 389 users mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/389-users