looking for the bitmap debug levels of ldap_library_debug_level....

[Spike White]

Sssd experts,

Our AD team is complaining that their AD controllers in our more
densely-populated datacenters are getting overloaded.  (up to 5K concurrent
LDAP connections per AD DC).

They find that 90% of the LDAP queries come from Linux servers -- which is
surprising as Linux servers are < 50% of the resident population.

We had previously found a coding bug in a vendor’s cron job that was
generating a lot of unnecessary LDAP queries.  That cybersecurity
monitoring software was on every Linux server, so it led to a thundering
herd problem when the cron job would kick off on every Linux server.

That thundering herd problem has been remediated, but we suspect something
similar occurring now.  We’d like to see what LDAP queries are being
executed by several random Linux servers, but we’ve had negative
experiences leaving sssd debug_level = 9 on for several days.  (Fills up
the /var/log filesystem after a few days).

I read in the sssd-ldap man page:

ldap_library_debug_level (integer)

Switches on libldap debugging with the given level. The libldap debug
messages will be written independent of the general debug_level.

OpenLDAP uses a bitmap to enable debugging for specific components, -1 will
enable full debug output.

Default: 0 (libldap debugging disabled)

This seems to be exactly what we want!  We want to see what LDAP queries
are being executed without all the other massive logging that arises from
debug_level 9.

However, I cannot find these bitmap values of debug levels for the openldap
library.  The most I can find is bitmapped debug values for a stand-alone
slapd daemon.

OpenLDAP debug level - Knowledge Base / Community - Univention Help
https://help.univention.com/t/openldap-debug-level/19301

Where may I find the documentation of the bitmapped debug values that can
be set via ldap_library_debug_level?

Spike White

[Alexey Tikhonov]

Hi,

I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
```
Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER,
LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE,  LDAP_DEBUG_PACKETS,  LDAP_DEBUG_PARSE,
 and  LDAP_DEBUG_TRACE
```

For values see
https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...

On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users <
[e-mail]> wrote:
> Sssd experts,
>
> Our AD team is complaining that their AD controllers in our more
> densely-populated datacenters are getting overloaded.  (up to 5K concurrent
> LDAP connections per AD DC).
>
> They find that 90% of the LDAP queries come from Linux servers -- which is
> surprising as Linux servers are < 50% of the resident population.
>
> We had previously found a coding bug in a vendor’s cron job that was
> generating a lot of unnecessary LDAP queries.  That cybersecurity
> monitoring software was on every Linux server, so it led to a thundering
> herd problem when the cron job would kick off on every Linux server.
>
> That thundering herd problem has been remediated, but we suspect something
> similar occurring now.  We’d like to see what LDAP queries are being
> executed by several random Linux servers, but we’ve had negative
> experiences leaving sssd debug_level = 9 on for several days.  (Fills up
> the /var/log filesystem after a few days).
>
> I read in the sssd-ldap man page:
>    ldap_library_debug_level (integer)
>
> Switches on libldap debugging with the given level. The libldap debug
> messages will be written independent of the general debug_level.
>
> OpenLDAP uses a bitmap to enable debugging for specific components, -1
> will enable full debug output.
>        Default: 0 (libldap debugging disabled)
>
>
>
> This seems to be exactly what we want!  We want to see what LDAP queries
> are being executed without all the other massive logging that arises from
> debug_level 9.
>
> However, I cannot find these bitmap values of debug levels for the
> openldap library.  The most I can find is bitmapped debug values for a
> stand-alone slapd daemon.
>
> OpenLDAP debug level - Knowledge Base / Community - Univention Help
> https://help.univention.com/t/openldap-debug-level/19301
>
> Where may I find the documentation of the bitmapped debug values that can
> be set via ldap_library_debug_level?
>
> Spike White
>
> --
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue

[Spike White]

Alexey,

Thank you.  That seems right.  But the openldap client lib content doesn't
seem like it's being written to the /var/log/sssd/*.log files.

I'm able to turn on debug_level = 5  in the domain section of the sssd.conf
file and see the fully-qualified user names requested.  But  even setting

ldap_library_debug_level = -1

seems to yield no additional LDAP content in /etc/sssd/*.log files.

Is it because my id_provider and auth_provider is ad and not ldap?  Or does
the openldap client lib write its output to another location?

Spike

On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov 
wrote:
> Hi,
>
> I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
> Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER,
> LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE,  LDAP_DEBUG_PACKETS,  LDAP_DEBUG_PARSE,
>  and  LDAP_DEBUG_TRACE
>
> For values see
> https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
>
> On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users <
> [e-mail]> wrote:
> > Sssd experts,
> >
> > Our AD team is complaining that their AD controllers in our more
> > densely-populated datacenters are getting overloaded.  (up to 5K concurrent
> > LDAP connections per AD DC).
> >
> > They find that 90% of the LDAP queries come from Linux servers -- which
> > is surprising as Linux servers are < 50% of the resident population.
> >
> > We had previously found a coding bug in a vendor’s cron job that was
> > generating a lot of unnecessary LDAP queries.  That cybersecurity
> > monitoring software was on every Linux server, so it led to a thundering
> > herd problem when the cron job would kick off on every Linux server.
> >
> > That thundering herd problem has been remediated, but we suspect
> > something similar occurring now.  We’d like to see what LDAP queries are
> > being executed by several random Linux servers, but we’ve had negative
> > experiences leaving sssd debug_level = 9 on for several days.  (Fills up
> > the /var/log filesystem after a few days).
> >
> > I read in the sssd-ldap man page:
> >    ldap_library_debug_level (integer)
> >
> > Switches on libldap debugging with the given level. The libldap debug
> > messages will be written independent of the general debug_level.
> >
> > OpenLDAP uses a bitmap to enable debugging for specific components, -1
> > will enable full debug output.
> >        Default: 0 (libldap debugging disabled)
> >
> >
> >
> > This seems to be exactly what we want!  We want to see what LDAP queries
> > are being executed without all the other massive logging that arises from
> > debug_level 9.
> >
> > However, I cannot find these bitmap values of debug levels for the
> > openldap library.  The most I can find is bitmapped debug values for a
> > stand-alone slapd daemon.
> >
> > OpenLDAP debug level - Knowledge Base / Community - Univention Help
> > https://help.univention.com/t/openldap-debug-level/19301
> >
> > Where may I find the documentation of the bitmapped debug values that can
> > be set via ldap_library_debug_level?
> >
> > Spike White
> >
> > --
> > _______________________________________________
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue

[Sumit Bose]

Am Thu, Jan 16, 2025 at 01:17:56PM -0600 schrieb Spike White via sssd-users:
> Alexey,
>
> Thank you.  That seems right.  But the openldap client lib content doesn't
> seem like it's being written to the /var/log/sssd/*.log files.
>
> I'm able to turn on debug_level = 5  in the domain section of the sssd.conf
> file and see the fully-qualified user names requested.  But  even setting
>
> ldap_library_debug_level = -1
>
> seems to yield no additional LDAP content in /etc/sssd/*.log files.
>
> Is it because my id_provider and auth_provider is ad and not ldap?  Or does
> the openldap client lib write its output to another location?

Hi,

no, unfortunately the man page (or the code) is wrong,
https://github.com/SSSD/sssd/blob/master/src/providers/ldap/sdap.c#L840
currently you need `debug_level = 9` to get the libldap debug logs.

I think it would be useful to have the behavior as described in the man
page, are you interested to provide a patch?

bye,
Sumit
> Spike
>
> On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov 
> wrote:
> > Hi,
> >
> > I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
> > Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER,
> > LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE,  LDAP_DEBUG_PACKETS,  LDAP_DEBUG_PARSE,
> >  and  LDAP_DEBUG_TRACE
> >
> > For values see
> > https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
> >
> > On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users <
> > [e-mail]> wrote:
> > > Sssd experts,
> > >
> > > Our AD team is complaining that their AD controllers in our more
> > > densely-populated datacenters are getting overloaded.  (up to 5K concurrent
> > > LDAP connections per AD DC).
> > >
> > > They find that 90% of the LDAP queries come from Linux servers -- which
> > > is surprising as Linux servers are < 50% of the resident population.
> > >
> > > We had previously found a coding bug in a vendor’s cron job that was
> > > generating a lot of unnecessary LDAP queries.  That cybersecurity
> > > monitoring software was on every Linux server, so it led to a thundering
> > > herd problem when the cron job would kick off on every Linux server.
> > >
> > > That thundering herd problem has been remediated, but we suspect
> > > something similar occurring now.  We’d like to see what LDAP queries are
> > > being executed by several random Linux servers, but we’ve had negative
> > > experiences leaving sssd debug_level = 9 on for several days.  (Fills up
> > > the /var/log filesystem after a few days).
> > >
> > > I read in the sssd-ldap man page:
> > >    ldap_library_debug_level (integer)
> > >
> > > Switches on libldap debugging with the given level. The libldap debug
> > > messages will be written independent of the general debug_level.
> > >
> > > OpenLDAP uses a bitmap to enable debugging for specific components, -1
> > > will enable full debug output.
> > >        Default: 0 (libldap debugging disabled)
> > >
> > >
> > >
> > > This seems to be exactly what we want!  We want to see what LDAP queries
> > > are being executed without all the other massive logging that arises from
> > > debug_level 9.
> > >
> > > However, I cannot find these bitmap values of debug levels for the
> > > openldap library.  The most I can find is bitmapped debug values for a
> > > stand-alone slapd daemon.
> > >
> > > OpenLDAP debug level - Knowledge Base / Community - Univention Help
> > > https://help.univention.com/t/openldap-debug-level/19301
> > >
> > > Where may I find the documentation of the bitmapped debug values that can
> > > be set via ldap_library_debug_level?
> > >
> > > Spike White
> > >
> > > --
> > > _______________________________________________
> > > sssd-users mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > > Do not reply to spam, report it:
> > > https://pagure.io/fedora-infrastructure/new_issue
> -- 
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Lukas Slebodnik]

On (17/01/25 09:03), Sumit Bose via sssd-users wrote:
> Am Thu, Jan 16, 2025 at 01:17:56PM -0600 schrieb Spike White via sssd-users:
> > Alexey,
> >
> > Thank you.  That seems right.  But the openldap client lib content doesn't
> > seem like it's being written to the /var/log/sssd/*.log files.
> >
> > I'm able to turn on debug_level = 5  in the domain section of the sssd.conf
> > file and see the fully-qualified user names requested.  But  even setting
> >
> > ldap_library_debug_level = -1
> >
> > seems to yield no additional LDAP content in /etc/sssd/*.log files.
> >
> > Is it because my id_provider and auth_provider is ad and not ldap?  Or does
> > the openldap client lib write its output to another location?
>
> Hi,
>
> no, unfortunately the man page (or the code) is wrong,
> https://github.com/SSSD/sssd/blob/master/src/providers/ldap/sdap.c#L840
> currently you need `debug_level = 9` to get the libldap debug logs.

`debug_level = 0x4000` should provide less logs than `debug_level = 9` :-)
https://github.com/SSSD/sssd/blob/85784e7614da54aeaed2e8a757b5bd3c9523125d/s...

LS
> I think it would be useful to have the behavior as described in the man
> page, are you interested to provide a patch?
>
> bye,
> Sumit
> > Spike
> >
> > On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov 
> > wrote:
> > > Hi,
> > >
> > > I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
> > > Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER,
> > > LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE,  LDAP_DEBUG_PACKETS,  LDAP_DEBUG_PARSE,
> > >  and  LDAP_DEBUG_TRACE
> > >
> > > For values see
> > > https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
> > >
> > > On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users <
> > > [e-mail]> wrote:
> > > > Sssd experts,
> > > >
> > > > Our AD team is complaining that their AD controllers in our more
> > > > densely-populated datacenters are getting overloaded.  (up to 5K concurrent
> > > > LDAP connections per AD DC).
> > > >
> > > > They find that 90% of the LDAP queries come from Linux servers -- which
> > > > is surprising as Linux servers are < 50% of the resident population.
> > > >
> > > > We had previously found a coding bug in a vendor’s cron job that was
> > > > generating a lot of unnecessary LDAP queries.  That cybersecurity
> > > > monitoring software was on every Linux server, so it led to a thundering
> > > > herd problem when the cron job would kick off on every Linux server.
> > > >
> > > > That thundering herd problem has been remediated, but we suspect
> > > > something similar occurring now.  We’d like to see what LDAP queries are
> > > > being executed by several random Linux servers, but we’ve had negative
> > > > experiences leaving sssd debug_level = 9 on for several days.  (Fills up
> > > > the /var/log filesystem after a few days).
> > > >
> > > > I read in the sssd-ldap man page:
> > > >    ldap_library_debug_level (integer)
> > > >
> > > > Switches on libldap debugging with the given level. The libldap debug
> > > > messages will be written independent of the general debug_level.
> > > >
> > > > OpenLDAP uses a bitmap to enable debugging for specific components, -1
> > > > will enable full debug output.
> > > >        Default: 0 (libldap debugging disabled)
> > > >
> > > >
> > > >
> > > > This seems to be exactly what we want!  We want to see what LDAP queries
> > > > are being executed without all the other massive logging that arises from
> > > > debug_level 9.
> > > >
> > > > However, I cannot find these bitmap values of debug levels for the
> > > > openldap library.  The most I can find is bitmapped debug values for a
> > > > stand-alone slapd daemon.
> > > >
> > > > OpenLDAP debug level - Knowledge Base / Community - Univention Help
> > > > https://help.univention.com/t/openldap-debug-level/19301
> > > >
> > > > Where may I find the documentation of the bitmapped debug values that can
> > > > be set via ldap_library_debug_level?
> > > >
> > > > Spike White
> > > >
> > > > --
> > > > _______________________________________________
> > > > sssd-users mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct:
> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
> > > > https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > > > Do not reply to spam, report it:
> > > > https://pagure.io/fedora-infrastructure/new_issue
> > -- 
> > _______________________________________________
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>
> -- 
> _______________________________________________
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue