Sssd experts,
Our AD team is complaining that their AD controllers in our more densely-populated datacenters are getting overloaded. (up to 5K concurrent LDAP connections per AD DC).
They find that 90% of the LDAP queries come from Linux servers -- which is surprising as Linux servers are < 50% of the resident population.
We had previously found a coding bug in a vendor’s cron job that was generating a lot of unnecessary LDAP queries. That cybersecurity monitoring software was on every Linux server, so it led to a thundering herd problem when the cron job would kick off on every Linux server.
That thundering herd problem has been remediated, but we suspect something similar occurring now. We’d like to see what LDAP queries are being executed by several random Linux servers, but we’ve had negative experiences leaving sssd debug_level = 9 on for several days. (Fills up the /var/log filesystem after a few days).
I read in the sssd-ldap man page:
ldap_library_debug_level (integer)
Switches on libldap debugging with the given level. The libldap debug messages will be written independent of the general debug_level.
OpenLDAP uses a bitmap to enable debugging for specific components, -1 will enable full debug output.
Default: 0 (libldap debugging disabled)
This seems to be exactly what we want! We want to see what LDAP queries are being executed without all the other massive logging that arises from debug_level 9.
However, I cannot find these bitmap values of debug levels for the openldap library. The most I can find is bitmapped debug values for a stand-alone slapd daemon.
OpenLDAP debug level - Knowledge Base / Community - Univention Help https://help.univention.com/t/openldap-debug-level/19301
Where may I find the documentation of the bitmapped debug values that can be set via ldap_library_debug_level?
Spike White
Hi,
I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`: ``` Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER, LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE, LDAP_DEBUG_PACKETS, LDAP_DEBUG_PARSE, and LDAP_DEBUG_TRACE ```
For values see https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users < sssd-users@lists.fedorahosted.org> wrote:
Sssd experts,
Our AD team is complaining that their AD controllers in our more densely-populated datacenters are getting overloaded. (up to 5K concurrent LDAP connections per AD DC).
They find that 90% of the LDAP queries come from Linux servers -- which is surprising as Linux servers are < 50% of the resident population.
We had previously found a coding bug in a vendor’s cron job that was generating a lot of unnecessary LDAP queries. That cybersecurity monitoring software was on every Linux server, so it led to a thundering herd problem when the cron job would kick off on every Linux server.
That thundering herd problem has been remediated, but we suspect something similar occurring now. We’d like to see what LDAP queries are being executed by several random Linux servers, but we’ve had negative experiences leaving sssd debug_level = 9 on for several days. (Fills up the /var/log filesystem after a few days).
I read in the sssd-ldap man page:
ldap_library_debug_level (integer)Switches on libldap debugging with the given level. The libldap debug messages will be written independent of the general debug_level.
OpenLDAP uses a bitmap to enable debugging for specific components, -1 will enable full debug output.
Default: 0 (libldap debugging disabled)This seems to be exactly what we want! We want to see what LDAP queries are being executed without all the other massive logging that arises from debug_level 9.
However, I cannot find these bitmap values of debug levels for the openldap library. The most I can find is bitmapped debug values for a stand-alone slapd daemon.
OpenLDAP debug level - Knowledge Base / Community - Univention Help https://help.univention.com/t/openldap-debug-level/19301
Where may I find the documentation of the bitmapped debug values that can be set via ldap_library_debug_level?
Spike White
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Alexey,
Thank you. That seems right. But the openldap client lib content doesn't seem like it's being written to the /var/log/sssd/*.log files.
I'm able to turn on debug_level = 5 in the domain section of the sssd.conf file and see the fully-qualified user names requested. But even setting
ldap_library_debug_level = -1
seems to yield no additional LDAP content in /etc/sssd/*.log files.
Is it because my id_provider and auth_provider is ad and not ldap? Or does the openldap client lib write its output to another location?
Spike
On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov atikhono@redhat.com wrote:
Hi,
I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER, LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE, LDAP_DEBUG_PACKETS, LDAP_DEBUG_PARSE, and LDAP_DEBUG_TRACEFor values see https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users < sssd-users@lists.fedorahosted.org> wrote:
Sssd experts,
Our AD team is complaining that their AD controllers in our more densely-populated datacenters are getting overloaded. (up to 5K concurrent LDAP connections per AD DC).
They find that 90% of the LDAP queries come from Linux servers -- which is surprising as Linux servers are < 50% of the resident population.
We had previously found a coding bug in a vendor’s cron job that was generating a lot of unnecessary LDAP queries. That cybersecurity monitoring software was on every Linux server, so it led to a thundering herd problem when the cron job would kick off on every Linux server.
That thundering herd problem has been remediated, but we suspect something similar occurring now. We’d like to see what LDAP queries are being executed by several random Linux servers, but we’ve had negative experiences leaving sssd debug_level = 9 on for several days. (Fills up the /var/log filesystem after a few days).
I read in the sssd-ldap man page:
ldap_library_debug_level (integer)Switches on libldap debugging with the given level. The libldap debug messages will be written independent of the general debug_level.
OpenLDAP uses a bitmap to enable debugging for specific components, -1 will enable full debug output.
Default: 0 (libldap debugging disabled)This seems to be exactly what we want! We want to see what LDAP queries are being executed without all the other massive logging that arises from debug_level 9.
However, I cannot find these bitmap values of debug levels for the openldap library. The most I can find is bitmapped debug values for a stand-alone slapd daemon.
OpenLDAP debug level - Knowledge Base / Community - Univention Help https://help.univention.com/t/openldap-debug-level/19301
Where may I find the documentation of the bitmapped debug values that can be set via ldap_library_debug_level?
Spike White
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Am Thu, Jan 16, 2025 at 01:17:56PM -0600 schrieb Spike White via sssd-users:
Alexey,
Thank you. That seems right. But the openldap client lib content doesn't seem like it's being written to the /var/log/sssd/*.log files.
I'm able to turn on debug_level = 5 in the domain section of the sssd.conf file and see the fully-qualified user names requested. But even setting
ldap_library_debug_level = -1
seems to yield no additional LDAP content in /etc/sssd/*.log files.
Is it because my id_provider and auth_provider is ad and not ldap? Or does the openldap client lib write its output to another location?
Hi,
no, unfortunately the man page (or the code) is wrong, https://github.com/SSSD/sssd/blob/master/src/providers/ldap/sdap.c#L840 currently you need `debug_level = 9` to get the libldap debug logs.
I think it would be useful to have the behavior as described in the man page, are you interested to provide a patch?
bye, Sumit
Spike
On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov atikhono@redhat.com wrote:
Hi,
I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER, LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE, LDAP_DEBUG_PACKETS, LDAP_DEBUG_PARSE, and LDAP_DEBUG_TRACEFor values see https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users < sssd-users@lists.fedorahosted.org> wrote:
Sssd experts,
Our AD team is complaining that their AD controllers in our more densely-populated datacenters are getting overloaded. (up to 5K concurrent LDAP connections per AD DC).
They find that 90% of the LDAP queries come from Linux servers -- which is surprising as Linux servers are < 50% of the resident population.
We had previously found a coding bug in a vendor’s cron job that was generating a lot of unnecessary LDAP queries. That cybersecurity monitoring software was on every Linux server, so it led to a thundering herd problem when the cron job would kick off on every Linux server.
That thundering herd problem has been remediated, but we suspect something similar occurring now. We’d like to see what LDAP queries are being executed by several random Linux servers, but we’ve had negative experiences leaving sssd debug_level = 9 on for several days. (Fills up the /var/log filesystem after a few days).
I read in the sssd-ldap man page:
ldap_library_debug_level (integer)Switches on libldap debugging with the given level. The libldap debug messages will be written independent of the general debug_level.
OpenLDAP uses a bitmap to enable debugging for specific components, -1 will enable full debug output.
Default: 0 (libldap debugging disabled)This seems to be exactly what we want! We want to see what LDAP queries are being executed without all the other massive logging that arises from debug_level 9.
However, I cannot find these bitmap values of debug levels for the openldap library. The most I can find is bitmapped debug values for a stand-alone slapd daemon.
OpenLDAP debug level - Knowledge Base / Community - Univention Help https://help.univention.com/t/openldap-debug-level/19301
Where may I find the documentation of the bitmapped debug values that can be set via ldap_library_debug_level?
Spike White
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On (17/01/25 09:03), Sumit Bose via sssd-users wrote:
Am Thu, Jan 16, 2025 at 01:17:56PM -0600 schrieb Spike White via sssd-users:
Alexey,
Thank you. That seems right. But the openldap client lib content doesn't seem like it's being written to the /var/log/sssd/*.log files.
I'm able to turn on debug_level = 5 in the domain section of the sssd.conf file and see the fully-qualified user names requested. But even setting
ldap_library_debug_level = -1
seems to yield no additional LDAP content in /etc/sssd/*.log files.
Is it because my id_provider and auth_provider is ad and not ldap? Or does the openldap client lib write its output to another location?
Hi,
no, unfortunately the man page (or the code) is wrong, https://github.com/SSSD/sssd/blob/master/src/providers/ldap/sdap.c#L840 currently you need `debug_level = 9` to get the libldap debug logs.
`debug_level = 0x4000` should provide less logs than `debug_level = 9` :-) https://github.com/SSSD/sssd/blob/85784e7614da54aeaed2e8a757b5bd3c9523125d/s...
LS
I think it would be useful to have the behavior as described in the man page, are you interested to provide a patch?
bye, Sumit
Spike
On Thu, Jan 16, 2025 at 11:45 AM Alexey Tikhonov atikhono@redhat.com wrote:
Hi,
I think this is `man ldap_set_option :: LDAP_OPT_DEBUG_LEVEL`:
Valid debug levels are LDAP_DEBUG_ANY, LDAP_DEBUG_ARGS, LDAP_DEBUG_BER, LDAP_DEBUG_CONNS, LDAP_DEBUG_NONE, LDAP_DEBUG_PACKETS, LDAP_DEBUG_PARSE, and LDAP_DEBUG_TRACEFor values see https://git.openldap.org/openldap/openldap/-/blob/master/include/ldap_log.h?...
On Thu, Jan 16, 2025 at 5:09 PM Spike White via sssd-users < sssd-users@lists.fedorahosted.org> wrote:
Sssd experts,
Our AD team is complaining that their AD controllers in our more densely-populated datacenters are getting overloaded. (up to 5K concurrent LDAP connections per AD DC).
They find that 90% of the LDAP queries come from Linux servers -- which is surprising as Linux servers are < 50% of the resident population.
We had previously found a coding bug in a vendor’s cron job that was generating a lot of unnecessary LDAP queries. That cybersecurity monitoring software was on every Linux server, so it led to a thundering herd problem when the cron job would kick off on every Linux server.
That thundering herd problem has been remediated, but we suspect something similar occurring now. We’d like to see what LDAP queries are being executed by several random Linux servers, but we’ve had negative experiences leaving sssd debug_level = 9 on for several days. (Fills up the /var/log filesystem after a few days).
I read in the sssd-ldap man page:
ldap_library_debug_level (integer)Switches on libldap debugging with the given level. The libldap debug messages will be written independent of the general debug_level.
OpenLDAP uses a bitmap to enable debugging for specific components, -1 will enable full debug output.
Default: 0 (libldap debugging disabled)This seems to be exactly what we want! We want to see what LDAP queries are being executed without all the other massive logging that arises from debug_level 9.
However, I cannot find these bitmap values of debug levels for the openldap library. The most I can find is bitmapped debug values for a stand-alone slapd daemon.
OpenLDAP debug level - Knowledge Base / Community - Univention Help https://help.univention.com/t/openldap-debug-level/19301
Where may I find the documentation of the bitmapped debug values that can be set via ldap_library_debug_level?
Spike White
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ sssd-users mailing list -- sssd-users@lists.fedorahosted.org To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.o... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
sssd-users@lists.fedorahosted.org