Re: FileDescriptors exhausted

[Howard Chu]

>  Date: Fri, 11 Nov 2022 14:25:57 +0000
>  From: Tobias Ernstberger <tobias.ernstberger(a)de.ibm.com&gt; 
>  Hello,
>  
>  we're observing the following error message:
>  "ERR - accept_and_configure - PR_Accept() failed, Netscape Portable Runtime error
> -5971 (Process open FD table is full.)"
>  Looks like the file descriptors are exhausted, probably mainly used by incoming TCP
> Connections (based on our investigation regarding open FDs). 
>  We've set (and checked using the runtime information in /proc/PID/limits) the ulimits
> and the nsslapd-maxdescriptors to many thousands (while having about 1000 open connection
> regularly)
>  
>  We are investigating in multiple directions here, and have some questions - any input is
> appreciated: 
Have you checked with e.g. lsof to see where all the FDs are being used? If you think
there's only 1000
incoming connections, probably there are other things going on e.g. DNS reverse lookups on
client
addresses, miscellaneous other files being opened, etc.
>  
>  1) We acknowledge that exhausted FDs prevent additional connections to be opened. But we
> also see, that existing connections are getting unusable, too. Is this a known behaviour?
> Can this be avoided?
>  2) Is there any chance to limit the number of open connections (lower than the max FDs)?
> (trying to achieve that existing connections still work)
>  3) What are best practice to prevent the ldap server from getting completely useless
> (until restart) if a client opens many connections? 
>  4) Any additional remarks to prevent this situation? 
Fyi, OpenLDAP has no issue with multiple thousands of FDs being served concurrently...
>  
>  
>  Kind regards
>  
>  Tobias Ernstberger
>  IBM Security
>  
>  IBM Deutschland GmbH
>  Vorsitzender des Aufsichtsrats: Sebastian Krause
>  Geschäftsführung: Gregor Pillen (Vorsitzender), Nicole Reimer, Gabriele Schwarenthorer,
> Christine Rupp, Frank Theisen 
>  Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 /
> WEEE-Reg.-Nr. DE 99369940
>  https://www.ibm.com/privacy/us/en/ 
-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

FileDescriptors exhausted

[Paul Robert Marino]

It sounds like you may have stale client connections. There are a lot
of bad LDAP clients that never close their connections properly.
the way to work around this is to set a TCP keepalive to a shorter
than default. here is a good article on the subject
https://www.looklinux.com/how-to-configure-tcp-keepalive-setting-in-linux/
i recommend that heartbeats be initiated after 60 seconds after the
connection becomes idle and only a few failures before they are
dropped.
Also it's important to know all 3 settings must be set for this to
work. If you don't set all 3 related settings or if any of them are
set to 0 it will not do anything.

an other thing is the open file limits they must be set in two places
first is explained here
https://access.redhat.com/solutions/1257953#:~:text=service.,descriptors%....
The other is in the dse.ldif file
https://docs.kolab.org/administrator-guide/tweaking-389ds.html#:~:text=A%....

On Fri, Nov 11, 2022 at 11:37 PM Howard Chu <hyc(a)symas.com&gt; wrote:
>
> > Date: Fri, 11 Nov 2022 14:25:57 +0000
> > From: Tobias Ernstberger <tobias.ernstberger(a)de.ibm.com&gt;
>
> > Hello,
> >
> > we're observing the following error message:
> > "ERR - accept_and_configure - PR_Accept() failed, Netscape Portable Runtime
error -5971 (Process open FD table is full.)"
> > Looks like the file descriptors are exhausted, probably mainly used by incoming
TCP Connections (based on our investigation regarding open FDs).
> > We've set (and checked using the runtime information in /proc/PID/limits)
the ulimits and the nsslapd-maxdescriptors to many thousands (while having about 1000 open
connection regularly)
> >
> > We are investigating in multiple directions here, and have some questions - any
input is appreciated:
>
> Have you checked with e.g. lsof to see where all the FDs are being used? If you think
there's only 1000
> incoming connections, probably there are other things going on e.g. DNS reverse
lookups on client
> addresses, miscellaneous other files being opened, etc.
> >
> > 1) We acknowledge that exhausted FDs prevent additional connections to be
opened. But we also see, that existing connections are getting unusable, too. Is this a
known behaviour? Can this be avoided?
> > 2) Is there any chance to limit the number of open connections (lower than the
max FDs)? (trying to achieve that existing connections still work)
> > 3) What are best practice to prevent the ldap server from getting completely
useless (until restart) if a client opens many connections?
> > 4) Any additional remarks to prevent this situation?
>
> Fyi, OpenLDAP has no issue with multiple thousands of FDs being served
concurrently...
> >
> >
> > Kind regards
> >
> > Tobias Ernstberger
> > IBM Security
> >
> > IBM Deutschland GmbH
> > Vorsitzender des Aufsichtsrats: Sebastian Krause
> > Geschäftsführung: Gregor Pillen (Vorsitzender), Nicole Reimer, Gabriele
Schwarenthorer, Christine Rupp, Frank Theisen
> > Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB
14562 / WEEE-Reg.-Nr. DE 99369940
> > https://www.ibm.com/privacy/us/en/
>
> --
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/
> _______________________________________________
> 389-users mailing list -- 389-users(a)lists.fedoraproject.org
> To unsubscribe send an email to 389-users-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue