How to tune the failover mecanism ?

[Olivier]

Hello everyone,

I use sssd to authenticate users on redhat boxes,
as a layer between pam and my ldap serverS.

User ldap accounts are announced but 3 replicated
(open)ldap servers.

I have declared my 3 servers in sssd.conf:

ldap_uri: ldap://ldap1.example.fr,ldap://ldap2.example.fr,ldap://ldap3.example.fr

That works, but I would like now to better tune the
sssd failover and cache function offered by sssd,
and I'm a bit stuck.

Questions :

1- about the failover mecanism, I read in sssd-ldap man page :

" If the resolution attempt succeeds, the back end tries to connect to
a service on this machine.
  If the service connection attempt fails, then only this particular
service is considered offline and
  the back end automatically switches over to the next service."

I there any way to tune the "timeout" after which the bakend considers
that the attempt
to connect to a server has failed and therefore that it's time to try
the next one ?

2- I'm a bit stuck to configure the behaviour of the boxes with regard to
    sssd cache (in anthother word, I have not perfectly understood under
    which condition sssd returns cached information rather than querying
    ldap and how to tune that).

Intuitively, I would something like the sssd cache for an entry to be returned
rather than querying ldap if :

-> the cache was "recently" refreshed for that entry ( not sur what I
should mean
    by "recently" )
-> no ldap server respond (I suspect that this would be tuned with
"ldap_search_timeout" ?)

Any help from anyone ?

Thanks,

---
Olivier

[Jakub Hrozek]

On Mon, Dec 03, 2012 at 04:35:00PM +0100, Olivier wrote:
>  Hello everyone,
>  
>  I use sssd to authenticate users on redhat boxes,
>  as a layer between pam and my ldap serverS.
>  
>  User ldap accounts are announced but 3 replicated
>  (open)ldap servers.
>  
>  I have declared my 3 servers in sssd.conf:
>  
>  ldap_uri: ldap://ldap1.example.fr,ldap://ldap2.example.fr,ldap://ldap3.example.fr
>  
>  That works, but I would like now to better tune the
>  sssd failover and cache function offered by sssd,
>  and I'm a bit stuck.
>  
>  Questions :
>  
>  1- about the failover mecanism, I read in sssd-ldap man page :
>  
>  " If the resolution attempt succeeds, the back end tries to connect to
>  a service on this machine.
>    If the service connection attempt fails, then only this particular
>  service is considered offline and
>    the back end automatically switches over to the next service."
>  
>  I there any way to tune the "timeout" after which the bakend considers
>  that the attempt
>  to connect to a server has failed and therefore that it's time to try
>  the next one ? 
It depends, there are several timeouts at play. There is the DNS
resolution timeout (dns_resolver_timeout) and there may be per-provider
timeouts such as ldap_network_timeout.

What particular problem you are seeing?

>  
>  2- I'm a bit stuck to configure the behaviour of the boxes with regard to
>      sssd cache (in anthother word, I have not perfectly understood under
>      which condition sssd returns cached information rather than querying
>      ldap and how to tune that).
>  
>  Intuitively, I would something like the sssd cache for an entry to be returned
>  rather than querying ldap if :
>  
>  -> the cache was "recently" refreshed for that entry ( not sur what I
>  should mean
>      by "recently" )
>  -> no ldap server respond (I suspect that this would be tuned with
>  "ldap_search_timeout" ?) 
That's pretty how the cache mechanism works.

In general, the entries are returned from on-disk cache for the time
specified in entry_cache_timeout or per-object entry timeouts (for
example entry_cache_user_timeout for users, entry_cache_group_timeout
for groups etc.).

If the cached entry has been saved more than entry_cache_timeout seconds
ago, the NSS responder queries LDAP and saves the entry again.

If the entry was present in the cache and was supposed to be refreshed
BUT no LDAP server is reachable, then the SSSD falls back to the expired
cache entry.

Additionally, there is a new cache layer introduced in 1.9 that is
entirely kept in-memory and is pretty fast. That memory cache is
controlled by the memcache_timeout option.

[Dmitri Pal]

On 12/03/2012 10:35 AM, Olivier wrote:
>  Hello everyone,
>
>  I use sssd to authenticate users on redhat boxes,
>  as a layer between pam and my ldap serverS.
>
>  User ldap accounts are announced but 3 replicated
>  (open)ldap servers.
>
>  I have declared my 3 servers in sssd.conf:
>
>  ldap_uri: ldap://ldap1.example.fr,ldap://ldap2.example.fr,ldap://ldap3.example.fr
>
>  That works, but I would like now to better tune the
>  sssd failover and cache function offered by sssd,
>  and I'm a bit stuck.
>
>  Questions :
>
>  1- about the failover mecanism, I read in sssd-ldap man page :
>
>  " If the resolution attempt succeeds, the back end tries to connect to
>  a service on this machine.
>    If the service connection attempt fails, then only this particular
>  service is considered offline and
>    the back end automatically switches over to the next service."
>
>  I there any way to tune the "timeout" after which the bakend considers
>  that the attempt
>  to connect to a server has failed and therefore that it's time to try
>  the next one ? 
I think the re-connection timeout is hardcoded as 30, I vaguely recall a
discussion about it four years ago.
If/when it is confirmed IMO would make sense to file an RFE.
I am leaning to it be configurable  and think that 5 or 10 second
timeout should be sufficient.


>  2- I'm a bit stuck to configure the behaviour of the boxes with regard to
>      sssd cache (in anthother word, I have not perfectly understood under
>      which condition sssd returns cached information rather than querying
>      ldap and how to tune that).
>
>  Intuitively, I would something like the sssd cache for an entry to be returned
>  rather than querying ldap if :
>
>  -> the cache was "recently" refreshed for that entry ( not sur what I
>  should mean
>      by "recently" )
>  -> no ldap server respond (I suspect that this would be tuned with
>  "ldap_search_timeout" ?)
>
>  Any help from anyone ? 
If the system is offline the data is served from the cache.
If the system is online but the entry is not available or the entry is
pretty old (defined by the cache expiration timeout) then the data is
fetched from LDAP and returned to the caller in parallel to updating cache.
If the system is online and entry is available from the cache and it is
younger than half of its life time then it is served from the cache.
If the system is online and entry is available from the cache and it is
older than half of its life time then it is served from the cache but an
async operation to fetch the entry from LDAP and update the cache is
launched in the background.

HTH


>  Thanks,
>
>  ---
>  Olivier
>  _______________________________________________
>  sssd-devel mailing list
>  sssd-devel(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-devel 

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Dmitri Pal]

On 12/03/2012 10:56 AM, Dmitri Pal wrote:
>  On 12/03/2012 10:35 AM, Olivier wrote:
> > Hello everyone,
> >
> > I use sssd to authenticate users on redhat boxes,
> > as a layer between pam and my ldap serverS.
> >
> > User ldap accounts are announced but 3 replicated
> > (open)ldap servers.
> >
> > I have declared my 3 servers in sssd.conf:
> >
> > ldap_uri: ldap://ldap1.example.fr,ldap://ldap2.example.fr,ldap://ldap3.example.fr
> >
> > That works, but I would like now to better tune the
> > sssd failover and cache function offered by sssd,
> > and I'm a bit stuck.
> >
> > Questions :
> >
> > 1- about the failover mecanism, I read in sssd-ldap man page :
> >
> > " If the resolution attempt succeeds, the back end tries to connect to
> > a service on this machine.
> >   If the service connection attempt fails, then only this particular
> > service is considered offline and
> >   the back end automatically switches over to the next service."
> >
> > I there any way to tune the "timeout" after which the bakend considers
> > that the attempt
> > to connect to a server has failed and therefore that it's time to try
> > the next one ?
>  I think the re-connection timeout is hardcoded as 30, I vaguely recall a
>  discussion about it four years ago.
>  If/when it is confirmed IMO would make sense to file an RFE.
>  I am leaning to it be configurable  and think that 5 or 10 second
>  timeout should be sufficient.
>
>
> > 2- I'm a bit stuck to configure the behaviour of the boxes with regard to
> >     sssd cache (in anthother word, I have not perfectly understood under
> >     which condition sssd returns cached information rather than querying
> >     ldap and how to tune that).
> >
> > Intuitively, I would something like the sssd cache for an entry to be returned
> > rather than querying ldap if :
> >
> > -> the cache was "recently" refreshed for that entry ( not sur what I
> > should mean
> >     by "recently" )
> > -> no ldap server respond (I suspect that this would be tuned with
> > "ldap_search_timeout" ?)
> >
> > Any help from anyone ?
>  If the system is offline the data is served from the cache.
>  If the system is online but the entry is not available or the entry is
>  pretty old (defined by the cache expiration timeout) then the data is
>  fetched from LDAP and returned to the caller in parallel to updating cache.
>  If the system is online and entry is available from the cache and it is
>  younger than half of its life time then it is served from the cache.
>  If the system is online and entry is available from the cache and it is
>  older than half of its life time then it is served from the cache but an
>  async operation to fetch the entry from LDAP and update the cache is
>  launched in the background.
>
>  HTH
>  
And yes this is the original logic of the SSSD cache not counting the
fast cache on top of it.

> > Thanks,
> >
> > ---
> > Olivier
> > _______________________________________________
> > sssd-devel mailing list
> > sssd-devel(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
>  

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Olivier]

Many thanks for yours answers, I will look at all this.

I would add a comment if I may :

>  I think the re-connection timeout is hardcoded as 30, I vaguely
> recall a
>  discussion about it four years ago.
>  If/when it is confirmed IMO would make sense to file an RFE.
>  I am leaning to it be configurable  and think that 5 or 10 second
>  timeout should be sufficient. 
I may be wrong but that is not exactly the way I read the man page :
http://linux.die.net/man/5/sssd-ldap

On failover section :

--> Further connection attempts are made to machines or services
marked as offline after a --> specified period of time; this is
currently hard coded to 30 seconds.

As I understand it, once a machine or service has been stamped "offline",
then it won't be queried before 30 second has passed.

The delay I would like to know (and to tune if possible) is the time after which
sssd query the next ldap server in ldap_uri list if the current one is
not reachable
(IMHO, 5 second would be more than enough and I think would even tune less
 if I could).

----
Olivier


2012/12/3 Dmitri Pal <dpal(a)redhat.com&gt;:
> On 12/03/2012 10:35 AM, Olivier wrote:
>> Hello everyone,
>>
>> I use sssd to authenticate users on redhat boxes,
>> as a layer between pam and my ldap serverS.
>>
>> User ldap accounts are announced but 3 replicated
>> (open)ldap servers.
>>
>> I have declared my 3 servers in sssd.conf:
>>
>> ldap_uri:
ldap://ldap1.example.fr,ldap://ldap2.example.fr,ldap://ldap3.example.fr
>>
>> That works, but I would like now to better tune the
>> sssd failover and cache function offered by sssd,
>> and I'm a bit stuck.
>>
>> Questions :
>>
>> 1- about the failover mecanism, I read in sssd-ldap man page :
>>
>> " If the resolution attempt succeeds, the back end tries to connect to
>> a service on this machine.
>>   If the service connection attempt fails, then only this particular
>> service is considered offline and
>>   the back end automatically switches over to the next service."
>>
>> I there any way to tune the "timeout" after which the bakend considers
>> that the attempt
>> to connect to a server has failed and therefore that it's time to try
>> the next one ?
>
>  I think the re-connection timeout is hardcoded as 30, I vaguely
> recall a
>  discussion about it four years ago.
>  If/when it is confirmed IMO would make sense to file an RFE.
>  I am leaning to it be configurable  and think that 5 or 10 second
>  timeout should be sufficient. 
>
>
>>
>> 2- I'm a bit stuck to configure the behaviour of the boxes with regard to
>>     sssd cache (in anthother word, I have not perfectly understood under
>>     which condition sssd returns cached information rather than querying
>>     ldap and how to tune that).
>>
>> Intuitively, I would something like the sssd cache for an entry to be returned
>> rather than querying ldap if :
>>
>> -> the cache was "recently" refreshed for that entry ( not sur what
I
>> should mean
>>     by "recently" )
>> -> no ldap server respond (I suspect that this would be tuned with
>> "ldap_search_timeout" ?)
>>
>> Any help from anyone ?
>
> If the system is offline the data is served from the cache.
> If the system is online but the entry is not available or the entry is
> pretty old (defined by the cache expiration timeout) then the data is
> fetched from LDAP and returned to the caller in parallel to updating cache.
> If the system is online and entry is available from the cache and it is
> younger than half of its life time then it is served from the cache.
> If the system is online and entry is available from the cache and it is
> older than half of its life time then it is served from the cache but an
> async operation to fetch the entry from LDAP and update the cache is
> launched in the background.
>
> HTH
>
>
>> Thanks,
>>
>> ---
>> Olivier
>> _______________________________________________
>> sssd-devel mailing list
>> sssd-devel(a)lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
>
>
> --
> Thank you,
> Dmitri Pal
>
> Sr. Engineering Manager for IdM portfolio
> Red Hat Inc.
>
>
> -------------------------------
> Looking to carve out IT costs?
> www.redhat.com/carveoutcosts/
>
>
>
> _______________________________________________
> sssd-devel mailing list
> sssd-devel(a)lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/sssd-devel

[Simo Sorce]

On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  I may be wrong but that is not exactly the way I read the man page :
>  http://linux.die.net/man/5/sssd-ldap
>  
>  On failover section :
>  
>  --> Further connection attempts are made to machines or services
>  marked as offline after a --> specified period of time; this is
>  currently hard coded to 30 seconds.
>  
>  As I understand it, once a machine or service has been stamped "offline",
>  then it won't be queried before 30 second has passed.
>  
>  The delay I would like to know (and to tune if possible) is the time after which
>  sssd query the next ldap server in ldap_uri list if the current one is
>  not reachable
>  (IMHO, 5 second would be more than enough and I think would even tune less
>   if I could). 
Ah I think you've been mislead by the timeout section.

That doesn't apply to a specific server but to a general attempt to go
online.

If I remember correctly, when SSSD tries to go online it will try the
first server, and if it fails it will immediately try the next and so on
until it finds one that works or goes back offline because none work.

If it goes back offline it will wait at least 30 sec. before going
online but will go online only if there is an event that requires data
after the 30 sec. has gone by.

HTH,
Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Dmitri Pal]

On 12/03/2012 03:21 PM, Simo Sorce wrote:
>  On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
> > I may be wrong but that is not exactly the way I read the man page :
> > http://linux.die.net/man/5/sssd-ldap
> >
> > On failover section :
> >
> > --> Further connection attempts are made to machines or services
> > marked as offline after a --> specified period of time; this is
> > currently hard coded to 30 seconds.
> >
> > As I understand it, once a machine or service has been stamped "offline",
> > then it won't be queried before 30 second has passed.
> >
> > The delay I would like to know (and to tune if possible) is the time after which
> > sssd query the next ldap server in ldap_uri list if the current one is
> > not reachable
> > (IMHO, 5 second would be more than enough and I think would even tune less
> >  if I could).
>  Ah I think you've been mislead by the timeout section.
>
>  That doesn't apply to a specific server but to a general attempt to go
>  online.
>
>  If I remember correctly, when SSSD tries to go online it will try the
>  first server, and if it fails it will immediately try the next and so on
>  until it finds one that works or goes back offline because none work.
>
>  If it goes back offline it will wait at least 30 sec. before going
>  online but will go online only if there is an event that requires data
>  after the 30 sec. has gone by.
>
>  HTH,
>  Simo.
>  
I think the question is for how long SSSD waits to detect that the
server is offline and whether this is configurable.
Jakub?

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Simo Sorce]

On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
>  On 12/03/2012 03:21 PM, Simo Sorce wrote:
>  > On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  >> I may be wrong but that is not exactly the way I read the man page :
>  >> http://linux.die.net/man/5/sssd-ldap
>  >>
>  >> On failover section :
>  >>
>  >> --> Further connection attempts are made to machines or services
>  >> marked as offline after a --> specified period of time; this is
>  >> currently hard coded to 30 seconds.
>  >>
>  >> As I understand it, once a machine or service has been stamped
> "offline",
>  >> then it won't be queried before 30 second has passed.
>  >>
>  >> The delay I would like to know (and to tune if possible) is the time after
> which
>  >> sssd query the next ldap server in ldap_uri list if the current one is
>  >> not reachable
>  >> (IMHO, 5 second would be more than enough and I think would even tune less
>  >>  if I could).
>  > Ah I think you've been mislead by the timeout section.
>  >
>  > That doesn't apply to a specific server but to a general attempt to go
>  > online.
>  >
>  > If I remember correctly, when SSSD tries to go online it will try the
>  > first server, and if it fails it will immediately try the next and so on
>  > until it finds one that works or goes back offline because none work.
>  >
>  > If it goes back offline it will wait at least 30 sec. before going
>  > online but will go online only if there is an event that requires data
>  > after the 30 sec. has gone by.
>  >
>  > HTH,
>  > Simo.
>  >
>  I think the question is for how long SSSD waits to detect that the
>  server is offline and whether this is configurable. 
It is not, SSSD will find out the first time it needs to resolve
something and will try to fetch data from the offline server.

At that point it will determine the server is offline.

What we may do is to retry immediately once, I do not recall if we do
that.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York

[Dmitri Pal]

On 12/03/2012 04:27 PM, Simo Sorce wrote:
>  On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
> > On 12/03/2012 03:21 PM, Simo Sorce wrote:
> >> On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
> >>> I may be wrong but that is not exactly the way I read the man page :
> >>> http://linux.die.net/man/5/sssd-ldap
> >>>
> >>> On failover section :
> >>>
> >>> --> Further connection attempts are made to machines or services
> >>> marked as offline after a --> specified period of time; this is
> >>> currently hard coded to 30 seconds.
> >>>
> >>> As I understand it, once a machine or service has been stamped
> "offline",
> >>> then it won't be queried before 30 second has passed.
> >>>
> >>> The delay I would like to know (and to tune if possible) is the time after
> which
> >>> sssd query the next ldap server in ldap_uri list if the current one is
> >>> not reachable
> >>> (IMHO, 5 second would be more than enough and I think would even tune less
> >>>  if I could).
> >> Ah I think you've been mislead by the timeout section.
> >>
> >> That doesn't apply to a specific server but to a general attempt to go
> >> online.
> >>
> >> If I remember correctly, when SSSD tries to go online it will try the
> >> first server, and if it fails it will immediately try the next and so on
> >> until it finds one that works or goes back offline because none work.
> >>
> >> If it goes back offline it will wait at least 30 sec. before going
> >> online but will go online only if there is an event that requires data
> >> after the 30 sec. has gone by.
> >>
> >> HTH,
> >> Simo.
> >>
> > I think the question is for how long SSSD waits to detect that the
> > server is offline and whether this is configurable.
>  It is not, SSSD will find out the first time it needs to resolve
>  something and will try to fetch data from the offline server.
>
>  At that point it will determine the server is offline.
>
>  What we may do is to retry immediately once, I do not recall if we do
>  that.
>
>  Simo.
>  
Let me try again...

SSSD sends a request to the central server because something triggered a
lookup.
How long SSSD waits to declare that server is not reachable before
trying another server?
What is the length of this timeout and is it configurable and how?

Does the question makes sense now?


-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Simo Sorce]

On Mon, 2012-12-03 at 16:54 -0500, Dmitri Pal wrote:
>  On 12/03/2012 04:27 PM, Simo Sorce wrote:
>  > On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
>  >> On 12/03/2012 03:21 PM, Simo Sorce wrote:
>  >>> On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  >>>> I may be wrong but that is not exactly the way I read the man page :
>  >>>> http://linux.die.net/man/5/sssd-ldap
>  >>>>
>  >>>> On failover section :
>  >>>>
>  >>>> --> Further connection attempts are made to machines or services
>  >>>> marked as offline after a --> specified period of time; this is
>  >>>> currently hard coded to 30 seconds.
>  >>>>
>  >>>> As I understand it, once a machine or service has been stamped
> "offline",
>  >>>> then it won't be queried before 30 second has passed.
>  >>>>
>  >>>> The delay I would like to know (and to tune if possible) is the time
> after which
>  >>>> sssd query the next ldap server in ldap_uri list if the current one is
>  >>>> not reachable
>  >>>> (IMHO, 5 second would be more than enough and I think would even tune
> less
>  >>>>  if I could).
>  >>> Ah I think you've been mislead by the timeout section.
>  >>>
>  >>> That doesn't apply to a specific server but to a general attempt to go
>  >>> online.
>  >>>
>  >>> If I remember correctly, when SSSD tries to go online it will try the
>  >>> first server, and if it fails it will immediately try the next and so on
>  >>> until it finds one that works or goes back offline because none work.
>  >>>
>  >>> If it goes back offline it will wait at least 30 sec. before going
>  >>> online but will go online only if there is an event that requires data
>  >>> after the 30 sec. has gone by.
>  >>>
>  >>> HTH,
>  >>> Simo.
>  >>>
>  >> I think the question is for how long SSSD waits to detect that the
>  >> server is offline and whether this is configurable.
>  > It is not, SSSD will find out the first time it needs to resolve
>  > something and will try to fetch data from the offline server.
>  >
>  > At that point it will determine the server is offline.
>  >
>  > What we may do is to retry immediately once, I do not recall if we do
>  > that.
>  >
>  > Simo.
>  >
>  Let me try again...
>  
>  SSSD sends a request to the central server because something triggered a
>  lookup.
>  How long SSSD waits to declare that server is not reachable before
>  trying another server?
>  What is the length of this timeout and is it configurable and how?
>  
>  Does the question makes sense now? 
It depends on a number of factors, as communication to the server
depends on various components like dns resolution.

For dns resolution we have a config option called dns_resolver_timeout
(default 5 sec), then for ldap we have other knobs like
ldap_search_timeout, ldap_enumeration_search_timeout,
ldap_network_timeout, ldap_opt_timeout, as well as
ldap_connection_expire_timeout

See the sssd-ldap manpage for details about these timeouts.

Simo.


-- 
Simo Sorce * Red Hat, Inc * New York

[Dmitri Pal]

On 12/03/2012 05:17 PM, Simo Sorce wrote:
>  On Mon, 2012-12-03 at 16:54 -0500, Dmitri Pal wrote:
> > On 12/03/2012 04:27 PM, Simo Sorce wrote:
> >> On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
> >>> On 12/03/2012 03:21 PM, Simo Sorce wrote:
> >>>> On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
> >>>>> I may be wrong but that is not exactly the way I read the man page :
> >>>>> http://linux.die.net/man/5/sssd-ldap
> >>>>>
> >>>>> On failover section :
> >>>>>
> >>>>> --> Further connection attempts are made to machines or services
> >>>>> marked as offline after a --> specified period of time; this is
> >>>>> currently hard coded to 30 seconds.
> >>>>>
> >>>>> As I understand it, once a machine or service has been stamped
> "offline",
> >>>>> then it won't be queried before 30 second has passed.
> >>>>>
> >>>>> The delay I would like to know (and to tune if possible) is the time
> after which
> >>>>> sssd query the next ldap server in ldap_uri list if the current one
> is
> >>>>> not reachable
> >>>>> (IMHO, 5 second would be more than enough and I think would even tune
> less
> >>>>>  if I could).
> >>>> Ah I think you've been mislead by the timeout section.
> >>>>
> >>>> That doesn't apply to a specific server but to a general attempt to
> go
> >>>> online.
> >>>>
> >>>> If I remember correctly, when SSSD tries to go online it will try the
> >>>> first server, and if it fails it will immediately try the next and so on
> >>>> until it finds one that works or goes back offline because none work.
> >>>>
> >>>> If it goes back offline it will wait at least 30 sec. before going
> >>>> online but will go online only if there is an event that requires data
> >>>> after the 30 sec. has gone by.
> >>>>
> >>>> HTH,
> >>>> Simo.
> >>>>
> >>> I think the question is for how long SSSD waits to detect that the
> >>> server is offline and whether this is configurable.
> >> It is not, SSSD will find out the first time it needs to resolve
> >> something and will try to fetch data from the offline server.
> >>
> >> At that point it will determine the server is offline.
> >>
> >> What we may do is to retry immediately once, I do not recall if we do
> >> that.
> >>
> >> Simo.
> >>
> > Let me try again...
> >
> > SSSD sends a request to the central server because something triggered a
> > lookup.
> > How long SSSD waits to declare that server is not reachable before
> > trying another server?
> > What is the length of this timeout and is it configurable and how?
> >
> > Does the question makes sense now?
>  It depends on a number of factors, as communication to the server
>  depends on various components like dns resolution.
>
>  For dns resolution we have a config option called dns_resolver_timeout
>  (default 5 sec), then for ldap we have other knobs like
>  ldap_search_timeout, ldap_enumeration_search_timeout,
>  ldap_network_timeout, ldap_opt_timeout, as well as
>  ldap_connection_expire_timeout
>
>  See the sssd-ldap manpage for details about these timeouts.
>
>  Simo.
>
>  
Sure. It is just hard to understand how they all play together.
This is why the person on the thread is asking.

-- 
Thank you,
Dmitri Pal

Sr. Engineering Manager for IdM portfolio
Red Hat Inc.


-------------------------------
Looking to carve out IT costs?
www.redhat.com/carveoutcosts/

[Jakub Hrozek]

On Mon, Dec 03, 2012 at 04:54:31PM -0500, Dmitri Pal wrote:
>  On 12/03/2012 04:27 PM, Simo Sorce wrote:
>  > On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
>  >> On 12/03/2012 03:21 PM, Simo Sorce wrote:
>  >>> On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  >>>> I may be wrong but that is not exactly the way I read the man page :
>  >>>> http://linux.die.net/man/5/sssd-ldap
>  >>>>
>  >>>> On failover section :
>  >>>>
>  >>>> --> Further connection attempts are made to machines or services
>  >>>> marked as offline after a --> specified period of time; this is
>  >>>> currently hard coded to 30 seconds.
>  >>>>
>  >>>> As I understand it, once a machine or service has been stamped
> "offline",
>  >>>> then it won't be queried before 30 second has passed.
>  >>>>
>  >>>> The delay I would like to know (and to tune if possible) is the time
> after which
>  >>>> sssd query the next ldap server in ldap_uri list if the current one is
>  >>>> not reachable
>  >>>> (IMHO, 5 second would be more than enough and I think would even tune
> less
>  >>>>  if I could).
>  >>> Ah I think you've been mislead by the timeout section.
>  >>>
>  >>> That doesn't apply to a specific server but to a general attempt to go
>  >>> online.
>  >>>
>  >>> If I remember correctly, when SSSD tries to go online it will try the
>  >>> first server, and if it fails it will immediately try the next and so on
>  >>> until it finds one that works or goes back offline because none work.
>  >>>
>  >>> If it goes back offline it will wait at least 30 sec. before going
>  >>> online but will go online only if there is an event that requires data
>  >>> after the 30 sec. has gone by.
>  >>>
>  >>> HTH,
>  >>> Simo.
>  >>>
>  >> I think the question is for how long SSSD waits to detect that the
>  >> server is offline and whether this is configurable.
>  > It is not, SSSD will find out the first time it needs to resolve
>  > something and will try to fetch data from the offline server.
>  >
>  > At that point it will determine the server is offline.
>  >
>  > What we may do is to retry immediately once, I do not recall if we do
>  > that.
>  >
>  > Simo.
>  >
>  Let me try again...
>  
>  SSSD sends a request to the central server because something triggered a
>  lookup.
>  How long SSSD waits to declare that server is not reachable before
>  trying another server?
>  What is the length of this timeout and is it configurable and how?
>   
30 seconds or when the networking changes as a result of a libnl
callback. The 30 seconds timeout is hardcoded right now.

>  Does the question makes sense now?

[Jakub Hrozek]

On Mon, Dec 03, 2012 at 04:27:30PM -0500, Simo Sorce wrote:
>  On Mon, 2012-12-03 at 16:16 -0500, Dmitri Pal wrote:
>  > On 12/03/2012 03:21 PM, Simo Sorce wrote:
>  > > On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  > >> I may be wrong but that is not exactly the way I read the man page :
>  > >> http://linux.die.net/man/5/sssd-ldap
>  > >>
>  > >> On failover section :
>  > >>
>  > >> --> Further connection attempts are made to machines or services
>  > >> marked as offline after a --> specified period of time; this is
>  > >> currently hard coded to 30 seconds.
>  > >>
>  > >> As I understand it, once a machine or service has been stamped
> "offline",
>  > >> then it won't be queried before 30 second has passed.
>  > >>
>  > >> The delay I would like to know (and to tune if possible) is the time after
> which
>  > >> sssd query the next ldap server in ldap_uri list if the current one is
>  > >> not reachable
>  > >> (IMHO, 5 second would be more than enough and I think would even tune less
>  > >>  if I could).
>  > > Ah I think you've been mislead by the timeout section.
>  > >
>  > > That doesn't apply to a specific server but to a general attempt to go
>  > > online.
>  > >
>  > > If I remember correctly, when SSSD tries to go online it will try the
>  > > first server, and if it fails it will immediately try the next and so on
>  > > until it finds one that works or goes back offline because none work.
>  > >
>  > > If it goes back offline it will wait at least 30 sec. before going
>  > > online but will go online only if there is an event that requires data
>  > > after the 30 sec. has gone by.
>  > >
>  > > HTH,
>  > > Simo.
>  > >
>  > I think the question is for how long SSSD waits to detect that the
>  > server is offline and whether this is configurable.
>  
>  It is not, SSSD will find out the first time it needs to resolve
>  something and will try to fetch data from the offline server.
>  
>  At that point it will determine the server is offline.
>  
>  What we may do is to retry immediately once, I do not recall if we do
>  that. 
We don't retry, once a server is offline, we either wait for 30 seconds
or until libnetlink notifies us that the networking changed.

Keep in mind that a "server" in failover concept is defined with
host:port, not just host. So we can mark a Kerberos "server" as offline
but happily talk to LDAP on the same machine. We have a special hack in
IPA to force talking to the same physical host for both LDAP and
Kerberos, but only in IPA.

[Jakub Hrozek]

On Mon, Dec 03, 2012 at 03:21:25PM -0500, Simo Sorce wrote:
>  On Mon, 2012-12-03 at 21:12 +0100, Olivier wrote:
>  > I may be wrong but that is not exactly the way I read the man page :
>  > http://linux.die.net/man/5/sssd-ldap
>  > 
>  > On failover section :
>  > 
>  > --> Further connection attempts are made to machines or services
>  > marked as offline after a --> specified period of time; this is
>  > currently hard coded to 30 seconds.
>  > 
>  > As I understand it, once a machine or service has been stamped "offline",
>  > then it won't be queried before 30 second has passed.
>  > 
>  > The delay I would like to know (and to tune if possible) is the time after which
>  > sssd query the next ldap server in ldap_uri list if the current one is
>  > not reachable
>  > (IMHO, 5 second would be more than enough and I think would even tune less
>  >  if I could).
>  
>  Ah I think you've been mislead by the timeout section.
>  
>  That doesn't apply to a specific server but to a general attempt to go
>  online.
>  
>  If I remember correctly, when SSSD tries to go online it will try the
>  first server, and if it fails it will immediately try the next and so on
>  until it finds one that works or goes back offline because none work.
>  
>  If it goes back offline it will wait at least 30 sec. before going
>  online but will go online only if there is an event that requires data
>  after the 30 sec. has gone by. 
That pretty much how it works with the single exception of libnl
callbacks. Those can clear up the offline state earlier.

I still think it would be more beneficial if the original poster
explained his problem, maybe it's not related to callbacks at all :-)

[Olivier]

Hello Jakub,

>  I still think it would be more beneficial if the original poster
>  explained his problem, maybe it's not related to callbacks at all :-) 
The goal of my question was more to seek for guidance to
configure and tune properly the authenticaton mecanism on
the client side than to fix any specific problem. However, if
I think about it I can explain a bit more in detail :

rarely and under certain circumstances I have noticed that I
was freezed when I tryed to log into certain machines that
were configured to use pam/sssd/ldap for users/groups.

I will investigate more but it sounds that this happened when
the first ldap service in ldap_uri list was not reachable. Since
the second ldap service in the list was available, I was wondering
if there were any sort of "flag" or "time_out" to set up to tune
properly the failover mecanism.

Thanks to all for the time spent on that issue,

---
Olivier

2012/12/4 Jakub Hrozek <jhrozek(a)redhat.com&gt;: