SL 6.x sssd-1.12.4-47 dereference processing failed I/O error

[John Desantis]

Hello all,

First off, a big thanks to the developers for providing this piece of
software!  Now, to the point!

I've recently run into the error(?) message below (/var/log/messages) on
some of our infrastructure nodes which have upgraded from sssd 1.9.x to
sssd-1.12.4-47:

sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error

Doing some online research and checking the list archives (2012-2015), I
found that other users with varied versions of sssd and Linux had run into
this issue as well.  It was suggested that they should use
"ldap_deref_threshold = 0".  A user also reported no errors after enabling
enumeration.  I've done both on a test node and the message persists.  I
even purged the db and cache without luck.  I am using "error(?)" because I
am not experiencing any user/group resolution errors.  All calls to getent
and id are successful.

A thread from February 2013 [1] had a suggestion to check LDAP with a deref
call and without.  On the affected nodes, it does return a result;  the OP
of that thread said that the deref call failed.

I also saw bug report for IPA 4.0 [2] that seems to reference the same
issue, but I'm not able to duplicate the problem.

There was an update to the LDAP servers via yum (minor bug fix revisions)
for 389ds and IPA, but nothing major.  All other nodes running sssd-1.9.x
are not manifesting this message.

We're using FreeIPA (IPA server 3.0.0-47) with 389ds 1.2.11.15-60.

I can produce detailed logs upon request, but before doing so I was hoping
that the community could tell me if the message is a red herring and can be
safely ignored, or if there something else that should be checked.  It's
just very odd that the older clients aren't manifesting the message and
these new clients are, even though nothing seems "broken".

[1]
https://lists.fedorahosted.org/pipermail/sssd-users/2013-February/000418....
[2] https://fedorahosted.org/freeipa/ticket/4389

Thanks for any information!
John DeSantis

[Lukas Slebodnik]

On (24/08/15 11:34), John Desantis wrote:
> Hello all,
>
> First off, a big thanks to the developers for providing this piece of
> software!  Now, to the point!
>
> I've recently run into the error(?) message below (/var/log/messages) on
> some of our infrastructure nodes which have upgraded from sssd 1.9.x to
> sssd-1.12.4-47:
>
> sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
> sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
>  
We will need to see log files around such error message.
There can be more reasons why it failed.

> Doing some online research and checking the list archives (2012-2015),
> I
> found that other users with varied versions of sssd and Linux had run into
> this issue as well.  It was suggested that they should use
> "ldap_deref_threshold = 0". 
It is just a workaround which completely
disable deref feature.
So in some cases there can be a performance penalty, but it will work correctly.

> A user also reported no errors after enabling
> enumeration.  I've done both on a test node and the message persists.  I
> even purged the db and cache without luck.  I am using "error(?)" because I
> am not experiencing any user/group resolution errors.  All calls to getent
> and id are successful.
>
> A thread from February 2013 [1] had a suggestion to check LDAP with a deref
> call and without.  On the affected nodes, it does return a result;  the OP
> of that thread said that the deref call failed.
>  
This could be different issue because in that time users could used sssd-1.9.x
and have issues with sssd-1.12.4-47.
There were many changes changes between these releases. 

> I also saw bug report for IPA 4.0 [2] that seems to reference the
> same
> issue, but I'm not able to duplicate the problem.
>  
IIRC, it can be caused in infrastructure with IPA 3.0 and replica to IPA 4.x
But It might have already been solved.

> There was an update to the LDAP servers via yum (minor bug fix
> revisions)
> for 389ds and IPA, but nothing major.  All other nodes running sssd-1.9.x
> are not manifesting this message.
>
> We're using FreeIPA (IPA server 3.0.0-47) with 389ds 1.2.11.15-60.
>
> I can produce detailed logs upon request, but before doing so I was hoping
> that the community could tell me if the message is a red herring and can be
> safely ignored, or if there something else that should be checked.  It's
> just very odd that the older clients aren't manifesting the message and
> these new clients are, even though nothing seems "broken".
>  
Yes, please provide log files with high debug_level (9)
https://fedorahosted.org/sssd/wiki/Troubleshooting
If there will be confidential data which cannot be sent to public mailing list
feel free to send them to my private mail.

LS

[John Desantis]

Lukas,

Thanks for your response.

I'll forward you the log snippets (debug_level 9) and also the ldapsearch
queries with and without the dereference flag.

John DeSantis



2015-08-25 5:27 GMT-04:00 Lukas Slebodnik <lslebodn(a)redhat.com&gt;:

>  On (24/08/15 11:34), John Desantis wrote:
>  >Hello all,
>  >
>  >First off, a big thanks to the developers for providing this piece of
>  >software!  Now, to the point!
>  >
>  >I've recently run into the error(?) message below (/var/log/messages) on
>  >some of our infrastructure nodes which have upgraded from sssd 1.9.x to
>  >sssd-1.12.4-47:
>  >
>  >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
>  >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
>  >
>  We will need to see log files around such error message.
>  There can be more reasons why it failed.
>
>  >Doing some online research and checking the list archives (2012-2015), I
>  >found that other users with varied versions of sssd and Linux had run into
>  >this issue as well.  It was suggested that they should use
>  >"ldap_deref_threshold = 0".
>  It is just a workaround which completely disable deref feature.
>  So in some cases there can be a performance penalty, but it will work
>  correctly.
>
>  >A user also reported no errors after enabling
>  >enumeration.  I've done both on a test node and the message persists.  I
>  >even purged the db and cache without luck.  I am using "error(?)" because
>  I
>  >am not experiencing any user/group resolution errors.  All calls to getent
>  >and id are successful.
>  >
>  >A thread from February 2013 [1] had a suggestion to check LDAP with a
>  deref
>  >call and without.  On the affected nodes, it does return a result;  the OP
>  >of that thread said that the deref call failed.
>  >
>  This could be different issue because in that time users could used
>  sssd-1.9.x
>  and have issues with sssd-1.12.4-47.
>  There were many changes changes between these releases.
>
>  >I also saw bug report for IPA 4.0 [2] that seems to reference the same
>  >issue, but I'm not able to duplicate the problem.
>  >
>  IIRC, it can be caused in infrastructure with IPA 3.0 and replica to IPA
>  4.x
>  But It might have already been solved.
>
>  >There was an update to the LDAP servers via yum (minor bug fix revisions)
>  >for 389ds and IPA, but nothing major.  All other nodes running sssd-1.9.x
>  >are not manifesting this message.
>  >
>  >We're using FreeIPA (IPA server 3.0.0-47) with 389ds 1.2.11.15-60.
>  >
>  >I can produce detailed logs upon request, but before doing so I was hoping
>  >that the community could tell me if the message is a red herring and can
>  be
>  >safely ignored, or if there something else that should be checked.  It's
>  >just very odd that the older clients aren't manifesting the message and
>  >these new clients are, even though nothing seems "broken".
>  >
>  Yes, please provide log files with high debug_level (9)
>  https://fedorahosted.org/sssd/wiki/Troubleshooting
>  If there will be confidential data which cannot be sent to public mailing
>  list
>  feel free to send them to my private mail.
>
>  LS
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>

[Sumit Bose]

On Tue, Aug 25, 2015 at 11:27:21AM +0200, Lukas Slebodnik wrote:
>  On (24/08/15 11:34), John Desantis wrote:
>  >Hello all,
>  >
>  >First off, a big thanks to the developers for providing this piece of
>  >software!  Now, to the point!
>  >
>  >I've recently run into the error(?) message below (/var/log/messages) on
>  >some of our infrastructure nodes which have upgraded from sssd 1.9.x to
>  >sssd-1.12.4-47:
>  >
>  >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
>  >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
>  >
>  We will need to see log files around such error message.
>  There can be more reasons why it failed.
>  
>  >Doing some online research and checking the list archives (2012-2015), I
>  >found that other users with varied versions of sssd and Linux had run into
>  >this issue as well.  It was suggested that they should use
>  >"ldap_deref_threshold = 0".
>  It is just a workaround which completely disable deref feature.
>  So in some cases there can be a performance penalty, but it will work correctly.
>  
>  >A user also reported no errors after enabling
>  >enumeration.  I've done both on a test node and the message persists.  I
>  >even purged the db and cache without luck.  I am using "error(?)" because
> I
>  >am not experiencing any user/group resolution errors.  All calls to getent
>  >and id are successful.
>  >
>  >A thread from February 2013 [1] had a suggestion to check LDAP with a deref
>  >call and without.  On the affected nodes, it does return a result;  the OP
>  >of that thread said that the deref call failed.
>  >
>  This could be different issue because in that time users could used sssd-1.9.x
>  and have issues with sssd-1.12.4-47.
>  There were many changes changes between these releases. 
>  
>  >I also saw bug report for IPA 4.0 [2] that seems to reference the same
>  >issue, but I'm not able to duplicate the problem.
>  >
>  IIRC, it can be caused in infrastructure with IPA 3.0 and replica to IPA 4.x
>  But It might have already been solved. 
The error might happen when checking if there is an idview assigned to
the given host. A dereference search is used here and different versions
of 389ds may return different errors and some might lead to the
'Input/output error' message. Nevertheless the message can in general be
ignored if you see no other issues. SSSD will handles this case as no
idview is assigned.

But to be sure more log context is needed.

bye,
Sumit

>  
>  >There was an update to the LDAP servers via yum (minor bug fix revisions)
>  >for 389ds and IPA, but nothing major.  All other nodes running sssd-1.9.x
>  >are not manifesting this message.
>  >
>  >We're using FreeIPA (IPA server 3.0.0-47) with 389ds 1.2.11.15-60.
>  >
>  >I can produce detailed logs upon request, but before doing so I was hoping
>  >that the community could tell me if the message is a red herring and can be
>  >safely ignored, or if there something else that should be checked.  It's
>  >just very odd that the older clients aren't manifesting the message and
>  >these new clients are, even though nothing seems "broken".
>  >
>  Yes, please provide log files with high debug_level (9)
>  https://fedorahosted.org/sssd/wiki/Troubleshooting
>  If there will be confidential data which cannot be sent to public mailing list
>  feel free to send them to my private mail.
>  
>  LS
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users

[Lukas Slebodnik]

On (28/08/15 11:00), Sumit Bose wrote:
> On Tue, Aug 25, 2015 at 11:27:21AM +0200, Lukas Slebodnik wrote:
> > On (24/08/15 11:34), John Desantis wrote:
> > >Hello all,
> > >
> > >First off, a big thanks to the developers for providing this piece of
> > >software!  Now, to the point!
> > >
> > >I've recently run into the error(?) message below (/var/log/messages) on
> > >some of our infrastructure nodes which have upgraded from sssd 1.9.x to
> > >sssd-1.12.4-47:
> > >
> > >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
> > >sssd[be[rc.usf.edu]]: dereference processing failed : Input/output error
> > >
> > We will need to see log files around such error message.
> > There can be more reasons why it failed.
> > 
> > >Doing some online research and checking the list archives (2012-2015), I
> > >found that other users with varied versions of sssd and Linux had run into
> > >this issue as well.  It was suggested that they should use
> > >"ldap_deref_threshold = 0".
> > It is just a workaround which completely disable deref feature.
> > So in some cases there can be a performance penalty, but it will work correctly.
> > 
> > >A user also reported no errors after enabling
> > >enumeration.  I've done both on a test node and the message persists.  I
> > >even purged the db and cache without luck.  I am using "error(?)"
> because I
> > >am not experiencing any user/group resolution errors.  All calls to getent
> > >and id are successful.
> > >
> > >A thread from February 2013 [1] had a suggestion to check LDAP with a deref
> > >call and without.  On the affected nodes, it does return a result;  the OP
> > >of that thread said that the deref call failed.
> > >
> > This could be different issue because in that time users could used sssd-1.9.x
> > and have issues with sssd-1.12.4-47.
> > There were many changes changes between these releases. 
> > 
> > >I also saw bug report for IPA 4.0 [2] that seems to reference the same
> > >issue, but I'm not able to duplicate the problem.
> > >
> > IIRC, it can be caused in infrastructure with IPA 3.0 and replica to IPA 4.x
> > But It might have already been solved.
>
> The error might happen when checking if there is an idview assigned to
> the given host. A dereference search is used here and different versions
> of 389ds may return different errors and some might lead to the
> 'Input/output error' message. Nevertheless the message can in general be
> ignored if you see no other issues. SSSD will handles this case as no
> idview is assigned.
>
> But to be sure more log context is needed.
>  
John confirmed in private conversation
there isn't any problem with functionality.
Just error messages in logs are annoying.

LS