Strangeness with groups returned using id user

[Max DiOrio]

So we are having issues with a couple servers where users suddenly won't be
able to log in.  All our auth is done through AD and not a thing has
changed.

On a working server, I can do 'id username' and get back the proper list of
groups the user is a member of.

On the non-working server, 'id username' returns *mostly* the same list.
However the one group that the user needs to be a member to log in is
missing.

There are some groups in both lists that that have a group ID, but not a
group name.  And the one non-working server has a single group entry
duplicated.  The results of 'id username' match throughout, except the
noted areas below and a few entries that are listed out of order between
the two.

Here are the differences "non-working" on top, "working" on bottom
(gs-technology is the group in question that I need on the non-working
server).  It doesn't make sense that 1002201991 is showing up twice in the
list.

1002201991
1002201991(fs01-technology-all(rw))
1002201620(infrastructureteam)
1002201620

1002201991
1002204761(gs-technology)

Thanks!

Max

[Lachlan Musicman]

On 24 April 2018 at 03:01, Max DiOrio  wrote:
> So we are having issues with a couple servers where users suddenly won't
> be able to log in.  All our auth is done through AD and not a thing has
> changed.
>
> On a working server, I can do 'id username' and get back the proper list
> of groups the user is a member of.
>
> On the non-working server, 'id username' returns *mostly* the same list.
> However the one group that the user needs to be a member to log in is
> missing.
>
> There are some groups in both lists that that have a group ID, but not a
> group name.  And the one non-working server has a single group entry
> duplicated.  The results of 'id username' match throughout, except the
> noted areas below and a few entries that are listed out of order between
> the two.
>
> Here are the differences "non-working" on top, "working" on bottom
> (gs-technology is the group in question that I need on the non-working
> server).  It doesn't make sense that 1002201991 is showing up twice in the
> list.
>
> 1002201991
> 1002201991(fs01-technology-all(rw))
> 1002201620(infrastructureteam)
> 1002201620
>
> 1002201991
> 1002204761(gs-technology)
>
> Thanks!

Max, Which version of SSSD are you using, and which OS?

Cheers
L.

[Max DiOrio]

I'll have to get back to you tomorrow on the version, but it's RHEL 7.4.

One server started working after flushing the cache a million times and
restarting sssd a couple hundred. The other server I believe is showing
even less groups.

I feel like we've had these issues a few times very randomly. And nothing I
do necessarily fixes it per se.

On Mon, Apr 23, 2018, 6:29 PM Lachlan Musicman  wrote:
> On 24 April 2018 at 03:01, Max DiOrio  wrote:
> > So we are having issues with a couple servers where users suddenly won't
> > be able to log in.  All our auth is done through AD and not a thing has
> > changed.
> >
> > On a working server, I can do 'id username' and get back the proper list
> > of groups the user is a member of.
> >
> > On the non-working server, 'id username' returns *mostly* the same list.
> > However the one group that the user needs to be a member to log in is
> > missing.
> >
> > There are some groups in both lists that that have a group ID, but not a
> > group name.  And the one non-working server has a single group entry
> > duplicated.  The results of 'id username' match throughout, except the
> > noted areas below and a few entries that are listed out of order between
> > the two.
> >
> > Here are the differences "non-working" on top, "working" on bottom
> > (gs-technology is the group in question that I need on the non-working
> > server).  It doesn't make sense that 1002201991 is showing up twice in the
> > list.
> >
> > 1002201991
> > 1002201991(fs01-technology-all(rw))
> > 1002201620(infrastructureteam)
> > 1002201620
> >
> > 1002201991
> > 1002204761(gs-technology)
> >
> > Thanks!
>
> Max, Which version of SSSD are you using, and which OS?
>
> Cheers
> L.
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Max DiOrio]

We’re running SSSD 1.15.2
> On Apr 23, 2018, at 6:29 PM, Lachlan Musicman  wrote:
>
> On 24 April 2018 at 03:01, Max DiOrio <[e-mail] > wrote:
> So we are having issues with a couple servers where users suddenly won't be able to log in.  All our auth is done through AD and not a thing has changed.
>
> On a working server, I can do 'id username' and get back the proper list of groups the user is a member of.
>
> On the non-working server, 'id username' returns *mostly* the same list.  However the one group that the user needs to be a member to log in is missing.
>
> There are some groups in both lists that that have a group ID, but not a group name.  And the one non-working server has a single group entry duplicated.  The results of 'id username' match throughout, except the noted areas below and a few entries that are listed out of order between the two.
>
> Here are the differences "non-working" on top, "working" on bottom (gs-technology is the group in question that I need on the non-working server).  It doesn't make sense that 1002201991 is showing up twice in the list.
>
> 1002201991
> 1002201991(fs01-technology-all(rw))
>
> 1002201620(infrastructureteam)
> 1002201620
>
> 1002201991
> 1002204761(gs-technology)
>
> Thanks!
>
> Max, Which version of SSSD are you using, and which OS?
>
> Cheers
> L.
>
> sssd-users mailing list -- [e-mail] 
> To unsubscribe send an email to [e-mail]

[Max DiOrio]

Any thoughts?  This issue seems to be rippling through all our AD Domain Joined servers.  The group randomly goes missing and nobody can log into the server.  After some time, it eventually starts working again.
> On Apr 24, 2018, at 9:08 AM, Max DiOrio  wrote:
>
> We’re running SSSD 1.15.2
> > On Apr 23, 2018, at 6:29 PM, Lachlan Musicman <[e-mail] > wrote:
> >
> > On 24 April 2018 at 03:01, Max DiOrio <[e-mail] > wrote:
> > So we are having issues with a couple servers where users suddenly won't be able to log in.  All our auth is done through AD and not a thing has changed.
> >
> > On a working server, I can do 'id username' and get back the proper list of groups the user is a member of.
> >
> > On the non-working server, 'id username' returns *mostly* the same list.  However the one group that the user needs to be a member to log in is missing.
> >
> > There are some groups in both lists that that have a group ID, but not a group name.  And the one non-working server has a single group entry duplicated.  The results of 'id username' match throughout, except the noted areas below and a few entries that are listed out of order between the two.
> >
> > Here are the differences "non-working" on top, "working" on bottom (gs-technology is the group in question that I need on the non-working server).  It doesn't make sense that 1002201991 is showing up twice in the list.
> >
> > 1002201991
> > 1002201991(fs01-technology-all(rw))
> >
> > 1002201620(infrastructureteam)
> > 1002201620
> >
> > 1002201991
> > 1002204761(gs-technology)
> >
> > Thanks!
> >
> > Max, Which version of SSSD are you using, and which OS?
> >
> > Cheers
> > L.
> >
> > sssd-users mailing list -- [e-mail] 
> > To unsubscribe send an email to [e-mail]

[Striker Leggette]

I would start by gathering debugging during the time that the issue happens:

1. Add "debug_level = 9" to all sections [sssd], [domain], etc.
   # vi /etc/sssd/sssd.conf

2. Restart SSSD:
   # systemctl stop sssd ; rm -rf /var/log/sssd/* /var/lib/sss/{db,mc}/* 
; systemctl start sssd

3. Replicate the issue.

4. Archive and upload logs to some location:
   # tar czvf /tmp/sssd-debug__$(hostname -s)__$(date +%F_%H%M%S).tar.gz 
/etc/{sssd,nsswitch*,krb5.conf,samba,pam.d} /var/log/{secure,messages,sssd}

Alternatively, you could also look into the logs alone 
(/var/log/sssd/sssd_example.com.log.  Try to find the location within 
the logs that correlates with the timestamp of when the issue happened 
and then diagnose from there.

Striker

On 05/04/2018 09:12 AM, Max DiOrio wrote:
> Any thoughts?  This issue seems to be rippling through all our AD 
> Domain Joined servers.  The group randomly goes missing and nobody can 
> log into the server.  After some time, it eventually starts working 
> again.
> > On Apr 24, 2018, at 9:08 AM, Max DiOrio <[e-mail] 
> > > wrote:
> >
> > We’re running SSSD 1.15.2
> > > On Apr 23, 2018, at 6:29 PM, Lachlan Musicman <[e-mail] 
> > > > wrote:
> > >
> > > On 24 April 2018 at 03:01, Max DiOrio<[e-mail] 
> > > >wrote:
> > > So we are having issues with a couple servers where users
> > > suddenly won't be able to log in.  All our auth is done through
> > > AD and not a thing has changed.
> > >
> > > On a working server, I can do 'id username' and get back the
> > > proper list of groups the user is a member of.
> > >
> > > On the non-working server, 'id username' returns *mostly* the
> > > same list.  However the one group that the user needs to be a
> > > member to log in is missing.
> > >
> > > There are some groups in both lists that that have a group ID,
> > > but not a group name.  And the one non-working server has a
> > > single group entry duplicated.  The results of 'id username'
> > > match throughout, except the noted areas below and a few entries
> > > that are listed out of order between the two.
> > >
> > > Here are the differences "non-working" on top, "working" on
> > > bottom (gs-technology is the group in question that I need on
> > > the non-working server).  It doesn't make sense that 1002201991
> > > is showing up twice in the list.
> > >
> > > 1002201991
> > > 1002201991(fs01-technology-all(rw))
> > >
> > >
> > > 1002201620(infrastructureteam)
> > > 1002201620
> > >
> > > 1002201991
> > > 1002204761(gs-technology)
> > >
> > >
> > >
> > > Thanks!
> > >
> > >
> > >
> > > Max, Which version of SSSD are you using, and which OS?
> > >
> > > Cheers
> > > L.
> > >
> > > sssd-users mailing list [e-mail] 
> > >
> > > To unsubscribe send an email 
> > > [e-mail] 
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Lachlan Musicman]

On 4 May 2018 at 23:31, Striker Leggette  wrote:
> I would start by gathering debugging during the time that the issue
> happens:
> Add "debug_level = 9" to all sections [sssd], [domain], etc.
>
> # vi /etc/sssd/sssd.conf
> Restart SSSD:
>
> # systemctl stop sssd ; rm -rf /var/log/sssd/* /var/lib/sss/{db,mc}/* ;
> systemctl start sssd
>
> Replicate the issue.
>
> Archive and upload logs to some location:
>
> # tar czvf /tmp/sssd-debug__$(hostname -s)__$(date +%F_%H%M%S).tar.gz
> /etc/{sssd,nsswitch*,krb5.conf,samba,pam.d} /var/log/{secure,messages,
> sssd}

Striker's advice is good. As a parallel solution while you are gathering
log data, you might want to test other versions. I am not an expert, but we
did see similar issues - logins failing because groups would go missing. We
found updating SSSD helped. I feel like we moved *to* 1.15.2, but maybe you
could try SSSD 1.16.x

We use CentOS, so the COPR repo was our friend. We have recently moved from
the 1.15 to 1.16 COPR repos and haven't had a problem for a while.

https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-15/
https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-16/

Cheers
L.
> On 05/04/2018 09:12 AM, Max DiOrio wrote:
>
> Any thoughts?  This issue seems to be rippling through all our AD Domain
> Joined servers.  The group randomly goes missing and nobody can log into
> the server.  After some time, it eventually starts working again.
>
> On Apr 24, 2018, at 9:08 AM, Max DiOrio  wrote:
>
> We’re running SSSD 1.15.2
>
> On Apr 23, 2018, at 6:29 PM, Lachlan Musicman  wrote:
>
> On 24 April 2018 at 03:01, Max DiOrio  wrote:
> > So we are having issues with a couple servers where users suddenly won't
> > be able to log in.  All our auth is done through AD and not a thing has
> > changed.
> >
> > On a working server, I can do 'id username' and get back the proper list
> > of groups the user is a member of.
> >
> > On the non-working server, 'id username' returns *mostly* the same list.
> > However the one group that the user needs to be a member to log in is
> > missing.
> >
> > There are some groups in both lists that that have a group ID, but not a
> > group name.  And the one non-working server has a single group entry
> > duplicated.  The results of 'id username' match throughout, except the
> > noted areas below and a few entries that are listed out of order between
> > the two.
> >
> > Here are the differences "non-working" on top, "working" on bottom
> > (gs-technology is the group in question that I need on the non-working
> > server).  It doesn't make sense that 1002201991 is showing up twice in the
> > list.
> >
> > 1002201991
> > 1002201991(fs01-technology-all(rw))
> > 1002201620(infrastructureteam)
> > 1002201620
> >
> > 1002201991
> > 1002204761(gs-technology)
> >
> > Thanks!
>
> Max, Which version of SSSD are you using, and which OS?
>
> Cheers
> L.
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>
> sssd-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Max DiOrio]

Interesting. Temporarily I used a different group that has been showing up
reliably so far, but the other one has been for about 6 months+ before it
stopped.

I'll check into loading the COPR repo into Spacewalk and give it a test.

On Fri, May 4, 2018, 7:49 PM Lachlan Musicman  wrote:
> On 4 May 2018 at 23:31, Striker Leggette  wrote:
> > I would start by gathering debugging during the time that the issue
> > happens:
> > Add "debug_level = 9" to all sections [sssd], [domain], etc.
> >
> > # vi /etc/sssd/sssd.conf
> > Restart SSSD:
> >
> > # systemctl stop sssd ; rm -rf /var/log/sssd/* /var/lib/sss/{db,mc}/* ;
> > systemctl start sssd
> >
> > Replicate the issue.
> >
> > Archive and upload logs to some location:
> >
> > # tar czvf /tmp/sssd-debug__$(hostname -s)__$(date +%F_%H%M%S).tar.gz
> > /etc/{sssd,nsswitch*,krb5.conf,samba,pam.d} /var/log/{secure,messages,sssd}
>
> Striker's advice is good. As a parallel solution while you are gathering
> log data, you might want to test other versions. I am not an expert, but we
> did see similar issues - logins failing because groups would go missing. We
> found updating SSSD helped. I feel like we moved *to* 1.15.2, but maybe you
> could try SSSD 1.16.x
>
> We use CentOS, so the COPR repo was our friend. We have recently moved
> from the 1.15 to 1.16 COPR repos and haven't had a problem for a while.
>
> https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-15/
> https://copr.fedorainfracloud.org/coprs/g/sssd/sssd-1-16/
>
> Cheers
> L.
> > On 05/04/2018 09:12 AM, Max DiOrio wrote:
> >
> > Any thoughts?  This issue seems to be rippling through all our AD Domain
> > Joined servers.  The group randomly goes missing and nobody can log into
> > the server.  After some time, it eventually starts working again.
> >
> > On Apr 24, 2018, at 9:08 AM, Max DiOrio  wrote:
> >
> > We’re running SSSD 1.15.2
> >
> > On Apr 23, 2018, at 6:29 PM, Lachlan Musicman  wrote:
> >
> > On 24 April 2018 at 03:01, Max DiOrio  wrote:
> > > So we are having issues with a couple servers where users suddenly won't
> > > be able to log in.  All our auth is done through AD and not a thing has
> > > changed.
> > >
> > > On a working server, I can do 'id username' and get back the proper list
> > > of groups the user is a member of.
> > >
> > > On the non-working server, 'id username' returns *mostly* the same
> > > list.  However the one group that the user needs to be a member to log in
> > > is missing.
> > >
> > > There are some groups in both lists that that have a group ID, but not a
> > > group name.  And the one non-working server has a single group entry
> > > duplicated.  The results of 'id username' match throughout, except the
> > > noted areas below and a few entries that are listed out of order between
> > > the two.
> > >
> > > Here are the differences "non-working" on top, "working" on bottom
> > > (gs-technology is the group in question that I need on the non-working
> > > server).  It doesn't make sense that 1002201991 is showing up twice in the
> > > list.
> > >
> > > 1002201991
> > > 1002201991(fs01-technology-all(rw))
> > > 1002201620(infrastructureteam)
> > > 1002201620
> > >
> > > 1002201991
> > > 1002204761(gs-technology)
> > >
> > > Thanks!
> >
> > Max, Which version of SSSD are you using, and which OS?
> >
> > Cheers
> > L.
> >
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> >
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> >
> > sssd-users mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]