SSSD seriously broken in RHEL 6.7 again?

[Ondrej Valousek]

Hi list,

Is sssd broken again in 6.7?
It was working fine for me as of 1.11.7 - but that was not official version. Now I
upgraded to the latest one (1.12.4-47) hoping I'll be on the official fixed version,
and it is even worse.

I am experiencing frequent messages like:
(Wed Aug 12 17:20:55 2015) [sssd[be[default]]] [be_run_offline_cb] (0x0080): Going
offline. Running callbacks.
(Wed Aug 12 17:20:55 2015) [sssd[be[default]]] [ad_subdomains_root_conn_done] (0x0080): No
AD server is available, cannot get the subdomain list while offline
(Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [fo_resolve_service_timeout] (0x0080):
Service resolving timeout reached
(Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [sdap_id_op_connect_done] (0x0020): Failed
to connect, going offline (5 [Input/output error])
(Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [be_ptask_enable] (0x0080): Task [Check if
online (periodic)]: already enabled
(Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [be_run_offline_cb] (0x0080): Going
offline. Running callbacks.
(Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [ad_subdomains_root_conn_done] (0x0080): No
AD server is available, cannot get the subdomain list

After cleaning cache, sssd refuses to list all groups for user, etc.

Is sssd going to be stable at some stage? This is really disappointing :(
Thanks,
Ondrej

-----

The information contained in this e-mail and in any attachments is confidential and is
designated solely for the attention of the intended recipient(s). If you are not an
intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or
any part thereof. If you have received this e-mail in error, please notify the sender by
return e-mail and delete all copies of this e-mail from your computer system(s). Please
direct any additional queries to: communications(a)s3group.com. Thank You. Silicon and
Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office:
South County Business Park, Leopardstown, Dublin 18.

[Jakub Hrozek]

On Wed, Aug 12, 2015 at 04:31:14PM +0000, Ondrej Valousek wrote:
>  Hi list,
>  
>  Is sssd broken again in 6.7?
>  It was working fine for me as of 1.11.7 - but that was not official version. Now I
> upgraded to the latest one (1.12.4-47) hoping I'll be on the official fixed version,
> and it is even worse.
>  
>  I am experiencing frequent messages like: 
Could you paste the lines before that to see why we are going offline?

>  (Wed Aug 12 17:20:55 2015) [sssd[be[default]]] [be_run_offline_cb]
> (0x0080): Going offline. Running callbacks.
>  (Wed Aug 12 17:20:55 2015) [sssd[be[default]]] [ad_subdomains_root_conn_done] (0x0080):
> No AD server is available, cannot get the subdomain list while offline
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [fo_resolve_service_timeout] (0x0080):
> Service resolving timeout reached
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [sdap_id_op_connect_done] (0x0020): Failed
> to connect, going offline (5 [Input/output error])
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [be_ptask_enable] (0x0080): Task [Check if
> online (periodic)]: already enabled
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [be_run_offline_cb] (0x0080): Going
> offline. Running callbacks.
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [ad_subdomains_root_conn_done] (0x0080):
> No AD server is available, cannot get the subdomain list
>  
>  After cleaning cache, sssd refuses to list all groups for user, etc.
>  
>  Is sssd going to be stable at some stage? This is really disappointing :( 
I'm sorry you're seeing issues. We didn't see them in our tests and
we'll work with you on fixing those.

I know this is not what you might expect from as stable distribution as
CentOS, but normally one of our upstream branches is very similar to
downstream. The 6.7 release is very similar to the sssd-1-12 branch. I
wonder if testing Lukas' COPR repositories with sssd-1-12 builds before
upgrading the machines would help catch the issues sooner?

[Lukas Slebodnik]

On (13/08/15 00:13), Jakub Hrozek wrote:
> On Wed, Aug 12, 2015 at 04:31:14PM +0000, Ondrej Valousek wrote:
> > Hi list,
> > 
> > Is sssd broken again in 6.7?
> > It was working fine for me as of 1.11.7 - but that was not official version.
Even the upstream 1.11.7 is not super stable. It contains bugs which caused
other issues for users with AD and there is not a workaround for them.
But they are fixed in 1.12.5
e.g. https://fedorahosted.org/sssd/ticket/2614
     https://fedorahosted.org/sssd/ticket/2592

> Now I upgraded to the latest one (1.12.4-47) hoping I'll be on the
> official fixed version, and it is even worse.
> > 
> > I am experiencing frequent messages like:
>
> Could you paste the lines before that to see why we are going offline?
>  
It would be good to provide debug messages with higher debug level;
ideal full (0xfff0)
@see also https://fedorahosted.org/sssd/wiki/Troubleshooting

LS

[Ondrej Valousek]

Ok,
I think I know what's going on here. I am hitting the bug I have submitted before:
https://fedorahosted.org/sssd/ticket/2702

But I am not behind a firewall now - some domain controllers are simply shut down.
If that's the case and service autodiscovery is enabled, it can easily happen that the
ad provider goes offline.

To me, this is really serious problem which renders the service discovery feature actually
unusable.
Ondrej

-----Original Message-----
From: sssd-users-bounces(a)lists.fedorahosted.org
[mailto:[e-mail]] On Behalf Of Jakub Hrozek
Sent: 13 August 2015 00:13
To: sssd-users(a)lists.fedorahosted.org
Subject: Re: [SSSD-users] SSSD seriously broken in RHEL 6.7 again?

On Wed, Aug 12, 2015 at 04:31:14PM +0000, Ondrej Valousek wrote:
>  Hi list,
>  
>  Is sssd broken again in 6.7?
>  It was working fine for me as of 1.11.7 - but that was not official version. Now I
> upgraded to the latest one (1.12.4-47) hoping I'll be on the official fixed version,
> and it is even worse.
>  
>  I am experiencing frequent messages like:
>  
Could you paste the lines before that to see why we are going offline?

>  (Wed Aug 12 17:20:55 2015) [sssd[be[default]]] [be_run_offline_cb]
> (0x0080): Going offline. Running callbacks.
>  (Wed Aug 12 17:20:55 2015) [sssd[be[default]]] 
>  [ad_subdomains_root_conn_done] (0x0080): No AD server is available, 
>  cannot get the subdomain list while offline (Wed Aug 12 17:21:01 2015) 
>  [sssd[be[default]]] [fo_resolve_service_timeout] (0x0080): Service 
>  resolving timeout reached (Wed Aug 12 17:21:01 2015) [sssd[be[default]]]
> [sdap_id_op_connect_done] (0x0020): Failed to connect, going offline (5 [Input/output
> error]) (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] [be_ptask_enable] (0x0080): Task
> [Check if online (periodic)]: already enabled (Wed Aug 12 17:21:01 2015)
> [sssd[be[default]]] [be_run_offline_cb] (0x0080): Going offline. Running callbacks.
>  (Wed Aug 12 17:21:01 2015) [sssd[be[default]]] 
>  [ad_subdomains_root_conn_done] (0x0080): No AD server is available, 
>  cannot get the subdomain list
>  
>  After cleaning cache, sssd refuses to list all groups for user, etc.
>  
>  Is sssd going to be stable at some stage? This is really disappointing 
>  :(
>  
I'm sorry you're seeing issues. We didn't see them in our tests and we'll
work with you on fixing those.

I know this is not what you might expect from as stable distribution as CentOS, but
normally one of our upstream branches is very similar to downstream. The 6.7 release is
very similar to the sssd-1-12 branch. I wonder if testing Lukas' COPR repositories
with sssd-1-12 builds before upgrading the machines would help catch the issues sooner?
_______________________________________________
sssd-users mailing list
sssd-users(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/sssd-users

-----

The information contained in this e-mail and in any attachments is confidential and is
designated solely for the attention of the intended recipient(s). If you are not an
intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or
any part thereof. If you have received this e-mail in error, please notify the sender by
return e-mail and delete all copies of this e-mail from your computer system(s). Please
direct any additional queries to: communications(a)s3group.com. Thank You. Silicon and
Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office:
South County Business Park, Leopardstown, Dublin 18.

[Jakub Hrozek]

On Thu, Aug 13, 2015 at 06:53:56AM +0000, Ondrej Valousek wrote:
>  Ok,
>  I think I know what's going on here. I am hitting the bug I have submitted before:
>  https://fedorahosted.org/sssd/ticket/2702
>  
>  But I am not behind a firewall now - some domain controllers are simply shut down.
>  If that's the case and service autodiscovery is enabled, it can easily happen that
> the ad provider goes offline.
>  
>  To me, this is really serious problem which renders the service discovery feature
> actually unusable.
>  Ondrej 
I see your point, but wouldn't this be solvable better with sites? ie
only advertise those DCs that the client can actually use rather than
working around it on the client side?

[Ondrej Valousek]

Sites won't help here  because of 2 reasons:

1. You start up the AD site discovery process sequentially connecting to ALL DCs that are
registered in SRV. This has to be done this way as you do not know yet to which site you
belong to.
If the random DC you pick up responds, you're lucky and you can finish the discovery
process and discover the site you belong to. 
If you are unlucky, the DC is unresponsive and you'll end up with AD provider
offline.

2. Once you discovered the site, there is still the risk that some DCs defined for the
site will be down or unresponsible - so the scenario above can happen as well.

Moreover it seems to me that SSSD is doing the site discovery too often. It should be done
once SSSD starts up or if we spot the IP address change - there is no reason to do it
otherwise. Nowadays I see it in logs on a regular basis.

Ondrej

>  To me, this is really serious problem which renders the service
> discovery feature actually unusable.
>  Ondrej
>  
I see your point, but wouldn't this be solvable better with sites? ie only advertise
those DCs that the client can actually use rather than working around it on the client
side?
_______________________________________________
sssd-users mailing list
sssd-users(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/sssd-users

-----

The information contained in this e-mail and in any attachments is confidential and is
designated solely for the attention of the intended recipient(s). If you are not an
intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or
any part thereof. If you have received this e-mail in error, please notify the sender by
return e-mail and delete all copies of this e-mail from your computer system(s). Please
direct any additional queries to: communications(a)s3group.com. Thank You. Silicon and
Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office:
South County Business Park, Leopardstown, Dublin 18.

[Jakub Hrozek]

>  On 13 Aug 2015, at 09:54, Ondrej Valousek
> <ondrej.valousek(a)s3group.com&gt; wrote:
>  
>  Sites won't help here  because of 2 reasons: 
Pavel, can you keep this issue in mind when you think about the refactoring of the
failover system?

>  
>  1. You start up the AD site discovery process sequentially connecting to ALL DCs that are
> registered in SRV. This has to be done this way as you do not know yet to which site you
> belong to.
>  If the random DC you pick up responds, you're lucky and you can finish the discovery
> process and discover the site you belong to. 
>  If you are unlucky, the DC is unresponsive and you'll end up with AD provider
> offline.
>  
>  2. Once you discovered the site, there is still the risk that some DCs defined for the
> site will be down or unresponsible - so the scenario above can happen as well.
>  
>  Moreover it seems to me that SSSD is doing the site discovery too often. It should be
> done once SSSD starts up or if we spot the IP address change - there is no reason to do it
> otherwise. Nowadays I see it in logs on a regular basis.
>  
>  Ondrej
>  
> > To me, this is really serious problem which renders the service discovery feature
> actually unusable.
> > Ondrej
>  
>  I see your point, but wouldn't this be solvable better with sites? ie only advertise
> those DCs that the client can actually use rather than working around it on the client
> side?
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>  
>  -----
>  
>  The information contained in this e-mail and in any attachments is confidential and is
> designated solely for the attention of the intended recipient(s). If you are not an
> intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or
> any part thereof. If you have received this e-mail in error, please notify the sender by
> return e-mail and delete all copies of this e-mail from your computer system(s). Please
> direct any additional queries to: communications(a)s3group.com. Thank You. Silicon and
> Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office:
> South County Business Park, Leopardstown, Dublin 18.
>  _______________________________________________
>  sssd-users mailing list
>  sssd-users(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/sssd-users

[Pavel Březina]

On 08/16/2015 06:22 PM, Jakub Hrozek wrote:
> > On 13 Aug 2015, at 09:54, Ondrej Valousek <ondrej.valousek(a)s3group.com&gt; wrote:
> >
> > Sites won't help here  because of 2 reasons:
>
>  Pavel, can you keep this issue in mind when you think about the refactoring of the
> failover system? 
Will do.
> >
> > 1. You start up the AD site discovery process sequentially connecting to ALL DCs that
> are registered in SRV. This has to be done this way as you do not know yet to which site
> you belong to.
> > If the random DC you pick up responds, you're lucky and you can finish the
> discovery process and discover the site you belong to.
> > If you are unlucky, the DC is unresponsive and you'll end up with AD provider
> offline.
> >
> > 2. Once you discovered the site, there is still the risk that some DCs defined for
> the site will be down or unresponsible - so the scenario above can happen as well.
> >
> > Moreover it seems to me that SSSD is doing the site discovery too often. It should be
> done once SSSD starts up or if we spot the IP address change - there is no reason to do it
> otherwise. Nowadays I see it in logs on a regular basis.
> >
> > Ondrej
> >
> >> To me, this is really serious problem which renders the service discovery feature
> actually unusable.
> >> Ondrej
> >
> > I see your point, but wouldn't this be solvable better with sites? ie only
> advertise those DCs that the client can actually use rather than working around it on the
> client side?
> > _______________________________________________
> > sssd-users mailing list
> > sssd-users(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/sssd-users
> >
> > -----
> >
> > The information contained in this e-mail and in any attachments is confidential and
> is designated solely for the attention of the intended recipient(s). If you are not an
> intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or
> any part thereof. If you have received this e-mail in error, please notify the sender by
> return e-mail and delete all copies of this e-mail from your computer system(s). Please
> direct any additional queries to: communications(a)s3group.com. Thank You. Silicon and
> Software Systems Limited (S3 Group). Registered in Ireland no. 378073. Registered Office:
> South County Business Park, Leopardstown, Dublin 18.
> > _______________________________________________
> > sssd-users mailing list
> > sssd-users(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/sssd-users
>