Does sssd (on RHEL8 and RHEL9) RPM have an implied dependency on sssd-kcm RPM?

[Spike White]

All,

Is there a packaging problem on the latest version of RHEL8 sssd?

On several of our RHEL8 servers during the last update cycle, sssd logins
start failing.  It appears to be when upgrading to version
sssd-2.9.1-4.0.1.el8_9.x86_64.

Upon deep dive, it turns out the sssd-kcm RPM is missing.

The sssd-kcm service fails to start, complaining about missing symbols in
/usr/libexec/sssd_kcm file.

Sure enough, there is a problem with that file:

[root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm

file /usr/libexec/sssd/sssd_kcm is not owned by any package


Once the sssd-kcm RPM is installed,  then this looks normal:


[root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm

sssd-kcm-2.9.1-4.0.1.el8_9.x86_64


Then sssd-kcm service will restart fine.  and then sssd logins again work
fine.


Also, it appears that a yum downgrade of sssd seems to work too --
presumably because this downgrade also triggers an install of sssd-kcm RPM.


I see that the sssd RPM has no direct explicit RPM dependency on sssd-kcm.
But is there some indirect or implied dependency that was missed on this
latest packaging for sssd-2.9.1-4.0.1.el8_9.x86_64?


We've run sssd for years and never had significant sssd-kcm problems until
last month.


BTW, another clue that it was KCM is that 'kinit -k' fails with error:


# knit -k

Cannot update default cache


But if you tell it to store creds under /tmp, it works:


# export KRB5CCNAME="FILE:/tmp/krb5cc_0"

# kinit -k


Spike

[Alejandro Lopez]

Hi Spike,

During the upgrade KCM was not updated. How was the upgrade done? Just a
simple `dnf upgrade` or something more complex?

There is a /usr/libexec/sssd/sssd_kcm file that belongs to no RPM. Are you
sure there were no errors during the update? No KCM RPM installed, even the
previous version?

Once the RPM is manually installed, KCM successfully starts. Does `kinit`
still fail at this point? If KCM fails to start, it is normal that `kinit`
also fails if it is configured to use KCM.

Alejandro

On Wed, Jan 10, 2024 at 11:02 PM Spike White <spikewhitetx(a)gmail.com&gt; wrote:

>  All,
>
>  Is there a packaging problem on the latest version of RHEL8 sssd?
>
>  On several of our RHEL8 servers during the last update cycle, sssd logins
>  start failing.  It appears to be when upgrading to version
>  sssd-2.9.1-4.0.1.el8_9.x86_64.
>
>  Upon deep dive, it turns out the sssd-kcm RPM is missing.
>
>  The sssd-kcm service fails to start, complaining about missing symbols in
>  /usr/libexec/sssd_kcm file.
>
>  Sure enough, there is a problem with that file:
>
>  [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>
>  file /usr/libexec/sssd/sssd_kcm is not owned by any package
>
>
>  Once the sssd-kcm RPM is installed,  then this looks normal:
>
>
>  [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>
>  sssd-kcm-2.9.1-4.0.1.el8_9.x86_64
>
>
>  Then sssd-kcm service will restart fine.  and then sssd logins again work
>  fine.
>
>
>  Also, it appears that a yum downgrade of sssd seems to work too --
>  presumably because this downgrade also triggers an install of sssd-kcm RPM.
>
>
>  I see that the sssd RPM has no direct explicit RPM dependency on
>  sssd-kcm.  But is there some indirect or implied dependency that was missed
>  on this latest packaging for sssd-2.9.1-4.0.1.el8_9.x86_64?
>
>
>  We've run sssd for years and never had significant sssd-kcm problems until
>  last month.
>
>
>  BTW, another clue that it was KCM is that 'kinit -k' fails with error:
>
>
>  # knit -k
>
>  Cannot update default cache
>
>
>  But if you tell it to store creds under /tmp, it works:
>
>
>  # export KRB5CCNAME="FILE:/tmp/krb5cc_0"
>
>  # kinit -k
>
>
>  Spike
>
>  --
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Alejandro

[Spike White]

Alejandro,

I don't have any failing servers right in front of me now, so this is from
memory.  (all reported servers fixed).

>  During the upgrade KCM was not updated. How was the upgrade done?
> Just a 
simple `dnf upgrade` or something more complex?

Yes, dnf update of all outstanding RPMs.

>  There is a /usr/libexec/sssd/sssd_kcm file that belongs to no RPM.
> Are 
you sure there were no errors during the update? No KCM RPM installed, even
the previous version?

Prior to latest round of patching, app teams were not complaining about any
sssd login issues.  After patching, they were.

>  Once the RPM is manually installed, KCM successfully starts. Does
> `kinit` 
still fail at this point? If KCM fails to start, it is normal that `kinit`
also fails if it is configured to use KCM.

Once sssd-kcm manually installed, kinit works.  there's a file
/etc/krb5.conf.d/kcm_default_ccache with this content:

# This file should normally be installed by your distribution into a
# directory that is included from the Kerberos configuration file
(/etc/krb5.conf)
# On Fedora/RHEL/CentOS, this is /etc/krb5.conf.d/
#
# To enable the KCM credential cache enable the KCM socket and the service:
#   systemctl enable sssd-kcm.socket
#   systemctl start sssd-kcm.socket
#
# To disable the KCM credential cache, comment out the following lines.

[libdefaults]
    default_ccache_name = KCM:


So with the sssd-kcm RPM missing, kinit -k fails.  I can make it succeed
(even with sssd-kcm service not running) via:

# export KRB5CCNAME="FILE:/tmp/krb5cc_0"

# kinit -k


This is overriding the setting in /etc/krb5.conf.d/kcm_default_ccache
above.

BTW, /etc/krb5.conf.d/kcm_default_ccache file comes from sssd-kcm RPM so
that tells me that certainly at one time (prior to patching) the sssd-kcm
RPM was on the server.

Spike


On Thu, Jan 11, 2024 at 3:27 AM Alejandro Lopez <allopez(a)redhat.com&gt; wrote:

> Hi Spike,
>
>  During the upgrade KCM was not updated. How was the upgrade done?
> Just a 
> simple `dnf upgrade` or something more complex?
>
> There is a /usr/libexec/sssd/sssd_kcm file that belongs to no RPM. Are you
> sure there were no errors during the update? No KCM RPM installed, even the
> previous version?
>
>  Once the RPM is manually installed, KCM successfully starts. Does
> `kinit` 
> still fail at this point? If KCM fails to start, it is normal that
`kinit`
> also fails if it is configured to use KCM.
>
> Alejandro
>
> On Wed, Jan 10, 2024 at 11:02 PM Spike White <spikewhitetx(a)gmail.com&gt;
> wrote:
>
>> All,
>>
>> Is there a packaging problem on the latest version of RHEL8 sssd?
>>
>> On several of our RHEL8 servers during the last update cycle, sssd logins
>> start failing.  It appears to be when upgrading to version
>> sssd-2.9.1-4.0.1.el8_9.x86_64.
>>
>> Upon deep dive, it turns out the sssd-kcm RPM is missing.
>>
>> The sssd-kcm service fails to start, complaining about missing symbols in
>> /usr/libexec/sssd_kcm file.
>>
>> Sure enough, there is a problem with that file:
>>
>> [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>>
>> file /usr/libexec/sssd/sssd_kcm is not owned by any package
>>
>>
>> Once the sssd-kcm RPM is installed,  then this looks normal:
>>
>>
>> [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>>
>> sssd-kcm-2.9.1-4.0.1.el8_9.x86_64
>>
>>
>> Then sssd-kcm service will restart fine.  and then sssd logins again work
>> fine.
>>
>>
>> Also, it appears that a yum downgrade of sssd seems to work too --
>> presumably because this downgrade also triggers an install of sssd-kcm RPM.
>>
>>
>> I see that the sssd RPM has no direct explicit RPM dependency on
>> sssd-kcm.  But is there some indirect or implied dependency that was missed
>> on this latest packaging for sssd-2.9.1-4.0.1.el8_9.x86_64?
>>
>>
>> We've run sssd for years and never had significant sssd-kcm problems
>> until last month.
>>
>>
>> BTW, another clue that it was KCM is that 'kinit -k' fails with error:
>>
>>
>> # knit -k
>>
>> Cannot update default cache
>>
>>
>> But if you tell it to store creds under /tmp, it works:
>>
>>
>> # export KRB5CCNAME="FILE:/tmp/krb5cc_0"
>>
>> # kinit -k
>>
>>
>> Spike
>>
>> --
>> _______________________________________________
>> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>> https://lists.fedorahosted.org/archives/list/[e-mail]...
>> Do not reply to spam, report it:
>> https://pagure.io/fedora-infrastructure/new_issue
>>
>
>
> --
> Alejandro
> --
> _______________________________________________
> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>

[Alejandro Lopez]

Spike,

The fact that sssd-kcm was present but didn't belong to no RPM means
something went wrong somewhere/sometime. Maybe, as Alexey said, the RPM was
once installed, but it was removed at some time. How did the files remain,
I cannot tell. That's not normal, but neither is it normal that it belongs
to no RPM. Maybe it was once installed by building from the sources without
using the RPMs?

My guess is that because the package was not installed, it was not updated;
but all the common libraries were. This explains the missing symbols. This
situation changed when the package was installed.

But all we can do is guess, because this cannot be reproduced and we don't
know the exact situation before all this happened.

HTH,
Alejandro.

On Thu, Jan 11, 2024 at 5:46 PM Spike White <spikewhitetx(a)gmail.com&gt; wrote:

>  Alejandro,
>
>  I don't have any failing servers right in front of me now, so this is from
>  memory.  (all reported servers fixed).
>
>  > During the upgrade KCM was not updated. How was the upgrade done? Just a
>  simple `dnf upgrade` or something more complex?
>
>  Yes, dnf update of all outstanding RPMs.
>
>  > There is a /usr/libexec/sssd/sssd_kcm file that belongs to no RPM. Are
>  you sure there were no errors during the update? No KCM RPM installed, even
>  the previous version?
>
>  Prior to latest round of patching, app teams were not complaining about
>  any sssd login issues.  After patching, they were.
>
>  > Once the RPM is manually installed, KCM successfully starts. Does
>  `kinit` still fail at this point? If KCM fails to start, it is normal that
>  `kinit` also fails if it is configured to use KCM.
>
>  Once sssd-kcm manually installed, kinit works.  there's a file
>  /etc/krb5.conf.d/kcm_default_ccache with this content:
>
>  # This file should normally be installed by your distribution into a
>  # directory that is included from the Kerberos configuration file
>  (/etc/krb5.conf)
>  # On Fedora/RHEL/CentOS, this is /etc/krb5.conf.d/
>  #
>  # To enable the KCM credential cache enable the KCM socket and the service:
>  #   systemctl enable sssd-kcm.socket
>  #   systemctl start sssd-kcm.socket
>  #
>  # To disable the KCM credential cache, comment out the following lines.
>
>  [libdefaults]
>      default_ccache_name = KCM:
>
>
>  So with the sssd-kcm RPM missing, kinit -k fails.  I can make it succeed
>  (even with sssd-kcm service not running) via:
>
>  # export KRB5CCNAME="FILE:/tmp/krb5cc_0"
>
>  # kinit -k
>
>
>  This is overriding the setting in /etc/krb5.conf.d/kcm_default_ccache
>  above.
>
>  BTW, /etc/krb5.conf.d/kcm_default_ccache file comes from sssd-kcm RPM so
>  that tells me that certainly at one time (prior to patching) the sssd-kcm
>  RPM was on the server.
>
>  Spike
>
>
>  On Thu, Jan 11, 2024 at 3:27 AM Alejandro Lopez <allopez(a)redhat.com&gt;
>  wrote:
>
> > Hi Spike,
> >
> > During the upgrade KCM was not updated. How was the upgrade done? Just a
> > simple `dnf upgrade` or something more complex?
> >
> > There is a /usr/libexec/sssd/sssd_kcm file that belongs to no RPM. Are
> > you sure there were no errors during the update? No KCM RPM installed, even
> > the previous version?
> >
> > Once the RPM is manually installed, KCM successfully starts. Does `kinit`
> > still fail at this point? If KCM fails to start, it is normal that `kinit`
> > also fails if it is configured to use KCM.
> >
> > Alejandro
> >
> > On Wed, Jan 10, 2024 at 11:02 PM Spike White <spikewhitetx(a)gmail.com&gt;
> > wrote:
> >
> >> All,
> >>
> >> Is there a packaging problem on the latest version of RHEL8 sssd?
> >>
> >> On several of our RHEL8 servers during the last update cycle, sssd
> >> logins start failing.  It appears to be when upgrading to version
> >> sssd-2.9.1-4.0.1.el8_9.x86_64.
> >>
> >> Upon deep dive, it turns out the sssd-kcm RPM is missing.
> >>
> >> The sssd-kcm service fails to start, complaining about missing symbols
> >> in /usr/libexec/sssd_kcm file.
> >>
> >> Sure enough, there is a problem with that file:
> >>
> >> [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
> >>
> >> file /usr/libexec/sssd/sssd_kcm is not owned by any package
> >>
> >>
> >> Once the sssd-kcm RPM is installed,  then this looks normal:
> >>
> >>
> >> [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
> >>
> >> sssd-kcm-2.9.1-4.0.1.el8_9.x86_64
> >>
> >>
> >> Then sssd-kcm service will restart fine.  and then sssd logins again
> >> work fine.
> >>
> >>
> >> Also, it appears that a yum downgrade of sssd seems to work too --
> >> presumably because this downgrade also triggers an install of sssd-kcm RPM.
> >>
> >>
> >> I see that the sssd RPM has no direct explicit RPM dependency on
> >> sssd-kcm.  But is there some indirect or implied dependency that was missed
> >> on this latest packaging for sssd-2.9.1-4.0.1.el8_9.x86_64?
> >>
> >>
> >> We've run sssd for years and never had significant sssd-kcm problems
> >> until last month.
> >>
> >>
> >> BTW, another clue that it was KCM is that 'kinit -k' fails with error:
> >>
> >>
> >> # knit -k
> >>
> >> Cannot update default cache
> >>
> >>
> >> But if you tell it to store creds under /tmp, it works:
> >>
> >>
> >> # export KRB5CCNAME="FILE:/tmp/krb5cc_0"
> >>
> >> # kinit -k
> >>
> >>
> >> Spike
> >>
> >> --
> >> _______________________________________________
> >> sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> >> To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam, report it:
> >> https://pagure.io/fedora-infrastructure/new_issue
> >>
> >
> >
> > --
> > Alejandro
> > --
> > _______________________________________________
> > sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam, report it:
> > https://pagure.io/fedora-infrastructure/new_issue
> >
>  --
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Alejandro

[Alexey Tikhonov]

Hi,

On Wed, Jan 10, 2024 at 11:02 PM Spike White <spikewhitetx(a)gmail.com&gt; wrote:
>  All,
>
>  Is there a packaging problem on the latest version of RHEL8 sssd?
>
>  On several of our RHEL8 servers during the last update cycle, sssd logins start failing. 
> It appears to be when upgrading to version sssd-2.9.1-4.0.1.el8_9.x86_64.
>
>  Upon deep dive, it turns out the sssd-kcm RPM is missing.
>
>  The sssd-kcm service fails to start, complaining about missing symbols in
> /usr/libexec/sssd_kcm file.
>
>  Sure enough, there is a problem with that file:
>
>  [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>
>  file /usr/libexec/sssd/sssd_kcm is not owned by any package
>
>
>  Once the sssd-kcm RPM is installed,  then this looks normal:
>
>
>  [root@mftplat1wplp105 ~]# rpm -qf /usr/libexec/sssd/sssd_kcm
>
>  sssd-kcm-2.9.1-4.0.1.el8_9.x86_64
>
>
>  Then sssd-kcm service will restart fine.  and then sssd logins again work fine.
>
>
>  Also, it appears that a yum downgrade of sssd seems to work too -- presumably because
> this downgrade also triggers an install of sssd-kcm RPM.
>
>
>  I see that the sssd RPM has no direct explicit RPM dependency on sssd-kcm.  But is there
> some indirect or implied dependency that was missed on this latest packaging for
> sssd-2.9.1-4.0.1.el8_9.x86_64? 
No, SSSD meta package doesn't require sssd_kcm:
https://gitlab.com/redhat/centos-stream/rpms/sssd/-/blob/c8s/sssd.spec?re...
and shouldn't - SSSD doesn't care what ccache type
('default_ccache_name') is used by libkrb5.

There is a dependency other way round though:
https://gitlab.com/redhat/centos-stream/rpms/sssd/-/blob/c8s/sssd.spec?re...
 - sssd-kcm uses some shared libraries with SSSD.

What you quote looks like sssd-kcm package was once installed, then
removed, but files were left (binary and
/etc/krb5.conf.d/kcm_default_ccache)

Can you reproduce this?

Maybe dnf logs can give a clue...

>  We've run sssd for years and never had significant sssd-kcm problems until last
> month.
>
>
>  BTW, another clue that it was KCM is that 'kinit -k' fails with error:
>
>
>  # knit -k
>
>  Cannot update default cache
>
>
>  But if you tell it to store creds under /tmp, it works:
>
>
>  # export KRB5CCNAME="FILE:/tmp/krb5cc_0"
>
>  # kinit -k
>
>
>  Spike
>