Am Wed, Sep 01, 2021 at 11:39:30AM -0500 schrieb Spike White:
> So to respond to my own email, but a co-worker did finally find some
> references to that bizarre name ZZZKBTDURBOL8.
>
> [root@nwpllv8bu100 post_install]# klist -kte
> Keytab name: FILE:/etc/krb5.keytab
> KVNO Timestamp           Principal
> ---- -------------------
> ------------------------------------------------------
>    2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 ZZZKBTDURBOL8$@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 host/
> zzzkbtdurbol8.amer.company.com@AMER.COMPANY.COM (aes128-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 host/
> zzzkbtdurbol8.amer.company.com@AMER.COMPANY.COM (aes256-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 host/ZZZKBTDURBOL8@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 host/ZZZKBTDURBOL8@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 RestrictedKrbHost/ZZZKBTDURBOL8@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 RestrictedKrbHost/ZZZKBTDURBOL8@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 RestrictedKrbHost/
> zzzkbtdurbol8.amer.company.com@AMER.COMPANY.COM (aes128-cts-hmac-sha1-96)
>    2 07/13/2021 16:42:17 RestrictedKrbHost/
> zzzkbtdurbol8.amer.company.com@AMER.COMPANY.COM (aes256-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM
> (DEPRECATED:arcfour-hmac)
>    3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 NWPLLV8BU100$@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM
> (DEPRECATED:arcfour-hmac)
>    3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 host/NWPLLV8BU100@AMER.COMPANY.COM
> (DEPRECATED:arcfour-hmac)
>    3 07/29/2021 13:38:27 host/NWPLLV8BU100@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 host/NWPLLV8BU100@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/NWPLLV8BU100@AMER.COMPANY.COM
> (DEPRECATED:arcfour-hmac)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/NWPLLV8BU100@AMER.COMPANY.COM
> (aes128-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/NWPLLV8BU100@AMER.COMPANY.COM
> (aes256-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/
> nwpllv8bu100.amer.company.com@AMER.COMPANY.COM (DEPRECATED:arcfour-hmac)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/
> nwpllv8bu100.amer.company.com@AMER.COMPANY.COM (aes128-cts-hmac-sha1-96)
>    3 07/29/2021 13:38:27 RestrictedKrbHost/
> nwpllv8bu100.amer.company.com@AMER.COMPANY.COM (aes256-cts-hmac-sha1-96)
>
> I realize this is reflecting the literal entries in the /etc/krb5.keytab
> file.  So it appears that when this VM was born (on July 13th), it was
> named zzzkbtdurbo18.amer.company.com.   (I see other supporting evidence
> for this).  On or before July 29th, it was renamed to final FQDN
> nwpllv8bu100.amer.company.com.  /etc/hostname, /etc/hosts,
> /etc/sysconfig/network etc were all updated and it was rejoined to AD.
>
> kinit -k works fine.  It picks up the current hostname and apparently uses
> host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM as its service
> principal.  Since there's a valid entry in /etc/krb5.keytab file for this,
> it uses this and all is good.  (I'm guessing it uses the 14th or 15th
> /etc/krb5.keytab file entry above.)
>
> sssd works, because it has this line:
>
> ldap_sasl_authid = host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM
>
> But when sssd invokes adcli update to refresh the machine account password,
> adcli update fails.
>
> Also, I see that adcli testjoin fails.
>
> [root@nwpllv8bu100 tmp]# adcli testjoin -D AMER.COMPANY.COM
> adcli: couldn't connect to AMER.COMPANY.COM domain: Couldn't authenticate
> as machine account: ZZZKBTDURBOL8: Preauthentication failed
> [root@nwpllv8bu100 tmp]#
>
> From a strace of this adcli testjoin, it appears that adcli is opening the
> /etc/krb5.keytab file to determine the default service principal to use and
> is pulling the old server name.  (instead of using the correct service
> principal, as kinit -k somehow does.)

Hi,

it looks like your environment is a bit special. I guess you have added
'host/nwpllv8bu100.amer.company.com@AMER.COMPANY.COM' to the
'userPrincipalName' LDAP attribute in the AD computer object for this
host.

# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
----
--------------------------------------------------------------------------
   2 MASTER$@CHILD.AD.VM
   2 MASTER$@CHILD.AD.VM
   2 host/MASTER@CHILD.AD.VM
   2 host/MASTER@CHILD.AD.VM
   2 host/master.client.vm@CHILD.AD.VM
   2 host/master.client.vm@CHILD.AD.VM
   2 RestrictedKrbHost/MASTER@CHILD.AD.VM
   2 RestrictedKrbHost/MASTER@CHILD.AD.VM
   2 RestrictedKrbHost/master.client.vm@CHILD.AD.VM
   2 RestrictedKrbHost/master.client.vm@CHILD.AD.VM
# kdestroy -A
#
#
# kinit -k
kinit: Client 'host/master.client.vm@CHILD.AD.VM' not found in Kerberos database while getting initial credentials
# kinit -k 'MASTER$@CHILD.AD.VM'


The reason is that 'kinit -k' constructs the principal by calling
gethostname() or similar, adding the 'host/' prefix and the realm. But
by default this principal in AD is only a service principal can cannot
be used to request a TGT as kinit does. AD only allows user principals
for request a TGT and this is by default 'SHORT$@AD.REALM'. If the
userPrincipalName attribute is set, this principal given here is allowed
as well.

That's why adcli is checking the keytab for a principal with a '$'
character and by default it uses the first it finds because it is
expected there is only one. Adding some heuristics in case there are
more '$' principals in the keytab like highest KVNO might help in some
cases but would fail in other cases so I think just using the first is
good enough.


>
> Maybe when sssd constructs this adcli update invocation, it's not passing
> the ldap_sasl_authid, so the adcli update is doing the above logic to pull
> the old server name?

Adding a new option to adcli and using the value from ldap_sasl_authid
might be a solution, I will think about it.

HTH

bye,
Sumit

>
> Sounds like an adcli problem.  Adcli should do as 'kinit -k' does when it's
> passed no explicit service principal.  Should dive into /etc/krb5.keytab
> file and use the most recent set of entries (KVNO = 3 in above example).
> Maybe derive the default service principal off the current FQDN and
> Kerberos realm?
>
> Spike
> PS As a general policy, we are not supposed to clone a VM and rename it to
> another FQDN/IP address.  I'll be trying to track down who did this and for
> what reason.
>
> On Wed, Sep 1, 2021 at 10:08 AM Spike White <spikewhitetx@gmail.com> wrote:
>
> > Ok, this is *very* illuminating!
> >
> > I see this in sssd_amer.company.com.log"
> >
> > (2021-09-01  3:44:46): [be[amer.company.com]]
> > [ad_machine_account_password_renewal_done] (0x1000): --- adcli output
> > start---
> > adcli: couldn't connect to amer.company.com domain: Couldn't authenticate
> > as machine account: ZZZKBTDURBOL8: Preauthentication failed
> > ---adcli output end---
> >
> > However, I don't find that host name ZZZKBTDURBOL8 anywhere on the
> > system.  (By company convention, servers named ZZZ* are test servers that
> > linux SEs spin up themselves).
> >
> > This server that's not renewing its creds is named:
> > nwpllv8bu100.amer.company.com.  it's a std dev server.  in
> > /etc/sssd/sssd.conf file, it has that as its sasl auth ID:
> >
> > [root@nwpllv8bu100 sssd]# grep sasl /etc/sssd/sssd.conf
> > ldap_sasl_authid = host/nwpllv8bu100.amer.dell.com@AMER.COMPANY.COM
> > [root@nwpllv8bu100 sssd]#
> >
> > If I do 'kinit -k',  the /etc/krb5.keytab file has that name as well:
> >
> > [root@nwpllv8bu100 sssd]# kinit -k
> > [root@nwpllv8bu100 sssd]# klist
> > Ticket cache: KCM:0
> > Default principal: host/nwpllv8bu100.amer.dell.com@AMER.COMPANY.COM
> >
> > Valid starting       Expires              Service principal
> > 09/01/2021 11:04:16  09/01/2021 21:04:16  krbtgt/
> > AMER.DELL.COM@AMER.COMPANY.COM
> >         renew until 09/08/2021 11:04:16
> > [root@nwpllv8bu100 sssd]#
> >
> > I searched /etc/sssd/sssd.conf -- no "zzz" or "ZZZ" string is anywhere in
> > there.   So where is sssd picking up this name ZZZKBTDURBOL8 and passing it
> > to adcli update?
> >
> > Spike
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Sep 1, 2021 at 2:46 AM Sumit Bose <sbose@redhat.com> wrote:
> >
> >> Am Tue, Aug 31, 2021 at 09:53:01PM +0200 schrieb Alexey Tikhonov:
> >> > On Tue, Aug 31, 2021 at 6:47 PM Spike White <spikewhitetx@gmail.com>
> >> wrote:
> >> >
> >> > > All,
> >> > >
> >> > > OK we have a query we run in AD for machine account passwords for a
> >> > > certain age.  In today's run, 31 - 32 days.  Then we verify it's
> >> pingable.
> >> > >
> >> > > We have found such one such suspicious candidate today (two actually,
> >> but
> >> > > the other Linux server is quite sick).  So one good research
> >> candidate.
> >> > > According to both AD and /etc/krb5.keytab file, the machine account
> >> > > password was last set on 7/29.  Today is 8/31, so that would be 32
> >> days.
> >> > > This 'automatic machine account keytab renewal'  background task
> >> should
> >> > > trigger again today.
> >> > >
> >> > > sssd service was last started 2 weeks ago and, by all appearances,
> >> appears
> >> > > healthy.  sssctl domain-status <domain> shows online, connected to AD
> >> > > servers (both domain and GC servers)..  All logins and group
> >> enumerations
> >> > > working as expected.
> >> > >
> >> > > Just now, we dynamically set the debug level to 9 with 'sssctl
> >> debug-level
> >> > > 9'.  This particular server is Oracle Linux 8.4,
> >> > > running sssd-*-2.4.0-9.0.1.el8_4.1.x86_64.   Installed July 13th,
> >> 2021.  So
> >> > > -- very recent sssd version.  (This problem occurs with both RHEL & OL
> >> > > 6/7/8, it's just today's candidate happens to be OL8.)
> >> > >
> >> > > We can't keep debug level 9 up for a great many days;  it swamps the
> >> > > /var/log filesystem.  But we can leave up for a few days.  We
> >> purposely did
> >> > > not restart sssd server as we know that would trigger a machine
> >> account
> >> > > renewal.
> >> > >
> >> > > Speaking of that -- from Sumit's sssd source code in
> >> > > ad_provider/ad_machine_pw_renewal.c, it appears that sssd is creating
> >> a
> >> > > back-end task to call external program /usr/sbin/adcli with certain
> >> args.
> >> > >  What string can I look for in which sssd log file (now that I have
> >> debug
> >> > > level 9 enabled) to tell me when this 'adcli update' task (aka
> >> 'automatic
> >> > > machine account keytab renewal')  is triggered?
> >> > >
> >> >
> >> > It seems SSSD itself only logs in case of errors. I didn't find any
> >> > explicit logs around `ad_machine_account_password_renewal_send()`.
> >> > But perhaps there will be something like "[be_ptask_execute] (0x0400):
> >> Task
> >> > [AD machine account password renewal]: executing task" from generic
> >> > be_ptask_* helpers in the sssd_$domain.log (I'm not sure).
> >> >
> >> > Also at this verbosity level `--verbose` should be supplied to adcli
> >> itself
> >> > and I guess output should be captured in sssd_$domain.log as well. I'm
> >> not
> >> > familiar with `adcli` internals, you can take a glance at
> >> > https://gitlab.freedesktop.org/realmd/adcli to find its log messages.
> >>
> >> Hi,
> >>
> >> if SSSD's debug_level is 7 or higher the '--verbose' option is set
> >> when calling adcli and the output is added to the backend logs. It will
> >> start with log message "--- adcli output start---".
> >>
> >> HTH
> >>
> >> bye,
> >> Sumit
> >>
> >> >
> >> >
> >> > >
> >> > > I'm less certain now that we've surveyed our env that this background
> >> > > 'adcli update' task is the reason behind 70 - 80 servers / month
> >> dropping
> >> > > off the domain.  It might be a slight contributor, but I find only a
> >> very
> >> > > few pingable servers with machine account last renewal date between
> >> 30 and
> >> > > 40 days.
> >> > >
> >> > > Yes, I can disable this default 30 day automatic update and roll my
> >> own
> >> > > 'adcli update' cron.  But that's a mass deployment, to fix what might
> >> not
> >> > > be the problem.   I want to verify this is the actual culprit before
> >> I take
> >> > > those drastic steps.
> >> > >
> >> > > Spike
> >> > >
> >> > >
> >>
> >> > _______________________________________________
> >> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> >> > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> >> > Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> > List Archives:
> >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> >> > Do not reply to spam on the list, report it:
> >> https://pagure.io/fedora-infrastructure
> >> _______________________________________________
> >> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> >> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >> https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> >> Do not reply to spam on the list, report it:
> >> https://pagure.io/fedora-infrastructure
> >>
> >

> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/sssd-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure