Sumit and Gordon,

You have given me much to think on and digest.  Thanks. 

Gordon, we religiously patch monthly. Except for sssd in July, where a new update sssd*-2.4.0-9.0.1.el8_4.1.x86_64  broke our env and we had to roll back the update to previous version sssd*-2.4.0-9.0.1.el8.x86_64 .  (We pushed a work-around and rolled out the new sssd version in Aug.)  This problem with automatic machine account renewal has been a problem for many months, server ops informs us.

I see Sumit is the author of SOURCES/sssd-x.x.x/src/providers/ad/ad_machine_pw_renewal.c.    I see it’s doing a be_ptask_create() to do the AD Machine PW renewal.

Sumit, we have dozens and dozens of dropped-off servers that we can survey.  All it takes is some slight time to find a machine account in our OU where 'passwordLastSet'  > 40 days.  For instance, using the powershell invocation that Gordon calls out.  (If it’s too old, chances are it’s a server decomm, where AD was not successfully cleaned up.  That happens – rarely.)

Yesterday, we surveyed one such server that had dropped of the domain.  We see that the msDS-KeyVersionNumber  (in AD) is one higher than the latest KVNO (in the local /etc/krb5.keytab file).   This indicates (to me) that AD was able to successfully update the machine account password, but this local be_task did not think AD successfully updated the entry, so it did not update the local /etc/krb5.keytab file with the new entry.

Again, this communication failure must be very infrequent, as it’s occurring only 0.3% of the time or so.  On random Linux servers – no discernible geographic or build location pattern.

BTW, we are not the AD admins.  We’re trying to get the content of the AD DC logs for the specified time period, but these logs are considered highly sensitive.  Also, I don’t know how frequently these logs cycle.

On a random test server, we set debug level == 5 and set ad_maximum_machine_account_password_age to less than passwordLastSet duration.   I.e., to trigger a machine account password renewal.  Then we restarted sssd.  Sure enough, 15 mins after sssd service restart, there was a machine account password renewal.  We saw it in the /etc/krb5.keytab entries.  But we didn’t see any indications of this in the sssd logs when we reviewed them.  (Of course, our password renewal was successful – so don’t expect much logging with successful be_ptasks).

Debug level 5 does not seem to fill up the /var/log filesystem to 100% like debug level 9 does.  We’ll keep monitoring disk space on this test server.  Until we track this down, we might set debug level 5 across all servers.

Spike