I don't know how but somehow /etc/krb5.keytab was deleted on my ipa server. As a result my sssd service is dead because the keytab file is missing
I understand the process to fix this is: 1. kinit admin and provide the password 2. ipa-getkeytab -s <FreeIPA server> -p host/<hostname>@REALM -k <keytab file>.
if command #2 is successful you should get a keytab file and you can run systemctl restart sssd and all will be happy again.
My problem is that when running command #2 I get an error if I specify -p host/ for the service prinicpal. SASL bind failed invalid credentials failed to bind to server Retrying with pre-4.0 keytab retrieval method...
If I specify -p ldap/ I can get it to complete and a keytab gets generated.
As a side note I have a bunch of ipa-healthcheck erors but maybe those are mostly related to the fact that sssd is not running and keytab is missing. The affected server is my primary but I do have a replica that is in good heatlh except for one healthcheck issue for dns.
Both servers contain the following roles: AD trust agent AD trust controller CA server DNS server
The IPA server also has a one way trust established with AD.
What is the correct principal to specify? Is it host or ldap or dns or something else?
Jeremy Tourville via FreeIPA-users wrote:
I don't know how but somehow /etc/krb5.keytab was deleted on my ipa server. As a result my sssd service is dead because the keytab file is missing
I understand the process to fix this is:
- kinit admin and provide the password
- ipa-getkeytab -s <FreeIPA server> -p host/<hostname>@REALM -k <keytab file>.
if command #2 is successful you should get a keytab file and you can run systemctl restart sssd and all will be happy again.
My problem is that when running command #2 I get an error if I specify -p host/ for the service prinicpal. SASL bind failed invalid credentials failed to bind to server Retrying with pre-4.0 keytab retrieval method...
If I specify -p ldap/ I can get it to complete and a keytab gets generated.
As a side note I have a bunch of ipa-healthcheck erors but maybe those are mostly related to the fact that sssd is not running and keytab is missing. The affected server is my primary but I do have a replica that is in good heatlh except for one healthcheck issue for dns.
Both servers contain the following roles: AD trust agent AD trust controller CA server DNS server
The IPA server also has a one way trust established with AD.
What is the correct principal to specify? Is it host or ldap or dns or something else?
host is the correct principal. Have you tried specifying the other server with -s in ipa-getkeytab?
Does the IPA server host entry still exist?
rob
Does the IPA server host entry still exist?
Yes, I see it under Identity>Hosts
Specifying the other server did work. SSSD is working again. [root@gsil-ipa01 etc]# ipa-getkeytab -s gsil-ipa02.idm.x.x -p host/gsil-ipa01.idm.x.x@IDM.x.x -k /etc/krb5.keytab
After doing this I restarted the server and tried to run health check again. Now healthcheck has a lot of errors. Where do you suggest I start?
I was also reading that "If your IdM topology contains an integrated CA, one server has the role of the Certificate revocation list (CRL) publisher server and one server has the role of the CA renewal server.
By default, the first CA server installed fulfills these two roles..." It was my first installed server that failed. Should I move the roles to my replica? (Assuming I can)
Here is the healthcheck- I presume that if I can fix dirsrv it will help clear up many of the other issues. Let me know if you think there is something more critical to be fixed first.
[root@gsil-ipa01 ~]# ipa-healthcheck --failures-only caSigningCert External CA not found, assuming 3rd party [ { "source": "ipahealthcheck.meta.services", "check": "dirsrv", "result": "ERROR", "uuid": "c7a2bf32-5878-44f5-b7a5-87b69e4149fa", "when": "20230316130703Z", "duration": "498.704933", "kw": { "status": false, "msg": "dirsrv: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "httpd", "result": "ERROR", "uuid": "edf67606-4326-4e37-b860-dc3992eb3bc7", "when": "20230316130703Z", "duration": "0.105259", "kw": { "status": false, "msg": "httpd: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "ipa_custodia", "result": "ERROR", "uuid": "87418dcb-c07c-4914-9296-ae5a2baca99d", "when": "20230316130703Z", "duration": "0.109029", "kw": { "status": false, "msg": "ipa-custodia: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "ipa_dnskeysyncd", "result": "ERROR", "uuid": "97d08885-7dac-4c60-a447-b669a4cc6e09", "when": "20230316130703Z", "duration": "0.100007", "kw": { "status": false, "msg": "ipa-dnskeysyncd: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "ipa_otpd", "result": "ERROR", "uuid": "c83f8cd2-6553-4934-99dd-de4cfa3f515f", "when": "20230316130703Z", "duration": "0.103143", "kw": { "status": false, "msg": "ipa-otpd: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "kadmin", "result": "ERROR", "uuid": "657130bf-396f-48a1-968f-3f89170313a3", "when": "20230316130703Z", "duration": "0.104496", "kw": { "status": false, "msg": "kadmin: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "krb5kdc", "result": "ERROR", "uuid": "5e82b86c-ac90-49f9-a8fe-1a8c35909f91", "when": "20230316130703Z", "duration": "0.097375", "kw": { "status": false, "msg": "krb5kdc: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "named", "result": "ERROR", "uuid": "3ca8751b-803a-46af-a849-fcb309eb65db", "when": "20230316130703Z", "duration": "0.093471", "kw": { "status": false, "msg": "named: not running" } }, { "source": "ipahealthcheck.meta.services", "check": "pki_tomcatd", "result": "ERROR", "uuid": "9c57a360-bd4c-48be-af8a-6e738c79c486", "when": "20230316130703Z", "duration": "0.002969", "kw": { "status": false, "msg": "pki_tomcatd: not running" } }, { "source": "ipahealthcheck.ipa.files", "check": "IPAFileCheck", "result": "WARNING", "uuid": "4a9001d5-1ad5-4a4c-99fd-48273dd9f822", "when": "20230316130707Z", "duration": "0.005756", "kw": { "key": "_var_log_ipaupgrade.log_mode", "path": "/var/log/ipaupgrade.log", "type": "mode", "expected": "0600", "got": "0644", "msg": "Permissions of /var/log/ipaupgrade.log are too permissive: 0644 and should be 0600" } } ]
UPDATE: I did a little more troubleshooting and was able to get dirsrv to start. Now I need to figure out why named service won't start. Here's the output from starting services and ipa-healthcheck. I presume several of the healthcheck failures are due to named service not running. Can anyone confirm?
[root@gsil-ipa01 ipa]# ipactl status Directory Service: STOPPED Directory Service must be running in order to obtain status of other services [root@gsil-ipa01 ipa]# ipactl start --ignore-service-failures Existing service file detected! Assuming stale, cleaning and proceeding Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Failed to start named Service Forced start, ignoring named Service, continuing normal operation Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting smb Service Starting winbind Service Starting ipa-otpd Service Starting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful [root@gsil-ipa01 ipa]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING 1 service(s) are not running [root@gsil-ipa01 ipa]# ipa-healthcheck --failures-only caSigningCert External CA not found, assuming 3rd party [ { "source": "ipahealthcheck.meta.services", "check": "named", "result": "ERROR", "uuid": "b5bfa450-77f4-4655-a4e2-fccbf88aa43a", "when": "20230316153125Z", "duration": "0.111160", "kw": { "status": false, "msg": "named: not running" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "CRITICAL", "uuid": "dcaa538c-a5e2-4247-9210-d6047a0d65f5", "when": "20230316153132Z", "duration": "0.281251", "kw": { "key": "DSREPLLE0001", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (metogsil-ipa02.idm.x.xl) under "dc=idm,dc=x,dc=x" is not in synchronization." } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "CRITICAL", "uuid": "556f572a-0ee9-42fa-8c06-b90e33ed961d", "when": "20230316153132Z", "duration": "0.281301", "kw": { "key": "DSREPLLE0001", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catogsil-ipa02.idm.x.x) under "o=ipaca" is not in synchronization." } }, { "source": "ipahealthcheck.ipa.dna", "check": "IPADNARangeCheck", "result": "CRITICAL", "uuid": "7b88f564-dac5-4191-96ec-b9ad922c0f5e", "when": "20230316153142Z", "duration": "0.027683", "kw": { "exception": "Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Preauthentication failed)" } }, { "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "WARNING", "uuid": "6b0bc0c1-d505-4f5a-944d-42dd044b2365", "when": "20230316153426Z", "duration": "164.364540", "kw": { "msg": "Got {count} ipa-ca A records, expected {expected}", "count": 1, "expected": 2 } }, { "source": "ipahealthcheck.ipa.files", "check": "IPAFileCheck", "result": "WARNING", "uuid": "ea3fcb5d-a280-4a29-ab5b-60abe15febdb", "when": "20230316153426Z", "duration": "0.003201", "kw": { "key": "_var_log_ipaupgrade.log_mode", "path": "/var/log/ipaupgrade.log", "type": "mode", "expected": "0600", "got": "0644", "msg": "Permissions of /var/log/ipaupgrade.log are too permissive: 0644 and should be 0600" } }, { "source": "ipahealthcheck.ipa.host", "check": "IPAHostKeytab", "result": "ERROR", "uuid": "9e43e0d9-7143-40b1-8411-c0aa4b53bb1e", "when": "20230316153426Z", "duration": "0.027001", "kw": { "msg": "Failed to obtain host TGT: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638936): Preauthentication failed" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustDomainsCheck", "result": "ERROR", "uuid": "a0ed3f4b-c409-42e4-b730-d9964ed46f64", "when": "20230316153427Z", "duration": "0.336395", "kw": { "key": "domain-list", "sssctl": "/usr/sbin/sssctl", "sssd_domains": "", "trust_domains": "gx.x", "msg": "{sssctl} {key} reports mismatch: sssd domains {sssd_domains} trust domains {trust_domains}" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustCatalogCheck", "result": "WARNING", "uuid": "fd1ff67b-48b3-49dd-a3b4-32631a51672f", "when": "20230316153427Z", "duration": "0.013619", "kw": { "key": "S-1-5-21-3568498085-2952124370-1649233135", "error": "returned nothing", "msg": "Look up of {key} {error}" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustCatalogCheck", "result": "ERROR", "uuid": "c478454c-f94c-4089-ade4-7c3bd73d6b65", "when": "20230316153427Z", "duration": "0.127239", "kw": { "key": "domain-status", "error": "CalledProcessError(Command ['/usr/sbin/sssctl', 'domain-status', 'gx.x', '--active-server'] returned non-zero exit status 1: 'Unable to get online status\n')", "msg": "Execution of {key} failed: {error}" } } ]
Jeremy Tourville via FreeIPA-users wrote:
UPDATE: I did a little more troubleshooting and was able to get dirsrv to start. Now I need to figure out why named service won't start. Here's the output from starting services and ipa-healthcheck. I presume several of the healthcheck failures are due to named service not running. Can anyone confirm?
It's likely. Kerberos and TLS rely on working name resolution. If your server has a valid entry in /etc/hosts that may mitigate some issues but but I'd still focus on getting named to start as a first step.
rob
[root@gsil-ipa01 ipa]# ipactl status Directory Service: STOPPED Directory Service must be running in order to obtain status of other services [root@gsil-ipa01 ipa]# ipactl start --ignore-service-failures Existing service file detected! Assuming stale, cleaning and proceeding Starting Directory Service Starting krb5kdc Service Starting kadmin Service Starting named Service Failed to start named Service Forced start, ignoring named Service, continuing normal operation Starting httpd Service Starting ipa-custodia Service Starting pki-tomcatd Service Starting smb Service Starting winbind Service Starting ipa-otpd Service Starting ipa-dnskeysyncd Service ipa: INFO: The ipactl command was successful [root@gsil-ipa01 ipa]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING smb Service: RUNNING winbind Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING 1 service(s) are not running [root@gsil-ipa01 ipa]# ipa-healthcheck --failures-only caSigningCert External CA not found, assuming 3rd party [ { "source": "ipahealthcheck.meta.services", "check": "named", "result": "ERROR", "uuid": "b5bfa450-77f4-4655-a4e2-fccbf88aa43a", "when": "20230316153125Z", "duration": "0.111160", "kw": { "status": false, "msg": "named: not running" } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "CRITICAL", "uuid": "dcaa538c-a5e2-4247-9210-d6047a0d65f5", "when": "20230316153132Z", "duration": "0.281251", "kw": { "key": "DSREPLLE0001", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (metogsil-ipa02.idm.x.xl) under "dc=idm,dc=x,dc=x" is not in synchronization." } }, { "source": "ipahealthcheck.ds.replication", "check": "ReplicationCheck", "result": "CRITICAL", "uuid": "556f572a-0ee9-42fa-8c06-b90e33ed961d", "when": "20230316153132Z", "duration": "0.281301", "kw": { "key": "DSREPLLE0001", "items": [ "Replication", "Agreement" ], "msg": "The replication agreement (catogsil-ipa02.idm.x.x) under "o=ipaca" is not in synchronization." } }, { "source": "ipahealthcheck.ipa.dna", "check": "IPADNARangeCheck", "result": "CRITICAL", "uuid": "7b88f564-dac5-4191-96ec-b9ad922c0f5e", "when": "20230316153142Z", "duration": "0.027683", "kw": { "exception": "Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Preauthentication failed)" } }, { "source": "ipahealthcheck.ipa.idns", "check": "IPADNSSystemRecordsCheck", "result": "WARNING", "uuid": "6b0bc0c1-d505-4f5a-944d-42dd044b2365", "when": "20230316153426Z", "duration": "164.364540", "kw": { "msg": "Got {count} ipa-ca A records, expected {expected}", "count": 1, "expected": 2 } }, { "source": "ipahealthcheck.ipa.files", "check": "IPAFileCheck", "result": "WARNING", "uuid": "ea3fcb5d-a280-4a29-ab5b-60abe15febdb", "when": "20230316153426Z", "duration": "0.003201", "kw": { "key": "_var_log_ipaupgrade.log_mode", "path": "/var/log/ipaupgrade.log", "type": "mode", "expected": "0600", "got": "0644", "msg": "Permissions of /var/log/ipaupgrade.log are too permissive: 0644 and should be 0600" } }, { "source": "ipahealthcheck.ipa.host", "check": "IPAHostKeytab", "result": "ERROR", "uuid": "9e43e0d9-7143-40b1-8411-c0aa4b53bb1e", "when": "20230316153426Z", "duration": "0.027001", "kw": { "msg": "Failed to obtain host TGT: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2529638936): Preauthentication failed" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustDomainsCheck", "result": "ERROR", "uuid": "a0ed3f4b-c409-42e4-b730-d9964ed46f64", "when": "20230316153427Z", "duration": "0.336395", "kw": { "key": "domain-list", "sssctl": "/usr/sbin/sssctl", "sssd_domains": "", "trust_domains": "gx.x", "msg": "{sssctl} {key} reports mismatch: sssd domains {sssd_domains} trust domains {trust_domains}" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustCatalogCheck", "result": "WARNING", "uuid": "fd1ff67b-48b3-49dd-a3b4-32631a51672f", "when": "20230316153427Z", "duration": "0.013619", "kw": { "key": "S-1-5-21-3568498085-2952124370-1649233135", "error": "returned nothing", "msg": "Look up of {key} {error}" } }, { "source": "ipahealthcheck.ipa.trust", "check": "IPATrustCatalogCheck", "result": "ERROR", "uuid": "c478454c-f94c-4089-ade4-7c3bd73d6b65", "when": "20230316153427Z", "duration": "0.127239", "kw": { "key": "domain-status", "error": "CalledProcessError(Command ['/usr/sbin/sssctl', 'domain-status', 'gx.x', '--active-server'] returned non-zero exit status 1: 'Unable to get online status\n')", "msg": "Execution of {key} failed: {error}" } } ] _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
I have noted that klist and kvno don't match for the keytab I fetched earlier. Could this cause issues with named or are those two separate issues? How do I get them to match?
[root@gsil-ipa01 data]# klist -ek /etc/krb5.keytab Keytab name: FILE:/etc/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes256-cts-hmac-sha384-192) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes128-cts-hmac-sha256-128) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes256-cts-hmac-sha1-96) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes128-cts-hmac-sha1-96)
[root@gsil-ipa01 data]# kvno host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL: kvno = 2
Jeremy Tourville via FreeIPA-users wrote:
I have noted that klist and kvno don't match for the keytab I fetched earlier. Could this cause issues with named or are those two separate issues? How do I get them to match?
[root@gsil-ipa01 data]# klist -ek /etc/krb5.keytab Keytab name: FILE:/etc/krb5.keytab KVNO Principal
5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes256-cts-hmac-sha384-192) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes128-cts-hmac-sha256-128) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes256-cts-hmac-sha1-96) 5 host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL (aes128-cts-hmac-sha1-96)
[root@gsil-ipa01 data]# kvno host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL host/gsil-ipa01.idm.gsil.smil@IDM.GSIL.SMIL: kvno = 2
Yes, it is basically: the passwords don't match. In a previous e-mail healthcehck reported that replication wasn't working, that might account for it.
rob
OK, but how do i get them to match again? Running ipa-getkeytab doesn't fix it. klist just keeps incrementing and kvno stays the same.
Hi,
On Fri, Mar 17, 2023 at 2:43 PM Jeremy Tourville via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
OK, but how do i get them to match again? Running ipa-getkeytab doesn't fix it. klist just keeps incrementing and kvno stays the same.
The command ipa-getkeytab creates a new key, unless it is called with --retrieve (in which case it downloads the existing keys to a keytab file).
In your case, a new key is generated for host/gsil-ipa01.idm.gsil.smil on server gsil-ipa02.idm.x.x. It means that the new key is updated in the LDAP entry on gsil-ipa02.idm.x.x. If the replication is broken, the LDAP entry on gsil-ipa01.idm.gsil.smil still contains the old key, and any kinit against this server will fail if using the new key.
You need to fix replication first, you may give a try at the command "ipa topologysegment-reinitialize".
flo
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
To make a long story short, we tried that command and it sort of appeared to work but ultimately failed when running the health check. Knowing to focus on topology was crucial in resolving things. There were several other commands we tried as well but just couldn't get things back in sync for whatever reason. Ultimately, we decided to remove the server altogether and rebuild it again. This allowed replication to work as needed once we had all the proper roles installed. Both servers are healthy and replicating properly! :) Thank you all for your help.
freeipa-users@lists.fedorahosted.org