AD Trust not authenticating users

[James Osbourn]

I have an inherited IPA domain that is a subdomain of an active directory domain, e.g. ipa.ad1.com as a child of ad1.com.  The IPA domain has AD Trust enabled and a one way domain trust to another AD sub domain, e.g. we want to use user logins from the AD domain users.ad2.com which is a child domain of ad2.com.  We are also using AD security group from the user.ad2.com domain to apply group based access control.  e.g. we are using simple authentication on SSSD to limit who can login and using AD groups to define sudo access.  This users domain and AD servers is managed by another team.

Everything was working for some time and then we started seeing intermittent problems with authentication, a quick restart of the IPA server would resolve the problem temporarily, but then it would stop again.  Even if we could login using SSH keys the sudo access would not work, it would appear to lose group membership details.

I have recently updated all of the IPA nodes to RHEL9 and made sure that DNS is updated correctly.

The sssd.conf configuration on the IPA server looks as follows

[domain/ipa.ad1.com]
debug_level = 6
id_provider = ipa
ipa_server = ipa-3.ipa.ad1.com
ipa_domain = ipa.ad1.com
ipa_hostname = ipa-3.ipa.ad1.com
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
cache_credentials = True
ldap_tls_cacert = /etc/ipa/ca.crt
krb5_store_password_if_offline = True
sudo_provider = ipa
autofs_provider = ipa
subdomains_provider = ipa
session_provider = ipa
hostid_provider = ipa
ipa_server_mode = True
subdomain_homedir = /home/%u
default_shell = /bin/bash
override_shell = /bin/bash
[sssd]
services = nss, pam, sudo, ifp

domains = ipa.ad1.com
domain_resolution_order = users.ad2.com
[nss]
homedir_substring = /home

[pam]

[sudo]

[autofs]

[ssh]

[pac]

[ifp]
allowed_uids = ipaapi, root

[session_recording]

I have debug level 6 enabled on SSSD and when I check the domain status I see the following more often than not.  The ad2.com forest domains are offline.  They go online and then as soon as someone tries to login again then either both or just the users.ad2.com domain go offline which causes the login to fail.

ipa.ad1.com Online status: Online
ad1.com Online status: Online
ad2.com Online status: Offline
users.ad2.com Online status: Offline

When I look at the SSSD domain logs I see the following (I have replaced internal domain names or hostname)

*  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_senders_lookup] (0x2000): Looking for identity of sender [sssd.ifp]
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain ipa.ad1.com is Active
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain ad1.com is Active
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain AD2.COM is Active
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400): sssd.DataProvider.Backend.IsOnline: Success
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_dispatch] (0x4000): Dispatching.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x1000): [RID#1162] the connection will expire at 1686152455
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9519] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x1000): [RID#1162] Waiting for child [9519].
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9519] finished successfully.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-1.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-1.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-3.ad1.com' as 'resolving name'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve AAAA record of 'ad1-dc-3.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162] No more address families to retry
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in DNS
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162] Deleting request watch
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-3.ad1.com' as 'name resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server ad1-dc-3.ad1.com: [172.28.8.7] TTL 3600
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed uri 'ldap://ad1-dc-3.ad1.com'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed GC uri 'ldap://ad1-dc-3.ad1.com:3268'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400): [RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162] calling ldap_search_ext with [(objectclass=*)][].
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400): [RID#1162] Search result: Success(0), no errmsg set
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Setting AD compatibility level to [7]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$, AD2.COM, 86400)
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server dcc01.ad2.com: [10.194.34.10] TTL 2107
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162] buffer size: 73
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162] Setting 8 seconds timeout for TGT child
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All data has been sent!
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
   *  ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9522] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
   *  ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-3.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-3.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-2.ad1.com' as 'resolving name'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve AAAA record of 'ad1-dc-2.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162] No more address families to retry
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in DNS
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162] Deleting request watch
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-2.ad1.com' as 'name resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server ad1-dc-2.ad1.com: [172.28.8.6] TTL 3600
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed uri 'ldap://ad1-dc-2.ad1.com'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed GC uri 'ldap://ad1-dc-2.ad1.com:3268'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400): [RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162] calling ldap_search_ext with [(objectclass=*)][].
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400): [RID#1162] Search result: Success(0), no errmsg set
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Setting AD compatibility level to [7]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$, AD2.COM, 86400)
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server dcc01.ad2.com: [10.194.34.10] TTL 2107
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162] buffer size: 73
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162] Setting 8 seconds timeout for TGT child
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All data has been sent!
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
   *  ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9523] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
   *  ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_be_fo_set_port_status] (0x8000): [RID#1162] Setting status: PORT_NOT_WORKING. Called from: src/providers/ldap/sdap_async_connection.c: sdap_cli_connect_recv: 2136
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_handle_release] (0x2000): [RID#1162] Trace: sh[0x556c865496d0], connected[1], ops[(nil)], ldap[0x556c865f89b0], destructor_lock[0], release_memory[0]
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [remove_connection_callback] (0x4000): [RID#1162] Successfully removed connection callback.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000): [RID#1162] attempting failover retry on op #1
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_step] (0x4000): [RID#1162] beginning to connect
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-1.ad1.com' is 'name resolved'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-1.ad1.com' is 'not working'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-3.ad1.com' is 'name resolved'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-3.ad1.com' is 'not working'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-2.ad1.com' is 'name resolved'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-2.ad1.com' is 'not working'
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0080): [RID#1162] Subdomain lookup failed, will try to reset subdomain.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trusted_dom_setup_1way] (0x0400): [RID#1162] Will re-fetch keytab for USERS.AD2.COM
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_getkeytab_send] (0x0400): [RID#1162] Retrieving keytab for AUTH$@AD2.COM from ipa-3.ipa.ad1.com into /var/lib/sss/keytabs/AD2.COM.keytabIw4R58 using ccache /var/lib/sss/db/ccache_AUTH.SSDIS.LOC
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400): sssd.DataProvider.Backend.IsOnline: Success
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9524] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Keytab successfully retrieved to /var/lib/sss/keytabs/AD2.COM.keytabIw4R58
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Keytab /var/lib/sss/keytabs/AD2.COM.keytabIw4R58 contains the expected principals
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Established trust context for USERS.AD2.COM
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_retried] (0x0400): [RID#1162] Subdomain re-set, will retry lookup
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcf01.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcf01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcf01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcs02.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcs02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcs02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcf02.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcf02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcf02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcc01.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcc01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcc01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcs03.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcs03.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcs03.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_gc_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 0 of server '(no name)' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_step] (0x0400): [RID#1162] Looking up AD account
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
   *  ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040): [RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_done] (0x1000): [RID#1162] Server [NULL] resolution failed: [5]: Input/output error
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000): [RID#1162] notify error to op #1: 5 [Input/output error]
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_dom_offline] (0x1000): [RID#1162] Marking subdomain USERS.AD2.COM offline
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_subdom_offline] (0x1000): [RID#1162] Marking subdomain USERS.AD2.COM as inactive
   *  (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040): [RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
********************** BACKTRACE DUMP ENDS HERE *********************************

As far as I can tell the users.ad2.com domain is "Active", but the bind fails to the domain which forces a domain check to take place.  Ultimately, this fails and the domain is flagged offline so the authentication fails.

There seems to be some odd lines where the users.ad2.com validation is trying to connect to servers from the ad1.com domain and global catalog validation which is failing.  Not sure why this would be taking place.

This seems to culminate in the line
(2023-06-07 16:25:55): [be[auth.ssdis.loc]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'

I can see actions taking place but cannot determine why they are taking place.  I can kinit as a users.ad2.com user successfully.  Likewise I can getent users and groups successfully and see the details correctly.  Yet authentication and authorisation is failing for SSH logins

Any help or guidance on resolving this problem would be appreciated.

[Sumit Bose]

Am Thu, Jun 08, 2023 at 11:48:58AM -0000 schrieb James Osbourn via FreeIPA-users:
> I have an inherited IPA domain that is a subdomain of an active directory domain, e.g. ipa.ad1.com as a child of ad1.com.  The IPA domain has AD Trust enabled and a one way domain trust to another AD sub domain, e.g. we want to use user logins from the AD domain users.ad2.com which is a child domain of ad2.com.  We are also using AD security group from the user.ad2.com domain to apply group based access control.  e.g. we are using simple authentication on SSSD to limit who can login and using AD groups to define sudo access.  This users domain and AD servers is managed by another team.
>
> Everything was working for some time and then we started seeing intermittent problems with authentication, a quick restart of the IPA server would resolve the problem temporarily, but then it would stop again.  Even if we could login using SSH keys the sudo access would not work, it would appear to lose group membership details.
>
> I have recently updated all of the IPA nodes to RHEL9 and made sure that DNS is updated correctly.

Hi,

this might be related to https://github.com/SSSD/sssd/issues/6600 but it
looks like not exactly the same issue.

Can you share /etc/krb5.conf and all files from /etc/krb5.conf.d/ and
/var/lib/sss/pubconf/krb5.include.d/

bye,
Sumit
> The sssd.conf configuration on the IPA server looks as follows
>
> [domain/ipa.ad1.com]
> debug_level = 6
> id_provider = ipa
> ipa_server = ipa-3.ipa.ad1.com
> ipa_domain = ipa.ad1.com
> ipa_hostname = ipa-3.ipa.ad1.com
> auth_provider = ipa
> chpass_provider = ipa
> access_provider = ipa
> cache_credentials = True
> ldap_tls_cacert = /etc/ipa/ca.crt
> krb5_store_password_if_offline = True
> sudo_provider = ipa
> autofs_provider = ipa
> subdomains_provider = ipa
> session_provider = ipa
> hostid_provider = ipa
> ipa_server_mode = True
> subdomain_homedir = /home/%u
> default_shell = /bin/bash
> override_shell = /bin/bash
> [sssd]
> services = nss, pam, sudo, ifp
>
> domains = ipa.ad1.com
> domain_resolution_order = users.ad2.com
> [nss]
> homedir_substring = /home
>
> [pam]
>
> [sudo]
>
> [autofs]
>
> [ssh]
>
> [pac]
>
> [ifp]
> allowed_uids = ipaapi, root
>
> [session_recording]
>
> I have debug level 6 enabled on SSSD and when I check the domain status I see the following more often than not.  The ad2.com forest domains are offline.  They go online and then as soon as someone tries to login again then either both or just the users.ad2.com domain go offline which causes the login to fail.
>
> ipa.ad1.com Online status: Online
> ad1.com Online status: Online
> ad2.com Online status: Offline
> users.ad2.com Online status: Offline
>
> When I look at the SSSD domain logs I see the following (I have replaced internal domain names or hostname)
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_senders_lookup] (0x2000): Looking for identity of sender [sssd.ifp]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain ipa.ad1.com is Active
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain ad1.com is Active
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain AD2.COM is Active
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400): sssd.DataProvider.Backend.IsOnline: Success
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_dispatch] (0x4000): Dispatching.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x1000): [RID#1162] the connection will expire at 1686152455
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
>
> ********************** BACKTRACE DUMP ENDS HERE *********************************
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9519] finished successfully.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
> ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x1000): [RID#1162] Waiting for child [9519].
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9519] finished successfully.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
>
> ********************** BACKTRACE DUMP ENDS HERE *********************************
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-1.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-1.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in files
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-3.ad1.com' as 'resolving name'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve AAAA record of 'ad1-dc-3.ad1.com' in files
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162] No more address families to retry
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in DNS
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162] Deleting request watch
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-3.ad1.com' as 'name resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server ad1-dc-3.ad1.com: [172.28.8.7] TTL 3600
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed uri 'ldap://ad1-dc-3.ad1.com'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed GC uri 'ldap://ad1-dc-3.ad1.com:3268'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400): [RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162] calling ldap_search_ext with [(objectclass=*)][].
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400): [RID#1162] Search result: Success(0), no errmsg set
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Setting AD compatibility level to [7]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$, AD2.COM, 86400)
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server dcc01.ad2.com: [10.194.34.10] TTL 2107
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162] buffer size: 73
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162] Setting 8 seconds timeout for TGT child
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All data has been sent!
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
> ... skipping repetitive backtrace ...
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9522] finished successfully.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
> ... skipping repetitive backtrace ...
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-3.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-3.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in files
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-2.ad1.com' as 'resolving name'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100): [RID#1162] Trying to resolve AAAA record of 'ad1-dc-2.ad1.com' in files
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162] No more address families to retry
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100): [RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in DNS
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162] Deleting request watch
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'ad1-dc-2.ad1.com' as 'name resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server ad1-dc-2.ad1.com: [172.28.8.6] TTL 3600
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed uri 'ldap://ad1-dc-2.ad1.com'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162] Constructed GC uri 'ldap://ad1-dc-2.ad1.com:3268'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400): [RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162] calling ldap_search_ext with [(objectclass=*)][].
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400): [RID#1162] Search result: Success(0), no errmsg set
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Setting AD compatibility level to [7]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100): [RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$, AD2.COM, 86400)
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The status of SRV lookup is resolved
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162] Found address for server dcc01.ad2.com: [10.194.34.10] TTL 2107
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162] buffer size: 73
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162] Setting 8 seconds timeout for TGT child
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All data has been sent!
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF received, client finished
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child responded: 0 [FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire timeout is 900
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing sasl bind mech: GSSAPI, user: AUTH$
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162] ldap_sasl_interactive_bind_s failed (-2)[Local error]
> ... skipping repetitive backtrace ...
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Server not found in Kerberos database)]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9523] finished successfully.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162] Unable to establish connection [1432158227]: Authentication Failed
> ... skipping repetitive backtrace ...
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
> ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_be_fo_set_port_status] (0x8000): [RID#1162] Setting status: PORT_NOT_WORKING. Called from: src/providers/ldap/sdap_async_connection.c: sdap_cli_connect_recv: 2136
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_handle_release] (0x2000): [RID#1162] Trace: sh[0x556c865496d0], connected[1], ops[(nil)], ldap[0x556c865f89b0], destructor_lock[0], release_memory[0]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [remove_connection_callback] (0x4000): [RID#1162] Successfully removed connection callback.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000): [RID#1162] attempting failover retry on op #1
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_step] (0x4000): [RID#1162] beginning to connect
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-1.ad1.com' is 'name resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-1.ad1.com' is 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-3.ad1.com' is 'name resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-3.ad1.com' is 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162] Status of server 'ad1-dc-2.ad1.com' is 'name resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port status of port 3268 for server 'ad1-dc-2.ad1.com' is 'not working'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
>
> ********************** BACKTRACE DUMP ENDS HERE *********************************
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0080): [RID#1162] Subdomain lookup failed, will try to reset subdomain.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trusted_dom_setup_1way] (0x0400): [RID#1162] Will re-fetch keytab for USERS.AD2.COM
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_getkeytab_send] (0x0400): [RID#1162] Retrieving keytab for AUTH$@AD2.COM from ipa-3.ipa.ad1.com into /var/lib/sss/keytabs/AD2.COM.keytabIw4R58 using ccache /var/lib/sss/db/ccache_AUTH.SSDIS.LOC
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400): sssd.DataProvider.Backend.IsOnline: Success
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child [9524] finished successfully.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Keytab successfully retrieved to /var/lib/sss/keytabs/AD2.COM.keytabIw4R58
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Keytab /var/lib/sss/keytabs/AD2.COM.keytabIw4R58 contains the expected principals
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400): [RID#1162] Established trust context for USERS.AD2.COM
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_retried] (0x0400): [RID#1162] Subdomain re-set, will retry lookup
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcf01.users.ad2.com' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcf01.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcf01.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcs02.users.ad2.com' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcs02.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcs02.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcf02.users.ad2.com' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcf02.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcf02.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcc01.users.ad2.com' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcc01.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcc01.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162] Marking server 'dcs03.users.ad2.com' as 'name not resolved'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 389 of server 'dcs03.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking port 389 of duplicate server 'dcs03.users.ad2.com' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162] Marking SRV lookup of service 'sd_gc_USERS.AD2.COM' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking port 0 of server '(no name)' as 'neutral'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_step] (0x0400): [RID#1162] Looking up AD account
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is unable to complete the full connection request, this internal status does not necessarily indicate network port issues.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
> ... skipping repetitive backtrace ...
>
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040): [RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
> ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_done] (0x1000): [RID#1162] Server [NULL] resolution failed: [5]: Input/output error
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162] Failed to connect to server, but ignore mark offline is enabled.
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000): [RID#1162] notify error to op #1: 5 [Input/output error]
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_dom_offline] (0x1000): [RID#1162] Marking subdomain USERS.AD2.COM offline
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_subdom_offline] (0x1000): [RID#1162] Marking subdomain USERS.AD2.COM as inactive
> (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040): [RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
>
> ********************** BACKTRACE DUMP ENDS HERE *********************************
>
> As far as I can tell the users.ad2.com domain is "Active", but the bind fails to the domain which forces a domain check to take place.  Ultimately, this fails and the domain is flagged offline so the authentication fails.
>
> There seems to be some odd lines where the users.ad2.com validation is trying to connect to servers from the ad1.com domain and global catalog validation which is failing.  Not sure why this would be taking place.
>
> This seems to culminate in the line
> (2023-06-07 16:25:55): [be[auth.ssdis.loc]] [fo_resolve_service_send] (0x0020): [RID#1162] No available servers for service 'sd_gc_AD2.COM'
>
> I can see actions taking place but cannot determine why they are taking place.  I can kinit as a users.ad2.com user successfully.  Likewise I can getent users and groups successfully and see the details correctly.  Yet authentication and authorisation is failing for SSH logins
>
> Any help or guidance on resolving this problem would be appreciated.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[James Osbourn]

Thanks I will take a look at the link.

The krb5.conf file looks as follows
includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = IPA.AD1.COM
 dns_lookup_realm = false
 dns_lookup_kdc = true
 rdns = false
 ticket_lifetime = 24h
 forwardable = true
 udp_preference_limit = 0
 default_ccache_name = KEYRING:persistent:%{uid}

[realms]
 IPA.AD1.COM = {
  kdc = ipa-3.ipa.ad1.com:88
  master_kdc = ipa-3.ipa.ad1.com:88
  kpasswd_server = ipa-3.ipa.ad1.com:464
  admin_server = ipa-3.ipa.ad1.com:749
  default_domain = ipa.ad1.com
  pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
  pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}

[domain_realm]
 .ipa.ad1.com = IPA.AD1.COM
 ipa.ad1.com = IPA.AD1.COM
 ipa-3.ipa.ad1.com = IPA.AD1.COM

[dbmodules]
  IPA.AD1.COM = {
    db_library = ipadb.so
  }

[plugins]
 certauth = {
  module = ipakdb:kdb/ipadb.so
  enable_only = ipakdb
 }

Under the /var/lib/sss/pubconf/krb5.include.d/ directory the files and contents are as follows
::::::::::::::
/var/lib/sss/pubconf/krb5.include.d/domain_realm_auth_ssdis_loc
::::::::::::::
[domain_realm]
.ssdis.loc = SSDIS.LOC
ssdis.loc = SSDIS.LOC
.ROOT.TES = ROOT.TES
ROOT.TES = ROOT.TES
.INTERNAL.ROOT.TES = INTERNAL.ROOT.TES
INTERNAL.ROOT.TES = INTERNAL.ROOT.TES
[capaths]
SSDIS.LOC = {
  AUTH.SSDIS.LOC = SSDIS.LOC
}
ROOT.TES = {
  AUTH.SSDIS.LOC = ROOT.TES
}
INTERNAL.ROOT.TES = {
  AUTH.SSDIS.LOC = ROOT.TES
}
AUTH.SSDIS.LOC = {
  SSDIS.LOC = SSDIS.LOC
  ROOT.TES = ROOT.TES
  INTERNAL.ROOT.TES = ROOT.TES
}
::::::::::::::
/var/lib/sss/pubconf/krb5.include.d/krb5_libdefaults
::::::::::::::
[libdefaults]
 canonicalize = true
::::::::::::::
/var/lib/sss/pubconf/krb5.include.d/localauth_plugin
::::::::::::::
[plugins]
 localauth = {
  module = sssd:/usr/lib64/sssd/modules/sssd_krb5_localauth_plugin.so
 }

I am still looking into my problem, a reboot of an IPA server seems to allow authentication and AD group authorisation to work for a period of time and then it stops.  Authentication will continue to work if the user is cached in the SSSD cache, but trying to use sudo fails as it can no longer get the membership details.

[Sumit Bose]

Am Thu, Jun 08, 2023 at 03:37:12PM -0000 schrieb James Osbourn via FreeIPA-users:
> Thanks I will take a look at the link.
>
> The krb5.conf file looks as follows
> includedir /etc/krb5.conf.d/
> includedir /var/lib/sss/pubconf/krb5.include.d/
>
> [logging]
>  default = FILE:/var/log/krb5libs.log
>  kdc = FILE:/var/log/krb5kdc.log
>  admin_server = FILE:/var/log/kadmind.log
>
> [libdefaults]
>  default_realm = IPA.AD1.COM
>  dns_lookup_realm = false
>  dns_lookup_kdc = true
>  rdns = false
>  ticket_lifetime = 24h
>  forwardable = true
>  udp_preference_limit = 0
>  default_ccache_name = KEYRING:persistent:%{uid}
>
> [realms]
>  IPA.AD1.COM = {
>   kdc = ipa-3.ipa.ad1.com:88
>   master_kdc = ipa-3.ipa.ad1.com:88
>   kpasswd_server = ipa-3.ipa.ad1.com:464
>   admin_server = ipa-3.ipa.ad1.com:749
>   default_domain = ipa.ad1.com
>   pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
>   pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
> }
>
> [domain_realm]
>  .ipa.ad1.com = IPA.AD1.COM
>  ipa.ad1.com = IPA.AD1.COM
>  ipa-3.ipa.ad1.com = IPA.AD1.COM

Hi,

assuming that auth.ssdis.loc is the domain with issues can you try if
adding

.auth.ssdis.loc = AUTH.SSDIS.LOC
   auth.ssdis.loc = AUTH.SSDIS.LOC

to the [domain_realm] of /etc/krb5.conf makes is more reliable?

bye,
Sumit
> [dbmodules]
>   IPA.AD1.COM = {
>     db_library = ipadb.so
>   }
>
> [plugins]
>  certauth = {
>   module = ipakdb:kdb/ipadb.so
>   enable_only = ipakdb
>  }
>
> Under the /var/lib/sss/pubconf/krb5.include.d/ directory the files and contents are as follows
> ::::::::::::::
> /var/lib/sss/pubconf/krb5.include.d/domain_realm_auth_ssdis_loc
> ::::::::::::::
> [domain_realm]
> .ssdis.loc = SSDIS.LOC
> ssdis.loc = SSDIS.LOC
> .ROOT.TES = ROOT.TES
> ROOT.TES = ROOT.TES
> .INTERNAL.ROOT.TES = INTERNAL.ROOT.TES
> INTERNAL.ROOT.TES = INTERNAL.ROOT.TES
> [capaths]
> SSDIS.LOC = {
>   AUTH.SSDIS.LOC = SSDIS.LOC
> }
> ROOT.TES = {
>   AUTH.SSDIS.LOC = ROOT.TES
> }
> INTERNAL.ROOT.TES = {
>   AUTH.SSDIS.LOC = ROOT.TES
> }
> AUTH.SSDIS.LOC = {
>   SSDIS.LOC = SSDIS.LOC
>   ROOT.TES = ROOT.TES
>   INTERNAL.ROOT.TES = ROOT.TES
> }
> ::::::::::::::
> /var/lib/sss/pubconf/krb5.include.d/krb5_libdefaults
> ::::::::::::::
> [libdefaults]
>  canonicalize = true
> ::::::::::::::
> /var/lib/sss/pubconf/krb5.include.d/localauth_plugin
> ::::::::::::::
> [plugins]
>  localauth = {
>   module = sssd:/usr/lib64/sssd/modules/sssd_krb5_localauth_plugin.so
>  }
>
> I am still looking into my problem, a reboot of an IPA server seems to allow authentication and AD group authorisation to work for a period of time and then it stops.  Authentication will continue to work if the user is cached in the SSSD cache, but trying to use sudo fails as it can no longer get the membership details.
> _______________________________________________
> FreeIPA-users mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue