IPA replica cannot lookup AD trust users (worked before)

[slek kus]

Hi, the only replica cannot retrieve AD trust users (one way trust). Trust agent had been
installed on this replica.
I noticed this issue, since clients that point to the replica started to fail
authenticating users. This replica worked OK before.
All functions and syncs except for the AD user lookup. overrides are synced over but
replica cannot find the user. 

Can't get it fixed. Is this repairable? Can I uninstall the replica and reinstall?

[root@idm01 ~]# ipa server-role-find
-----------------------
10 server roles matched
-----------------------
  Server name: idm01.linux.redacted.domain
  Role name: AD trust agent
  Role status: enabled

  Server name: idm02.linux.redacted.domain
  Role name: AD trust agent
  Role status: enabled

  Server name: idm01.linux.redacted.domain
  Role name: AD trust controller
  Role status: enabled

  Server name: idm02.linux.redacted.domain
  Role name: AD trust controller
  Role status: enabled

<...>

On the main server, the AD user can be looked up. On the "replica" it returns
empty.

working on main server:
[root@idm01 ~]# getent passwd testuser(a)subdoma.redacted.domain
[e-mail]:*:683005154:683005154:CHANGED:/home/testuser:/usr/bin/bash



Checking the sssd_doamin.log of the replica, I see the message that the domain is not
active while fetching ad user. Further in the same log there's mention of another
subdomain be inactive. 
The trust is wirth a AD forest with 2 subdomains. 
-----
(2024-04-25 16:40:11): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done] (0x0040):
[RID#34] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
   *  ... skipping repetitive backtrace ...
   
<...>

(2024-04-25 16:39:44): [be[linux.redacted.domain]] [resolv_discover_srv_done] (0x0040):
[RID#33] SRV query failed [11]: Could not contact DNS servers
   *  ... skipping repetitive backtrace ...
(2024-04-25 16:39:44): [be[linux.redacted.domain]] [fo_discover_servers_primary_done]
(0x0040): [RID#33] Unable to retrieve primary servers [1432158238]: SRV lookup error
   *  ... skipping repetitive backtrace ...
(2024-04-25 16:39:44): [be[linux.redacted.domain]] [resolve_srv_done] (0x0040): [RID#33]
Unable to resolve SRV [1432158238]: SRV lookup error
   *  ... skipping repetitive backtrace ...
(2024-04-25 16:39:44): [be[linux.redacted.domain]] [fo_resolve_service_send] (0x0020):
[RID#33] No available servers for service 'sd_SUBDOMB.redacted.domain'
   *  ... skipping repetitive backtrace ...
(2024-04-25 16:39:44): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done] (0x0040):
[RID#33] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_resolve_server_done]
(0x1000): [RID#33] Server [NULL] resolution failed: [5]: Input/output error
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [sdap_id_op_connect_done]
(0x0400): [RID#33] Failed to connect to server, but ignore mark offline is enabled.
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [sdap_id_op_connect_done]
(0x4000): [RID#33] notify error to op #1: 5 [Input/output error]
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_mark_dom_offline] (0x1000):
[RID#33] Marking subdomain SUBDOMB.redacted.domain offline
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_mark_subdom_offline]
(0x1000): [RID#33] Marking subdomain SUBDOMB.redacted.domain as inactive
   *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done]
(0x0040): [RID#33] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
********************** BACKTRACE DUMP ENDS HERE *********************************

There are not replication issues:
----
[root@idm01 ~]# ipa-healthcheck --source=ipahealthcheck.ds.replication
[
  {
    "source": "ipahealthcheck.ds.replication",
    "check": "ReplicationCheck",
    "result": "WARNING",
    "uuid": "4a5341db-bf65-4350-bf2c-c81872db536b",
    "when": "20240425145134Z",
    "duration": "0.391402",
    "kw": {
      "key": "DSREPLLE0002",
      "items": [
        "Replication",
        "Conflict Entries"
      ],
      "msg": "There were 1 conflict entries found under the replication
suffix \"dc=linux,dc=redacted,dc=domain\"."
    }
  }
]

[Sumit Bose]

Am Thu, Apr 25, 2024 at 03:03:41PM -0000 schrieb slek kus via FreeIPA-users:
>  Hi, the only replica cannot retrieve AD trust users (one way trust).
> Trust agent had been installed on this replica.
>  I noticed this issue, since clients that point to the replica started to fail
> authenticating users. This replica worked OK before.
>  All functions and syncs except for the AD user lookup. overrides are synced over but
> replica cannot find the user. 
>  
>  Can't get it fixed. Is this repairable? Can I uninstall the replica and reinstall?
>  
>  [root@idm01 ~]# ipa server-role-find
>  -----------------------
>  10 server roles matched
>  -----------------------
>    Server name: idm01.linux.redacted.domain
>    Role name: AD trust agent
>    Role status: enabled
>  
>    Server name: idm02.linux.redacted.domain
>    Role name: AD trust agent
>    Role status: enabled
>  
>    Server name: idm01.linux.redacted.domain
>    Role name: AD trust controller
>    Role status: enabled
>  
>    Server name: idm02.linux.redacted.domain
>    Role name: AD trust controller
>    Role status: enabled
>  
>  <...>
>  
>  On the main server, the AD user can be looked up. On the "replica" it returns
> empty.
>  
>  working on main server:
>  [root@idm01 ~]# getent passwd testuser(a)subdoma.redacted.domain
>
> [e-mail]:*:683005154:683005154:CHANGED:/home/testuser:/usr/bin/bash
>  
>  
>  
>  Checking the sssd_doamin.log of the replica, I see the message that the domain is not
> active while fetching ad user. Further in the same log there's mention of another
> subdomain be inactive. 
>  The trust is wirth a AD forest with 2 subdomains. 
>  -----
>  (2024-04-25 16:40:11): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done]
> (0x0040): [RID#34] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
>     *  ... skipping repetitive backtrace ...
>     
>  <...>
>  
>  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [resolv_discover_srv_done] (0x0040):
> [RID#33] SRV query failed [11]: Could not contact DNS servers 
Hi,

looks like DNS issues, does

    host -t SRV _ldap._tcp.SUBDOMB.redacted.domain

return anything?

bye,
Sumit
 
>     *  ... skipping repetitive backtrace ...
>  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [fo_discover_servers_primary_done]
> (0x0040): [RID#33] Unable to retrieve primary servers [1432158238]: SRV lookup error
>     *  ... skipping repetitive backtrace ...
>  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [resolve_srv_done] (0x0040): [RID#33]
> Unable to resolve SRV [1432158238]: SRV lookup error
>     *  ... skipping repetitive backtrace ...
>  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [fo_resolve_service_send] (0x0020):
> [RID#33] No available servers for service 'sd_SUBDOMB.redacted.domain'
>     *  ... skipping repetitive backtrace ...
>  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done]
> (0x0040): [RID#33] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
>  ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_resolve_server_done]
> (0x1000): [RID#33] Server [NULL] resolution failed: [5]: Input/output error
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [sdap_id_op_connect_done]
> (0x0400): [RID#33] Failed to connect to server, but ignore mark offline is enabled.
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [sdap_id_op_connect_done]
> (0x4000): [RID#33] notify error to op #1: 5 [Input/output error]
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_mark_dom_offline] (0x1000):
> [RID#33] Marking subdomain SUBDOMB.redacted.domain offline
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [be_mark_subdom_offline]
> (0x1000): [RID#33] Marking subdomain SUBDOMB.redacted.domain as inactive
>     *  (2024-04-25 16:39:44): [be[linux.redacted.domain]] [ipa_srv_ad_acct_lookup_done]
> (0x0040): [RID#33] ipa_get_*_acct request failed: [1432158277]: Subdomain is inactive.
>  ********************** BACKTRACE DUMP ENDS HERE *********************************
>  
>  There are not replication issues:
>  ----
>  [root@idm01 ~]# ipa-healthcheck --source=ipahealthcheck.ds.replication
>  [
>    {
>      "source": "ipahealthcheck.ds.replication",
>      "check": "ReplicationCheck",
>      "result": "WARNING",
>      "uuid": "4a5341db-bf65-4350-bf2c-c81872db536b",
>      "when": "20240425145134Z",
>      "duration": "0.391402",
>      "kw": {
>        "key": "DSREPLLE0002",
>        "items": [
>          "Replication",
>          "Conflict Entries"
>        ],
>        "msg": "There were 1 conflict entries found under the replication
> suffix \"dc=linux,dc=redacted,dc=domain\"."
>      }
>    }
>  ]
>  
>    
>     
>  
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[slek kus]

Hi Sumit, that does not return anything good on the replica. See below. 

On the main IPA node node:
----
 [alma@idm01 ~]$ host -t SRV _ldap._tcp.redacted.domain
_ldap._tcp.redacted.domain has SRV record 0 100 389 dc01.redacted.domain.
_ldap._tcp.redacted.domain has SRV record 0 100 389 dc01.redacted.domain.

[alma@idm01 ~]$ host -t SRV _ldap._tcp.domaina.redacted.domain
_ldap._tcp.domaina.redacted.domain has SRV record 0 100 389
windc-dc01.domaina.redacted.domain.
_ldap._tcp.domaina.redacted.domain has SRV record 0 100 389
windc-dc02.domaina.redacted.domain.

[alma@idm01 ~]$ host -t SRV _ldap._tcp.domainb.redacted.domain
_ldap._tcp.domainb.redacted.domain has SRV record 0 100 389
windc-dc02.domainb.redacted.domain.
_ldap._tcp.domainb.redacted.domain has SRV record 0 100 389
windc-dc01.domainb.redacted.domain.

On the secondary (replica):
----
[alma@idm02 ~]$  host -t SRV _ldap._tcp.redacted.domain
Host _ldap._tcp.redacted.domain not found: 2(SERVFAIL)

[alma@idm02 ~]$  host -t SRV _ldap._tcp.domainb.redacted.domain
Host _ldap._tcp.domainb.redacted.domain not found: 2(SERVFAIL)

[alma@idm02 ~]$  host -t SRV _ldap._tcp.domaina.redacted.domain
Host _ldap._tcp.domaina.redacted.domain not found: 2(SERVFAIL)

The DNS zone seems replicated and OK on the replica. The record is present there too.

On the main IPA node node:
----
[alma@idm01 ~]$ ipa dnsrecord-find linux.redacted.domain.
Record name: _ldap._tcp
  SRV record: 0 100 389 idm01.linux.redacted.domain., 0 200 389
idm02.linux.redacted.domain.

On the secondary (replica):
-----
[alma@idm02 ~]$ ipa dnsrecord-find linux.redacted.domain.
Record name: _ldap._tcp
  SRV record: 0 100 389 idm01.linux.redacted.domain., 0 200 389
idm02.linux.redacted.domain.

[Alexander Bokovoy]

On Пят, 26 кра 2024, slek kus via FreeIPA-users wrote:
> Hi Sumit, that does not return anything good on the replica. See
> below.
>
> On the main IPA node node:
> ----
>  [alma@idm01 ~]$ host -t SRV _ldap._tcp.redacted.domain
> _ldap._tcp.redacted.domain has SRV record 0 100 389 dc01.redacted.domain.
> _ldap._tcp.redacted.domain has SRV record 0 100 389 dc01.redacted.domain.
>
> [alma@idm01 ~]$ host -t SRV _ldap._tcp.domaina.redacted.domain
> _ldap._tcp.domaina.redacted.domain has SRV record 0 100 389
> windc-dc01.domaina.redacted.domain.
> _ldap._tcp.domaina.redacted.domain has SRV record 0 100 389
> windc-dc02.domaina.redacted.domain.
>
> [alma@idm01 ~]$ host -t SRV _ldap._tcp.domainb.redacted.domain
> _ldap._tcp.domainb.redacted.domain has SRV record 0 100 389
> windc-dc02.domainb.redacted.domain.
> _ldap._tcp.domainb.redacted.domain has SRV record 0 100 389
> windc-dc01.domainb.redacted.domain.
>
> On the secondary (replica):
> ----
> [alma@idm02 ~]$  host -t SRV _ldap._tcp.redacted.domain
> Host _ldap._tcp.redacted.domain not found: 2(SERVFAIL)
>
> [alma@idm02 ~]$  host -t SRV _ldap._tcp.domainb.redacted.domain
> Host _ldap._tcp.domainb.redacted.domain not found: 2(SERVFAIL)
>
> [alma@idm02 ~]$  host -t SRV _ldap._tcp.domaina.redacted.domain
> Host _ldap._tcp.domaina.redacted.domain not found: 2(SERVFAIL)
>
> The DNS zone seems replicated and OK on the replica. The record is present there too.
What is used as a DNS server for the idm02? Are you running idm02 with
an integrated DNS server or it is some other machine that resolves the
queries?

SERVFAIL means DNS server did return an error when processing your
request. Judging that this error happens for IPA domain's DNS zone and
for others too, I wonder if you have a generic DNS resolution issue from
idm02? For example, if idm01 is used as a DNS server there and idm02 is
in a different IP network, then BIND on idm01 will not allow DNS client
from idm02 to perform DNS queries. You'd need to add an ACL to allow
that.

Or it could be a DNSSEC error where a client is configured to have
DNSSEC validation but the DNS server responds without DNSSEC.

> On the main IPA node node:
> ----
> [alma@idm01 ~]$ ipa dnsrecord-find linux.redacted.domain.
> Record name: _ldap._tcp
>   SRV record: 0 100 389 idm01.linux.redacted.domain., 0 200 389
> idm02.linux.redacted.domain.
>
> On the secondary (replica):
> -----
> [alma@idm02 ~]$ ipa dnsrecord-find linux.redacted.domain.
> Record name: _ldap._tcp
>   SRV record: 0 100 389 idm01.linux.redacted.domain., 0 200 389
> idm02.linux.redacted.domain.
> --
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue 


-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[slek kus]

Hi Alexander, according to /etc/resolv.conf it is integrated and points to localhost, but
nmcli says DNS is set to idm01. 
A bit strange, since resolv.conf is generated by networkmanager. 
----
 [root@idm02 ~]# nmcli dev show | grep DNS
IP4.DNS[1]:                             172.16.27.10   <---- this is idm01
[root@idm02 ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search linux.redacted.domain
nameserver 127.0.0.1
----

Both servers are in the same nertwork.

On idm02, I can resolve the ipa domain it is the AD domains that fail:
----
[root@idm02 ~]# host -t SRV _ldap._tcp.linux.redacted.domain
_ldap._tcp.linux.redacted.domain has SRV record 0 200 389 idm02.linux.redacted.domain.
_ldap._tcp.linux.redacted.domain has SRV record 0 100 389 idm01.linux.redacted.domain.

[Alexander Bokovoy]

On Пят, 26 кра 2024, slek kus via FreeIPA-users wrote:
> Hi Alexander, according to /etc/resolv.conf it is integrated and
> points to localhost, but nmcli says DNS is set to idm01.
> A bit strange, since resolv.conf is generated by networkmanager.
> ----
>  [root@idm02 ~]# nmcli dev show | grep DNS
> IP4.DNS[1]:                             172.16.27.10   <---- this is idm01
> [root@idm02 ~]# cat /etc/resolv.conf
> # Generated by NetworkManager
> search linux.redacted.domain
> nameserver 127.0.0.1
> ----
>
> Both servers are in the same nertwork.
>
> On idm02, I can resolve the ipa domain it is the AD domains that fail:
> ----
> [root@idm02 ~]# host -t SRV _ldap._tcp.linux.redacted.domain
> _ldap._tcp.linux.redacted.domain has SRV record 0 200 389 idm02.linux.redacted.domain.
> _ldap._tcp.linux.redacted.domain has SRV record 0 100 389 idm01.linux.redacted.domain.
Do you have DNSSEC validation enforced on BIND side?

# grep dnssec /etc/named/ipa-options-ext.conf
/* dnssec-enable is obsolete and 'yes' by default */
dnssec-validation no;

If dnssec-validation is set to yes, that would explain because your AD
DNS server most likely is not using DNSSEC at all.

-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland

[slek kus]

Thanks much. dnssec-validation was set to yes on the replica. No idea how that happened.
Works now.

Something else and not related I wondered about, is why some clients point to a certain
server (in my case the failing server). 
This is seen with `sssctl domain-status <ipa domain>` under "Active
servers". Is the weight added to the SRV records only when server/dns service is down
and not when misconfigured/malfunctioning?