Replica re-initialization failing Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid credentials) ()

[Tania Hagan]

Hi Freeipa users, 

I have a replica that has been failing replication for a while, so I have tried the
following command to re-initialize (a back up of the server did not work):
ipa-replica-manage -vd re-initialize --from healthly.ipa.server

On the replica that I run this command I just see Update in progress, 1606 seconds elapsed
from the above command. 

I see no errors in /var/log/dirsrv/slapd/errors on the replica, but on the
healthy.ipa.server after 1000 seconds elapsed I see: ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=healthy.ipa.server-to-unhealthly.ipa.server"
(unhealty:389) - Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
credentials) ()

Any ideas how I can overcome this issue? 

Many Thanks, 
Tania

[Tania Hagan]

Further troubleshooting. 

If I run:
kinit -k -t /etc/dirsrv/ds.keytab ldap/ipa-unhealthly.ipa.server before the re-initialise
it complete successfully and a klist shows Default principal: ldap/unhealthly.ipa.server

After the LDAP error shows and the re-initialise is cancelled I see kinit: Generic error
(see e-text) while getting initial credentials.

In the healthy server if I look at /var/log/krb5kdc.log I see when the re-initialise in
progress: 
TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20),
camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19),
camellia128-cts-cmac(25)}) 10.100.104.7: ISSUE: authtime 1714662555, etypes
{rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
ses=aes256-cts-hmac-sha1-96(18)}, ldap/healthy.ipa.server for ldap/unhealthy.ipa.server

Thanks,

[Rob Crittenden]

Tania Hagan via FreeIPA-users wrote:
>  Further troubleshooting. 
>  
>  If I run:
>  kinit -k -t /etc/dirsrv/ds.keytab ldap/ipa-unhealthly.ipa.server before the re-initialise
> it complete successfully and a klist shows Default principal: ldap/unhealthly.ipa.server
>  
>  After the LDAP error shows and the re-initialise is cancelled I see kinit: Generic error
> (see e-text) while getting initial credentials.
>  
>  In the healthy server if I look at /var/log/krb5kdc.log I see when the re-initialise in
> progress: 
>  TGS_REQ (6 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20),
> camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19),
> camellia128-cts-cmac(25)}) 10.100.104.7: ISSUE: authtime 1714662555, etypes
> {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18),
> ses=aes256-cts-hmac-sha1-96(18)}, ldap/healthy.ipa.server for ldap/unhealthy.ipa.server
On the healthy server I'd run: kvno ldap/unhealthy.ipa.server

On the unhealthy server: klist -kt /etc/dirrv/ds.keytab

Compare the version numbers. I'm guessing they are different.

rob

[Tania Hagan]

Hi Rob, 

Turns out this was a DNS issue, thank you for responding. 

We had our /etc/resolv.conf pointing to local host and adding another ipa server as the
top nameserver solved the issue. This begs the question by default installing with the
ansible playbook it adds the localhost has the nameserver, which is the preferred setup?

Many Thanks, 
Tania

[Sam Morris]

On 08/05/2024 16:16, Tania Hagan via FreeIPA-users wrote:
>  Turns out this was a DNS issue, thank you for responding.
>  
>  We had our /etc/resolv.conf pointing to local host and adding another ipa server as the
> top nameserver solved the issue. This begs the question by default installing with the
> ansible playbook it adds the localhost has the nameserver, which is the preferred setup?
Assuming your IPA server runs the integrated DNS service, it should have 
127.0.0.1 as the only nameserver.

It's then the job of BIND on that server to provide recursive DNS 
service for other stuff running on the server. Typically you'd configure 
global and/or per-server forwarders so that queries for names outside of 
your domain's configured DNS zones are forwarded elsewhere.

So if DNS resolution on a server breaks, BIND is the component to 
investigate & fix.

-- 
Sam Morris <https://robots.org.uk/>
PGP: rsa4096/CAAA AA1A CA69 A83A 892B  1855 D20B 4202 5CDA 27B9

[David Harvey]

Sorry if this is thread hijack (happy to start another) but further to
this, is the single resolver 127.0.0.1 the blessed / recommended setup?
We've had some chicken and egg situations recently where dirsrv being sad
has broken local DNS resolution, and then krb behaviours and lookup for the
other IPA servers has been impaired as a result.

If using local bind is the recommended state, should we be putting the
other IPA servers in /etc/hosts or anything to make sure they can identify
one and other in the case of dirsrv sadness?

Thanks in advance,

David

On Wed, 8 May 2024 at 16:34, Sam Morris via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  On 08/05/2024 16:16, Tania Hagan via FreeIPA-users wrote:
>  > Turns out this was a DNS issue, thank you for responding.
>  >
>  > We had our /etc/resolv.conf pointing to local host and adding another
>  ipa server as the top nameserver solved the issue. This begs the question
>  by default installing with the ansible playbook it adds the localhost has
>  the nameserver, which is the preferred setup?
>
>  Assuming your IPA server runs the integrated DNS service, it should have
>  127.0.0.1 as the only nameserver.
>
>  It's then the job of BIND on that server to provide recursive DNS
>  service for other stuff running on the server. Typically you'd configure
>  global and/or per-server forwarders so that queries for names outside of
>  your domain's configured DNS zones are forwarded elsewhere.
>
>  So if DNS resolution on a server breaks, BIND is the component to
>  investigate & fix.
>
>  --
>  Sam Morris <https://robots.org.uk/>
>  PGP: rsa4096/CAAA AA1A CA69 A83A 892B  1855 D20B 4202 5CDA 27B9
>  --
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>