1 server not syncing with the others

[Russell Jones]

Hi all,

I have a setup of 4 FreeIPA servers, version 4.6.5, all on CentOS 7.

I've discovered that #4 is not syncing a new "video" group I created, while
the other 3 all have the group.

When looking at dirsrv error log, I am seeing the following after running
an ipactl stop / ipactl start:

[27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds - Could not
get initial credentials for principal
[ldap/freeipa4.cluster(a)US.EP.CORP.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon - slapd
started.  Listening on All Interfaces port 389 for LDAP requests
[27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon - Listening on
All Interfaces port 636 for LDAPS requests
[27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon - Listening on
/var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI requests
[27/Jan/2022:11:35:55.235218894 -0600] - ERR - schema-compat-plugin -
schema-compat-plugin tree scan will start in about 5 seconds!
[27/Jan/2022:11:35:58.368835716 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
credentials) ()


I am unsure what the issue is or how to resolve this. Could I get some
assistance with being pointed in the right direction?

Thank you!

[Florence Blanc-Renaud]

Hi,
you can find troubleshooting tips in
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...

HTH,
flo

On Thu, Jan 27, 2022 at 6:54 PM Russell Jones via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org&gt; wrote:

>  Hi all,
>
>  I have a setup of 4 FreeIPA servers, version 4.6.5, all on CentOS 7.
>
>  I've discovered that #4 is not syncing a new "video" group I created,
>  while the other 3 all have the group.
>
>  When looking at dirsrv error log, I am seeing the following after running
>  an ipactl stop / ipactl start:
>
>  [27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds - Could not
>  get initial credentials for principal
>  [ldap/freeipa4.cluster(a)US.EP.CORP.LOCAL] in keytab
>  [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
>  requested realm)
>  [27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon - slapd
>  started.  Listening on All Interfaces port 389 for LDAP requests
>  [27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon - Listening
>  on All Interfaces port 636 for LDAPS requests
>  [27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon - Listening
>  on /var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI requests
>  [27/Jan/2022:11:35:55.235218894 -0600] - ERR - schema-compat-plugin -
>  schema-compat-plugin tree scan will start in about 5 seconds!
>  [27/Jan/2022:11:35:58.368835716 -0600] - ERR - NSMMReplicationPlugin -
>  bind_and_check_pwp - agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
>  Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
>  credentials) ()
>
>
>  I am unsure what the issue is or how to resolve this. Could I get some
>  assistance with being pointed in the right direction?
>
>  Thank you!
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>

[Russell Jones]

Thanks,

I ended up finding the issue from another mailing list post. ntpd was not
running on this host and the time got skewed too much from the other
masters.

For what it's worth, the ipa-healthcheck script did not catch this issue.
Might be something to add?

On Fri, Jan 28, 2022 at 2:49 AM Florence Blanc-Renaud <flo(a)redhat.com&gt;
wrote:

>  Hi,
>  you can find troubleshooting tips in
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...
>
>  HTH,
>  flo
>
>  On Thu, Jan 27, 2022 at 6:54 PM Russell Jones via FreeIPA-users <
>  freeipa-users(a)lists.fedorahosted.org&gt; wrote:
>
> > Hi all,
> >
> > I have a setup of 4 FreeIPA servers, version 4.6.5, all on CentOS 7.
> >
> > I've discovered that #4 is not syncing a new "video" group I created,
> > while the other 3 all have the group.
> >
> > When looking at dirsrv error log, I am seeing the following after running
> > an ipactl stop / ipactl start:
> >
> > [27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds - Could not
> > get initial credentials for principal
> > [ldap/freeipa4.cluster(a)US.EP.CORP.LOCAL] in keytab
> > [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
> > requested realm)
> > [27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon - slapd
> > started.  Listening on All Interfaces port 389 for LDAP requests
> > [27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon - Listening
> > on All Interfaces port 636 for LDAPS requests
> > [27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon - Listening
> > on /var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI requests
> > [27/Jan/2022:11:35:55.235218894 -0600] - ERR - schema-compat-plugin -
> > schema-compat-plugin tree scan will start in about 5 seconds!
> > [27/Jan/2022:11:35:58.368835716 -0600] - ERR - NSMMReplicationPlugin -
> > bind_and_check_pwp - agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389)
> -
> > Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
> > credentials) ()
> >
> >
> > I am unsure what the issue is or how to resolve this. Could I get some
> > assistance with being pointed in the right direction?
> >
> > Thank you!
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
> > Do not reply to spam on the list, report it:
> > https://pagure.io/fedora-infrastructure
> >
>

[Rob Crittenden]

Russell Jones via FreeIPA-users wrote:
>  Thanks,
>  
>  I ended up finding the issue from another mailing list post. ntpd was
>  not running on this host and the time got skewed too much from the other
>  masters.
>  
>  For what it's worth, the ipa-healthcheck script did not catch this
>  issue. Might be something to add? 
It would be nice but syncing time can be quite slow and, AFAIK, there is
no way in advance to know if there is a time source available. So check
against what?

rob

>  
>  On Fri, Jan 28, 2022 at 2:49 AM Florence Blanc-Renaud <flo(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>  
>      Hi,
>      you can find troubleshooting tips in
>     
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...
>  
>      HTH,
>      flo
>  
>      On Thu, Jan 27, 2022 at 6:54 PM Russell Jones via FreeIPA-users
>      <freeipa-users(a)lists.fedorahosted.org
>      <mailto:[e-mail]>> wrote:
>  
>          Hi all,
>  
>          I have a setup of 4 FreeIPA servers, version 4.6.5, all on CentOS 7.
>  
>          I've discovered that #4 is not syncing a new "video" group I
>          created, while the other 3 all have the group.
>  
>          When looking at dirsrv error log, I am seeing the following
>          after running an ipactl stop / ipactl start:
>  
>          [27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds -
>          Could not get initial credentials for principal
>          [ldap/freeipa4.cluster(a)US.EP.CORP.LOCAL] in keytab
>          [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>          KDC for requested realm)
>          [27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon -
>          slapd started.  Listening on All Interfaces port 389 for LDAP
>          requests
>          [27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon -
>          Listening on All Interfaces port 636 for LDAPS requests
>          [27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon -
>          Listening on /var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI
>          requests
>          [27/Jan/2022:11:35:55.235218894 -0600] - ERR -
>          schema-compat-plugin - schema-compat-plugin tree scan will start
>          in about 5 seconds!
>          [27/Jan/2022:11:35:58.368835716 -0600] - ERR -
>          NSMMReplicationPlugin - bind_and_check_pwp -
>          agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
>          Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
>          credentials) ()
>  
>  
>          I am unsure what the issue is or how to resolve this. Could I
>          get some assistance with being pointed in the right direction?
>  
>          Thank you!
>          _______________________________________________
>          FreeIPA-users mailing list --
>          freeipa-users(a)lists.fedorahosted.org
>          <mailto:[e-mail]>
>          To unsubscribe send an email to
>          freeipa-users-leave(a)lists.fedorahosted.org
>          <mailto:[e-mail]>
>          Fedora Code of Conduct:
>          https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>          List Guidelines:
>          https://fedoraproject.org/wiki/Mailing_list_guidelines
>          List Archives:
>         
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>          Do not reply to spam on the list, report it:
>          https://pagure.io/fedora-infrastructure
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Russell Jones]

Not necessarily that the time was off, but that replication was not
happening. The error logs had this error in them, but the healthcheck
script was not picking up on it:

"[27/Jan/2022:11:35:58.368835716 -0600] - ERR - NSMMReplicationPlugin -
bind_and_check_pwp - agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
credentials) ()"

On Fri, Jan 28, 2022 at 9:23 AM Rob Crittenden <rcritten(a)redhat.com&gt; wrote:

>  Russell Jones via FreeIPA-users wrote:
>  > Thanks,
>  >
>  > I ended up finding the issue from another mailing list post. ntpd was
>  > not running on this host and the time got skewed too much from the other
>  > masters.
>  >
>  > For what it's worth, the ipa-healthcheck script did not catch this
>  > issue. Might be something to add?
>
>  It would be nice but syncing time can be quite slow and, AFAIK, there is
>  no way in advance to know if there is a time source available. So check
>  against what?
>
>  rob
>
>  >
>  > On Fri, Jan 28, 2022 at 2:49 AM Florence Blanc-Renaud <flo(a)redhat.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >     Hi,
>  >     you can find troubleshooting tips in
>  >
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...
>  >
>  >     HTH,
>  >     flo
>  >
>  >     On Thu, Jan 27, 2022 at 6:54 PM Russell Jones via FreeIPA-users
>  >     <freeipa-users(a)lists.fedorahosted.org
>  >     <mailto:[e-mail]>> wrote:
>  >
>  >         Hi all,
>  >
>  >         I have a setup of 4 FreeIPA servers, version 4.6.5, all on
>  CentOS 7.
>  >
>  >         I've discovered that #4 is not syncing a new "video" group I
>  >         created, while the other 3 all have the group.
>  >
>  >         When looking at dirsrv error log, I am seeing the following
>  >         after running an ipactl stop / ipactl start:
>  >
>  >         [27/Jan/2022:11:35:55.158724429 -0600] - ERR - set_krb5_creds -
>  >         Could not get initial credentials for principal
>  >         [ldap/freeipa4.cluster(a)US.EP.CORP.LOCAL] in keytab
>  >         [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any
>  >         KDC for requested realm)
>  >         [27/Jan/2022:11:35:55.169790450 -0600] - INFO - slapd_daemon -
>  >         slapd started.  Listening on All Interfaces port 389 for LDAP
>  >         requests
>  >         [27/Jan/2022:11:35:55.173079823 -0600] - INFO - slapd_daemon -
>  >         Listening on All Interfaces port 636 for LDAPS requests
>  >         [27/Jan/2022:11:35:55.175096801 -0600] - INFO - slapd_daemon -
>  >         Listening on /var/run/slapd-US-EP-CORP-LOCAL.socket for LDAPI
>  >         requests
>  >         [27/Jan/2022:11:35:55.235218894 -0600] - ERR -
>  >         schema-compat-plugin - schema-compat-plugin tree scan will start
>  >         in about 5 seconds!
>  >         [27/Jan/2022:11:35:58.368835716 -0600] - ERR -
>  >         NSMMReplicationPlugin - bind_and_check_pwp -
>  >         agmt="cn=meTofreeipa.us.ep.corp.local" (freeipa:389) -
>  >         Replication bind with GSSAPI auth failed: LDAP error 49 (Invalid
>  >         credentials) ()
>  >
>  >
>  >         I am unsure what the issue is or how to resolve this. Could I
>  >         get some assistance with being pointed in the right direction?
>  >
>  >         Thank you!
>  >         _______________________________________________
>  >         FreeIPA-users mailing list --
>  >         freeipa-users(a)lists.fedorahosted.org
>  >         <mailto:[e-mail]>
>  >         To unsubscribe send an email to
>  >         freeipa-users-leave(a)lists.fedorahosted.org
>  >         <mailto:[e-mail]>
>  >         Fedora Code of Conduct:
>  >         https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  >         List Guidelines:
>  >         https://fedoraproject.org/wiki/Mailing_list_guidelines
>  >         List Archives:
>  >
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  >         Do not reply to spam on the list, report it:
>  >         https://pagure.io/fedora-infrastructure
>  >
>  >
>  > _______________________________________________
>  > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  > To unsubscribe send an email to
>  freeipa-users-leave(a)lists.fedorahosted.org
>  > Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  > Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>  >
>
>