[Freeipa-users] Re: Kerberos appears to be broken on a FreeIPA server on CentOS 7.8

[Rob Crittenden]

Vinícius Ferrão wrote:
>  Hi Rob.
>  
> > On 15 Feb 2021, at 10:58, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:[e-mail]>> wrote:
> >
> > Vinícius Ferrão wrote:
> >> Hi Rob.
> >>
> >> Actually nothing that relies on Kerberos Keytabs is working.
> >
> > Kerberos is working. The kinit was successful.
>  
>  Sorry perhaps I didn’t say it correctly. In fact Kerberos is working (I
>  can kinit) but anything that relies on Keytabs, specifically Keytabs,
>  aren’t working.
>  
>  named-pkcs11 does not start without the hack that I’ve mentioned. Please
>  correct me if I’m wrong about this.
>  
>  Every other service fails with “insufficient credentials”; dogtag,
>  gssproxy, etc. 
Looping in the Kerberos maintainer. You'll note that later in the output
there is a reference to credential cache is empty. I wonder if gssproxy
is having issues.

rob

>  
> >> I can properly issue kinit’s and login, but I can’t use ‘ipa’ commands
> >> for instance. named-pkcs11 is only starting up because I’ve changed the
> >> authentication method on /etc/named.conf:
> >>
> >> /* WARNING: This part of the config file is IPA-managed.
> >>  * Modifications may break IPA setup or upgrades.
> >>  */
> >> dyndb "ipa" "/usr/lib64/bind/ldap.so" {
> >> uri "ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket";
> >> base "cn=dns, dc=cluster,dc=cetene,dc=gov,dc=br";
> >> server_id "neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>";
> >> #auth_method "sasl";
> >> #sasl_mech "GSSAPI";
> >> #sasl_user "DNS/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>";
> >> /* Desespero */
> >> auth_method "simple";
> >> bind_dn
> >> "uid=admin,cn=users,cn=accounts,dc=cluster,dc=cetene,dc=gov,dc=br";
> >> password “REDACTED";
> >> };
> >> /* End of IPA-managed part. */
> >>
> >> I’ve done the test that you’ve asked, and was a no go:
> >>
> >> [root@neumann2 ~]# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> [root@neumann2 ~]# klist
> >> Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
> >> Default principal:
> >> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>
> <mailto:ipa-dnskeysyncd/[e-mail]>
> >>
> <mailto:ipa-dnskeysyncd/[e-mail]>
> >>
> >> Valid starting       Expires              Service principal
> >> 02/12/2021 22:42:03  02/13/2021 22:42:03 
> >> krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> >> <mailto:krbtgt/[e-mail]>
> >> <mailto:krbtgt/[e-mail]>
> >> [root@neumann2 ~]# ipa user-show admin
> >> ipa: ERROR: Insufficient access:  Invalid credentials
> >> [root@neumann2 ~]# ipa -v user-show admin
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 1]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 2]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 3]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 4]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 5]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: ERROR: cannot connect to
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
> >> number of tries to forward a request.
> >>
> >> I never seen this on FreeIPA.
> >>
> >> Subsequent queries of IPA commands just returns the same error:
> >>
> >> [root@neumann2 ~]# ipa user-show admin
> >> ipa: ERROR: cannot connect to
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
> >> number of tries to forward a request.
> >
> > Did you get a HTTP service ticket? (klist)
>  
>  I issued and admin ticket as I usually do:
>  
>  [root@neumann2 ~]# kinit admin
>  Password for admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:[e-mail]>: 
>  [root@neumann2 ~]# klist
>  Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
>  Default principal: admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:[e-mail]>
>  
>  Valid starting       Expires              Service principal
>  02/15/2021 13:09:04  02/16/2021 13:09:04 
>  krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
>  <mailto:krbtgt/[e-mail]>
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: Insufficient access:  Invalid credentials
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  
>  But I can recover the HTTP ticket and kinit:
>  
>  [root@neumann2 ~]# klist -kt /var/lib/ipa/gssproxy/http.keytab
>  Keytab name: FILE:/var/lib/ipa/gssproxy/http.keytab
>  KVNO Timestamp           Principal
>  ---- -------------------
>  ------------------------------------------------------
>     3 02/10/2021 22:52:34
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>     3 02/10/2021 22:52:34
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>  [root@neumann2 ~]# kinit -kt /var/lib/ipa/gssproxy/http.keytab
>  HTTP/neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  [root@neumann2 ~]# klist
>  Ticket cache: KEYRING:persistent:0:krb_ccache_JRv9hJN
>  Default principal:
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>  
>  Valid starting       Expires              Service principal
>  02/15/2021 13:13:47  02/16/2021 13:13:47 
>  krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
>  <mailto:krbtgt/[e-mail]>
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: Insufficient access:  Invalid credentials
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  
>  But again it didn’t work.
>  
>  On /var/log/httpd/error_log there basically this:
>  
>  [Wed Feb 10 17:34:19.129505 2021] [:error] [pid 13912] ipa: INFO: 401
>  Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI
>  Error: Unspecified GSS failure.  Minor code may provide more information
>  (Credential cache is empty)
>  [Wed Feb 10 17:34:19.151811 2021] [auth_gssapi:error] [pid 13917]
>  [client 172.26.255.254:48758] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:34:31.982562 2021] [:error] [pid 13913] ipa: INFO: 401
>  Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI
>  Error: Unspecified GSS failure.  Minor code may provide more information
>  (Credential cache is empty)
>  [Wed Feb 10 17:34:32.015893 2021] [auth_gssapi:error] [pid 13914]
>  [client 172.26.255.254:49020] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:35:08.037058 2021] [auth_gssapi:error] [pid 13915]
>  [client 172.26.255.254:49624] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:38:08.183222 2021] [:warn] [pid 13916] [client
>  172.26.255.254:52646] failed to set perms (3140) on file
>  (/var/run/ipa/ccaches/admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:var/run/ipa/ccaches/[e-mail]>)!, referer:
>  https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:38:08.213367 2021] [:error] [pid 13911] ipa: INFO: 401
>  Unauthorized: Insufficient access:  Invalid credentials
>  [Wed Feb 10 17:38:08.256346 2021] [:error] [pid 13912] ipa: INFO: 401
>  Unauthorized: Insufficient access:  Invalid credentials
>  [Wed Feb 10 17:38:08.278769 2021] [:warn] [pid 13917] [client
>  172.26.255.254:52654] failed to set perms (3140) on file
>  (/var/run/ipa/ccaches/admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:var/run/ipa/ccaches/[e-mail]>)!, referer:
>  https://neumann2.cluster.cetene.gov.br/ipa/xml
>  
>  Just for the completude, removing the /etc/named.conf hack; this happens:
>  
>  [root@neumann2 ~]# ipactl start
>  Starting Directory Service
>  Starting krb5kdc Service
>  Starting kadmin Service
>  Starting named Service
>  Failed to start named Service
>  Shutting down
>  Hint: You can use --ignore-service-failure option for forced start in
>  case that a non-critical service failed
>  Aborting ipactl
>  
>  On /var/log/messages:
>  
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: set up managed keys zone
>  for view _default, file '/var/named/dynamic/managed-keys.bind'
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading DynDB instance
>  'ipa' driver '/usr/lib64/bind/ldap.so'
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: bind-dyndb-ldap version
>  11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red Hat
>  4.8.5-39)
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: LDAP error: Invalid
>  credentials: bind to LDAP server failed
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: couldn't establish
>  connection in LDAP connection pool: permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: dynamic database 'ipa'
>  configuration failed: permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading configuration:
>  permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: exiting (due to fatal error)
>  Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service: control process
>  exited, code=exited status=1
>  Feb 15 13:18:52 neumann2 systemd: Failed to start Berkeley Internet Name
>  Domain (DNS) with native PKCS#11.
>  Feb 15 13:18:52 neumann2 systemd: Unit named-pkcs11.service entered
>  failed state.
>  Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service failed.
>  Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 KDC...
>  Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 KDC.
>  Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 Password-changing
>  and Administration...
>  Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 Password-changing
>  and Administration.
>  Feb 15 13:18:52 neumann2 systemd: Stopping 389 Directory Server
>  CLUSTER-CETENE-GOV-BR....
>  
>  Thats it Rob.
>  
>  If there’s anything more that I should try or you need to see please let
>  me know.
>  
>  Thank you.
>  
> >
> > Check the Apache error log for more details.
> >
> > rob
> >
> >>
> >> Thank you.
> >>
> >>
> >>> On 12 Feb 2021, at 18:11, Rob Crittenden <rcritten(a)redhat.com
> >>> <mailto:[e-mail]>
> >>> <mailto:[e-mail]>> wrote:
> >>>
> >>> Just to confirm, the system is working with the exception of
> >>> ipa-dnskeysyncd.service?
> >>>
> >>> Does this work?
> >>>
> >>> # kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >>> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >>> <http://neumann2.cluster.cetene.gov.br/>
> >>> <http://neumann2.cluster.cetene.gov.br/>
> >>> # ipa user-show admin
> >>>
> >>> This will get a ticket and then use that ticket.
> >>>
> >>> rob
> >>>
> >>> Vinícius Ferrão via FreeIPA-users wrote:
> >>>> Hello,
> >>>>
> >>>> I still not sure of what is happening but, I got some interesting error
> >>>> message on ipa-healthcheck:
> >>>>
> >>>> [root@neumann2 keytabs]# ipa-healthcheck --failures-only
> >>>> --output-type human
> >>>> CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient
> >>>> access: 
> >>>> Invalid credentials
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/lib/ipa/backup/: free space percentage under threshold: 16% <
> 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /tmp:
> >>>> free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/log/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/tmp/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/log/audit/: free space percentage under threshold: 16% < 20%
> >>>>
> >>>> I tried to search for the critical message but nothing comes up.
> >>>> There’s
> >>>> a lot of GSSAPI errors on all logs.
> >>>>
> >>>> I tried to regenerate all keytabs of the system but it was a no go
> >>>> either:
> >>>> # gssproxy
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'HTTP/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /var/lib/ipa/gssproxy/http.keytab
> >>>>
> >>>> # Dogtag
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'dogtag/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /etc/pki/pki-tomcat/dogtag.keytab
> >>>>
> >>>> # DNSKeySync
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >>>>
> >>>> # Host Keytab
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'host/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/krb5.keytab
> >>>>
> >>>> # named
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'DNS/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/named.keytab
> >>>>
> >>>> # 389ds
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'ldap/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/dirsrv/ds.keytab
> >>>>
> >>>> Some error messages:
> >>>>
> >>>> [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49
> >>>> tag=97
> >>>> nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
> >>>> Unspecified GSS failure.  Minor code may provide more information
> >>>> (Cannot create replay cache file /var/tmp/ldap_389: Operation not
> >>>> permitted)
> >>>>
> >>>> ==> /var/log/messages <==
> >>>> Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
> >>>> over, scheduling restart.
> >>>> Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
> >>>> Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO    
> >>>> LDAP
> >>>> bind...
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR   
> >>>> Login to LDAP server failed: {'desc': 'Invalid
> credentials'}
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call
> >>>> last):
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
> >>>> ldap_connection.sasl_interactive_bind_s("",
> ipaldap.SASL_GSSAPI)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 850, in
> >>>> sasl_interactive_bind_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
> >>>>
> self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 818, in
> >>>> _apply_method_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >>>> func(self,*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 229, in
> >>>> sasl_interactive_bind_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >>>>
> self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 99, in
> >>>> _ldap_call
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS:
> {'desc':
> >>>> 'Invalid credentials'}
> >>>> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
> >>>> exited, code=exited, status=1/FAILURE
> >>>> Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
> >>>> failed state.
> >>>> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.
> >>>>
> >>>> Thanks,
> >>>>
> >>>>> On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
> >>>>> <freeipa-users(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it
> by
> >>>>> myself. After reading a lot of threads here on the list, it appears
> >>>>> that I’ve the same issue as this
> >>>>>
> topic: https://www.mail-archive.com/[e-mail]/...
> >>>>>
> >>>>> Since Kerberos is apparently not working as expected, I cannot use
> >>>>> FreeIPA and none of the services are working correctly. Following
> the
> >>>>> debug guide I was able to at least start named with single
> >>>>> authentication to further debug. (Workaround 1
> >>>>>
> of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)
> >>>>>
> >>>>> And now I’m stuck on item 5 of the same manual.
> >>>>>
> >>>>> [root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
> >>>>>
> 'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
> >>>>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
> >>>>> <ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y
> GSSAPI
> >>>>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
> >>>>> SASL/GSSAPI authentication started
> >>>>> [6588] 1612932571.244080: ccselect module realm chose cache
> >>>>> KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> for
> >>>>> server principal
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> [6588] 1612932571.244081: Getting credentials
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>>>> [6588] 1612932571.244082: Retrieving
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
> >>>>> [6588] 1612932571.244084: Creating authenticator for
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>,
> >>>>> seqnum 1040975659, subkey aes256-cts/48E9, session key
> aes256-cts/DF1E
> >>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
> >>>>>
> >>>>> [root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all
> --raw
> >>>>> ipa: ERROR: Insufficient access:  Invalid credentials
> >>>>>
> >>>>> [root@neumann2 ~]# klist
> >>>>> Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>>>> Default principal:
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> >>>>> Valid starting       Expires              Service principal
> >>>>> 02/10/2021 01:52:43  02/11/2021 01:49:04 
> >>>>> HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>> 02/10/2021 01:49:16  02/11/2021 01:49:04 
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> 02/10/2021 01:49:04  02/11/2021 01:49:04 
> >>>>> krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>>
> >>>>> Any ideia on how to fix this?
> >>>>>
> >>>>> Thanks,
> >>>>> Vinícius.
> >>>>>
> >>>>> PS: Before the workaround named-pkcs11 fails to start with the
> >>>>> following error:
> >>>>>
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys
> zone
> >>>>> for view _default, file
> '/var/named/dynamic/managed-keys.bind'
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
> >>>>> 'ipa' driver '/usr/lib64/bind/ldap.so'
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
> >>>>> 11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
> >>>>> Hat 4.8.5-39)
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
> >>>>> credentials: bind to LDAP server failed
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
> >>>>> connection in LDAP connection pool: permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database
> 'ipa'
> >>>>> configuration failed: permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
> >>>>> permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal
> >>>>> error)
> >>>>> Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
> >>>>> process exited, code=exited status=1
> >>>>> Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
> >>>>> Name Domain (DNS) with native PKCS#11.
> >>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> Fedora Code of Conduct:
> >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>>> List
> >>>>> Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>>> List Archives:
> >>>>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>> <mailto:[e-mail]>
> >>>> <mailto:[e-mail]>
> >>>> To unsubscribe send an email to
> >>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>> <mailto:[e-mail]>
> >>>> <mailto:[e-mail]>
> >>>> Fedora Code of Conduct:
> >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>> List Archives:
> >>>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>> Do not reply to spam on the list, report it:
> >>>> https://pagure.io/fedora-infrastructure
>