Kerberos appears to be broken on a FreeIPA server on CentOS 7.8

[Vinícius Ferrão]

Hello,

FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by myself. After
reading a lot of threads here on the list, it appears that I’ve the same issue as this
topic: https://www.mail-archive.com/[e-mail]/msg0550...

Since Kerberos is apparently not working as expected, I cannot use FreeIPA and none of the
services are working correctly. Following the debug guide I was able to at least start
named with single authentication to further debug. (Workaround 1 of
https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)

And now I’m stuck on item 5 of the same manual.

[root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI -b
'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br<ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket'%20-Y%20GSSAPI%20-b%20'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
SASL/GSSAPI authentication started
[6588] 1612932571.244080: ccselect module realm chose cache
KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
DNS/[e-mail]<mailto:DNS/[e-mail]>
for server principal
ldap/[e-mail]<mailto:ldap/[e-mail]>
[6588] 1612932571.244081: Getting credentials
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>
using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
[6588] 1612932571.244082: Retrieving
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>
from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
[6588] 1612932571.244084: Creating authenticator for
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>,
seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
ldap_sasl_interactive_bind_s: Invalid credentials (49)

[root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
ipa: ERROR: Insufficient access:  Invalid credentials

[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
Default principal:
DNS/[e-mail]<mailto:DNS/[e-mail]>

Valid starting       Expires              Service principal
02/10/2021 01:52:43  02/11/2021 01:49:04 
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
02/10/2021 01:49:16  02/11/2021 01:49:04 
ldap/[e-mail]<mailto:ldap/[e-mail]>
02/10/2021 01:49:04  02/11/2021 01:49:04 
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>

Any ideia on how to fix this?

Thanks,
Vinícius.

PS: Before the workaround named-pkcs11 fails to start with the following error:

Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone for view _default,
file '/var/named/dynamic/managed-keys.bind'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance 'ipa' driver
'/usr/lib64/bind/ldap.so'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version 11.1 compiled at
02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red Hat 4.8.5-39)
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid credentials: bind to LDAP
server failed
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish connection in LDAP
connection pool: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa' configuration
failed: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal error)
Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control process exited,
code=exited status=1
Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet Name Domain (DNS) with
native PKCS#11.

[Vinícius Ferrão]

Hello,

I still not sure of what is happening but, I got some interesting error message on
ipa-healthcheck:

[root@neumann2 keytabs]# ipa-healthcheck --failures-only --output-type human
CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient access:  Invalid
credentials
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /var/lib/ipa/backup/:
free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /tmp: free space
percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /var/lib/dirsrv/: free
space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /var/log/: free space
percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /var/tmp/: free space
percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /var/log/audit/: free
space percentage under threshold: 16% < 20%

I tried to search for the critical message but nothing comes up. There’s a lot of GSSAPI
errors on all logs.

I tried to regenerate all keytabs of the system but it was a no go either:
# gssproxy
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'HTTP/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /var/lib/ipa/gssproxy/http.keytab

# Dogtag
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'dogtag/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /etc/pki/pki-tomcat/dogtag.keytab

# DNSKeySync
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /etc/ipa/dnssec/ipa-dnskeysyncd.keytab

# Host Keytab
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'host/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /etc/krb5.keytab

# named
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'DNS/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /etc/named.keytab

# 389ds
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br> -p
'ldap/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>'
-r -k /etc/dirsrv/ds.keytab

Some error messages:

[10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97 nentries=0
etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. 
Minor code may provide more information (Cannot create replay cache file
/var/tmp/ldap_389: Operation not permitted)

==> /var/log/messages <==
Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time over, scheduling
restart.
Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO     LDAP bind...
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR    Login to LDAP server
failed: {'desc': 'Invalid credentials'}
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call last):
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in
_apply_method_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return func(self,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in
_ldap_call
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc':
'Invalid credentials'}
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process exited,
code=exited, status=1/FAILURE
Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered failed state.
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.

Thanks,

On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
<[e-mail]<mailto:[e-mail]>>
wrote:

Hello,

FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by myself. After
reading a lot of threads here on the list, it appears that I’ve the same issue as this
topic: https://www.mail-archive.com/[e-mail]/msg0550...

Since Kerberos is apparently not working as expected, I cannot use FreeIPA and none of the
services are working correctly. Following the debug guide I was able to at least start
named with single authentication to further debug. (Workaround 1 of
https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)

And now I’m stuck on item 5 of the same manual.

[root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI -b
'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br<ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket'%20-Y%20GSSAPI%20-b%20'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
SASL/GSSAPI authentication started
[6588] 1612932571.244080: ccselect module realm chose cache
KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
DNS/[e-mail]<mailto:DNS/[e-mail]>
for server principal
ldap/[e-mail]<mailto:ldap/[e-mail]>
[6588] 1612932571.244081: Getting credentials
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>
using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
[6588] 1612932571.244082: Retrieving
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>
from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
[6588] 1612932571.244084: Creating authenticator for
DNS/[e-mail]<mailto:DNS/[e-mail]>
->
ldap/[e-mail]<mailto:ldap/[e-mail]>,
seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
ldap_sasl_interactive_bind_s: Invalid credentials (49)

[root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
ipa: ERROR: Insufficient access:  Invalid credentials

[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
Default principal:
DNS/[e-mail]<mailto:DNS/[e-mail]>

Valid starting       Expires              Service principal
02/10/2021 01:52:43  02/11/2021 01:49:04 
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
02/10/2021 01:49:16  02/11/2021 01:49:04 
ldap/[e-mail]<mailto:ldap/[e-mail]>
02/10/2021 01:49:04  02/11/2021 01:49:04 
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>

Any ideia on how to fix this?

Thanks,
Vinícius.

PS: Before the workaround named-pkcs11 fails to start with the following error:

Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone for view _default,
file '/var/named/dynamic/managed-keys.bind'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance 'ipa' driver
'/usr/lib64/bind/ldap.so'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version 11.1 compiled at
02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red Hat 4.8.5-39)
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid credentials: bind to LDAP
server failed
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish connection in LDAP
connection pool: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa' configuration
failed: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal error)
Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control process exited,
code=exited status=1
Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet Name Domain (DNS) with
native PKCS#11.

_______________________________________________
FreeIPA-users mailing list --
[e-mail]<mailto:[e-mail]>
To unsubscribe send an email to
[e-mail]<mailto:[e-mail]>
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...

[Rob Crittenden]

Just to confirm, the system is working with the exception of
ipa-dnskeysyncd.service?

Does this work?

# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
# ipa user-show admin

This will get a ticket and then use that ticket.

rob

Vinícius Ferrão via FreeIPA-users wrote:
>  Hello,
>  
>  I still not sure of what is happening but, I got some interesting error
>  message on ipa-healthcheck:
>  
>  [root@neumann2 keytabs]# ipa-healthcheck --failures-only --output-type human
>  CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient access: 
>  Invalid credentials
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
>  /var/lib/ipa/backup/: free space percentage under threshold: 16% < 20%
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /tmp:
>  free space percentage under threshold: 16% < 20%
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
>  /var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
>  /var/log/: free space percentage under threshold: 16% < 20%
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
>  /var/tmp/: free space percentage under threshold: 16% < 20%
>  ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
>  /var/log/audit/: free space percentage under threshold: 16% < 20%
>  
>  I tried to search for the critical message but nothing comes up. There’s
>  a lot of GSSAPI errors on all logs.
>  
>  I tried to regenerate all keytabs of the system but it was a no go either:
>  # gssproxy
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'HTTP/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k
>  /var/lib/ipa/gssproxy/http.keytab
>  
>  # Dogtag
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'dogtag/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k
>  /etc/pki/pki-tomcat/dogtag.keytab
>  
>  # DNSKeySync
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k
>  /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
>  
>  # Host Keytab
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'host/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k /etc/krb5.keytab
>  
>  # named
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'DNS/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k /etc/named.keytab
>  
>  # 389ds
>  ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
>  neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  -p 'ldap/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>' -r -k /etc/dirsrv/ds.keytab
>  
>  Some error messages:
>  
>  [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97
>  nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
>  Unspecified GSS failure.  Minor code may provide more information
>  (Cannot create replay cache file /var/tmp/ldap_389: Operation not permitted)
>  
>  ==> /var/log/messages <==
>  Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
>  over, scheduling restart.
>  Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
>  Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO     LDAP
>  bind...
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR   
>  Login to LDAP server failed: {'desc': 'Invalid credentials'}
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call last):
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
>  "/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
>  ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI)
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
>  "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in
>  sasl_interactive_bind_s
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
>  self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
>  "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in
>  _apply_method_s
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return func(self,*args,**kwargs)
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
>  "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in
>  sasl_interactive_bind_s
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
>
> self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
>  "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in
>  _ldap_call
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
>  Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc':
>  'Invalid credentials'}
>  Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
>  exited, code=exited, status=1/FAILURE
>  Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
>  failed state.
>  Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.
>  
>  Thanks,
>  
> > On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
> > <freeipa-users(a)lists.fedorahosted.org
> > <mailto:[e-mail]>> wrote:
> >
> > Hello,
> >
> > FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by
> > myself. After reading a lot of threads here on the list, it appears
> > that I’ve the same issue as this
> >
> topic: https://www.mail-archive.com/[e-mail]/...
> >
> > Since Kerberos is apparently not working as expected, I cannot use
> > FreeIPA and none of the services are working correctly. Following the
> > debug guide I was able to at least start named with single
> > authentication to further debug. (Workaround 1
> > of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)
> >
> > And now I’m stuck on item 5 of the same manual.
> >
> > [root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
> > 'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
> > -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
> > <ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
> > -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
> > SASL/GSSAPI authentication started
> > [6588] 1612932571.244080: ccselect module realm chose cache
> > KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
> > DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:DNS/[e-mail]> for
> > server principal
> > ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:ldap/[e-mail]>
> > [6588] 1612932571.244081: Getting credentials
> > DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:DNS/[e-mail]> ->
> > ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:ldap/[e-mail]>
> > using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
> > [6588] 1612932571.244082: Retrieving
> > DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:DNS/[e-mail]> ->
> > ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:ldap/[e-mail]>
> > from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
> > [6588] 1612932571.244084: Creating authenticator for
> > DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:DNS/[e-mail]> ->
> > ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:ldap/[e-mail]>,
> > seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
> > ldap_sasl_interactive_bind_s: Invalid credentials (49)
> >
> > [root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
> > ipa: ERROR: Insufficient access:  Invalid credentials
> >
> > [root@neumann2 ~]# klist
> > Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
> > Default principal:
> > DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:DNS/[e-mail]>
> >
> > Valid starting       Expires              Service principal
> > 02/10/2021 01:52:43  02/11/2021 01:49:04 
> > HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:HTTP/[e-mail]>
> > 02/10/2021 01:49:16  02/11/2021 01:49:04 
> > ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> > <mailto:ldap/[e-mail]>
> > 02/10/2021 01:49:04  02/11/2021 01:49:04 
> > krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> > <mailto:krbtgt/[e-mail]>
> >
> > Any ideia on how to fix this?
> >
> > Thanks,
> > Vinícius.
> >
> > PS: Before the workaround named-pkcs11 fails to start with the
> > following error:
> >
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone
> > for view _default, file '/var/named/dynamic/managed-keys.bind'
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
> > 'ipa' driver '/usr/lib64/bind/ldap.so'
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
> > 11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
> > Hat 4.8.5-39)
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
> > credentials: bind to LDAP server failed
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
> > connection in LDAP connection pool: permission denied
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa'
> > configuration failed: permission denied
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
> > permission denied
> > Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal error)
> > Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
> > process exited, code=exited status=1
> > Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
> > Name Domain (DNS) with native PKCS#11.
> >
> > _______________________________________________
> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > <mailto:[e-mail]>
> > To unsubscribe send an email to
> > freeipa-users-leave(a)lists.fedorahosted.org
> > <mailto:[e-mail]>
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedorahosted.org/archives/list/[e-mail]...
>  
>  
>  _______________________________________________
>  FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
>

[Vinícius Ferrão]

Hi Rob.

Actually nothing that relies on Kerberos Keytabs is working.

I can properly issue kinit’s and login, but I can’t use ‘ipa’ commands for instance.
named-pkcs11 is only starting up because I’ve changed the authentication method on
/etc/named.conf:

/* WARNING: This part of the config file is IPA-managed.
 * Modifications may break IPA setup or upgrades.
 */
dyndb "ipa" "/usr/lib64/bind/ldap.so" {
uri "ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket";
base "cn=dns, dc=cluster,dc=cetene,dc=gov,dc=br";
server_id
"neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>";
#auth_method "sasl";
#sasl_mech "GSSAPI";
#sasl_user
"DNS/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>";
/* Desespero */
auth_method "simple";
bind_dn "uid=admin,cn=users,cn=accounts,dc=cluster,dc=cetene,dc=gov,dc=br";
password “REDACTED";
};
/* End of IPA-managed part. */

I’ve done the test that you’ve asked, and was a no go:

[root@neumann2 ~]# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>
[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
Default principal:
ipa-dnskeysyncd/[e-mail]<mailto:ipa-dnskeysyncd/[e-mail]>

Valid starting       Expires              Service principal
02/12/2021 22:42:03  02/13/2021 22:42:03 
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
[root@neumann2 ~]# ipa user-show admin
ipa: ERROR: Insufficient access:  Invalid credentials
[root@neumann2 ~]# ipa -v user-show admin
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 1]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 2]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 3]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 4]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 5]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.

I never seen this on FreeIPA.

Subsequent queries of IPA commands just returns the same error:

[root@neumann2 ~]# ipa user-show admin
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.

Thank you.


On 12 Feb 2021, at 18:11, Rob Crittenden
<[e-mail]<mailto:[e-mail]>> wrote:

Just to confirm, the system is working with the exception of
ipa-dnskeysyncd.service?

Does this work?

# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
# ipa user-show admin

This will get a ticket and then use that ticket.

rob

Vinícius Ferrão via FreeIPA-users wrote:
Hello,

I still not sure of what is happening but, I got some interesting error
message on ipa-healthcheck:

[root@neumann2 keytabs]# ipa-healthcheck --failures-only --output-type human
CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient access:
Invalid credentials
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/lib/ipa/backup/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /tmp:
free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/log/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/tmp/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/log/audit/: free space percentage under threshold: 16% < 20%

I tried to search for the critical message but nothing comes up. There’s
a lot of GSSAPI errors on all logs.

I tried to regenerate all keytabs of the system but it was a no go either:
# gssproxy
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p
'HTTP/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k
/var/lib/ipa/gssproxy/http.keytab

# Dogtag
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p
'dogtag/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k
/etc/pki/pki-tomcat/dogtag.keytab

# DNSKeySync
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p
'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k
/etc/ipa/dnssec/ipa-dnskeysyncd.keytab

# Host Keytab
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p
'host/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k /etc/krb5.keytab

# named
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p 'DNS/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k /etc/named.keytab

# 389ds
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-p
'ldap/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
-r -k /etc/dirsrv/ds.keytab

Some error messages:

[10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97
nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
Unspecified GSS failure.  Minor code may provide more information
(Cannot create replay cache file /var/tmp/ldap_389: Operation not permitted)

==> /var/log/messages <==
Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
over, scheduling restart.
Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO     LDAP
bind...
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR
Login to LDAP server failed: {'desc': 'Invalid credentials'}
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call last):
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in
_apply_method_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return func(self,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in
_ldap_call
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc':
'Invalid credentials'}
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
exited, code=exited, status=1/FAILURE
Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
failed state.
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.

Thanks,

On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
<[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>> wrote:

Hello,

FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by
myself. After reading a lot of threads here on the list, it appears
that I’ve the same issue as this
topic: https://www.mail-archive.com/[e-mail]/msg0550...

Since Kerberos is apparently not working as expected, I cannot use
FreeIPA and none of the services are working correctly. Following the
debug guide I was able to at least start named with single
authentication to further debug. (Workaround 1
of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)

And now I’m stuck on item 5 of the same manual.

[root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
-b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
<ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
-b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
SASL/GSSAPI authentication started
[6588] 1612932571.244080: ccselect module realm chose cache
KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> for
server principal
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
[6588] 1612932571.244081: Getting credentials
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
[6588] 1612932571.244082: Retrieving
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
[6588] 1612932571.244084: Creating authenticator for
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>,
seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
ldap_sasl_interactive_bind_s: Invalid credentials (49)

[root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
ipa: ERROR: Insufficient access:  Invalid credentials

[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
Default principal:
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>

Valid starting       Expires              Service principal
02/10/2021 01:52:43  02/11/2021 01:49:04
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
<mailto:HTTP/[e-mail]>
02/10/2021 01:49:16  02/11/2021 01:49:04
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
02/10/2021 01:49:04  02/11/2021 01:49:04
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
<mailto:krbtgt/[e-mail]>

Any ideia on how to fix this?

Thanks,
Vinícius.

PS: Before the workaround named-pkcs11 fails to start with the
following error:

Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone
for view _default, file '/var/named/dynamic/managed-keys.bind'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
'ipa' driver '/usr/lib64/bind/ldap.so'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
Hat 4.8.5-39)
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
credentials: bind to LDAP server failed
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
connection in LDAP connection pool: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa'
configuration failed: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal error)
Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
process exited, code=exited status=1
Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
Name Domain (DNS) with native PKCS#11.

_______________________________________________
FreeIPA-users mailing list --
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
To unsubscribe send an email to
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...


_______________________________________________
FreeIPA-users mailing list --
[e-mail]<mailto:[e-mail]>
To unsubscribe send an email to
[e-mail]<mailto:[e-mail]>
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Rob Crittenden]

Vinícius Ferrão wrote:
>  Hi Rob.
>  
>  Actually nothing that relies on Kerberos Keytabs is working. 
Kerberos is working. The kinit was successful.

>  I can properly issue kinit’s and login, but I can’t use ‘ipa’
> commands
>  for instance. named-pkcs11 is only starting up because I’ve changed the
>  authentication method on /etc/named.conf:
>  
>  /* WARNING: This part of the config file is IPA-managed.
>   * Modifications may break IPA setup or upgrades.
>   */
>  dyndb "ipa" "/usr/lib64/bind/ldap.so" {
>  uri "ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket";
>  base "cn=dns, dc=cluster,dc=cetene,dc=gov,dc=br";
>  server_id "neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>";
>  #auth_method "sasl";
>  #sasl_mech "GSSAPI";
>  #sasl_user "DNS/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>";
>  /* Desespero */
>  auth_method "simple";
>  bind_dn "uid=admin,cn=users,cn=accounts,dc=cluster,dc=cetene,dc=gov,dc=br";
>  password “REDACTED";
>  };
>  /* End of IPA-managed part. */
>  
>  I’ve done the test that you’ve asked, and was a no go:
>  
>  [root@neumann2 ~]# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
>  ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
>  <http://neumann2.cluster.cetene.gov.br>
>  [root@neumann2 ~]# klist
>  Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
>  Default principal:
>  ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:ipa-dnskeysyncd/[e-mail]>
>  
>  Valid starting       Expires              Service principal
>  02/12/2021 22:42:03  02/13/2021 22:42:03 
>  krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
>  <mailto:krbtgt/[e-mail]>
>  [root@neumann2 ~]# ipa user-show admin
>  ipa: ERROR: Insufficient access:  Invalid credentials
>  [root@neumann2 ~]# ipa -v user-show admin
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: INFO: [try 1]: Forwarding 'schema' to json server
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: INFO: [try 2]: Forwarding 'schema' to json server
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: INFO: [try 3]: Forwarding 'schema' to json server
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: INFO: [try 4]: Forwarding 'schema' to json server
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: INFO: [try 5]: Forwarding 'schema' to json server
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
>  ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  
>  I never seen this on FreeIPA.
>  
>  Subsequent queries of IPA commands just returns the same error:
>  
>  [root@neumann2 ~]# ipa user-show admin
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request. 
Did you get a HTTP service ticket? (klist)

Check the Apache error log for more details.

rob

>  
>  Thank you.
>  
>  
> > On 12 Feb 2021, at 18:11, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:[e-mail]>> wrote:
> >
> > Just to confirm, the system is working with the exception of
> > ipa-dnskeysyncd.service?
> >
> > Does this work?
> >
> > # kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> > ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> > <http://neumann2.cluster.cetene.gov.br/>
> > # ipa user-show admin
> >
> > This will get a ticket and then use that ticket.
> >
> > rob
> >
> > Vinícius Ferrão via FreeIPA-users wrote:
> >> Hello,
> >>
> >> I still not sure of what is happening but, I got some interesting error
> >> message on ipa-healthcheck:
> >>
> >> [root@neumann2 keytabs]# ipa-healthcheck --failures-only
> >> --output-type human
> >> CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient access: 
> >> Invalid credentials
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >> /var/lib/ipa/backup/: free space percentage under threshold: 16% < 20%
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /tmp:
> >> free space percentage under threshold: 16% < 20%
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >> /var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >> /var/log/: free space percentage under threshold: 16% < 20%
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >> /var/tmp/: free space percentage under threshold: 16% < 20%
> >> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >> /var/log/audit/: free space percentage under threshold: 16% < 20%
> >>
> >> I tried to search for the critical message but nothing comes up. There’s
> >> a lot of GSSAPI errors on all logs.
> >>
> >> I tried to regenerate all keytabs of the system but it was a no go
> >> either:
> >> # gssproxy
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'HTTP/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >> /var/lib/ipa/gssproxy/http.keytab
> >>
> >> # Dogtag
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'dogtag/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >> /etc/pki/pki-tomcat/dogtag.keytab
> >>
> >> # DNSKeySync
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >> /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >>
> >> # Host Keytab
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'host/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k /etc/krb5.keytab
> >>
> >> # named
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'DNS/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k /etc/named.keytab
> >>
> >> # 389ds
> >> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
> >> neumann2.cluster.cetene.gov.br
> >>
> <http://neumann2.cluster.cetene.gov.br/> <http://neumann2.cluster.ce...
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> -p 'ldap/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/dirsrv/ds.keytab
> >>
> >> Some error messages:
> >>
> >> [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97
> >> nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
> >> Unspecified GSS failure.  Minor code may provide more information
> >> (Cannot create replay cache file /var/tmp/ldap_389: Operation not
> >> permitted)
> >>
> >> ==> /var/log/messages <==
> >> Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
> >> over, scheduling restart.
> >> Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
> >> Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO     LDAP
> >> bind...
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR   
> >> Login to LDAP server failed: {'desc': 'Invalid credentials'}
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call
> >> last):
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >> "/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
> >> ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI)
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in
> >> sasl_interactive_bind_s
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
> >> self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in
> >> _apply_method_s
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >> func(self,*args,**kwargs)
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in
> >> sasl_interactive_bind_s
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >>
> self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in
> >> _ldap_call
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
> >> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc':
> >> 'Invalid credentials'}
> >> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
> >> exited, code=exited, status=1/FAILURE
> >> Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
> >> failed state.
> >> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.
> >>
> >> Thanks,
> >>
> >>> On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
> >>> <freeipa-users(a)lists.fedorahosted.org
> >>> <mailto:[e-mail]>
> >>> <mailto:[e-mail]>> wrote:
> >>>
> >>> Hello,
> >>>
> >>> FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by
> >>> myself. After reading a lot of threads here on the list, it appears
> >>> that I’ve the same issue as this
> >>>
> topic: https://www.mail-archive.com/[e-mail]/...
> >>>
> >>> Since Kerberos is apparently not working as expected, I cannot use
> >>> FreeIPA and none of the services are working correctly. Following the
> >>> debug guide I was able to at least start named with single
> >>> authentication to further debug. (Workaround 1
> >>> of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)
> >>>
> >>> And now I’m stuck on item 5 of the same manual.
> >>>
> >>> [root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
> >>> 'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y
> GSSAPI
> >>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
> >>> <ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
> >>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
> >>> SASL/GSSAPI authentication started
> >>> [6588] 1612932571.244080: ccselect module realm chose cache
> >>> KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
> >>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:DNS/[e-mail]>
> >>> <mailto:DNS/[e-mail]> for
> >>> server principal
> >>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:ldap/[e-mail]>
> >>> <mailto:ldap/[e-mail]>
> >>> [6588] 1612932571.244081: Getting credentials
> >>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:DNS/[e-mail]>
> >>> <mailto:DNS/[e-mail]>
> ->
> >>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:ldap/[e-mail]>
> >>> <mailto:ldap/[e-mail]>
> >>> using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>> [6588] 1612932571.244082: Retrieving
> >>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:DNS/[e-mail]>
> >>> <mailto:DNS/[e-mail]>
> ->
> >>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:ldap/[e-mail]>
> >>> <mailto:ldap/[e-mail]>
> >>> from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
> >>> [6588] 1612932571.244084: Creating authenticator for
> >>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:DNS/[e-mail]>
> >>> <mailto:DNS/[e-mail]>
> ->
> >>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:ldap/[e-mail]>
> >>> <mailto:ldap/[e-mail]>,
> >>> seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
> >>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
> >>>
> >>> [root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
> >>> ipa: ERROR: Insufficient access:  Invalid credentials
> >>>
> >>> [root@neumann2 ~]# klist
> >>> Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>> Default principal:
> >>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:DNS/[e-mail]>
> >>> <mailto:DNS/[e-mail]>
> >>>
> >>> Valid starting       Expires              Service principal
> >>> 02/10/2021 01:52:43  02/11/2021 01:49:04 
> >>> HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:HTTP/[e-mail]>
> >>> <mailto:HTTP/[e-mail]>
> >>> 02/10/2021 01:49:16  02/11/2021 01:49:04 
> >>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:ldap/[e-mail]>
> >>> <mailto:ldap/[e-mail]>
> >>> 02/10/2021 01:49:04  02/11/2021 01:49:04 
> >>> krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> >>> <mailto:krbtgt/[e-mail]>
> >>> <mailto:krbtgt/[e-mail]>
> >>>
> >>> Any ideia on how to fix this?
> >>>
> >>> Thanks,
> >>> Vinícius.
> >>>
> >>> PS: Before the workaround named-pkcs11 fails to start with the
> >>> following error:
> >>>
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone
> >>> for view _default, file '/var/named/dynamic/managed-keys.bind'
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
> >>> 'ipa' driver '/usr/lib64/bind/ldap.so'
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
> >>> 11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
> >>> Hat 4.8.5-39)
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
> >>> credentials: bind to LDAP server failed
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
> >>> connection in LDAP connection pool: permission denied
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa'
> >>> configuration failed: permission denied
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
> >>> permission denied
> >>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal
> >>> error)
> >>> Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
> >>> process exited, code=exited status=1
> >>> Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
> >>> Name Domain (DNS) with native PKCS#11.
> >>>
> >>> _______________________________________________
> >>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>> <mailto:[e-mail]>
> >>> <mailto:[e-mail]>
> >>> To unsubscribe send an email to
> >>> freeipa-users-leave(a)lists.fedorahosted.org
> >>> <mailto:[e-mail]>
> >>> <mailto:[e-mail]>
> >>> Fedora Code of Conduct:
> >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> List Archives:
> >>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>
> >>
> >> _______________________________________________
> >> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >> <mailto:[e-mail]>
> >> To unsubscribe send an email to
> >> freeipa-users-leave(a)lists.fedorahosted.org
> >> <mailto:[e-mail]>
> >> Fedora Code of Conduct:
> >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >> List Archives:
> >>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >> Do not reply to spam on the list, report it:
> >> https://pagure.io/fedora-infrastructure
>

[Vinícius Ferrão]

Hi Rob.

On 15 Feb 2021, at 10:58, Rob Crittenden
<[e-mail]<mailto:[e-mail]>> wrote:

Vinícius Ferrão wrote:
Hi Rob.

Actually nothing that relies on Kerberos Keytabs is working.

Kerberos is working. The kinit was successful.

Sorry perhaps I didn’t say it correctly. In fact Kerberos is working (I can kinit) but
anything that relies on Keytabs, specifically Keytabs, aren’t working.

named-pkcs11 does not start without the hack that I’ve mentioned. Please correct me if I’m
wrong about this.

Every other service fails with “insufficient credentials”; dogtag, gssproxy, etc.

I can properly issue kinit’s and login, but I can’t use ‘ipa’ commands
for instance. named-pkcs11 is only starting up because I’ve changed the
authentication method on /etc/named.conf:

/* WARNING: This part of the config file is IPA-managed.
 * Modifications may break IPA setup or upgrades.
 */
dyndb "ipa" "/usr/lib64/bind/ldap.so" {
uri "ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket";
base "cn=dns, dc=cluster,dc=cetene,dc=gov,dc=br";
server_id
"neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...;
#auth_method "sasl";
#sasl_mech "GSSAPI";
#sasl_user
"DNS/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...;
/* Desespero */
auth_method "simple";
bind_dn "uid=admin,cn=users,cn=accounts,dc=cluster,dc=cetene,dc=gov,dc=br";
password “REDACTED";
};
/* End of IPA-managed part. */

I’ve done the test that you’ve asked, and was a no go:

[root@neumann2 ~]# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
Default principal:
ipa-dnskeysyncd/[e-mail]<mailto:ipa-dnskeysyncd/[e-mail]>
<mailto:ipa-dnskeysyncd/[e-mail]>

Valid starting       Expires              Service principal
02/12/2021 22:42:03  02/13/2021 22:42:03
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
<mailto:krbtgt/[e-mail]>
[root@neumann2 ~]# ipa user-show admin
ipa: ERROR: Insufficient access:  Invalid credentials
[root@neumann2 ~]# ipa -v user-show admin
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 1]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 2]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 3]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 4]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: INFO: [try 5]: Forwarding 'schema' to json server
'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
number of tries to forward a request.

I never seen this on FreeIPA.

Subsequent queries of IPA commands just returns the same error:

[root@neumann2 ~]# ipa user-show admin
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
number of tries to forward a request.

Did you get a HTTP service ticket? (klist)

I issued and admin ticket as I usually do:

[root@neumann2 ~]# kinit admin
Password for [e-mail]<mailto:[e-mail]>:
[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
Default principal: [e-mail]<mailto:[e-mail]>

Valid starting       Expires              Service principal
02/15/2021 13:09:04  02/16/2021 13:09:04 
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
[root@neumann2 ~]# ipa user-list
ipa: ERROR: Insufficient access:  Invalid credentials
[root@neumann2 ~]# ipa user-list
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.
[root@neumann2 ~]# ipa user-list
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.

But I can recover the HTTP ticket and kinit:

[root@neumann2 ~]# klist -kt /var/lib/ipa/gssproxy/http.keytab
Keytab name: FILE:/var/lib/ipa/gssproxy/http.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   3 02/10/2021 22:52:34
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
   3 02/10/2021 22:52:34
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
[root@neumann2 ~]# kinit -kt /var/lib/ipa/gssproxy/http.keytab
HTTP/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br>
[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_JRv9hJN
Default principal:
HTTP/[e-mail]<mailto:HTTP/[e-mail]>

Valid starting       Expires              Service principal
02/15/2021 13:13:47  02/16/2021 13:13:47 
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
[root@neumann2 ~]# ipa user-list
ipa: ERROR: Insufficient access:  Invalid credentials
[root@neumann2 ~]# ipa user-list
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.
[root@neumann2 ~]# ipa user-list
ipa: ERROR: cannot connect to
'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded number of
tries to forward a request.

But again it didn’t work.

On /var/log/httpd/error_log there basically this:

[Wed Feb 10 17:34:19.129505 2021] [:error] [pid 13912] ipa: INFO: 401 Unauthorized:
Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. 
Minor code may provide more information (Credential cache is empty)
[Wed Feb 10 17:34:19.151811 2021] [auth_gssapi:error] [pid 13917] [client
172.26.255.254:48758] GSS ERROR gss_acquire_cred[_from]() failed to get server creds:
[Unspecified GSS failure.  Minor code may provide more information ( SPNEGO cannot find
mechanisms to negotiate)], referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
[Wed Feb 10 17:34:31.982562 2021] [:error] [pid 13913] ipa: INFO: 401 Unauthorized:
Insufficient access: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. 
Minor code may provide more information (Credential cache is empty)
[Wed Feb 10 17:34:32.015893 2021] [auth_gssapi:error] [pid 13914] [client
172.26.255.254:49020] GSS ERROR gss_acquire_cred[_from]() failed to get server creds:
[Unspecified GSS failure.  Minor code may provide more information ( SPNEGO cannot find
mechanisms to negotiate)], referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
[Wed Feb 10 17:35:08.037058 2021] [auth_gssapi:error] [pid 13915] [client
172.26.255.254:49624] GSS ERROR gss_acquire_cred[_from]() failed to get server creds:
[Unspecified GSS failure.  Minor code may provide more information ( SPNEGO cannot find
mechanisms to negotiate)], referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
[Wed Feb 10 17:38:08.183222 2021] [:warn] [pid 13916] [client 172.26.255.254:52646] failed
to set perms (3140) on file
(/var/run/ipa/ccaches/[e-mail]<mailto:var/run/ipa/ccaches/[e-mail]>)!,
referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
[Wed Feb 10 17:38:08.213367 2021] [:error] [pid 13911] ipa: INFO: 401 Unauthorized:
Insufficient access:  Invalid credentials
[Wed Feb 10 17:38:08.256346 2021] [:error] [pid 13912] ipa: INFO: 401 Unauthorized:
Insufficient access:  Invalid credentials
[Wed Feb 10 17:38:08.278769 2021] [:warn] [pid 13917] [client 172.26.255.254:52654] failed
to set perms (3140) on file
(/var/run/ipa/ccaches/[e-mail]<mailto:var/run/ipa/ccaches/[e-mail]>)!,
referer: https://neumann2.cluster.cetene.gov.br/ipa/xml

Just for the completude, removing the /etc/named.conf hack; this happens:

[root@neumann2 ~]# ipactl start
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Failed to start named Service
Shutting down
Hint: You can use --ignore-service-failure option for forced start in case that a
non-critical service failed
Aborting ipactl

On /var/log/messages:

Feb 15 13:18:52 neumann2 named-pkcs11[32027]: set up managed keys zone for view _default,
file '/var/named/dynamic/managed-keys.bind'
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading DynDB instance 'ipa' driver
'/usr/lib64/bind/ldap.so'
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: bind-dyndb-ldap version 11.1 compiled at
02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red Hat 4.8.5-39)
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: LDAP error: Invalid credentials: bind to
LDAP server failed
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: couldn't establish connection in LDAP
connection pool: permission denied
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: dynamic database 'ipa' configuration
failed: permission denied
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading configuration: permission denied
Feb 15 13:18:52 neumann2 named-pkcs11[32027]: exiting (due to fatal error)
Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service: control process exited,
code=exited status=1
Feb 15 13:18:52 neumann2 systemd: Failed to start Berkeley Internet Name Domain (DNS) with
native PKCS#11.
Feb 15 13:18:52 neumann2 systemd: Unit named-pkcs11.service entered failed state.
Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service failed.
Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 KDC...
Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 KDC.
Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 Password-changing and
Administration...
Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 Password-changing and
Administration.
Feb 15 13:18:52 neumann2 systemd: Stopping 389 Directory Server
CLUSTER-CETENE-GOV-BR....

Thats it Rob.

If there’s anything more that I should try or you need to see please let me know.

Thank you.


Check the Apache error log for more details.

rob


Thank you.


On 12 Feb 2021, at 18:11, Rob Crittenden
<[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>> wrote:

Just to confirm, the system is working with the exception of
ipa-dnskeysyncd.service?

Does this work?

# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
# ipa user-show admin

This will get a ticket and then use that ticket.

rob

Vinícius Ferrão via FreeIPA-users wrote:
Hello,

I still not sure of what is happening but, I got some interesting error
message on ipa-healthcheck:

[root@neumann2 keytabs]# ipa-healthcheck --failures-only
--output-type human
CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient access:
Invalid credentials
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/lib/ipa/backup/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck: /tmp:
free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/log/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/tmp/: free space percentage under threshold: 16% < 20%
ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
/var/log/audit/: free space percentage under threshold: 16% < 20%

I tried to search for the critical message but nothing comes up. There’s
a lot of GSSAPI errors on all logs.

I tried to regenerate all keytabs of the system but it was a no go
either:
# gssproxy
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p
'HTTP/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k
/var/lib/ipa/gssproxy/http.keytab

# Dogtag
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p
'dogtag/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k
/etc/pki/pki-tomcat/dogtag.keytab

# DNSKeySync
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p
'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k
/etc/ipa/dnssec/ipa-dnskeysyncd.keytab

# Host Keytab
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p
'host/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k /etc/krb5.keytab

# named
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p 'DNS/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k /etc/named.keytab

# 389ds
ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr -s
neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>
-p
'ldap/neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br/>
<http://neumann2.cluster.cetene.gov.br<http://neumann2.cluster.cetene.g...
<http://neumann2.cluster.cetene.gov.br/>>' -r -k /etc/dirsrv/ds.keytab

Some error messages:

[10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97
nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
Unspecified GSS failure.  Minor code may provide more information
(Cannot create replay cache file /var/tmp/ldap_389: Operation not
permitted)

==> /var/log/messages <==
Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
over, scheduling restart.
Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO     LDAP
bind...
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR
Login to LDAP server failed: {'desc': 'Invalid credentials'}
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call
last):
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
ldap_connection.sasl_interactive_bind_s("", ipaldap.SASL_GSSAPI)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 850, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 818, in
_apply_method_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
func(self,*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 229, in
sasl_interactive_bind_s
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
"/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line 99, in
_ldap_call
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS: {'desc':
'Invalid credentials'}
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
exited, code=exited, status=1/FAILURE
Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
failed state.
Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.

Thanks,

On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
<[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
<mailto:[e-mail]>> wrote:

Hello,

FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it by
myself. After reading a lot of threads here on the list, it appears
that I’ve the same issue as this
topic: https://www.mail-archive.com/[e-mail]/msg0550...

Since Kerberos is apparently not working as expected, I cannot use
FreeIPA and none of the services are working correctly. Following the
debug guide I was able to at least start named with single
authentication to further debug. (Workaround 1
of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)

And now I’m stuck on item 5 of the same manual.

[root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
-b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
<ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
-b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
SASL/GSSAPI authentication started
[6588] 1612932571.244080: ccselect module realm chose cache
KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> for
server principal
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
[6588] 1612932571.244081: Getting credentials
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
[6588] 1612932571.244082: Retrieving
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
[6588] 1612932571.244084: Creating authenticator for
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]> ->
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>,
seqnum 1040975659, subkey aes256-cts/48E9, session key aes256-cts/DF1E
ldap_sasl_interactive_bind_s: Invalid credentials (49)

[root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all --raw
ipa: ERROR: Insufficient access:  Invalid credentials

[root@neumann2 ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
Default principal:
DNS/[e-mail]<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>
<mailto:DNS/[e-mail]>

Valid starting       Expires              Service principal
02/10/2021 01:52:43  02/11/2021 01:49:04
HTTP/[e-mail]<mailto:HTTP/[e-mail]>
<mailto:HTTP/[e-mail]>
<mailto:HTTP/[e-mail]>
02/10/2021 01:49:16  02/11/2021 01:49:04
ldap/[e-mail]<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
<mailto:ldap/[e-mail]>
02/10/2021 01:49:04  02/11/2021 01:49:04
krbtgt/[e-mail]<mailto:krbtgt/[e-mail]>
<mailto:krbtgt/[e-mail]>
<mailto:krbtgt/[e-mail]>

Any ideia on how to fix this?

Thanks,
Vinícius.

PS: Before the workaround named-pkcs11 fails to start with the
following error:

Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys zone
for view _default, file '/var/named/dynamic/managed-keys.bind'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
'ipa' driver '/usr/lib64/bind/ldap.so'
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
Hat 4.8.5-39)
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
credentials: bind to LDAP server failed
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
connection in LDAP connection pool: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database 'ipa'
configuration failed: permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
permission denied
Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal
error)
Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
process exited, code=exited status=1
Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
Name Domain (DNS) with native PKCS#11.

_______________________________________________
FreeIPA-users mailing list --
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
<mailto:[e-mail]>
To unsubscribe send an email to
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
<mailto:[e-mail]>
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...


_______________________________________________
FreeIPA-users mailing list --
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
To unsubscribe send an email to
[e-mail]<mailto:[e-mail]>
<mailto:[e-mail]>
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/[e-mail]...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure

[Rob Crittenden]

Vinícius Ferrão wrote:
>  Hi Rob.
>  
> > On 15 Feb 2021, at 10:58, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:[e-mail]>> wrote:
> >
> > Vinícius Ferrão wrote:
> >> Hi Rob.
> >>
> >> Actually nothing that relies on Kerberos Keytabs is working.
> >
> > Kerberos is working. The kinit was successful.
>  
>  Sorry perhaps I didn’t say it correctly. In fact Kerberos is working (I
>  can kinit) but anything that relies on Keytabs, specifically Keytabs,
>  aren’t working.
>  
>  named-pkcs11 does not start without the hack that I’ve mentioned. Please
>  correct me if I’m wrong about this.
>  
>  Every other service fails with “insufficient credentials”; dogtag,
>  gssproxy, etc. 
Looping in the Kerberos maintainer. You'll note that later in the output
there is a reference to credential cache is empty. I wonder if gssproxy
is having issues.

rob

>  
> >> I can properly issue kinit’s and login, but I can’t use ‘ipa’ commands
> >> for instance. named-pkcs11 is only starting up because I’ve changed the
> >> authentication method on /etc/named.conf:
> >>
> >> /* WARNING: This part of the config file is IPA-managed.
> >>  * Modifications may break IPA setup or upgrades.
> >>  */
> >> dyndb "ipa" "/usr/lib64/bind/ldap.so" {
> >> uri "ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket";
> >> base "cn=dns, dc=cluster,dc=cetene,dc=gov,dc=br";
> >> server_id "neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>";
> >> #auth_method "sasl";
> >> #sasl_mech "GSSAPI";
> >> #sasl_user "DNS/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>";
> >> /* Desespero */
> >> auth_method "simple";
> >> bind_dn
> >> "uid=admin,cn=users,cn=accounts,dc=cluster,dc=cetene,dc=gov,dc=br";
> >> password “REDACTED";
> >> };
> >> /* End of IPA-managed part. */
> >>
> >> I’ve done the test that you’ve asked, and was a no go:
> >>
> >> [root@neumann2 ~]# kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>
> >> <http://neumann2.cluster.cetene.gov.br
> >> <http://neumann2.cluster.cetene.gov.br/>>
> >> [root@neumann2 ~]# klist
> >> Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
> >> Default principal:
> >> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>
> <mailto:ipa-dnskeysyncd/[e-mail]>
> >>
> <mailto:ipa-dnskeysyncd/[e-mail]>
> >>
> >> Valid starting       Expires              Service principal
> >> 02/12/2021 22:42:03  02/13/2021 22:42:03 
> >> krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> >> <mailto:krbtgt/[e-mail]>
> >> <mailto:krbtgt/[e-mail]>
> >> [root@neumann2 ~]# ipa user-show admin
> >> ipa: ERROR: Insufficient access:  Invalid credentials
> >> [root@neumann2 ~]# ipa -v user-show admin
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 1]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 2]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 3]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 4]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: INFO: [try 5]: Forwarding 'schema' to json server
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json'
> >> ipa: INFO: trying https://neumann2.cluster.cetene.gov.br/ipa/session/json
> >> ipa: ERROR: cannot connect to
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
> >> number of tries to forward a request.
> >>
> >> I never seen this on FreeIPA.
> >>
> >> Subsequent queries of IPA commands just returns the same error:
> >>
> >> [root@neumann2 ~]# ipa user-show admin
> >> ipa: ERROR: cannot connect to
> >> 'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
> >> number of tries to forward a request.
> >
> > Did you get a HTTP service ticket? (klist)
>  
>  I issued and admin ticket as I usually do:
>  
>  [root@neumann2 ~]# kinit admin
>  Password for admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:[e-mail]>: 
>  [root@neumann2 ~]# klist
>  Ticket cache: KEYRING:persistent:0:krb_ccache_dngnA1P
>  Default principal: admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:[e-mail]>
>  
>  Valid starting       Expires              Service principal
>  02/15/2021 13:09:04  02/16/2021 13:09:04 
>  krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
>  <mailto:krbtgt/[e-mail]>
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: Insufficient access:  Invalid credentials
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  
>  But I can recover the HTTP ticket and kinit:
>  
>  [root@neumann2 ~]# klist -kt /var/lib/ipa/gssproxy/http.keytab
>  Keytab name: FILE:/var/lib/ipa/gssproxy/http.keytab
>  KVNO Timestamp           Principal
>  ---- -------------------
>  ------------------------------------------------------
>     3 02/10/2021 22:52:34
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>     3 02/10/2021 22:52:34
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>  [root@neumann2 ~]# kinit -kt /var/lib/ipa/gssproxy/http.keytab
>  HTTP/neumann2.cluster.cetene.gov.br <http://neumann2.cluster.cetene.gov.br>
>  [root@neumann2 ~]# klist
>  Ticket cache: KEYRING:persistent:0:krb_ccache_JRv9hJN
>  Default principal:
>  HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
>  <mailto:HTTP/[e-mail]>
>  
>  Valid starting       Expires              Service principal
>  02/15/2021 13:13:47  02/16/2021 13:13:47 
>  krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
>  <mailto:krbtgt/[e-mail]>
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: Insufficient access:  Invalid credentials
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  [root@neumann2 ~]# ipa user-list
>  ipa: ERROR: cannot connect to
>  'https://neumann2.cluster.cetene.gov.br/ipa/session/json': Exceeded
>  number of tries to forward a request.
>  
>  But again it didn’t work.
>  
>  On /var/log/httpd/error_log there basically this:
>  
>  [Wed Feb 10 17:34:19.129505 2021] [:error] [pid 13912] ipa: INFO: 401
>  Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI
>  Error: Unspecified GSS failure.  Minor code may provide more information
>  (Credential cache is empty)
>  [Wed Feb 10 17:34:19.151811 2021] [auth_gssapi:error] [pid 13917]
>  [client 172.26.255.254:48758] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:34:31.982562 2021] [:error] [pid 13913] ipa: INFO: 401
>  Unauthorized: Insufficient access: SASL(-1): generic failure: GSSAPI
>  Error: Unspecified GSS failure.  Minor code may provide more information
>  (Credential cache is empty)
>  [Wed Feb 10 17:34:32.015893 2021] [auth_gssapi:error] [pid 13914]
>  [client 172.26.255.254:49020] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:35:08.037058 2021] [auth_gssapi:error] [pid 13915]
>  [client 172.26.255.254:49624] GSS ERROR gss_acquire_cred[_from]() failed
>  to get server creds: [Unspecified GSS failure.  Minor code may provide
>  more information ( SPNEGO cannot find mechanisms to negotiate)],
>  referer: https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:38:08.183222 2021] [:warn] [pid 13916] [client
>  172.26.255.254:52646] failed to set perms (3140) on file
>  (/var/run/ipa/ccaches/admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:var/run/ipa/ccaches/[e-mail]>)!, referer:
>  https://neumann2.cluster.cetene.gov.br/ipa/xml
>  [Wed Feb 10 17:38:08.213367 2021] [:error] [pid 13911] ipa: INFO: 401
>  Unauthorized: Insufficient access:  Invalid credentials
>  [Wed Feb 10 17:38:08.256346 2021] [:error] [pid 13912] ipa: INFO: 401
>  Unauthorized: Insufficient access:  Invalid credentials
>  [Wed Feb 10 17:38:08.278769 2021] [:warn] [pid 13917] [client
>  172.26.255.254:52654] failed to set perms (3140) on file
>  (/var/run/ipa/ccaches/admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:var/run/ipa/ccaches/[e-mail]>)!, referer:
>  https://neumann2.cluster.cetene.gov.br/ipa/xml
>  
>  Just for the completude, removing the /etc/named.conf hack; this happens:
>  
>  [root@neumann2 ~]# ipactl start
>  Starting Directory Service
>  Starting krb5kdc Service
>  Starting kadmin Service
>  Starting named Service
>  Failed to start named Service
>  Shutting down
>  Hint: You can use --ignore-service-failure option for forced start in
>  case that a non-critical service failed
>  Aborting ipactl
>  
>  On /var/log/messages:
>  
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: set up managed keys zone
>  for view _default, file '/var/named/dynamic/managed-keys.bind'
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading DynDB instance
>  'ipa' driver '/usr/lib64/bind/ldap.so'
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: bind-dyndb-ldap version
>  11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red Hat
>  4.8.5-39)
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: LDAP error: Invalid
>  credentials: bind to LDAP server failed
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: couldn't establish
>  connection in LDAP connection pool: permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: dynamic database 'ipa'
>  configuration failed: permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: loading configuration:
>  permission denied
>  Feb 15 13:18:52 neumann2 named-pkcs11[32027]: exiting (due to fatal error)
>  Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service: control process
>  exited, code=exited status=1
>  Feb 15 13:18:52 neumann2 systemd: Failed to start Berkeley Internet Name
>  Domain (DNS) with native PKCS#11.
>  Feb 15 13:18:52 neumann2 systemd: Unit named-pkcs11.service entered
>  failed state.
>  Feb 15 13:18:52 neumann2 systemd: named-pkcs11.service failed.
>  Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 KDC...
>  Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 KDC.
>  Feb 15 13:18:52 neumann2 systemd: Stopping Kerberos 5 Password-changing
>  and Administration...
>  Feb 15 13:18:52 neumann2 systemd: Stopped Kerberos 5 Password-changing
>  and Administration.
>  Feb 15 13:18:52 neumann2 systemd: Stopping 389 Directory Server
>  CLUSTER-CETENE-GOV-BR....
>  
>  Thats it Rob.
>  
>  If there’s anything more that I should try or you need to see please let
>  me know.
>  
>  Thank you.
>  
> >
> > Check the Apache error log for more details.
> >
> > rob
> >
> >>
> >> Thank you.
> >>
> >>
> >>> On 12 Feb 2021, at 18:11, Rob Crittenden <rcritten(a)redhat.com
> >>> <mailto:[e-mail]>
> >>> <mailto:[e-mail]>> wrote:
> >>>
> >>> Just to confirm, the system is working with the exception of
> >>> ipa-dnskeysyncd.service?
> >>>
> >>> Does this work?
> >>>
> >>> # kinit -kt /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >>> ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >>> <http://neumann2.cluster.cetene.gov.br/>
> >>> <http://neumann2.cluster.cetene.gov.br/>
> >>> # ipa user-show admin
> >>>
> >>> This will get a ticket and then use that ticket.
> >>>
> >>> rob
> >>>
> >>> Vinícius Ferrão via FreeIPA-users wrote:
> >>>> Hello,
> >>>>
> >>>> I still not sure of what is happening but, I got some interesting error
> >>>> message on ipa-healthcheck:
> >>>>
> >>>> [root@neumann2 keytabs]# ipa-healthcheck --failures-only
> >>>> --output-type human
> >>>> CRITICAL: ipahealthcheck.ipa.dna.IPADNARangeCheck: Insufficient
> >>>> access: 
> >>>> Invalid credentials
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/lib/ipa/backup/: free space percentage under threshold: 16% <
> 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /tmp:
> >>>> free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/lib/dirsrv/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/log/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/tmp/: free space percentage under threshold: 16% < 20%
> >>>> ERROR: ipahealthcheck.system.filesystemspace.FileSystemSpaceCheck:
> >>>> /var/log/audit/: free space percentage under threshold: 16% < 20%
> >>>>
> >>>> I tried to search for the critical message but nothing comes up.
> >>>> There’s
> >>>> a lot of GSSAPI errors on all logs.
> >>>>
> >>>> I tried to regenerate all keytabs of the system but it was a no go
> >>>> either:
> >>>> # gssproxy
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'HTTP/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /var/lib/ipa/gssproxy/http.keytab
> >>>>
> >>>> # Dogtag
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'dogtag/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /etc/pki/pki-tomcat/dogtag.keytab
> >>>>
> >>>> # DNSKeySync
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'ipa-dnskeysyncd/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> >>>> /etc/ipa/dnssec/ipa-dnskeysyncd.keytab
> >>>>
> >>>> # Host Keytab
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'host/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/krb5.keytab
> >>>>
> >>>> # named
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'DNS/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/named.keytab
> >>>>
> >>>> # 389ds
> >>>> ipa-getkeytab -D "cn=directory manager" -w 86dNCxFFCpNMLEf6kr
> -s
> >>>> neumann2.cluster.cetene.gov.br
> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>
> >>>> -p 'ldap/neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br
> >>>> <http://neumann2.cluster.cetene.gov.br/>
> >>>> <http://neumann2.cluster.cetene.gov.br/>>' -r -k
> /etc/dirsrv/ds.keytab
> >>>>
> >>>> Some error messages:
> >>>>
> >>>> [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49
> >>>> tag=97
> >>>> nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error:
> >>>> Unspecified GSS failure.  Minor code may provide more information
> >>>> (Cannot create replay cache file /var/tmp/ldap_389: Operation not
> >>>> permitted)
> >>>>
> >>>> ==> /var/log/messages <==
> >>>> Feb 10 23:05:14 neumann2 systemd: ipa-dnskeysyncd.service holdoff time
> >>>> over, scheduling restart.
> >>>> Feb 10 23:05:14 neumann2 systemd: Stopped IPA key daemon.
> >>>> Feb 10 23:05:14 neumann2 systemd: Started IPA key daemon.
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: INFO    
> >>>> LDAP
> >>>> bind...
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: ipa-dnskeysyncd: ERROR   
> >>>> Login to LDAP server failed: {'desc': 'Invalid
> credentials'}
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: Traceback (most recent call
> >>>> last):
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/libexec/ipa/ipa-dnskeysyncd", line 96, in <module>
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd:
> >>>> ldap_connection.sasl_interactive_bind_s("",
> ipaldap.SASL_GSSAPI)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 850, in
> >>>> sasl_interactive_bind_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: res =
> >>>>
> self._apply_method_s(SimpleLDAPObject.sasl_interactive_bind_s,*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 818, in
> >>>> _apply_method_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >>>> func(self,*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 229, in
> >>>> sasl_interactive_bind_s
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: return
> >>>>
> self._ldap_call(self._l.sasl_interactive_bind_s,who,auth,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls),sasl_flags)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: File
> >>>> "/usr/lib64/python2.7/site-packages/ldap/ldapobject.py", line
> 99, in
> >>>> _ldap_call
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: result = func(*args,**kwargs)
> >>>> Feb 10 23:05:16 neumann2 ipa-dnskeysyncd: INVALID_CREDENTIALS:
> {'desc':
> >>>> 'Invalid credentials'}
> >>>> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service: main process
> >>>> exited, code=exited, status=1/FAILURE
> >>>> Feb 10 23:05:16 neumann2 systemd: Unit ipa-dnskeysyncd.service entered
> >>>> failed state.
> >>>> Feb 10 23:05:16 neumann2 systemd: ipa-dnskeysyncd.service failed.
> >>>>
> >>>> Thanks,
> >>>>
> >>>>> On 10 Feb 2021, at 02:01, Vinícius Ferrão via FreeIPA-users
> >>>>> <freeipa-users(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>> wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> FreeIPA on CentOS 7.8 just stopped working and I’m unable to fix it
> by
> >>>>> myself. After reading a lot of threads here on the list, it appears
> >>>>> that I’ve the same issue as this
> >>>>>
> topic: https://www.mail-archive.com/[e-mail]/...
> >>>>>
> >>>>> Since Kerberos is apparently not working as expected, I cannot use
> >>>>> FreeIPA and none of the services are working correctly. Following
> the
> >>>>> debug guide I was able to at least start named with single
> >>>>> authentication to further debug. (Workaround 1
> >>>>>
> of https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html)
> >>>>>
> >>>>> And now I’m stuck on item 5 of the same manual.
> >>>>>
> >>>>> [root@neumann2 ~]# KRB5_TRACE=/dev/stderr ldapsearch -H
> >>>>>
> 'ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y GSSAPI
> >>>>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br
> >>>>> <ldapi://%2fvar%2frun%2fslapd-CLUSTER-CETENE-GOV-BR.socket' -Y
> GSSAPI
> >>>>> -b 'cn=dns,dc=cluster,dc=cetene,dc=gov,dc=br>'
> >>>>> SASL/GSSAPI authentication started
> >>>>> [6588] 1612932571.244080: ccselect module realm chose cache
> >>>>> KEYRING:persistent:0:krb_ccache_UuVdVRC with client principal
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> for
> >>>>> server principal
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> [6588] 1612932571.244081: Getting credentials
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> using ccache KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>>>> [6588] 1612932571.244082: Retrieving
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> from KEYRING:persistent:0:krb_ccache_UuVdVRC with result: 0/Success
> >>>>> [6588] 1612932571.244084: Creating authenticator for
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]> ->
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>,
> >>>>> seqnum 1040975659, subkey aes256-cts/48E9, session key
> aes256-cts/DF1E
> >>>>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
> >>>>>
> >>>>> [root@neumann2 ~]# ipa privilege-show 'DNS Servers' --all
> --raw
> >>>>> ipa: ERROR: Insufficient access:  Invalid credentials
> >>>>>
> >>>>> [root@neumann2 ~]# klist
> >>>>> Ticket cache: KEYRING:persistent:0:krb_ccache_UuVdVRC
> >>>>> Default principal:
> >>>>> DNS/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> <mailto:DNS/[e-mail]>
> >>>>>
> >>>>> Valid starting       Expires              Service principal
> >>>>> 02/10/2021 01:52:43  02/11/2021 01:49:04 
> >>>>> HTTP/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>>
> <mailto:HTTP/[e-mail]>
> >>>>> 02/10/2021 01:49:16  02/11/2021 01:49:04 
> >>>>> ldap/neumann2.cluster.cetene.gov.br(a)CLUSTER.CETENE.GOV.BR
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>>
> <mailto:ldap/[e-mail]>
> >>>>> 02/10/2021 01:49:04  02/11/2021 01:49:04 
> >>>>> krbtgt/CLUSTER.CETENE.GOV.BR(a)CLUSTER.CETENE.GOV.BR
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>> <mailto:krbtgt/[e-mail]>
> >>>>>
> >>>>> Any ideia on how to fix this?
> >>>>>
> >>>>> Thanks,
> >>>>> Vinícius.
> >>>>>
> >>>>> PS: Before the workaround named-pkcs11 fails to start with the
> >>>>> following error:
> >>>>>
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: set up managed keys
> zone
> >>>>> for view _default, file
> '/var/named/dynamic/managed-keys.bind'
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading DynDB instance
> >>>>> 'ipa' driver '/usr/lib64/bind/ldap.so'
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: bind-dyndb-ldap version
> >>>>> 11.1 compiled at 02:16:24 Apr  1 2020, compiler 4.8.5 20150623 (Red
> >>>>> Hat 4.8.5-39)
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: LDAP error: Invalid
> >>>>> credentials: bind to LDAP server failed
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: couldn't establish
> >>>>> connection in LDAP connection pool: permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: dynamic database
> 'ipa'
> >>>>> configuration failed: permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: loading configuration:
> >>>>> permission denied
> >>>>> Feb 10 01:40:46 neumann2 named-pkcs11[4532]: exiting (due to fatal
> >>>>> error)
> >>>>> Feb 10 01:40:46 neumann2 systemd: named-pkcs11.service: control
> >>>>> process exited, code=exited status=1
> >>>>> Feb 10 01:40:46 neumann2 systemd: Failed to start Berkeley Internet
> >>>>> Name Domain (DNS) with native PKCS#11.
> >>>>>
> >>>>> _______________________________________________
> >>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> To unsubscribe send an email to
> >>>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> <mailto:[e-mail]>
> >>>>> Fedora Code of Conduct:
> >>>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>>> List
> >>>>> Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>>> List Archives:
> >>>>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> >>>> <mailto:[e-mail]>
> >>>> <mailto:[e-mail]>
> >>>> To unsubscribe send an email to
> >>>> freeipa-users-leave(a)lists.fedorahosted.org
> >>>> <mailto:[e-mail]>
> >>>> <mailto:[e-mail]>
> >>>> Fedora Code of Conduct:
> >>>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>>> List Archives:
> >>>>
> https://lists.fedorahosted.org/archives/list/[e-mail]...
> >>>> Do not reply to spam on the list, report it:
> >>>> https://pagure.io/fedora-infrastructure
>

[Robbie Harwood]

Vinícius Ferrão writes:

>  [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49
> tag=97 nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error: Unspecified
> GSS failure.  Minor code may provide more information (Cannot create replay cache file
> /var/tmp/ldap_389: Operation not permitted) 
Well, this looks suspicious.  Any idea why it can't create that?
SELinux maybe?

Thanks,
--Robbie

[Vinícius Ferrão]

Hi Robbie.

>  On 15 Feb 2021, at 18:45, Robbie Harwood <rharwood(a)redhat.com&gt;
> wrote:
>  
>  Vinícius Ferrão writes:
>  
> > [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97 nentries=0
> etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. 
> Minor code may provide more information (Cannot create replay cache file
> /var/tmp/ldap_389: Operation not permitted)
>  
>  Well, this looks suspicious.  Any idea why it can't create that?
>  SELinux maybe?
>  
I was suspecting of SELinux too, so I’ve issued setenforce 0 to check of it will work but
no success either.

Thanks,
V.

>  
>  Thanks,
>  --Robbie
>

[Rob Crittenden]

Vinícius Ferrão via FreeIPA-users wrote:
>  Hi Robbie.
>  
> > On 15 Feb 2021, at 18:45, Robbie Harwood <rharwood(a)redhat.com&gt; wrote:
> >
> > Vinícius Ferrão writes:
> >
> >> [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97
> nentries=0 etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
> failure.  Minor code may provide more information (Cannot create replay cache file
> /var/tmp/ldap_389: Operation not permitted)
> >
> > Well, this looks suspicious.  Any idea why it can't create that?
> > SELinux maybe?
>  
>  I was suspecting of SELinux too, so I’ve issued setenforce 0 to check of it will work but
> no success either. 
What is the mode of /var/tmp?

rob

[Vinícius Ferrão]

Hi guys! Good news.

On 15 Feb 2021, at 20:11, Rob Crittenden
<[e-mail]<mailto:[e-mail]>> wrote:

Vinícius Ferrão via FreeIPA-users wrote:
Hi Robbie.

On 15 Feb 2021, at 18:45, Robbie Harwood
<[e-mail]<mailto:[e-mail]>> wrote:

Vinícius Ferrão writes:

[10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49 tag=97 nentries=0
etime=0.001927716 - SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. 
Minor code may provide more information (Cannot create replay cache file
/var/tmp/ldap_389: Operation not permitted)

Well, this looks suspicious.  Any idea why it can't create that?
SELinux maybe?

I was suspecting of SELinux too, so I’ve issued setenforce 0 to check of it will work but
no success either.

What is the mode of /var/tmp?

:)

You figured out.

For reason that I don’t know yet - you’ll try to discover why this happened - /var/tmp was
with UID and GID permissions for a random user:

[root@neumann2 ~]# ls -l /var | grep tmp
drwxrwxrwt.  7 depaula depaula 4096 Feb 15 21:21 tmp

Since sticky bit is enabled we got some bizarre things like this:

[root@neumann2 ~]# ls -l /var/tmp/
total 12
-rw-------. 1 root    root       6 Feb  6 11:21 host_0
-rw-------. 1 root    root       6 Feb  9 19:42 kadmin_0
-rw-------. 1 depaula depaula 2738 Feb  2 08:36 ldap_389

So yeah. February 2nd matches with the start of the issue.

I’ve immediately stopped IPA, removed the files, fixed the permissions, reverted back my
/etc/named.conf hack and IPA started without any apparent issue.

I was able to properly issue commands after kinit’ing as admin.

Guys, thank you so much. It’s really good to have help from smart guys. Thanks!!!

Best regards,
Vinicius

PS: Just to confirm:

[root@neumann2 ~]# ipa user-find | head
----------------
74 users matched
----------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: [e-mail]<mailto:[e-mail]>
  UID: 917400000
  GID: 917400000


rob

[Rob Crittenden]

Vinícius Ferrão wrote:
>  Hi guys! Good news.
>  
> > On 15 Feb 2021, at 20:11, Rob Crittenden <rcritten(a)redhat.com
> > <mailto:[e-mail]>> wrote:
> >
> > Vinícius Ferrão via FreeIPA-users wrote:
> >> Hi Robbie.
> >>
> >>> On 15 Feb 2021, at 18:45, Robbie Harwood <rharwood(a)redhat.com
> >>> <mailto:[e-mail]>> wrote:
> >>>
> >>> Vinícius Ferrão writes:
> >>>
> >>>> [10/Feb/2021:23:05:57.501853962 -0300] conn=92 op=1 RESULT err=49
> >>>> tag=97 nentries=0 etime=0.001927716 - SASL(-1): generic failure:
> >>>> GSSAPI Error: Unspecified GSS failure.  Minor code may provide more
> >>>> information (Cannot create replay cache file /var/tmp/ldap_389:
> >>>> Operation not permitted)
> >>>
> >>> Well, this looks suspicious.  Any idea why it can't create that?
> >>> SELinux maybe?
> >>
> >> I was suspecting of SELinux too, so I’ve issued setenforce 0 to check
> >> of it will work but no success either.
> >
> > What is the mode of /var/tmp?
>  
>  :)
>  
>  You figured out.
>  
>  For reason that I don’t know yet - you’ll try to discover why this
>  happened - /var/tmp was with UID and GID permissions for a random user:
>  
>  [root@neumann2 ~]# ls -l /var | grep tmp
>  drwxrwxrwt.  7 depaula depaula 4096 Feb 15 21:21 tmp
>  
>  Since sticky bit is enabled we got some bizarre things like this:
>  
>  [root@neumann2 ~]# ls -l /var/tmp/
>  total 12
>  -rw-------. 1 root    root       6 Feb  6 11:21 host_0
>  -rw-------. 1 root    root       6 Feb  9 19:42 kadmin_0
>  -rw-------. 1 depaula depaula 2738 Feb  2 08:36 ldap_389
>  
>  So yeah. February 2nd matches with the start of the issue.
>  
>  I’ve immediately stopped IPA, removed the files, fixed the permissions,
>  reverted back my /etc/named.conf hack and IPA started without any
>  apparent issue.
>  
>  I was able to properly issue commands after kinit’ing as admin.
>  
>  Guys, thank you so much. It’s really good to have help from smart guys. 
Awesome, great news. Glad you got it working and thanks for closing the
loop.

rob

>  Thanks!!!
>  
>  Best regards,
>  Vinicius
>  
>  PS: Just to confirm:
>  
>  [root@neumann2 ~]# ipa user-find | head
>  ----------------
>  74 users matched
>  ----------------
>    User login: admin
>    Last name: Administrator
>    Home directory: /home/admin
>    Login shell: /bin/bash
>    Principal alias: admin(a)CLUSTER.CETENE.GOV.BR
>  <mailto:[e-mail]>
>    UID: 917400000
>    GID: 917400000
>  
> >
> > rob
> >
>