New subject: ipa CLI doesn't work due to revoked TGT following S4U2PROXY_NO_HEADER_PAC

26 Oct 2023

      On Чцв, 26 кас 2023, Kroon PC, Peter wrote:
...
Hi Alexander and Rob,
many thanks for your prompt responses :)
I made a new lxc machine and restored a backup so at least I have a working environment again. I kept the borken one for further investigation which I'll use to provide more information.
I'm not super comfortable using mailing lists, and I'm not sure whether my mail client (outlook) will mangle my inline responses.
Peter

Van: Alexander Bokovoy abokovoy@redhat.com
Verzonden: woensdag 25 oktober 2023 20:49
Aan: Rob Crittenden
CC: FreeIPA users list; Kroon PC, Peter
Onderwerp: Re: [Freeipa-users] Re: ipa CLI doesn't work due to revoked TGT following S4U2PROXY_NO_HEADER_PAC
On ���, 25 ��� 2023, Rob Crittenden wrote:
...
Alexander Bokovoy via FreeIPA-users wrote:
...
On ���, 25 ��� 2023, Kroon PC, Peter via FreeIPA-users wrote:
...
Hi all,
After upgrading to Rocky linux 9.2 I'm running into issues with my IPA
server (4.10.1-9.el9_2). In particular, my IPA CLI seems FUBARred:
$ kinit admin
Password for admin@EXAMPLE.COM:
$ ipa show-user admin
ipa: ERROR: Insufficient access: SASL(-1): generic failure: GSSAPI
Error: No credentials were supplied, or the credentials were
unavailable or inaccessible (Credential cache is empty)
/var/log/krb5kdc.log:
okt 24 16:17:48 freeipa.example.com krb5kdc[10493]: TGS_REQ (4 etypes
{aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20),
aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17)})
192.168.12.57: S4U2PROXY_NO_HEADER_PAC: authtime 0, etypes
{rep=UNSUPPORTED:(0)} HTTP/freeipa.example.com@EXAMPLE.COM for
ldap/freeipa.example.com@EXAMPLE.COM, TGT has been revoked
As the log shows, the KDC states there is no PAC, and therefore revokes
the TGT (note, I had to RTFS to decipher the S4U2PROXY_NO_HEADER_PAC).
Because of this, the web gui also doesn't work.
That is correct description of the reason why it does not work.
...
$ ldapsearch -Y GSSAPI -b cn=users,cn=accounts,dc=example,dc=nl
"ipaNTSecurityIdentifier=*" uid ipaNTSecurityIdentifier
SASL/GSSAPI authentication started
SASL username: admin@EXAMPLE.COM
SASL SSF: 256
SASL data security layer installed.
# extended LDIF
#
# LDAPv3
# base <cn=users,cn=accounts,dc=example,dc=com> with scope subtree
# filter: ipaNTSecurityIdentifier=*
# requesting: uid ipaNTSecurityIdentifier
#
# admin, users, accounts, example.com
dn: uid=admin,cn=users,cn=accounts,dc=example,dc=com
uid: admin
ipaNTSecurityIdentifier: S-1-5-21-3777974847-1414448952-306354440-500
# search result
search: 4
result: 0 Success
# numResponses: 2
# numEntries: 1
Out of the ~200 or so users only the admin user has a
ipaNTSecurityIdentifier, but I don't know if it's correct...
I can't run `ipa config-mod --enable-sid --add-sids`, since my ipa CLI
is broken. I do still have LDAP access fortunately.
You can run it, see below. If you'd run, do you have any error messages in
the dirsrv errors log related to sidgen plugin?
...
I tried to set `disable_pac = true` in /var/kerberos/krb5kdc/kdc.conf,
but that results in the exact same error.  Setting ipaKrbAuthzData=None
in cn=ipaConfig also has no effect.
No, one cannot disable PAC globally in FreeIPA. S4U operations
require PAC presence since last year, so for any real Kerberos service
that uses S4U (like IPA API or web UI) one cannot disable PAC
enforcement.
This is useful information :)
...
...
Look at your ID range and SID configuration. You can avoid admin issue
currently by running 'ipa' tool on IPA server as root with '-e
in_server=true' option. This will force the tool to simulate direct
access (as if it is running within httpd) and talk directly to LDAPI
socket.
Something like below:
# KRB5CACHE=/dev/null ipa -e in_server=true trustconfig-show
ipa: WARNING: API Version number was not sent, forward compatibility not
guaranteed. Assuming server's API version, 2.253
  Domain: ipa1.test
  Security Identifier: S-1-5-21-790702333-3825749031-3739951824
  NetBIOS name: IPA1
  Domain GUID: 529fcbe9-3e34-436d-a541-6ffa88e7dac1
  Fallback primary group: Default SMB Group
  IPA AD trust agents: master1.ipa1.test
  IPA AD trust controllers: master1.ipa1.test
KRB5CACHE=/dev/null ipa -e in_server=true trustconfig-show
ipa: ERROR: : trust configuration not found
Ok, let's try differently. Can you provide output of
# ldapsearch -Y EXTERNAL -H ldapi://%2Frun%2Fslapd-EXAMPLE-COM.socket \
     -b  cn=ad,cn=etc,dc=example,dc=com
(replace EXAMPLE-COM and dc=example,dc=com by your domain data)
...
...
...
# KRB5CACHE=/dev/null ipa -e in_server=true idrange-find
ipa: WARNING: API Version number was not sent, forward compatibility not
guaranteed. Assuming server's API version, 2.253

5 ranges matched
Range name: IPA1.TEST_id_range
  First Posix ID of the range: 1055600000
  Number of IDs in the range: 200000
  First RID of the corresponding RID range: 1000
  First RID of the secondary RID range: 100000000
  Range type: local domain range
... [ skip ] ...
ipa: WARNING: API Version number was not sent, forward compatibility not guaranteed. Assuming server's API version, 2.251
2 ranges matched
Range name: EXAMPLE.COM_id_range
 First Posix ID of the range: 1000
 Number of IDs in the range: 4000
 Range type: local domain range
This one is definitely not configured to handle SIDs. Also, see my
comment at the bottom of this email.
...
Range name: EXAMPLE.COM_subid_range
 First Posix ID of the range: 2147483648
 Number of IDs in the range: 2147352576
 First RID of the corresponding RID range: 2147479648
 Domain SID of the trusted domain: S-1-5-21-738065-838566-2966017632
 Range type: Active Directory domain range

Number of entries returned 2
...
In my testing you can't run config-mod without a principal, and running
in-server does not have a principal.
# KRB5CACHE=/dev/null ipa -e in_server=true config-mod --add-sids
--enable-sid
[snip]
 File "/usr/lib/python3.11/site-packages/ipaserver/plugins/config.py",
line 701, in pre_callback
   self._enable_sid(ldap, options)
 File "/usr/lib/python3.11/site-packages/ipaserver/plugins/config.py",
line 512, in _enable_sid
   if not principal_has_privilege(self.api, context.principal, privilege):
                                            ^^^^^^^^^^^^^^^^^
AttributeError: '_thread._local' object has no attribute 'principal'
ipa: ERROR: an internal error has occurred
Thank you, Rob. I did not check that part.
On IPA master one can run the oddjobd-activated script directly:
# /usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid --add-sids
$ /usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid --add-sids
Configuring SID generation
 [1/8]: creating samba domain object
 [error] TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
('Tuple_to_LDAPMod(): expected a byte string in the list', None)
The ipa-enable-sid command failed. See /var/log/ipaserver-enable-sid.log for more information
Python traceback from the log:
2023-10-26T13:24:21Z DEBUG Traceback (most recent call last):
 File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 686, in start_creation
   run_step(full_msg, method)
 File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 672, in run_step
   method()
 File "/usr/lib/python3.9/site-packages/ipaserver/install/adtrustinstance.py", line 485, in __create_samba_domain_object
   api.Backend.ldap2.add_entry(entry)
 File "/usr/lib/python3.9/site-packages/ipapython/ipaldap.py", line 1893, in add_entry
   super(LDAPCache, self).add_entry(entry)
 File "/usr/lib/python3.9/site-packages/ipapython/ipaldap.py", line 1659, in add_entry
   self.conn.add_s(str(entry.dn), list(attrs.items()))
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 236, in add_s
   return self.add_ext_s(dn,modlist,None,None)
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 221, in add_ext_s
   msgid = self.add_ext(dn,modlist,serverctrls,clientctrls)
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 218, in add_ext
   return self._ldap_call(self._l.add_ext,dn,modlist,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 128, in _ldap_call
   result = func(*args,**kwargs)
TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
2023-10-26T13:24:21Z DEBUG   [error] TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
2023-10-26T13:24:21Z DEBUG Destroyed connection context.ldap2_140617190554016
2023-10-26T13:24:21Z DEBUG   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
   return_value = self.run()
 File "/usr/libexec/ipa/oddjob/org.freeipa.server.config-enable-sid", line 68, in run
   smb.create_instance()
 File "/usr/lib/python3.9/site-packages/ipaserver/install/adtrustinstance.py", line 913, in create_instance
   self.start_creation(show_service_name=False)
 File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 686, in start_creation
   run_step(full_msg, method)
 File "/usr/lib/python3.9/site-packages/ipaserver/install/service.py", line 672, in run_step
   method()
 File "/usr/lib/python3.9/site-packages/ipaserver/install/adtrustinstance.py", line 485, in __create_samba_domain_object
   api.Backend.ldap2.add_entry(entry)
 File "/usr/lib/python3.9/site-packages/ipapython/ipaldap.py", line 1893, in add_entry
   super(LDAPCache, self).add_entry(entry)
 File "/usr/lib/python3.9/site-packages/ipapython/ipaldap.py", line 1659, in add_entry
   self.conn.add_s(str(entry.dn), list(attrs.items()))
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 236, in add_s
   return self.add_ext_s(dn,modlist,None,None)
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 221, in add_ext_s
   msgid = self.add_ext(dn,modlist,serverctrls,clientctrls)
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 218, in add_ext
   return self._ldap_call(self._l.add_ext,dn,modlist,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
 File "/usr/lib64/python3.9/site-packages/ldap/ldapobject.py", line 128, in _ldap_call
   result = func(*args,**kwargs)
2023-10-26T13:24:21Z DEBUG The ipa-enable-sid command failed, exception: TypeError: ('Tuple_to_LDAPMod(): expected a byte string in the list', None)
I still need to see ID range and trustconfig-show output to understand
the state of this deployment. Also, dirsrv errors log would be helpful
if there was an attempt to run sidgen in past.
I went through the dirsrv logs, and found the following:
[24/Oct/2023:10:25:34.071341978 +0000] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 194]: Sidgen task starts ...
[24/Oct/2023:10:25:34.300104111 +0000] - ERR - find_sid_for_ldap_entry - [file ipa_sidgen_common.c, line 522]: Cannot convert Posix ID [52021] into an unused SID.
[24/Oct/2023:10:25:34.300266490 +0000] - ERR - do_work - [file ipa_sidgen_task.c, line 154]: Cannot add SID to existing entry.
[24/Oct/2023:10:25:34.303536359 +0000] - ERR - sidgen_task_thread - [file ipa_sidgen_task.c, line 199]: Sidgen task finished [32].
You have a range that defines UID/GID space of [1000...5000] but IDs are
outside this range. This is pretty much wrong regardless of whether we
enforce SIDs or not ;)
You need to create a separate ID range that would cover your existing
IDs. Before that, we need to create a configuration to be used for SID
generation -- if the ldapsearch above would show us that the entry in
cn=ad,cn=etc,$SUFFIX does not exist.
Since ipa-enable-sid has failed, probably the entry indeed does not exist and
it would be easier to construct it with ipa-ldap-updater tool:
----
dn: cn=${DOMAIN},cn=ad,cn=etc,${SUFFIX}
default:objectClass: ipaNTDomainAttrs
default:objectClass: nsContainer
default:objectClass: top
default:cn: ${DOMAIN}
default:ipaNTFlatName: NETBIOSNAME
default:ipaNTSecurityIdentifier: S-1-5-21-3777974847-1414448952-306354440
default:ipaNTDomainGUID: 529fcbe9-3e34-3122-a541-6786236014c1
default:ipaNTFallbackPrimaryGroup: cn=Default SMB Group,cn=groups,cn=accounts,${SUFFIX}
----
Change 'NETBIOSNAME' above to some name. By default that would be a
first part of your Kerberos realm, e.g. for IPA1.TEST that would be
IPA.
The SID value (S-1-5-21-...) is the one that your admin user has,
without the last part (relative identifier, RID, which is -500 for
administrator case).
Save this to a file named '90-somefile.update' and run as root
# ipa-ldap-updater ./90-somefile.update
-- 
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland