Internal credentials cache error while getting initial credentials

[Albert Szostkiewicz]

Hey, 
Need some help here, I am unable to log-in. when trying to use kinit on my user, I am
getting an error:
kinit: Failed to store credentials: Internal credentials cache error while getting initial
credentials

sssd runs. log shows:
Oct 13 20:32:59 user.mydomain.com krb5_child[4846]: Internal credentials cache error


sssd_kcm.log states:
  *  (2023-10-13 21:17:43): [kcm] [local_db_check_peruid_number_of_secrets] (0x0040):
[CID#8708] Cannot store any more secrets for this client (basedn
cn=1907400001,cn=persistent,cn=kcm) as the maximum allowed limit (66) has been reached
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-10-13 21:17:43): [kcm] [sss_sec_update] (0x0040): [CID#8708]
local_db_check_number_of_secrets failed [1432158289]: The maximum number of stored secrets
has been reached
(2023-10-13 21:17:43): [kcm] [sec_update] (0x0040): [CID#8708] Cannot write the secret
[1432158289]: The maximum number of stored secrets has been reached
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-10-13 21:17:43): [kcm] [sss_sec_update] (0x0040): [CID#8708]
local_db_check_number_of_secrets failed [1432158289]: The maximum number of stored secrets
has been reached
   *  (2023-10-13 21:17:43): [kcm] [sec_update] (0x0040): [CID#8708] Cannot write the
secret [1432158289]: The maximum number of stored secrets has been reached
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-10-13 21:17:43): [kcm] [kcm_ccdb_mod_done] (0x0040): [CID#8708] Failed to create
ccache [1432158289]: The maximum number of stored secrets has been reached
(2023-10-13 21:17:43): [kcm] [kcm_op_set_kdc_offset_mod_done] (0x0040): [CID#8708] Cannot
modify ccache [1432158289]: The maximum number of stored secrets has been reached
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-10-13 21:17:43): [kcm] [kcm_ccdb_mod_done] (0x0040): [CID#8708] Failed to
create ccache [1432158289]: The maximum number of stored secrets has been reached
   *  (2023-10-13 21:17:43): [kcm] [kcm_op_set_kdc_offset_mod_done] (0x0040): [CID#8708]
Cannot modify ccache [1432158289]: The maximum number of stored secrets has been reached
********************** BACKTRACE DUMP ENDS HERE *********************************

(2023-10-13 21:17:43): [kcm] [kcm_cmd_done] (0x0040): [CID#8708] op receive function
failed [1432158289]: The maximum number of stored secrets has been reached
(2023-10-13 21:17:43): [kcm] [kcm_cmd_request_done] (0x0040): [CID#8708] KCM operation
failed [1432158289]: The maximum number of stored secrets has been reached
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
   *  (2023-10-13 21:17:43): [kcm] [kcm_cmd_done] (0x0040): [CID#8708] op receive function
failed [1432158289]: The maximum number of stored secrets has been reached
   *  (2023-10-13 21:17:43): [kcm] [kcm_cmd_request_done] (0x0040): [CID#8708] KCM
operation failed [1432158289]: The maximum number of stored secrets has been reached
********************** BACKTRACE DUMP ENDS HERE *********************************

KRB5_TRACE=/dev/stderr ipa --debug ping

ipa: DEBUG: importing plugin module ipaclient.plugins.trust
ipa: DEBUG: importing plugin module ipaclient.plugins.user
ipa: DEBUG: importing plugin module ipaclient.plugins.vault
ipa: DEBUG: trying https://workstation.mydomain.com/ipa/json
ipa: DEBUG: Created connection context.rpcclient_140066561958480
ipa: DEBUG: raw: ping(version='2.252')
ipa: DEBUG: ping(version='2.252')
ipa: DEBUG: [try 1]: Forwarding 'ping/1' to json server
'https://workstation.mydomain.com/ipa/json'
ipa: DEBUG: New HTTP connection (workstation.mydomain.com)
ipa: DEBUG: HTTP connection destroyed (workstation.mydomain.com)
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 644, in
get_auth_info
    response = self._sec_context.step()
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/gssapi/_utils.py", line 165, in
check_last_err
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/gssapi/_utils.py", line 131, in
catch_and_return_token
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/gssapi/sec_contexts.py", line 584,
in step
    return self._initiator_step(token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/gssapi/sec_contexts.py", line 606,
in _initiator_step
    res = rsec_contexts.init_sec_context(self._target_name, self._creds,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "gssapi/raw/sec_contexts.pyx", line 188, in
gssapi.raw.sec_contexts.init_sec_context
gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No credentials were
supplied, or the credentials were unavailable or inaccessible, Minor (2529639053): No
Kerberos credentials available (default cache: KCM:)

During the handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 697, in
single_request
    self.get_auth_info()
  File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 646, in
get_auth_info
    self._handle_exception(e, service=service)
  File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 603, in
_handle_exception
    raise errors.CCacheError()
ipalib.errors.CCacheError: did not receive Kerberos credentials
ipa: DEBUG: Destroyed connection context.rpcclient_140066561958480
ipa: ERROR: did not receive Kerberos credentials

I appreciate if anyone have some ideas. Thank you!

[Albert Szostkiewicz]

tried kdestroy as well. that did not help.

But one thing that did help now. I did kinit admin, and then kinit user (which also had
admin role), this time it worked and all came back to normal.
Thanks!

[Alejandro Lopez]

HI,

>  Cannot store any more secrets for this client (basedn
cn=1907400001,cn=persistent,cn=kcm) as the maximum allowed limit (66) has
been reached

This is the key. You have stored too many credentials in KCM. Try removing
them with "kdestroy -A"

HTH.

On Sat, Oct 14, 2023 at 6:30 AM Albert Szostkiewicz <tmdag(a)tmdag.com&gt; wrote:

>  Hey,
>  Need some help here, I am unable to log-in. when trying to use kinit on my
>  user, I am getting an error:
>  kinit: Failed to store credentials: Internal credentials cache error while
>  getting initial credentials
>
>  sssd runs. log shows:
>  Oct 13 20:32:59 user.mydomain.com krb5_child[4846]: Internal credentials
>  cache error
>
>
>  sssd_kcm.log states:
>    *  (2023-10-13 21:17:43): [kcm]
>  [local_db_check_peruid_number_of_secrets] (0x0040): [CID#8708] Cannot store
>  any more secrets for this client (basedn
>  cn=1907400001,cn=persistent,cn=kcm) as the maximum allowed limit (66) has
>  been reached
>  ********************** BACKTRACE DUMP ENDS HERE
>  *********************************
>
>  (2023-10-13 21:17:43): [kcm] [sss_sec_update] (0x0040): [CID#8708]
>  local_db_check_number_of_secrets failed [1432158289]: The maximum number of
>  stored secrets has been reached
>  (2023-10-13 21:17:43): [kcm] [sec_update] (0x0040): [CID#8708] Cannot
>  write the secret [1432158289]: The maximum number of stored secrets has
>  been reached
>  ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING
>  BACKTRACE:
>     *  (2023-10-13 21:17:43): [kcm] [sss_sec_update] (0x0040): [CID#8708]
>  local_db_check_number_of_secrets failed [1432158289]: The maximum number of
>  stored secrets has been reached
>     *  (2023-10-13 21:17:43): [kcm] [sec_update] (0x0040): [CID#8708]
>  Cannot write the secret [1432158289]: The maximum number of stored secrets
>  has been reached
>  ********************** BACKTRACE DUMP ENDS HERE
>  *********************************
>
>  (2023-10-13 21:17:43): [kcm] [kcm_ccdb_mod_done] (0x0040): [CID#8708]
>  Failed to create ccache [1432158289]: The maximum number of stored secrets
>  has been reached
>  (2023-10-13 21:17:43): [kcm] [kcm_op_set_kdc_offset_mod_done] (0x0040):
>  [CID#8708] Cannot modify ccache [1432158289]: The maximum number of stored
>  secrets has been reached
>  ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING
>  BACKTRACE:
>     *  (2023-10-13 21:17:43): [kcm] [kcm_ccdb_mod_done] (0x0040):
>  [CID#8708] Failed to create ccache [1432158289]: The maximum number of
>  stored secrets has been reached
>     *  (2023-10-13 21:17:43): [kcm] [kcm_op_set_kdc_offset_mod_done]
>  (0x0040): [CID#8708] Cannot modify ccache [1432158289]: The maximum number
>  of stored secrets has been reached
>  ********************** BACKTRACE DUMP ENDS HERE
>  *********************************
>
>  (2023-10-13 21:17:43): [kcm] [kcm_cmd_done] (0x0040): [CID#8708] op
>  receive function failed [1432158289]: The maximum number of stored secrets
>  has been reached
>  (2023-10-13 21:17:43): [kcm] [kcm_cmd_request_done] (0x0040): [CID#8708]
>  KCM operation failed [1432158289]: The maximum number of stored secrets has
>  been reached
>  ********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING
>  BACKTRACE:
>     *  (2023-10-13 21:17:43): [kcm] [kcm_cmd_done] (0x0040): [CID#8708] op
>  receive function failed [1432158289]: The maximum number of stored secrets
>  has been reached
>     *  (2023-10-13 21:17:43): [kcm] [kcm_cmd_request_done] (0x0040):
>  [CID#8708] KCM operation failed [1432158289]: The maximum number of stored
>  secrets has been reached
>  ********************** BACKTRACE DUMP ENDS HERE
>  *********************************
>
>  KRB5_TRACE=/dev/stderr ipa --debug ping
>
>  ipa: DEBUG: importing plugin module ipaclient.plugins.trust
>  ipa: DEBUG: importing plugin module ipaclient.plugins.user
>  ipa: DEBUG: importing plugin module ipaclient.plugins.vault
>  ipa: DEBUG: trying https://workstation.mydomain.com/ipa/json
>  ipa: DEBUG: Created connection context.rpcclient_140066561958480
>  ipa: DEBUG: raw: ping(version='2.252')
>  ipa: DEBUG: ping(version='2.252')
>  ipa: DEBUG: [try 1]: Forwarding 'ping/1' to json server '
>  https://workstation.mydomain.com/ipa/json'
>  ipa: DEBUG: New HTTP connection (workstation.mydomain.com)
>  ipa: DEBUG: HTTP connection destroyed (workstation.mydomain.com)
>  Traceback (most recent call last):
>    File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 644, in
>  get_auth_info
>      response = self._sec_context.step()
>                 ^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib/python3.11/site-packages/decorator.py", line 232, in fun
>      return caller(func, *(extras + args), **kw)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib64/python3.11/site-packages/gssapi/_utils.py", line 165,
>  in check_last_err
>      return func(self, *args, **kwargs)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib/python3.11/site-packages/decorator.py", line 232, in fun
>      return caller(func, *(extras + args), **kw)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib64/python3.11/site-packages/gssapi/_utils.py", line 131,
>  in catch_and_return_token
>      return func(self, *args, **kwargs)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib64/python3.11/site-packages/gssapi/sec_contexts.py", line
>  584, in step
>      return self._initiator_step(token=token)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "/usr/lib64/python3.11/site-packages/gssapi/sec_contexts.py", line
>  606, in _initiator_step
>      res = rsec_contexts.init_sec_context(self._target_name, self._creds,
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>    File "gssapi/raw/sec_contexts.pyx", line 188, in
>  gssapi.raw.sec_contexts.init_sec_context
>  gssapi.raw.exceptions.MissingCredentialsError: Major (458752): No
>  credentials were supplied, or the credentials were unavailable or
>  inaccessible, Minor (2529639053): No Kerberos credentials available
>  (default cache: KCM:)
>
>  During the handling of the above exception, another exception occurred:
>
>  Traceback (most recent call last):
>    File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 697, in
>  single_request
>      self.get_auth_info()
>    File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 646, in
>  get_auth_info
>      self._handle_exception(e, service=service)
>    File "/usr/lib/python3.11/site-packages/ipalib/rpc.py", line 603, in
>  _handle_exception
>      raise errors.CCacheError()
>  ipalib.errors.CCacheError: did not receive Kerberos credentials
>  ipa: DEBUG: Destroyed connection context.rpcclient_140066561958480
>  ipa: ERROR: did not receive Kerberos credentials
>
>  I appreciate if anyone have some ideas. Thank you!
>  _______________________________________________
>  sssd-users mailing list -- sssd-users(a)lists.fedorahosted.org
>  To unsubscribe send an email to sssd-users-leave(a)lists.fedorahosted.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedorahosted.org/archives/list/[e-mail]...
>  Do not reply to spam, report it:
>  https://pagure.io/fedora-infrastructure/new_issue
>  

-- 
Alejandro

[Albert Szostkiewicz]

Thank you!

'kdestroy -A' does help!

But I found that I am running into the same issue every now and then. What might be
causing it?

cheers, 
Albert

[Alexey Tikhonov]

On Wed, Feb 21, 2024 at 5:58 PM Albert Szostkiewicz <tmdag(a)tmdag.com&gt; wrote:
>  Thank you!
>
>  'kdestroy -A' does help!
>
>  But I found that I am running into the same issue every now and then. What might be
> causing it? 
`klist -A` and see what is there?