ipa failed to start after reboot

[Paul-Henri Hons]

Hello,

I'm on Centos8 with freeipa installed from several month in lxc container (2 containers with replication). I've intalled custom certificates from letsencrypt for httpd and slapd and they're valid till january 2021. Yesterday, I restarted the containers and on both, Directory service failed to start. The log is below. Can someone help me to find the right direction to solve it ? All my services heavely depends on it :-(

Thanks by advance,
Paul-Henri

[30/Nov/2020:08:16:06.423512539 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
[30/Nov/2020:08:16:06.440854922 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.469627909 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
[30/Nov/2020:08:16:06.499234923 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.526831242 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
[30/Nov/2020:08:16:06.555048556 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
[30/Nov/2020:08:16:06.591310772 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.653648267 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
[30/Nov/2020:08:16:06.686970459 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.716504472 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
[30/Nov/2020:08:16:06.773674674 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
[30/Nov/2020:08:16:06.807784636 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.848156076 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
[30/Nov/2020:08:16:06.881073427 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
[30/Nov/2020:08:16:06.910055086 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
[30/Nov/2020:08:16:06.974353372 +0000] - ERR - schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!
[30/Nov/2020:08:16:07.039826294 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=groups,cn=compat,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.152097703 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=computers,cn=compat,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.172262353 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=ng,cn=compat,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.204863801 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target ou=sudoers,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.215156151 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=users,cn=compat,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.216821135 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.219650834 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.238011898 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.249040534 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.274750517 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.283165976 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.290449211 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.309211301 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.344580813 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.371243332 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.381258115 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.442193236 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.464066203 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.479286324 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=hoah,dc=ch does not exist
[30/Nov/2020:08:16:07.594646290 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[30/Nov/2020:08:16:07.629034110 +0000] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hoah,dc=ch--no CoS Templates found, which should be added before the CoS Definition.
[30/Nov/2020:08:16:07.651839151 +0000] - ERR - ipalockout_get_global_config - [file ipa_lockout.c, line 178]: krb5_init_context failed (-1429577697)
[30/Nov/2020:08:16:07.685167130 +0000] - ERR - ipaenrollment_start - [file ipa_enrollment.c, line 398]: krb5_init_context failed
[30/Nov/2020:08:16:07.713369817 +0000] - ERR - ipapwd_start - [file ipa_pwd_extop.c, line 1857]: krb5_init_context failed

[Alexander Bokovoy]

On ma, 30 marras 2020, Paul-Henri Hons via FreeIPA-users wrote:
> Hello,
>
> I'm on Centos8 with freeipa installed from several month in lxc
> container (2 containers with replication). I've intalled custom
> certificates from letsencrypt for httpd and slapd and they're valid
> till january 2021. Yesterday, I restarted the containers and on both,
> Directory service failed to start. The log is below. Can someone help
> me to find the right direction to solve it ? All my services heavely
> depends on it :-(

Just a note, FreeIPA upstream does not test with LXC and we do not in
general support this configuration ourselves for this reason.
> Thanks by advance,
> Paul-Henri
>
> [30/Nov/2020:08:16:06.423512539 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
> [30/Nov/2020:08:16:06.440854922 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.469627909 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
> [30/Nov/2020:08:16:06.499234923 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.526831242 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
> [30/Nov/2020:08:16:06.555048556 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
> [30/Nov/2020:08:16:06.591310772 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.653648267 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
> [30/Nov/2020:08:16:06.686970459 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.716504472 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
> [30/Nov/2020:08:16:06.773674674 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher AES
> [30/Nov/2020:08:16:06.807784636 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.848156076 +0000] - ERR - attrcrypt_unwrap_key - Failed to unwrap key for cipher 3DES
> [30/Nov/2020:08:16:06.881073427 +0000] - ERR - attrcrypt_cipher_init - Symmetric key failed to unwrap with the private key; Cert might have been renewed since the key is wrapped.  To recover the encrypted contents, keep the wrapped symmetric key value.
> [30/Nov/2020:08:16:06.910055086 +0000] - ERR - attrcrypt_init - All prepared ciphers are not available. Please disable attribute encryption.
> [30/Nov/2020:08:16:06.974353372 +0000] - ERR - schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!
> [30/Nov/2020:08:16:07.039826294 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=groups,cn=compat,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.152097703 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=computers,cn=compat,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.172262353 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=ng,cn=compat,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.204863801 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target ou=sudoers,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.215156151 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=users,cn=compat,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.216821135 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.219650834 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.238011898 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.249040534 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.274750517 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.283165976 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.290449211 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.309211301 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.344580813 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.371243332 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.381258115 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=vaults,cn=kra,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.442193236 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=ad,cn=etc,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.464066203 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.479286324 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=casigningcert cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=hoah,dc=ch does not exist
> [30/Nov/2020:08:16:07.594646290 +0000] - WARN - NSACLPlugin - acl_parse - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
> [30/Nov/2020:08:16:07.629034110 +0000] - ERR - cos-plugin - cos_dn_defs_cb - Skipping CoS Definition cn=Password Policy,cn=accounts,dc=hoah,dc=ch--no CoS Templates found, which should be added before the CoS Definition.
> [30/Nov/2020:08:16:07.651839151 +0000] - ERR - ipalockout_get_global_config - [file ipa_lockout.c, line 178]: krb5_init_context failed (-1429577697)

This line is your issue. It means libkrb5 cannot find the default realm
configuration from /etc/krb5.conf. Typically, you would have something
like this in krb5.conf on IPA master:

[libdefaults]
  default_realm = IPA.TEST

..

[realms]
  IPA.TEST = {
   kdc = master.ipa.test:88
   master_kdc = master.ipa.test:88
   admin_server = master.ipa.test:749
   default_domain = ipa.test
   pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
   pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}

..

[domain_realm]
  .ipa.test = IPA.TEST
  ipa.test = IPA.TEST
  master.ipa.test = IPA.TEST

[Paul-Henri Hons]

Hello,

Thank you for your fast answer. My my krb5.conf seems correct, based on info you gave me :

includedir /etc/krb5.conf.d/
includedir /var/lib/sss/pubconf/krb5.include.d/

[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = HOAH.CH
 dns_lookup_realm = false
 dns_lookup_kdc = true
 rdns = false
 ticket_lifetime = 24h
 forwardable = true
 udp_preference_limit = 0

[realms]
 HOAH.CH = {
  kdc = ipa.hoah.ch:88
  master_kdc = ipa.hoah.ch:88
  admin_server = ipa.hoah.ch:749
  default_domain = hoah.ch
  pkinit_anchors = FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
  pkinit_pool = FILE:/var/lib/ipa-client/pki/ca-bundle.pem
}

[domain_realm]
 .hoah.ch = HOAH.CH
 hoah.ch = HOAH.CH
 ipa.hoah.ch = HOAH.CH

[dbmodules]
  HOAH.CH = {
    db_library = ipadb.so
  }

[plugins]
 certauth = {
  module = ipakdb:kdb/ipadb.so
  enable_only = ipakdb
 }

Do you have another hint ?