roy liang via FreeIPA-users wrote:
Like I've said, there is no documentation for this, a system that is unrenewable because of a missing library.
I do have another suggestion on something to try. It's a bit half-baked and who knows, you may have already tried it.
I'd strongly urge trying this on a clone of your production CA.
IIRC you can go back in time where all the certs are valid and the CA is operational, right? If so, do that. If not you're still going to be stuck and you can stop reading.
Bring up a new server one running CentOS or RHEL, and set time back on it as well. Preferably running 4.6.8 (RHEL 7). This is the closest to your current version.
Install it as a client with -N to skip syncing time, then run ipa-replica-install -N for the same reason. If you get that far, try running ipa-ca-install. This may well give you a working CA. At that point you'd set it as a the CA renewal master, etc (see the RHEL docs) and you'd be back in business.
There would be more to do afterward but lets not get ahead of ourselves.
rob
After libnsspem.so is added to Ubuntu16.04, all expired certificates pass the change time and the test is renewed normally. However, there are new problems during the IPA-replica-install test. The details are as follows:
ipa-client-install --domain=hiido.host.yydevops.com --realm=YYDEVOPS.COM --server=ipa-test-65-188.hiido.host.yydevops.com Everything is all right ....
root@fs-hiido-dn-12-65-18:/home/liangrui# ipa-replica-install Run connection check to master Connection check OK Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd Done configuring NTP daemon (ntpd). Configuring directory server (dirsrv). Estimated time: 1 minute [1/43]: creating directory server user [2/43]: creating directory server instance [3/43]: restarting directory server [4/43]: adding default schema [5/43]: enabling memberof plugin [6/43]: enabling winsync plugin [7/43]: configuring replication version plugin [8/43]: enabling IPA enrollment plugin [9/43]: enabling ldapi [10/43]: configuring uniqueness plugin [11/43]: configuring uuid plugin [12/43]: configuring modrdn plugin [13/43]: configuring DNS plugin [14/43]: enabling entryUSN plugin [15/43]: configuring lockout plugin [16/43]: configuring topology plugin [17/43]: creating indices [18/43]: enabling referential integrity plugin [19/43]: configuring certmap.conf [20/43]: configure autobind for root [21/43]: configure new location for managed entries [22/43]: configure dirsrv ccache [23/43]: enabling SASL mapping fallback [24/43]: restarting directory server [25/43]: creating DS keytab [26/43]: retrieving DS Certificate [27/43]: restarting directory server ipa : CRITICAL Failed to restart the directory server. See the installation log for details. [error] SystemExit: 1 ipa.ipapython.install.cli.install_tool(Replica): ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#cat /var/log/ipareplica-install.log .... 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab
2022-08-08T09:14:29Z DEBUG duration: 1 seconds 2022-08-08T09:14:29Z DEBUG [26/43]: retrieving DS Certificate 2022-08-08T09:14:29Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=255 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -N -f /etc/dirsrv/slapd-YYDEVOPS-COM//pwdfile.txt 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1) 2022-08-08T09:14:34Z DEBUG certmonger request is in state dbus.String(u'CA_UNREACHABLE', variant_level=1) 2022-08-08T09:14:34Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket from SchemaCache 2022-08-08T09:14:34Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f36a4433e60> 2022-08-08T09:14:34Z DEBUG duration: 5 seconds 2022-08-08T09:14:34Z DEBUG [27/43]: restarting directory server 2022-08-08T09:14:34Z DEBUG Starting external process 2022-08-08T09:14:34Z DEBUG args=/bin/systemctl --system daemon-reload 2022-08-08T09:14:35Z DEBUG Process finished, return code=0 2022-08-08T09:14:35Z DEBUG stdout= 2022-08-08T09:14:35Z DEBUG stderr= 2022-08-08T09:14:35Z DEBUG Starting external process 2022-08-08T09:14:35Z DEBUG args=/bin/systemctl restart dirsrv@YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=0 2022-08-08T09:14:36Z DEBUG stdout= 2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv@YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv@YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z CRITICAL Failed to restart the directory server. See the installation log for details. 2022-08-08T09:14:36Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e SystemExit: 1
2022-08-08T09:14:36Z DEBUG [error] SystemExit: 1 2022-08-08T09:14:36Z DEBUG File "/usr/lib/python2.7/dist-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/cli.py", line 318, in run cfgr.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 310, in run self.execute() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 332, in execute for nothing in self._executor(): File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 586, in _configure next(executor) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 449, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 446, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/common.py", line 63, in _install for nothing in self._installer(self.parent): File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1652, in main promote(self) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 375, in decorated func(installer) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1359, in promote promote=True, pkcs12_info=dirsrv_pkcs12_info) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 125, in install_replica_ds promote=promote, File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 399, in create_replica self.start_creation(runtime=60) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e
2022-08-08T09:14:36Z DEBUG The ipa-replica-install command failed, exception: SystemExit: 1 2022-08-08T09:14:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#less /var/log/dirsrv/slapd-YYDEVOPS-COM/errors [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL failure: None of the cipher are valid [08/Aug/2022:17:14:36 +0800] - ERROR: SSL2 Initialization Failed. Disabling SSL2. [08/Aug/2022:17:14:36 +0800] - 389-Directory/1.3.4.9 B2016.109.158 starting up [08/Aug/2022:17:14:36 +0800] - Can't find certificate Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Can't get private key from cert Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Error: unable to initialize attrcrypt system for userRoot [08/Aug/2022:17:14:36 +0800] - start: Failed to start databases, err=-1 BDB0092 Unknown error: -1 [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - WARNING: ldbm instance userRoot already exists [08/Aug/2022:17:14:36 +0800] - ldbm_config_read_instance_entries: failed to add instance entry cn=userRoot,cn=ldbm database,cn=plugins,cn=config [08/Aug/2022:17:14:36 +0800] - ldbm_config_load_dse_info: failed to read instance entries [08/Aug/2022:17:14:36 +0800] - start: Loading database configuration failed [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - Error: Failed to resolve plugin dependencies [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin 7-bit check is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin Account Usability Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: accesscontrol plugin ACL Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ACL preoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Auto Membership Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Class of Service is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin deref is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin HTTP Client is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA DNS is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Lockout is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin IPA MODRDN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Topology Configuration is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA UUID is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipa-winsync is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin ipa_enrollment_extop is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipaUniqueID uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbCanonicalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbPrincipalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: database plugin ldbm database is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Legacy Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Linked Attributes is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Managed Entries is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin MemberOf Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Multimaster Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin netgroup uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin referential integrity postoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Roles Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin sudorule name uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin USN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Views is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin whoami is not started
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
YYDEVOPS.COM IPA CA CT,C,C YYDEVOPS.COM IPA CA CT,C,C
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a certutil: Could not find cert: YYDEVOPS.COM : PR_FILE_NOT_FOUND_ERROR: File not found root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n 'YYDEVOPS.COM IPA CA' -a -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBEzANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNzExMzlaFw00MDA3MzExNzExMzlaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MB0GA1UdDgQWBBS5OsQGMUGy nqugqE6mmggFelSXPzAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAWQ27Ct/fKQ6AUg4szZ5zvoQ3H94GCxExQZRPhkx48XJnHF2mrAkd zlvUBOZ1HSAaB7ym4svjnrjVIC/BhjXH2k7BvfSCDJlkm5IP7J2DIJ+czvduRftz c+4TXOIJ14u5PY+Bcn4BHQ1iR1erR1LGaHa6G9IzbYVtNmY5gWHokFOcRbQmduLl ddZPlkdujWU8WxdXzuULBgfnHSFoNB8SATFo686RTmflAPG0So72LhzF4ElFm1An dUIftRc4PvS7DtQD7VVSc86VhCJVIGTCOx/BfbI05JP8HXQDYjBSUIezCH8rjOhu HA89ijC2ULSXBOdmtOddGxuc72wSjeqMVQ== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNjU2NDZaFw00MDA3MzExNjU2NDZaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MA8GA1UdEwEB/wQFMAMBAf8w DgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBS5OsQGMUGynqugqE6mmggFelSXPzBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAQcgq+Tm9Mqxy0Kk1eX/E7/7B0sa8WoeNFTpIweyeZEQdJyxQwe3T gQeDBZsP6meqscWTgsmxNdm9bCpPlBnPThbGNgHsdmLzCQvpLDU1cn7BQs+jFoNJ YC9g+eIzhFAw3E63WG//0VJyPkOOXrXc3o2QCqKHBZFrnn2YpYqXJN/bqN2rLwHS s5NOuK7Q70kq6etz+T9o+s5uM2A3RYTiPen4SY9kKkcMJ1CKyh6YatRUV0o7kTvA 0it2cFc74mIdsqb91VgYL+kzKTIIWH88OZYaMIWxj60gGBntKyF61RlCnhW94GQw SkdKwEAIXTJTMJwk849tbGwi7Tk4MOT5pA== -----END CERTIFICATE----- root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM#
According to the log output, are the quotes missing, so the name cannot be found, or are there two (YYDEVOPS.COM IPA CA) names, so the service cannot be replicated?
/var/log/ipareplica-install.log 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
After libnsspem.so is added to Ubuntu16.04, all expired certificates pass the change time and the test is renewed normally. However, there are new problems during the IPA-replica-install test. The details are as follows:
ipa-client-install --domain=hiido.host.yydevops.com --realm=YYDEVOPS.COM --server=ipa-test-65-188.hiido.host.yydevops.com Everything is all right ....
root@fs-hiido-dn-12-65-18:/home/liangrui# ipa-replica-install Run connection check to master Connection check OK Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd Done configuring NTP daemon (ntpd). Configuring directory server (dirsrv). Estimated time: 1 minute [1/43]: creating directory server user [2/43]: creating directory server instance [3/43]: restarting directory server [4/43]: adding default schema [5/43]: enabling memberof plugin [6/43]: enabling winsync plugin [7/43]: configuring replication version plugin [8/43]: enabling IPA enrollment plugin [9/43]: enabling ldapi [10/43]: configuring uniqueness plugin [11/43]: configuring uuid plugin [12/43]: configuring modrdn plugin [13/43]: configuring DNS plugin [14/43]: enabling entryUSN plugin [15/43]: configuring lockout plugin [16/43]: configuring topology plugin [17/43]: creating indices [18/43]: enabling referential integrity plugin [19/43]: configuring certmap.conf [20/43]: configure autobind for root [21/43]: configure new location for managed entries [22/43]: configure dirsrv ccache [23/43]: enabling SASL mapping fallback [24/43]: restarting directory server [25/43]: creating DS keytab [26/43]: retrieving DS Certificate [27/43]: restarting directory server ipa : CRITICAL Failed to restart the directory server. See the installation log for details. [error] SystemExit: 1 ipa.ipapython.install.cli.install_tool(Replica): ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#cat /var/log/ipareplica-install.log .... 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab
2022-08-08T09:14:29Z DEBUG duration: 1 seconds 2022-08-08T09:14:29Z DEBUG [26/43]: retrieving DS Certificate 2022-08-08T09:14:29Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=255 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -N -f /etc/dirsrv/slapd-YYDEVOPS-COM//pwdfile.txt 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1) 2022-08-08T09:14:34Z DEBUG certmonger request is in state dbus.String(u'CA_UNREACHABLE', variant_level=1) 2022-08-08T09:14:34Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket from SchemaCache 2022-08-08T09:14:34Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f36a4433e60> 2022-08-08T09:14:34Z DEBUG duration: 5 seconds 2022-08-08T09:14:34Z DEBUG [27/43]: restarting directory server 2022-08-08T09:14:34Z DEBUG Starting external process 2022-08-08T09:14:34Z DEBUG args=/bin/systemctl --system daemon-reload 2022-08-08T09:14:35Z DEBUG Process finished, return code=0 2022-08-08T09:14:35Z DEBUG stdout= 2022-08-08T09:14:35Z DEBUG stderr= 2022-08-08T09:14:35Z DEBUG Starting external process 2022-08-08T09:14:35Z DEBUG args=/bin/systemctl restart dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=0 2022-08-08T09:14:36Z DEBUG stdout= 2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z CRITICAL Failed to restart the directory server. See the installation log for details. 2022-08-08T09:14:36Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e SystemExit: 1
2022-08-08T09:14:36Z DEBUG [error] SystemExit: 1 2022-08-08T09:14:36Z DEBUG File "/usr/lib/python2.7/dist-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/cli.py", line 318, in run cfgr.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 310, in run self.execute() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 332, in execute for nothing in self._executor(): File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 586, in _configure next(executor) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 449, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 446, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/common.py", line 63, in _install for nothing in self._installer(self.parent): File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1652, in main promote(self) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 375, in decorated func(installer) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1359, in promote promote=True, pkcs12_info=dirsrv_pkcs12_info) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 125, in install_replica_ds promote=promote, File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 399, in create_replica self.start_creation(runtime=60) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e
2022-08-08T09:14:36Z DEBUG The ipa-replica-install command failed, exception: SystemExit: 1 2022-08-08T09:14:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#less /var/log/dirsrv/slapd-YYDEVOPS-COM/errors [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL failure: None of the cipher are valid [08/Aug/2022:17:14:36 +0800] - ERROR: SSL2 Initialization Failed. Disabling SSL2. [08/Aug/2022:17:14:36 +0800] - 389-Directory/1.3.4.9 B2016.109.158 starting up [08/Aug/2022:17:14:36 +0800] - Can't find certificate Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Can't get private key from cert Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Error: unable to initialize attrcrypt system for userRoot [08/Aug/2022:17:14:36 +0800] - start: Failed to start databases, err=-1 BDB0092 Unknown error: -1 [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - WARNING: ldbm instance userRoot already exists [08/Aug/2022:17:14:36 +0800] - ldbm_config_read_instance_entries: failed to add instance entry cn=userRoot,cn=ldbm database,cn=plugins,cn=config [08/Aug/2022:17:14:36 +0800] - ldbm_config_load_dse_info: failed to read instance entries [08/Aug/2022:17:14:36 +0800] - start: Loading database configuration failed [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - Error: Failed to resolve plugin dependencies [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin 7-bit check is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin Account Usability Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: accesscontrol plugin ACL Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ACL preoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Auto Membership Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Class of Service is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin deref is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin HTTP Client is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA DNS is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Lockout is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin IPA MODRDN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Topology Configuration is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA UUID is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipa-winsync is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin ipa_enrollment_extop is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipaUniqueID uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbCanonicalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbPrincipalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: database plugin ldbm database is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Legacy Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Linked Attributes is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Managed Entries is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin MemberOf Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Multimaster Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin netgroup uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin referential integrity postoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Roles Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin sudorule name uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin USN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Views is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin whoami is not started
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
YYDEVOPS.COM IPA CA CT,C,C YYDEVOPS.COM IPA CA CT,C,C
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a certutil: Could not find cert: YYDEVOPS.COM : PR_FILE_NOT_FOUND_ERROR: File not found root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n 'YYDEVOPS.COM IPA CA' -a -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBEzANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNzExMzlaFw00MDA3MzExNzExMzlaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MB0GA1UdDgQWBBS5OsQGMUGy nqugqE6mmggFelSXPzAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAWQ27Ct/fKQ6AUg4szZ5zvoQ3H94GCxExQZRPhkx48XJnHF2mrAkd zlvUBOZ1HSAaB7ym4svjnrjVIC/BhjXH2k7BvfSCDJlkm5IP7J2DIJ+czvduRftz c+4TXOIJ14u5PY+Bcn4BHQ1iR1erR1LGaHa6G9IzbYVtNmY5gWHokFOcRbQmduLl ddZPlkdujWU8WxdXzuULBgfnHSFoNB8SATFo686RTmflAPG0So72LhzF4ElFm1An dUIftRc4PvS7DtQD7VVSc86VhCJVIGTCOx/BfbI05JP8HXQDYjBSUIezCH8rjOhu HA89ijC2ULSXBOdmtOddGxuc72wSjeqMVQ== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNjU2NDZaFw00MDA3MzExNjU2NDZaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MA8GA1UdEwEB/wQFMAMBAf8w DgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBS5OsQGMUGynqugqE6mmggFelSXPzBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAQcgq+Tm9Mqxy0Kk1eX/E7/7B0sa8WoeNFTpIweyeZEQdJyxQwe3T gQeDBZsP6meqscWTgsmxNdm9bCpPlBnPThbGNgHsdmLzCQvpLDU1cn7BQs+jFoNJ YC9g+eIzhFAw3E63WG//0VJyPkOOXrXc3o2QCqKHBZFrnn2YpYqXJN/bqN2rLwHS s5NOuK7Q70kq6etz+T9o+s5uM2A3RYTiPen4SY9kKkcMJ1CKyh6YatRUV0o7kTvA 0it2cFc74mIdsqb91VgYL+kzKTIIWH88OZYaMIWxj60gGBntKyF61RlCnhW94GQw SkdKwEAIXTJTMJwk849tbGwi7Tk4MOT5pA== -----END CERTIFICATE----- root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM#
According to the log output, are the quotes missing, so the name cannot be found, or are there two (YYDEVOPS.COM IPA CA) names, so the service cannot be replicated?
/var/log/ipareplica-install.log 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
Strangely, after a few days, I tried IPA-certupDate again Then execute on the new node Ipa - up - install, debug Ipa - up - install, setup - ca - the debug It all worked. The data was replicated.The main reason seems to be libnsspem.so
roy liang via FreeIPA-users wrote:
After libnsspem.so is added to Ubuntu16.04, all expired certificates pass the change time and the test is renewed normally. However, there are new problems during the IPA-replica-install test. The details are as follows:
ipa-client-install --domain=hiido.host.yydevops.com --realm=YYDEVOPS.COM --server=ipa-test-65-188.hiido.host.yydevops.com Everything is all right ....
root@fs-hiido-dn-12-65-18:/home/liangrui# ipa-replica-install Run connection check to master Connection check OK Configuring NTP daemon (ntpd) [1/4]: stopping ntpd [2/4]: writing configuration [3/4]: configuring ntpd to start on boot [4/4]: starting ntpd Done configuring NTP daemon (ntpd). Configuring directory server (dirsrv). Estimated time: 1 minute [1/43]: creating directory server user [2/43]: creating directory server instance [3/43]: restarting directory server [4/43]: adding default schema [5/43]: enabling memberof plugin [6/43]: enabling winsync plugin [7/43]: configuring replication version plugin [8/43]: enabling IPA enrollment plugin [9/43]: enabling ldapi [10/43]: configuring uniqueness plugin [11/43]: configuring uuid plugin [12/43]: configuring modrdn plugin [13/43]: configuring DNS plugin [14/43]: enabling entryUSN plugin [15/43]: configuring lockout plugin [16/43]: configuring topology plugin [17/43]: creating indices [18/43]: enabling referential integrity plugin [19/43]: configuring certmap.conf [20/43]: configure autobind for root [21/43]: configure new location for managed entries [22/43]: configure dirsrv ccache [23/43]: enabling SASL mapping fallback [24/43]: restarting directory server [25/43]: creating DS keytab [26/43]: retrieving DS Certificate [27/43]: restarting directory server ipa : CRITICAL Failed to restart the directory server. See the installation log for details. [error] SystemExit: 1 ipa.ipapython.install.cli.install_tool(Replica): ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#cat /var/log/ipareplica-install.log .... 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/dirsrv/ds.keytab
2022-08-08T09:14:29Z DEBUG duration: 1 seconds 2022-08-08T09:14:29Z DEBUG [26/43]: retrieving DS Certificate 2022-08-08T09:14:29Z DEBUG Loading Index file from '/var/lib/ipa/sysrestore/sysrestore.index' 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=255 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -N -f /etc/dirsrv/slapd-YYDEVOPS-COM//pwdfile.txt 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG Starting external process 2022-08-08T09:14:29Z DEBUG args=/usr/bin/certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -A -n YYDEVOPS.COM IPA CA -t CT,C,C -a 2022-08-08T09:14:29Z DEBUG Process finished, return code=0 2022-08-08T09:14:29Z DEBUG stdout= 2022-08-08T09:14:29Z DEBUG stderr= 2022-08-08T09:14:29Z DEBUG certmonger request is in state dbus.String(u'GENERATING_KEY_PAIR', variant_level=1) 2022-08-08T09:14:34Z DEBUG certmonger request is in state dbus.String(u'CA_UNREACHABLE', variant_level=1) 2022-08-08T09:14:34Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket from SchemaCache 2022-08-08T09:14:34Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-YYDEVOPS-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f36a4433e60> 2022-08-08T09:14:34Z DEBUG duration: 5 seconds 2022-08-08T09:14:34Z DEBUG [27/43]: restarting directory server 2022-08-08T09:14:34Z DEBUG Starting external process 2022-08-08T09:14:34Z DEBUG args=/bin/systemctl --system daemon-reload 2022-08-08T09:14:35Z DEBUG Process finished, return code=0 2022-08-08T09:14:35Z DEBUG stdout= 2022-08-08T09:14:35Z DEBUG stderr= 2022-08-08T09:14:35Z DEBUG Starting external process 2022-08-08T09:14:35Z DEBUG args=/bin/systemctl restart dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=0 2022-08-08T09:14:36Z DEBUG stdout= 2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z DEBUG Starting external process 2022-08-08T09:14:36Z DEBUG args=/bin/systemctl is-active dirsrv(a)YYDEVOPS-COM.service 2022-08-08T09:14:36Z DEBUG Process finished, return code=3 2022-08-08T09:14:36Z DEBUG stdout=failed
2022-08-08T09:14:36Z DEBUG stderr= 2022-08-08T09:14:36Z CRITICAL Failed to restart the directory server. See the installation log for details. 2022-08-08T09:14:36Z DEBUG Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e SystemExit: 1
2022-08-08T09:14:36Z DEBUG [error] SystemExit: 1 2022-08-08T09:14:36Z DEBUG File "/usr/lib/python2.7/dist-packages/ipapython/admintool.py", line 171, in execute return_value = self.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/cli.py", line 318, in run cfgr.run() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 310, in run self.execute() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 332, in execute for nothing in self._executor(): File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 586, in _configure next(executor) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 372, in __runner self._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 449, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 446, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 394, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 362, in __runner step() File "/usr/lib/python2.7/dist-packages/ipapython/install/core.py", line 359, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/dist-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/dist-packages/ipapython/install/common.py", line 63, in _install for nothing in self._installer(self.parent): File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1652, in main promote(self) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 375, in decorated func(installer) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 1359, in promote promote=True, pkcs12_info=dirsrv_pkcs12_info) File "/usr/lib/python2.7/dist-packages/ipaserver/install/server/replicainstall.py", line 125, in install_replica_ds promote=promote, File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 399, in create_replica self.start_creation(runtime=60) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 447, in start_creation run_step(full_msg, method) File "/usr/lib/python2.7/dist-packages/ipaserver/install/service.py", line 437, in run_step method() File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 625, in __restart_instance self.restart(self.serverid) File "/usr/lib/python2.7/dist-packages/ipaserver/install/dsinstance.py", line 619, in restart raise e
2022-08-08T09:14:36Z DEBUG The ipa-replica-install command failed, exception: SystemExit: 1 2022-08-08T09:14:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
#less /var/log/dirsrv/slapd-YYDEVOPS-COM/errors [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Can't find certificate (Server-Cert) for family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL alert: Security Initialization: Unable to retrieve private key for cert Server-Cert of family cn=RSA,cn=encryption,cn=config (Netscape Portable Runtime error -8174 - security library: bad database.) [08/Aug/2022:17:14:36 +0800] - SSL failure: None of the cipher are valid [08/Aug/2022:17:14:36 +0800] - ERROR: SSL2 Initialization Failed. Disabling SSL2. [08/Aug/2022:17:14:36 +0800] - 389-Directory/1.3.4.9 B2016.109.158 starting up [08/Aug/2022:17:14:36 +0800] - Can't find certificate Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Can't get private key from cert Server-Cert in attrcrypt_fetch_private_key: -8174 - security library: bad database. [08/Aug/2022:17:14:36 +0800] - Error: unable to initialize attrcrypt system for userRoot [08/Aug/2022:17:14:36 +0800] - start: Failed to start databases, err=-1 BDB0092 Unknown error: -1 [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - WARNING: ldbm instance userRoot already exists [08/Aug/2022:17:14:36 +0800] - ldbm_config_read_instance_entries: failed to add instance entry cn=userRoot,cn=ldbm database,cn=plugins,cn=config [08/Aug/2022:17:14:36 +0800] - ldbm_config_load_dse_info: failed to read instance entries [08/Aug/2022:17:14:36 +0800] - start: Loading database configuration failed [08/Aug/2022:17:14:36 +0800] - Failed to start database plugin ldbm database [08/Aug/2022:17:14:36 +0800] - Error: Failed to resolve plugin dependencies [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin 7-bit check is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin Account Usability Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: accesscontrol plugin ACL Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ACL preoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Auto Membership Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Class of Service is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin deref is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin HTTP Client is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA DNS is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Lockout is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin IPA MODRDN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin IPA Topology Configuration is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin IPA UUID is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipa-winsync is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin ipa_enrollment_extop is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin ipaUniqueID uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbCanonicalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin krbPrincipalName uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: database plugin ldbm database is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Legacy Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Linked Attributes is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpreoperation plugin Managed Entries is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin MemberOf Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Multimaster Replication Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin netgroup uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: betxnpostoperation plugin referential integrity postoperation is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Roles Plugin is not started [08/Aug/2022:17:14:36 +0800] - Error: preoperation plugin sudorule name uniqueness is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin USN is not started [08/Aug/2022:17:14:36 +0800] - Error: object plugin Views is not started [08/Aug/2022:17:14:36 +0800] - Error: extendedop plugin whoami is not started
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L
Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI
YYDEVOPS.COM IPA CA CT,C,C YYDEVOPS.COM IPA CA CT,C,C
root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n YYDEVOPS.COM IPA CA -a certutil: Could not find cert: YYDEVOPS.COM : PR_FILE_NOT_FOUND_ERROR: File not found root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM# certutil -d /etc/dirsrv/slapd-YYDEVOPS-COM/ -L -n 'YYDEVOPS.COM IPA CA' -a -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBEzANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNzExMzlaFw00MDA3MzExNzExMzlaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MB0GA1UdDgQWBBS5OsQGMUGy nqugqE6mmggFelSXPzAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBxjBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAWQ27Ct/fKQ6AUg4szZ5zvoQ3H94GCxExQZRPhkx48XJnHF2mrAkd zlvUBOZ1HSAaB7ym4svjnrjVIC/BhjXH2k7BvfSCDJlkm5IP7J2DIJ+czvduRftz c+4TXOIJ14u5PY+Bcn4BHQ1iR1erR1LGaHa6G9IzbYVtNmY5gWHokFOcRbQmduLl ddZPlkdujWU8WxdXzuULBgfnHSFoNB8SATFo686RTmflAPG0So72LhzF4ElFm1An dUIftRc4PvS7DtQD7VVSc86VhCJVIGTCOx/BfbI05JP8HXQDYjBSUIezCH8rjOhu HA89ijC2ULSXBOdmtOddGxuc72wSjeqMVQ== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDpTCCAo2gAwIBAgIBATANBgkqhkiG9w0BAQsFADA3MRUwEwYDVQQKDAxZWURF Vk9QUy5DT00xHjAcBgNVBAMMFUNlcnRpZmljYXRlIEF1dGhvcml0eTAeFw0yMDA3 MzExNjU2NDZaFw00MDA3MzExNjU2NDZaMDcxFTATBgNVBAoMDFlZREVWT1BTLkNP TTEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAvKlmpaCuohS3WQgnG2Ppzr56MCpjTyJgPifDZpvC NkRCS+MtqaRKC2NX2E8oZjQAqbkUaeVfduuTL7BmTQgblm29mfKEGWtQiezNbp2k X20xzRqRV85P7Vz1H+mGLUFb3WbKcFPFlWNqKwxPcpQi49ajACwjHaXBu+dtjT5D wTuV1tQskwl17x1r858DoW1L9OwwXT08f7zIWwdUaENwZKBhVBntA4se1Zow0euC KQOy1z9x1PQPhmVuHf8xqZnqHC7de95/k1JWBe8pa0k8EKKJ0SckI8siX7cSViKx rSC/yR5pn7Q4GuN6cT7epayO/voWStaKK0NnjMO/Ue6ShQIDAQABo4G7MIG4MB8G A1UdIwQYMBaAFLk6xAYxQbKeq6CoTqaaCAV6VJc/MA8GA1UdEwEB/wQFMAMBAf8w DgYDVR0PAQH/BAQDAgHGMB0GA1UdDgQWBBS5OsQGMUGynqugqE6mmggFelSXPzBV BggrBgEFBQcBAQRJMEcwRQYIKwYBBQUHMAGGOWh0dHA6Ly9pcGEtdGVzdC02NS0x ODguaGlpZG8uaG9zdC55eWRldm9wcy5jb206ODAvY2Evb2NzcDANBgkqhkiG9w0B AQsFAAOCAQEAQcgq+Tm9Mqxy0Kk1eX/E7/7B0sa8WoeNFTpIweyeZEQdJyxQwe3T gQeDBZsP6meqscWTgsmxNdm9bCpPlBnPThbGNgHsdmLzCQvpLDU1cn7BQs+jFoNJ YC9g+eIzhFAw3E63WG//0VJyPkOOXrXc3o2QCqKHBZFrnn2YpYqXJN/bqN2rLwHS s5NOuK7Q70kq6etz+T9o+s5uM2A3RYTiPen4SY9kKkcMJ1CKyh6YatRUV0o7kTvA 0it2cFc74mIdsqb91VgYL+kzKTIIWH88OZYaMIWxj60gGBntKyF61RlCnhW94GQw SkdKwEAIXTJTMJwk849tbGwi7Tk4MOT5pA== -----END CERTIFICATE----- root@fs-hiido-dn-12-65-18:/var/log/dirsrv/slapd-YYDEVOPS-COM#
According to the log output, are the quotes missing, so the name cannot be found, or are there two (YYDEVOPS.COM IPA CA) names, so the service cannot be replicated?
/var/log/ipareplica-install.log 2022-08-08T09:14:29Z DEBUG stderr=certutil: Could not find cert: YYDEVOPS.COM IPA CA : PR_FILE_NOT_FOUND_ERROR: File not found
Strangely, after a few days, I tried IPA-certupDate again Then execute on the new node Ipa - up - install, debug Ipa - up - install, setup - ca - the debug It all worked. The data was replicated.The main reason seems to be libnsspem.so
libnsspem is a PKCS#11 driver that allows PEM files to be used by NSS database applications.
The IPA RA cert used to communicate with the CA is stored as PEM files. So in order to do the renewal this file had to be loaded and since libnsspem was missing it was not possible.
So glad you got things working. Be sure that you have at least two CAs so if one suffers catastrophic failure (hardware fault, for example) you'll have the ability to recover.
rob
freeipa-users@lists.fedorahosted.org
-
Rob Crittenden -
roy liang