On 6/30/20 10:24 AM, Mariusz Stolarczyk via FreeIPA-users wrote:
> All,
>
> I did a routine server updates last night on my IPA server. After the
> reboot I first noticed the DNS was not resolving and the ipa.service
> failed. The ipa.service failed to start so I ran the following:
>
>
> # ipactl start
> IPA version error: data needs to be upgraded (expected version
> '4.6.6-11.el7.centos', current version '4.6.5-11.el7.centos.4')
> Automatically running upgrade, for details see /var/log/ipaupgrade.log
> Be patient, this may take a few minutes.
> Automatic upgrade failed: Update complete
> Upgrading the configuration of the IPA services
> [Verifying that root certificate is published]
> [Migrate CRL publish directory]
> CRL tree already moved
> [Verifying that CA proxy configuration is correct]
> [Verifying that KDC configuration is using ipa-kdb backend]
> [Fix DS schema file syntax]
> Syntax already fixed
> [Removing RA cert from DS NSS database]
> RA cert already removed
> [Enable sidgen and extdom plugins by default]
> [Updating HTTPD service IPA configuration]
> [Updating HTTPD service IPA WSGI configuration]
> Nothing to do for configure_httpd_wsgi_conf
> [Updating mod_nss protocol versions]
> Protocol versions already updated
> [Updating mod_nss cipher suite]
> [Updating mod_nss enabling OCSP]
> [Fixing trust flags in /etc/httpd/alias]
> Trust flags already processed
> [Moving HTTPD service keytab to gssproxy]
> [Removing self-signed CA]
> [Removing Dogtag 9 CA]
> [Checking for deprecated KDC configuration files]
> [Checking for deprecated backups of Samba configuration files]
> [Remove FILE: prefix from 'dedicated keytab file' in Samba configuration]
> [Update 'max smbd processes' in Samba configuration to prevent unlimited
> SMBLoris attack amplification]
> [Add missing CA DNS records]
> IPA CA DNS records already processed
> [Removing deprecated DNS configuration options]
> [Ensuring minimal number of connections]
> [Updating GSSAPI configuration in DNS]
> [Updating pid-file configuration in DNS]
> [Checking global forwarding policy in named.conf to avoid conflicts with
> automatic empty zones]
> Changes to named.conf have been made, restart named
> [Upgrading CA schema]
> CA schema update complete (no changes)
> [Verifying that CA audit signing cert has 2 year validity]
> [Update certmonger certificate renewal configuration]
> Certmonger certificate renewal configuration already up-to-date
> [Enable PKIX certificate path discovery and validation]
> PKIX already enabled
> [Authorizing RA Agent to modify profiles]
> [Authorizing RA Agent to manage lightweight CAs]
> [Ensuring Lightweight CAs container exists in Dogtag database]
> [Adding default OCSP URI configuration]
> [Ensuring CA is using LDAPProfileSubsystem]
> [Migrating certificate profiles to LDAP]
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> command ipa-server-upgrade manually.
> Unexpected error - see /var/log/ipaupgrade.log for details:
> NetworkError: cannot connect to
> '
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300777903&sdata=FEc7EdbY6TKtCQlwtF39um4xgRPGVsxcMB08SpP1eRQ%3D&reserved=0':
> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> more information
>
> See the upgrade log for more details and/or run
> /usr/sbin/ipa-server-upgrade again
> Aborting ipactl
>
>
> The end of the /var/log/ipaupgrade.log file:
>
> 2020-06-29T22:43:38Z DEBUG stderr=
> 2020-06-29T22:43:38Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
> 2020-06-29T22:43:38Z DEBUG Starting external process
> 2020-06-29T22:43:38Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/pki/pki-tomcat/alias -L -f /etc/pki/pki-tomcat/alias/pwdfile.txt
> 2020-06-29T22:43:38Z DEBUG Process finished, return code=0
> 2020-06-29T22:43:38Z DEBUG stdout=
> Certificate Nickname                     Trust
> Attributes
>
> Â SSL,S/MIME,JAR/XPI
>
> caSigningCert cert-pki-ca                   CTu,Cu,Cu
> subsystemCert cert-pki-ca                   u,u,u
> Server-Cert cert-pki-ca                    u,u,u
> ocspSigningCert cert-pki-ca                  u,u,u
> auditSigningCert cert-pki-ca                 u,u,Pu
>
> 2020-06-29T22:43:38Z DEBUG stderr=
> 2020-06-29T22:43:38Z INFO Certmonger certificate renewal configuration
> already up-to-date
> 2020-06-29T22:43:38Z INFO [Enable PKIX certificate path discovery and
> validation]
> 2020-06-29T22:43:38Z DEBUG Loading StateFile from
> '/var/lib/ipa/sysupgrade/sysupgrade.state'
> 2020-06-29T22:43:38Z INFO PKIX already enabled
> 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to modify profiles]
> 2020-06-29T22:43:38Z INFO [Authorizing RA Agent to manage lightweight CAs]
> 2020-06-29T22:43:38Z INFO [Ensuring Lightweight CAs container exists in
> Dogtag database]
> 2020-06-29T22:43:38Z DEBUG Created connection context.ldap2_140346851657552
> 2020-06-29T22:43:38Z DEBUG flushing
> ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache
> 2020-06-29T22:43:38Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50c3e8e60>
> 2020-06-29T22:43:39Z DEBUG Destroyed connection
> context.ldap2_140346851657552
> 2020-06-29T22:43:39Z INFO [Adding default OCSP URI configuration]
> 2020-06-29T22:43:39Z INFO [Ensuring CA is using LDAPProfileSubsystem]
> 2020-06-29T22:43:39Z INFO [Migrating certificate profiles to LDAP]
> 2020-06-29T22:43:39Z DEBUG Created connection context.ldap2_140346825804304
> 2020-06-29T22:43:39Z DEBUG flushing
> ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket from SchemaCache
> 2020-06-29T22:43:39Z DEBUG retrieving schema for SchemaCache
> url=ldapi://%2fvar%2frun%2fslapd-FAKE-IPA-DOMAIN-LAN.socket
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7fa50ac19b90>
> 2020-06-29T22:43:39Z DEBUG Destroyed connection
> context.ldap2_140346825804304
> 2020-06-29T22:43:39Z DEBUG request GET
>
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0
> 2020-06-29T22:43:39Z DEBUG request body ''
> 2020-06-29T22:43:39Z DEBUG httplib request failed:
> Traceback (most recent call last):
> Â File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line
> 220, in _httplib_request
> Â Â conn.request(method, path, body=request_body, headers=headers)
> Â File "/usr/lib64/python2.7/httplib.py", line 1056, in request
> Â Â self._send_request(method, url, body, headers)
> Â File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request
> Â Â self.endheaders(body)
> Â File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders
> Â Â self._send_output(message_body)
> Â File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output
> Â Â self.send(msg)
> Â File "/usr/lib64/python2.7/httplib.py", line 852, in send
> Â Â self.connect()
> Â File "/usr/lib64/python2.7/httplib.py", line 1275, in connect
> Â Â server_hostname=sni_hostname)
> Â File "/usr/lib64/python2.7/ssl.py", line 348, in wrap_socket
> Â Â _context=self)
> Â File "/usr/lib64/python2.7/ssl.py", line 609, in __init__
> Â Â self.do_handshake()
> Â File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
> Â Â self._sslobj.do_handshake()
> SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
> (_ssl.c:618)
> 2020-06-29T22:43:39Z ERROR IPA server upgrade failed: Inspect
> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
> 2020-06-29T22:43:39Z DEBUG Â File
> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in
> execute
> Â Â return_value = self.run()
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> line 54, in run
> Â Â server.upgrade()
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 2166, in upgrade
> Â Â upgrade_configuration()
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 2038, in upgrade_configuration
> Â Â ca_enable_ldap_profile_subsystem(ca)
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py",
> line 425, in ca_enable_ldap_profile_subsystem
> Â Â cainstance.migrate_profiles_to_ldap()
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 2027, in migrate_profiles_to_ldap
> Â Â _create_dogtag_profile(profile_id, profile_data, overwrite=False)
> Â File
> "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line
> 2033, in _create_dogtag_profile
> Â Â with api.Backend.ra_certprofile as profile_api:
> Â File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py",
> line 1311, in __enter__
> Â Â method='GET'
> Â File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line
> 167, in https_request
> Â Â method=method, headers=headers)
> Â File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line
> 229, in _httplib_request
> Â Â raise NetworkError(uri=uri, error=str(e))
>
> 2020-06-29T22:43:39Z DEBUG The ipa-server-upgrade command failed,
> exception: NetworkError: cannot connect to
> '
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0':
> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
> 2020-06-29T22:43:39Z ERROR Unexpected error - see
> /var/log/ipaupgrade.log for details:
> NetworkError: cannot connect to
> '
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffake-ipa-host.fake-ipa-domain.lan%3A8443%2Fca%2Frest%2Faccount%2Flogin&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=iaoyf6VSEUgKUjT9%2F%2Bp6EFAqL5BEXGxhSdx59V1W%2BnE%3D&reserved=0':
> [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
> 2020-06-29T22:43:39Z ERROR The ipa-server-upgrade command failed. See
> /var/log/ipaupgrade.log for more information
>
>
> What should be my next debug steps?
>
Hi,
I would check whether any certificate expired:
$ getcert list
Look specifically for the "status: " and "expires: " labels. If some
certs have expired, you will need to find the CA renewal master and fix
this host first. To find the CA renewal master:
$ kinit admin
$ ipa config-show | grep "CA renewal"
If you need help, please mention:
- the output of "ipa server-role-find"
- the output of "getcert list" on all the server nodes
- are the httpd and ldap server certificates issued by IPA CA or by an
external Certificate Authority?
HTH,
flo
> Thanks in advance,
> -ms
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.fedoraproject.org%2Fen-US%2Fproject%2Fcode-of-conduct%2F&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=eZhJR06t5Pi280VE7SCAfBX6AzurzSA3e5qcbSNGHiE%3D&reserved=0
> List Guidelines:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffedoraproject.org%2Fwiki%2FMailing_list_guidelines&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=UvxQm1OecFaStjKLSSIMoIJ72IZgDnjv8Pmq9uPeL9s%3D&reserved=0
> List Archives:
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.fedorahosted.org%2Farchives%2Flist%2Ffreeipa-users%40lists.fedorahosted.org&data=02%7C01%7C%7C474697e47e794ce1189c08d81cd1f156%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637291035300787897&sdata=ItMCi20mfQLIoEorvQ20Fau0PGGFmRpgVAvbkgvAhMY%3D&reserved=0
>