From flo at redhat.com Thu Nov 21 09:08:12 2019 Content-Type: multipart/mixed; boundary="===============8066564604462172680==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: yum update problem Date: Thu, 21 Nov 2019 10:07:31 +0100 Message-ID: <3366276b-3852-a1d1-983c-0a401a70a0f4@redhat.com> In-Reply-To: CAHBEJzWGNKHEAODXjjnk6vTy2UsQRnX1ZpJFop+2Lp=fRH8xGw@mail.gmail.com --===============8066564604462172680== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 11/20/19 8:13 PM, Natxo Asenjo via FreeIPA-users wrote: > = > hi, > = > after patching our centos 7 hosts to the latest version today, one of = > the two replicas is having trouble. > = > [root(a)kdc2 ~]# ipactl status > Directory Service: RUNNING > krb5kdc Service: STOPPED > kadmin Service: STOPPED > named Service: STOPPED > httpd Service: RUNNING > ipa-custodia Service: STOPPED > ntpd Service: STOPPED > pki-tomcatd Service: RUNNING > smb Service: STOPPED > winbind Service: STOPPED > ipa-otpd Service: STOPPED > ipa-dnskeysyncd Service: STOPPED > ipa: INFO: The ipactl command was successful > = > and after digging in the logs I come across this in /var/log/ipaupgrade.l= og: > = > 2019-11-20T18:18:29Z DEBUG stderr=3D > 2019-11-20T18:18:31Z INFO Certmonger certificate renewal configuration = > already up-to-date > 2019-11-20T18:18:31Z INFO [Enable PKIX certificate path discovery and = > validation] > 2019-11-20T18:18:31Z DEBUG Loading StateFile from = > '/var/lib/ipa/sysupgrade/sysupgrade.state' > 2019-11-20T18:18:31Z INFO PKIX already enabled > 2019-11-20T18:18:31Z INFO [Authorizing RA Agent to modify profiles] > 2019-11-20T18:18:31Z INFO [Authorizing RA Agent to manage lightweight CAs] > 2019-11-20T18:18:31Z INFO [Ensuring Lightweight CAs container exists in = > Dogtag database] > 2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_1397401625474= 72 > 2019-11-20T18:18:31Z DEBUG flushing = > ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache > 2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache = > url=3Dldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket = > conn=3D > 2019-11-20T18:18:31Z DEBUG Destroyed connection = > context.ldap2_139740162547472 > 2019-11-20T18:18:31Z INFO [Adding default OCSP URI configuration] > 2019-11-20T18:18:31Z INFO [Ensuring CA is using LDAPProfileSubsystem] > 2019-11-20T18:18:31Z INFO [Migrating certificate profiles to LDAP] > 2019-11-20T18:18:31Z DEBUG Created connection context.ldap2_1397401600216= 48 > 2019-11-20T18:18:31Z DEBUG flushing = > ldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket from SchemaCache > 2019-11-20T18:18:31Z DEBUG retrieving schema for SchemaCache = > url=3Dldapi://%2fvar%2frun%2fslapd-L-DOMAIN-IT.socket = > conn=3D > 2019-11-20T18:18:31Z DEBUG Destroyed connection = > context.ldap2_139740160021648 > 2019-11-20T18:18:31Z DEBUG request GET = > https://kdc2.l.domain.it:8443/ca/rest/account/login > 2019-11-20T18:18:31Z DEBUG request body '' > 2019-11-20T18:18:31Z DEBUG response status 401 > 2019-11-20T18:18:31Z DEBUG response headers Server: Apache-Coyote/1.1 > Cache-Control: private > Expires: Thu, 01 Jan 1970 01:00:00 CET > WWW-Authenticate: Basic realm=3D"Certificate Authority" > Content-Type: text/html;charset=3Dutf-8 > Content-Language: en > Content-Length: 951 > Date: Wed, 20 Nov 2019 18:18:31 GMT > = > 2019-11-20T18:18:31Z DEBUG response body 'Apache = > Tomcat/7.0.76 - Error report = >

HTTP Status 401 -

noshade=3D"noshade">

type Status report

message = >

description This request requires HTTP = > authentication.

Apache = > Tomcat/7.0.76

' > 2019-11-20T18:18:31Z ERROR IPA server upgrade failed: Inspect = > /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. > 2019-11-20T18:18:31Z DEBUG =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in = > execute > =C2=A0 =C2=A0 return_value =3D self.run() > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py= ", = > line 54, in run > =C2=A0 =C2=A0 server.upgrade() > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", = > line 2146, in upgrade > =C2=A0 =C2=A0 upgrade_configuration() > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", = > line 2018, in upgrade_configuration > =C2=A0 =C2=A0 ca_enable_ldap_profile_subsystem(ca) > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", = > line 406, in ca_enable_ldap_profile_subsystem > =C2=A0 =C2=A0 cainstance.migrate_profiles_to_ldap() > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line = > 2027, in migrate_profiles_to_ldap > =C2=A0 =C2=A0 _create_dogtag_profile(profile_id, profile_data, overwrite= =3DFalse) > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line = > 2033, in _create_dogtag_profile > =C2=A0 =C2=A0 with api.Backend.ra_certprofile as profile_api: > =C2=A0 File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.p= y", = > line 1315, in __enter__ > =C2=A0 =C2=A0 raise errors.RemoteRetrieveError(reason=3D_('Failed to aut= henticate = > to CA REST API')) > = > 2019-11-20T18:18:31Z DEBUG The ipa-server-upgrade command failed, = > exception: RemoteRetrieveError: Failed to authenticate to CA REST API > 2019-11-20T18:18:31Z ERROR Unexpected error - see = > /var/log/ipaupgrade.log for details: > RemoteRetrieveError: Failed to authenticate to CA REST API > = The authentication between IPA and dogtag is done using the ra-agent = cert located in /var/lib/ipa/ra-agent.pem. As its expiration date is = near, it's possible that the renewal process for this cert started but = did not complete successfully. You need to check the following: - note the serial ID of the cert, its subject and issuer: $ openssl x509 -noout -text -in /var/lib/ipa/ra-agent.pem You can also check if it has already been renewed (look at the date Not = Before / Not After). If it has been renewed, check the content of the entry = uid=3Dipara,ou=3Dpeople,o=3Dipaca: $ ldapsearch -D cn=3Ddirectory\ manager -W -b uid=3Dipara,ou=3Dpeople,o=3Di= paca There are 2 things to check: - The userCertificate attribute must contain the cert (same value as in = ra-agent.pem, in a single line and without the -----BEGIN = CERTIFICATE----- and -----END CERTIFICATE---- lines). - The description attribute must have the foollowing value: description: 2;;; If it's not the case, it's likely that the renewal failed to update the = entry and that may be causing your issue. You will need to manually fix = the entry using ldapmodify. After that, restart ipa with ipactl stop / ipactl start and check if = certmonger is able to renew the other certs that will expire soon. HTH, flo > = > In this kdc I see these errors in getcert list: > = > Request ID '20190220182014': > =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: MONITORING > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ca-error: Invalid cookie: u'' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 stuck: no > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'auditSign= ingCert = > cert-pki-ca',token=3D'NSS Certificate DB',pin set > =C2=A0 =C2=A0 =C2=A0 =C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'auditSign= ingCert = > cert-pki-ca',token=3D'NSS Certificate DB' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 CA: dogtag-ipa-ca-renew-agent > =C2=A0 =C2=A0 =C2=A0 =C2=A0 issuer: CN=3DCertificate Authority,O=3DL.DOM= AIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 subject: CN=3DCA Audit,O=3DL.DOMAIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 expires: 2019-12-05 13:58:24 UTC > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key usage: digitalSignature,nonRepudiation > =C2=A0 =C2=A0 =C2=A0 =C2=A0 pre-save command: /usr/libexec/ipa/certmonge= r/stop_pkicad > =C2=A0 =C2=A0 =C2=A0 =C2=A0 post-save command: /usr/libexec/ipa/certmong= er/renew_ca_cert = > "auditSigningCert cert-pki-ca" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 track: yes > =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto-renew: yes > Request ID '20190220182015': > =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: MONITORING > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ca-error: Invalid cookie: u'' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 stuck: no > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB',pin set > =C2=A0 =C2=A0 =C2=A0 =C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 CA: dogtag-ipa-ca-renew-agent > =C2=A0 =C2=A0 =C2=A0 =C2=A0 issuer: CN=3DCertificate Authority,O=3DL.DOM= AIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 subject: CN=3DOCSP Subsystem,O=3DL.DOMAIN.IT= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 expires: 2019-12-05 13:58:24 UTC > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key usage: digitalSignature,nonRepudiation,k= eyCertSign,cRLSign > =C2=A0 =C2=A0 =C2=A0 =C2=A0 eku: id-kp-OCSPSigning > =C2=A0 =C2=A0 =C2=A0 =C2=A0 pre-save command: /usr/libexec/ipa/certmonge= r/stop_pkicad > =C2=A0 =C2=A0 =C2=A0 =C2=A0 post-save command: /usr/libexec/ipa/certmong= er/renew_ca_cert = > "ocspSigningCert cert-pki-ca" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 track: yes > =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto-renew: yes > Request ID '20190220182016': > =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: MONITORING > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ca-error: Invalid cookie: u'' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 stuck: no > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'subsystem= Cert = > cert-pki-ca',token=3D'NSS Certificate DB',pin set > =C2=A0 =C2=A0 =C2=A0 =C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'subsystem= Cert = > cert-pki-ca',token=3D'NSS Certificate DB' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 CA: dogtag-ipa-ca-renew-agent > =C2=A0 =C2=A0 =C2=A0 =C2=A0 issuer: CN=3DCertificate Authority,O=3DL.DOM= AIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 subject: CN=3DCA Subsystem,O=3DL.DOMAIN.IT <= http://L.DOMAIN.IT> > =C2=A0 =C2=A0 =C2=A0 =C2=A0 expires: 2019-12-05 13:58:24 UTC > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > =C2=A0 =C2=A0 =C2=A0 =C2=A0 eku: id-kp-serverAuth,id-kp-clientAuth > =C2=A0 =C2=A0 =C2=A0 =C2=A0 pre-save command: /usr/libexec/ipa/certmonge= r/stop_pkicad > =C2=A0 =C2=A0 =C2=A0 =C2=A0 post-save command: /usr/libexec/ipa/certmong= er/renew_ca_cert = > "subsystemCert cert-pki-ca" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 track: yes > =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto-renew: yes > = > Request ID '20190220182018': > =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: MONITORING > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ca-error: Invalid cookie: u'' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 stuck: no > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key pair storage: type=3DFILE,location=3D'/v= ar/lib/ipa/ra-agent.key' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 certificate: type=3DFILE,location=3D'/var/li= b/ipa/ra-agent.pem' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 CA: dogtag-ipa-ca-renew-agent > =C2=A0 =C2=A0 =C2=A0 =C2=A0 issuer: CN=3DCertificate Authority,O=3DL.DOM= AIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 subject: CN=3DIPA RA,O=3DL.DOMAIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 expires: 2019-12-05 13:58:44 UTC > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > =C2=A0 =C2=A0 =C2=A0 =C2=A0 eku: id-kp-serverAuth,id-kp-clientAuth > =C2=A0 =C2=A0 =C2=A0 =C2=A0 pre-save command: /usr/libexec/ipa/certmonge= r/renew_ra_cert_pre > =C2=A0 =C2=A0 =C2=A0 =C2=A0 post-save command: /usr/libexec/ipa/certmong= er/renew_ra_cert > =C2=A0 =C2=A0 =C2=A0 =C2=A0 track: yes > =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto-renew: yes > Request ID '20190220182019': > =C2=A0 =C2=A0 =C2=A0 =C2=A0 status: MONITORING > =C2=A0 =C2=A0 =C2=A0 =C2=A0 ca-error: Server at = > "https://kdc2.l.domain.it:8443/ca/agent/ca/profileProcess" replied: 1: = > Invalid Credential. > =C2=A0 =C2=A0 =C2=A0 =C2=A0 stuck: no > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'Server-Ce= rt = > cert-pki-ca',token=3D'NSS Certificate DB',pin set > =C2=A0 =C2=A0 =C2=A0 =C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'Server-Ce= rt = > cert-pki-ca',token=3D'NSS Certificate DB' > =C2=A0 =C2=A0 =C2=A0 =C2=A0 CA: dogtag-ipa-ca-renew-agent > =C2=A0 =C2=A0 =C2=A0 =C2=A0 issuer: CN=3DCertificate Authority,O=3DL.DOM= AIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 subject: CN=3Dkdc2.l.domain.it = > ,O=3DL.DOMAIN.IT > =C2=A0 =C2=A0 =C2=A0 =C2=A0 expires: 2019-12-10 10:57:52 UTC > =C2=A0 =C2=A0 =C2=A0 =C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > =C2=A0 =C2=A0 =C2=A0 =C2=A0 eku: id-kp-serverAuth,id-kp-clientAuth,id-kp= -emailProtection > =C2=A0 =C2=A0 =C2=A0 =C2=A0 pre-save command: /usr/libexec/ipa/certmonge= r/stop_pkicad > =C2=A0 =C2=A0 =C2=A0 =C2=A0 post-save command: /usr/libexec/ipa/certmong= er/renew_ca_cert = > "Server-Cert cert-pki-ca" > =C2=A0 =C2=A0 =C2=A0 =C2=A0 track: yes > =C2=A0 =C2=A0 =C2=A0 =C2=A0 auto-renew: yes > = > I still have a working replica, so I could just reinstall and have a = > working set in a couple of minutes, but I would like to find out what = > has gone wrong. > = > The systems are running ipa-server-4.6.5-11.el7.centos.3.x86_64 > = > Any help welcome ;-) > = > Thanks, > = > -- > Groeten, > natxo > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org >=20 --===============8066564604462172680==--