Sorry for accidentally dropping freeipa-users.

I was impatient so went back in time before your answer, but I did chose a good date 

Before this, I had the following two entries with an expired date:

Request ID '20150316184508':
status: NEED_TO_SUBMIT
ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm.

Request ID '20150316184529':
status: CA_UNREACHABLE
ca-error: Error setting up ccache for "host" service on client using default keytab: Cannot contact any KDC for requested realm.

After restarting certmonger, I have:

Request ID '20150316184508':
        status: MONITORING
        ca-error: Server at https://spinque04.hq.spinque.com/ipa/xml denied our request, giving up: 2100 (RPC failed at server.  Insufficient access: Principal 'ldap/spinque04.hq.spinque.com@HQ.SPINQUE.COM' is not permitted to use CA '.' with profile 'caIPAserviceCert' for certificate issuance.).

Request ID '20150316184529':
        status: MONITORING
        ca-error: Server at https://spinque04.hq.spinque.com/ipa/xml denied our request, giving up: 2100 (RPC failed at server.  Insufficient access: Principal 'HTTP/spinque04.hq.spinque.com@HQ.SPINQUE.COM' is not permitted to use CA '.' with profile 'caIPAserviceCert' for certificate issuance.).


The journal shows continuous Tomcat errors such as this:

Mar 01 00:09:28 spinque04.hq.spinque.com named-pkcs11[5692]: validating docs.oracle.com/CNAME: bad cache hit (oracle.com/DS)
Mar 01 00:09:28 spinque04.hq.spinque.com named-pkcs11[5692]: broken trust chain resolving 'docs.oracle.com/A/IN': 8.8.8.8#53
Mar 01 00:09:28 spinque04.hq.spinque.com named-pkcs11[5692]: validating docs.oracle.com/CNAME: bad cache hit (oracle.com/DS)
Mar 01 00:09:28 spinque04.hq.spinque.com named-pkcs11[5692]: broken trust chain resolving 'docs.oracle.com/AAAA/IN': 8.8.8.8#53
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: Mar 01, 2017 12:09:32 AM org.apache.catalina.core.ContainerBase backgroundProcess
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@4077b502 background process
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: java.lang.NullPointerException
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:109)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5697)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
Mar 01 00:09:32 spinque04.hq.spinque.com server[5912]: at java.lang.Thread.run(Thread.java:745)

On Wed, 7 Jun 2017 at 16:22 Rob Crittenden <rcritten@redhat.com> wrote:
Adding freeipa-users back.

Roberto Cornacchia wrote:
> Just to confirm, this is indeed the expired certificate
>
> $ getcert list -d /etc/httpd/alias -n Server-Cert
> Number of certificates and requests being tracked: 8.
> Request ID '20150316184529':
> status: CA_UNREACHABLE
> ca-error: Error setting up ccache for "host" service on client using
> default keytab: Cannot contact any KDC for requested realm.
> stuck: no
> key pair storage:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
> certificate:
> type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
> Certificate DB'
> CA: IPA
> issuer: CN=Certificate Authority,O=HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
> subject: CN=spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com>,O=HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
> expires: 2017-03-16 18:45:29 UTC
> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command: /usr/lib64/ipa/certmonger/restart_httpd
> track: yes
> auto-renew: yes

Since you have Apache up and limping along I'd first try:

# getcert resubmit -i 20150316184529

It is very possible that the tool will reject the server cert since it
is expired. If that happens then you'll need to set time back.

I'd encourage you to run: getcert list | grep expires

This will show you what certs need to be renewed and should give some
insight into the window of opportunity for setting the date back.

Assuming it's just the Apache cert that is expired I'd try setting the
date back to March 15, restart IPA, then I'd restart certmonger or try
the about resubmit command again.

Things can get tricky when some certs have renewed and some haven't.

rob

>
>
> On Wed, 7 Jun 2017 at 15:36 Roberto Cornacchia
> <roberto.cornacchia@gmail.com <mailto:roberto.cornacchia@gmail.com>> wrote:
>
>     Thanks Rob,
>
>     I've seen in similar posts that you recommend to go back in time and
>     restart certmonger. Would it work in this case?
>
>
>     On Wed, 7 Jun 2017 at 15:30 Rob Crittenden <rcritten@redhat.com
>     <mailto:rcritten@redhat.com>> wrote:
>
>         Roberto Cornacchia via FreeIPA-users wrote:
>         > OK, I did so and httpd restarts.
>         >
>         > $ openssl s_client -connect 127.0.0.1:443
>         <http://127.0.0.1:443> <http://127.0.0.1:443> -showcerts
>         > CONNECTED(00000003)
>         > depth=1 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>         <http://HQ.SPINQUE.COM>, CN = Certificate
>         > Authority
>         > verify return:1
>         > depth=0 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>         <http://HQ.SPINQUE.COM>, CN =
>         > spinque04.hq.spinque.com <http://spinque04.hq.spinque.com>
>         <http://spinque04.hq.spinque.com>
>         > verify error:num=10:certificate has expired
>         > notAfter=Mar 16 18:45:29 2017 GMT
>         > verify return:1
>         > depth=0 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>         <http://HQ.SPINQUE.COM>, CN =
>         > spinque04.hq.spinque.com <http://spinque04.hq.spinque.com>
>         <http://spinque04.hq.spinque.com>
>         > notAfter=Mar 16 18:45:29 2017 GMT
>         > verify return:1
>         > ---
>         > Certificate chain
>         >  0 s:/O=HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com
>         <http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>         > <http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>         >    i:/O=HQ.SPINQUE.COM/CN=Certificate
>         <http://HQ.SPINQUE.COM/CN=Certificate>
>         > <http://HQ.SPINQUE.COM/CN=Certificate> Authority
>         > ...
>         >
>         > Fair enough, but why does this say it expires in 2019? Are
>         they two
>         > different certificates?
>         >
>         > $ getcert list -d /etc/httpd/alias -n ipaCert
>         > Number of certificates and requests being tracked: 8.
>         > Request ID '20160501114633':
>         > status: MONITORING
>         > stuck: no
>         > key pair storage:
>         >
>         type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>         > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>         > certificate:
>         >
>         type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>         > Certificate DB'
>         > CA: dogtag-ipa-ca-renew-agent
>         > issuer: CN=Certificate Authority,O=HQ.SPINQUE.COM
>         <http://HQ.SPINQUE.COM> <http://HQ.SPINQUE.COM>
>         > subject: CN=IPA RA,O=HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>         <http://HQ.SPINQUE.COM>
>         > expires: 2019-01-26 19:41:51 UTC
>         > key usage:
>         digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>         > eku: id-kp-serverAuth,id-kp-clientAuth
>         > pre-save command: /usr/lib64/ipa/certmonger/renew_ra_cert_pre
>         > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
>         > track: yes
>         > auto-renew: yes
>         >
>         > What's the right way to solve this?
>
>         You're looking at the wrong cert.
>
>         # getcert list -d /etc/httpd/alias -n Server-Cert
>
>         And really, you should examine all certificate status, not just
>         a single
>         one.
>
>         I was also strongly urge you to wait until all problems are resolved
>         before attempting to update packages in the future (unless a package
>         claims to fix a specific problem), particularly when it comes to
>         certificates.
>
>         rob
>