On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>
>
> On 26-10-18 18:00, Timo Aaltonen wrote:
>> On 26.10.2018 18.59, Kees Bakker wrote:
>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>> Kees Bakker wrote:
>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote:
>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote:
>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>> Could it be that this error already
existed since we started? Notice
>>>>>>>>>>>> the Request ID of 2016..., and the
expires: 2018-10-24.
>>>>>>>>>>>>
>>>>>>>>>>>> # getcert list -n ipaCert | sed blabla
>>>>>>>>>>>> Number of certificates and requests being
tracked: 8.
>>>>>>>>>>>> Request ID '20161103094546':
>>>>>>>>>>>> status: CA_UNREACHABLE
>>>>>>>>>>>> ca-error: Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>> stuck: no
>>>>>>>>>>>> key pair storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent
>>>>>>>>>>>> issuer: CN=Certificate
Authority,O=MYDOMAIN
>>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN
>>>>>>>>>>>> expires: 2018-10-24 08:45:40 UTC
>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>> pre-save command:
/usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>> post-save command:
/usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>> track: yes
>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>
>>>>>>>>>>>> In other words, is this the same issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>> The problem is your certs expired yesterday
so connections won't work
>>>>>>>>>>> (the code and message don't come from
within certmonger).
>>>>>>>>>>>
>>>>>>>>>>> certmonger _should_ have renewed them. Try
killing ntpd, going back a
>>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd and
the CA then certmonger and
>>>>>>>>>>> see what happens.
>>>>>>>>>>>
>>>>>>>>>> Easy for you to say. You know what you're
doing :-)
>>>>>>>>>> For me it's all magic.
>>>>>>>>>>
>>>>>>>>>> Anyway, I'll try it. I'm just scared to
set the clock back, because there may
>>>>>>>>>> be clients in the network that use this server as
a NTP server.
>>>>>>>>>>
>>>>>>>>>> Another thing I want to mention is that the error
started showing up two days
>>>>>>>>>> ago, on Oct 22, while the expiration is today,
Oct 24.
>>>>>>>>>>
>>>>>>>>> It shouldn't take more than a few minutes to roll
back time, restart
>>>>>>>>> services and see what happens. I think your NTP
clients will be able to
>>>>>>>>> recover ok if the server is not available for a few
minutes.
>>>>>>>>>
>>>>>>>>> certmonger logs to syslog so you probably want to
look at that to see if
>>>>>>>>> you can find a reason the certs weren't renewed
automatically.
>>>>>>>>>
>>>>>>>> No, that didn't help.
>>>>>>>> And in the syslog there was nothing more than this. (I
had to stop the
>>>>>>>> nameserver because it was spitting out lots of
messages.)
>>>>>>>>
>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been changed
>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been
changed
>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping Certificate
monitoring and PKI enrollment...
>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped Certificate
monitoring and PKI enrollment.
>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting Certificate
monitoring and PKI enrollment...
>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started Certificate
monitoring and PKI enrollment.
>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11
06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>> Review: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit:
Forwarding request to dogtag-ipa-renew-agent
>>>>>>>> Oct 11 06:00:07 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11
06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit:
Forwarding request to dogtag-ipa-renew-agent
>>>>>>>> Oct 11 06:00:17 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11
06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview:
Problem with the SSL CA cert (path? access rights?).
>>>>>>>>
>>>>>>> Ok, I think I know what is going on. This is Ubuntu which
AFAIK still
>>>>>>> lacks nss-pem. That is probably why it can't connect to
renew the certs.
>>>>>>>
>>>>>>> I don't know if there is a workaround. Timo, do you
know?
>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on it.
I've
>>>>>> never tested cert renewal though.
>>>>>>
>>>>> Does that mean, I'm screwed? What options do I have?
>>>>> Live with it?
>>>>> Migrate to, say Centos?
>>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty whether
it will work)?
>>>>> Something else?
>>>> Stock 18.04 has other issues, there's an updated version on
>>>> ppa:freeipa/staging which is backported from 18.10 and should be fine
>>>> and hopefully provided as a stable update on 18.04 later on.
>>>>
>>>> But you could try pulling libnsspem from 18.04, and *then* roll back
time?
>>>>
>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>
>>> Then I stopped ntp (and bind).
>>> Set the time back to Oct 11
>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger
>>> (in that order).
>>>
>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request
to dogtag-ipa-renew-agent
>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error
60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate
cannot be authenticated with given CA certificates.
>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327] Error
60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate
cannot be authenticated with given CA certificates.
>>>
>>> :-(
>>>
>>> Rob said also to restart CA.
>>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger"
>>> I don't know which service that is. Does that matter?
>> systemctl restart ipa?
>>
>>
> I'm a bit scared to restart service ipa, because it also restarts several other
services,
> link bind, and perhaps ntp. The latter is the one that I want to be absolutely in
control
> of not starting.
And you're right! The CA is pki-tomcatd, so you already restarted it.
> It's getting too late now, time for weekend. I'll give it another try on
Monday.
> Meanwhile I want to point at the changed message. In case that rings a bell for
> someone.
>
> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60
connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate
cannot be authenticated with given CA certificates.
>
You can have a look at Rob's blog for additional items to check:
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite
understand all details.
I really need some guidance what to do next. I tried the date trick, I installed
libnsspem (from Ubunu 18.04). The certmonger error message changed from
Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d background
process
java.lang.NullPointerException
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last
time?
--
Kees