On 26-10-18 18:20, Florence Blanc-Renaud wrote:
> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>
>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users wrote:
>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>>> Kees Bakker wrote:
>>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote:
>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden wrote:
>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>>> Could it be that this error already
existed since we started? Notice
>>>>>>>>>>>>> the Request ID of 2016..., and the
expires: 2018-10-24.
>>>>>>>>>>>>>
>>>>>>>>>>>>> # getcert list -n ipaCert | sed
blabla
>>>>>>>>>>>>> Number of certificates and requests
being tracked: 8.
>>>>>>>>>>>>> Request ID '20161103094546':
>>>>>>>>>>>>> status: CA_UNREACHABLE
>>>>>>>>>>>>> ca-error: Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>> key pair storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>> CA: dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>> issuer: CN=Certificate
Authority,O=MYDOMAIN
>>>>>>>>>>>>> subject: CN=IPA RA,O=MYDOMAIN
>>>>>>>>>>>>> expires: 2018-10-24 08:45:40
UTC
>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>> pre-save command:
/usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>> post-save command:
/usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>>
>>>>>>>>>>>>> In other words, is this the same
issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>> The problem is your certs expired
yesterday so connections won't work
>>>>>>>>>>>> (the code and message don't come from
within certmonger).
>>>>>>>>>>>>
>>>>>>>>>>>> certmonger _should_ have renewed them.
Try killing ntpd, going back a
>>>>>>>>>>>> few days, restart krb5kdc, dirsrv, httpd
and the CA then certmonger and
>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>
>>>>>>>>>>> Easy for you to say. You know what you're
doing :-)
>>>>>>>>>>> For me it's all magic.
>>>>>>>>>>>
>>>>>>>>>>> Anyway, I'll try it. I'm just scared
to set the clock back, because there may
>>>>>>>>>>> be clients in the network that use this
server as a NTP server.
>>>>>>>>>>>
>>>>>>>>>>> Another thing I want to mention is that the
error started showing up two days
>>>>>>>>>>> ago, on Oct 22, while the expiration is
today, Oct 24.
>>>>>>>>>>>
>>>>>>>>>> It shouldn't take more than a few minutes to
roll back time, restart
>>>>>>>>>> services and see what happens. I think your NTP
clients will be able to
>>>>>>>>>> recover ok if the server is not available for a
few minutes.
>>>>>>>>>>
>>>>>>>>>> certmonger logs to syslog so you probably want to
look at that to see if
>>>>>>>>>> you can find a reason the certs weren't
renewed automatically.
>>>>>>>>>>
>>>>>>>>> No, that didn't help.
>>>>>>>>> And in the syslog there was nothing more than this.
(I had to stop the
>>>>>>>>> nameserver because it was spitting out lots of
messages.)
>>>>>>>>>
>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time has been
changed
>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]: Time has been
changed
>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopping
Certificate monitoring and PKI enrollment...
>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Stopped
Certificate monitoring and PKI enrollment.
>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Starting
Certificate monitoring and PKI enrollment...
>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]: Started
Certificate monitoring and PKI enrollment.
>>>>>>>>> Oct 11 06:00:05 ipasrv certmonger[131018]: 2018-10-11
06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>> Review: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>> Oct 11 06:00:07 ipasrv certmonger[131018]: 2018-10-11
06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>> Oct 11 06:00:17 ipasrv certmonger[131018]: 2018-10-11
06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview:
Problem with the SSL CA cert (path? access rights?).
>>>>>>>>>
>>>>>>>> Ok, I think I know what is going on. This is Ubuntu which
AFAIK still
>>>>>>>> lacks nss-pem. That is probably why it can't connect
to renew the certs.
>>>>>>>>
>>>>>>>> I don't know if there is a workaround. Timo, do you
know?
>>>>>>> Ubuntu 18.04 and up have libnsspem, and certmonger depends on
it. I've
>>>>>>> never tested cert renewal though.
>>>>>>>
>>>>>> Does that mean, I'm screwed? What options do I have?
>>>>>> Live with it?
>>>>>> Migrate to, say Centos?
>>>>>> Try to upgrade the server to Ubuntu 18.04 (with uncertainty
whether it will work)?
>>>>>> Something else?
>>>>> Stock 18.04 has other issues, there's an updated version on
>>>>> ppa:freeipa/staging which is backported from 18.10 and should be
fine
>>>>> and hopefully provided as a stable update on 18.04 later on.
>>>>>
>>>>> But you could try pulling libnsspem from 18.04, and *then* roll back
time?
>>>>>
>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>
>>>> Then I stopped ntp (and bind).
>>>> Set the time back to Oct 11
>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd, certmonger
>>>> (in that order).
>>>>
>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding
request to dogtag-ipa-renew-agent
>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327]
Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer
certificate cannot be authenticated with given CA certificates.
>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11 06:08:12 [168327]
Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer
certificate cannot be authenticated with given CA certificates.
>>>>
>>>> :-(
>>>>
>>>> Rob said also to restart CA.
>>>> "restart krb5kdc, dirsrv, httpd and the CA then certmonger"
>>>> I don't know which service that is. Does that matter?
>>> systemctl restart ipa?
>>>
>>>
>> I'm a bit scared to restart service ipa, because it also restarts several
other services,
>> link bind, and perhaps ntp. The latter is the one that I want to be absolutely in
control
>> of not starting.
> And you're right! The CA is pki-tomcatd, so you already restarted it.
>
>> It's getting too late now, time for weekend. I'll give it another try on
Monday.
>> Meanwhile I want to point at the changed message. In case that rings a bell for
>> someone.
>>
>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03 [168327] Error 60
connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate
cannot be authenticated with given CA certificates.
>>
> You can have a look at Rob's blog for additional items to check:
>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
Thanks, I just stumbled on it myself. Interesting read, although I don't quite
understand all details.
I really need some guidance what to do next. I tried the date trick, I installed
libnsspem (from Ubunu 18.04). The certmonger error message changed from
Error 77 into Error 60, but the problem remained.
Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase backgroundProcess
WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@5417a64d
background process
java.lang.NullPointerException
at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
at java.lang.Thread.run(Thread.java:748)
I could do the date trick again, but then the question is, why didn't it work last
time?
I tried it once more to set the date back and to restart the renewal process.
I keep getting Error 60 with
Peer certificate cannot be authenticated with given CA certificates.
Also, I had a look at Rob's blog. But I'm lost at what to do with
"... the fix was to reset the NSS trust flags in the Apache NSS database"
The curl command (with the date set back) seems to connect but it gives a
gnutls_handshake() failed: Illegal parameter
And then I don't know what it looks like if
" You should get client certificate not found."
So I didn't try the certutil -M commands.
BTW. While setting the date back I have bind (DNS) switched off because it
gave a lot of messages in my /var/log/syslog.
--
Kees