On 29-10-18 19:30, Rob Crittenden wrote:
> Kees Bakker via FreeIPA-users wrote:
>> On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
>>> On 26-10-18 18:20, Florence Blanc-Renaud wrote:
>>>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>>>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>>>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>>>>> On 26.10.2018 09:59, Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>>>>> On 25.10.2018 21.44, Rob Crittenden wrote:
>>>>>>>>>>> Kees Bakker wrote:
>>>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden wrote:
>>>>>>>>>>>>> Kees Bakker via FreeIPA-users wrote:
>>>>>>>>>>>>>> On 25-10-18 14:18, Rob Crittenden
wrote:
>>>>>>>>>>>>>>> Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>>>>>>>>> Could it be that this
error already existed since we started? Notice
>>>>>>>>>>>>>>>> the Request ID of
2016..., and the expires: 2018-10-24.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> # getcert list -n ipaCert
| sed blabla
>>>>>>>>>>>>>>>> Number of certificates
and requests being tracked: 8.
>>>>>>>>>>>>>>>> Request ID
'20161103094546':
>>>>>>>>>>>>>>>> status:
CA_UNREACHABLE
>>>>>>>>>>>>>>>> ca-error: Error 77
connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL
CA cert (path? access rights?).
>>>>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>>>>> key pair storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>>>>> CA:
dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>>>>> issuer:
CN=Certificate Authority,O=MYDOMAIN
>>>>>>>>>>>>>>>> subject: CN=IPA
RA,O=MYDOMAIN
>>>>>>>>>>>>>>>> expires: 2018-10-24
08:45:40 UTC
>>>>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>>>>> pre-save command:
/usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>>>>> post-save command:
/usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>>>>> auto-renew: yes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In other words, is this
the same issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>>>>> The problem is your certs
expired yesterday so connections won't work
>>>>>>>>>>>>>>> (the code and message
don't come from within certmonger).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> certmonger _should_ have
renewed them. Try killing ntpd, going back a
>>>>>>>>>>>>>>> few days, restart krb5kdc,
dirsrv, httpd and the CA then certmonger and
>>>>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Easy for you to say. You know
what you're doing :-)
>>>>>>>>>>>>>> For me it's all magic.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyway, I'll try it. I'm
just scared to set the clock back, because there may
>>>>>>>>>>>>>> be clients in the network that
use this server as a NTP server.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another thing I want to mention
is that the error started showing up two days
>>>>>>>>>>>>>> ago, on Oct 22, while the
expiration is today, Oct 24.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> It shouldn't take more than a few
minutes to roll back time, restart
>>>>>>>>>>>>> services and see what happens. I
think your NTP clients will be able to
>>>>>>>>>>>>> recover ok if the server is not
available for a few minutes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> certmonger logs to syslog so you
probably want to look at that to see if
>>>>>>>>>>>>> you can find a reason the certs
weren't renewed automatically.
>>>>>>>>>>>>>
>>>>>>>>>>>> No, that didn't help.
>>>>>>>>>>>> And in the syslog there was nothing more
than this. (I had to stop the
>>>>>>>>>>>> nameserver because it was spitting out
lots of messages.)
>>>>>>>>>>>>
>>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[1]: Time
has been changed
>>>>>>>>>>>> Oct 11 06:00:00 ipasrv systemd[52167]:
Time has been changed
>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Stopping Certificate monitoring and PKI enrollment...
>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Stopped Certificate monitoring and PKI enrollment.
>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Starting Certificate monitoring and PKI enrollment...
>>>>>>>>>>>> Oct 11 06:00:04 ipasrv systemd[1]:
Started Certificate monitoring and PKI enrollment.
>>>>>>>>>>>> Oct 11 06:00:05 ipasrv
certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>>>>> Review: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>>>>>
>>>>>>>>>>> Ok, I think I know what is going on. This is
Ubuntu which AFAIK still
>>>>>>>>>>> lacks nss-pem. That is probably why it
can't connect to renew the certs.
>>>>>>>>>>>
>>>>>>>>>>> I don't know if there is a workaround.
Timo, do you know?
>>>>>>>>>> Ubuntu 18.04 and up have libnsspem, and
certmonger depends on it. I've
>>>>>>>>>> never tested cert renewal though.
>>>>>>>>>>
>>>>>>>>> Does that mean, I'm screwed? What options do I
have?
>>>>>>>>> Live with it?
>>>>>>>>> Migrate to, say Centos?
>>>>>>>>> Try to upgrade the server to Ubuntu 18.04 (with
uncertainty whether it will work)?
>>>>>>>>> Something else?
>>>>>>>> Stock 18.04 has other issues, there's an updated
version on
>>>>>>>> ppa:freeipa/staging which is backported from 18.10 and
should be fine
>>>>>>>> and hopefully provided as a stable update on 18.04 later
on.
>>>>>>>>
>>>>>>>> But you could try pulling libnsspem from 18.04, and
*then* roll back time?
>>>>>>>>
>>>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>>>>
>>>>>>> Then I stopped ntp (and bind).
>>>>>>> Set the time back to Oct 11
>>>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2, pki-tomcatd,
certmonger
>>>>>>> (in that order).
>>>>>>>
>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
Forwarding request to dogtag-ipa-renew-agent
>>>>>>> Oct 11 06:08:03 ipasrv dogtag-ipa-ca-renew-agent-submit:
dogtag-ipa-renew-agent returned 3
>>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11
06:08:03 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11
06:08:12 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>
>>>>>>> :-(
>>>>>>>
>>>>>>> Rob said also to restart CA.
>>>>>>> "restart krb5kdc, dirsrv, httpd and the CA then
certmonger"
>>>>>>> I don't know which service that is. Does that matter?
>>>>>> systemctl restart ipa?
>>>>>>
>>>>>>
>>>>> I'm a bit scared to restart service ipa, because it also restarts
several other services,
>>>>> link bind, and perhaps ntp. The latter is the one that I want to be
absolutely in control
>>>>> of not starting.
>>>> And you're right! The CA is pki-tomcatd, so you already restarted
it.
>>>>
>>>>> It's getting too late now, time for weekend. I'll give it
another try on Monday.
>>>>> Meanwhile I want to point at the changed message. In case that rings
a bell for
>>>>> someone.
>>>>>
>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11 06:08:03
[168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview:
Peer certificate cannot be authenticated with given CA certificates.
>>>>>
>>>> You can have a look at Rob's blog for additional items to check:
>>>>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
>>> Thanks, I just stumbled on it myself. Interesting read, although I don't
quite
>>> understand all details.
>>>
>>> I really need some guidance what to do next. I tried the date trick, I
installed
>>> libnsspem (from Ubunu 18.04). The certmonger error message changed from
>>> Error 77 into Error 60, but the problem remained.
>>>
>>> Futhermore I noticed that pki-tomcat spits out a warning every 10 seconds
>>>
>>> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase
backgroundProcess
>>> WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@5417a64d background process
>>> java.lang.NullPointerException
>>> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
>>> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>> I could do the date trick again, but then the question is, why didn't it
work last time?
>> I tried it once more to set the date back and to restart the renewal process.
>> I keep getting Error 60 with
>> Peer certificate cannot be authenticated with given CA certificates.
>>
>> Also, I had a look at Rob's blog. But I'm lost at what to do with
>> "... the fix was to reset the NSS trust flags in the Apache NSS
database"
>> The curl command (with the date set back) seems to connect but it gives a
>> gnutls_handshake() failed: Illegal parameter
>>
>> And then I don't know what it looks like if
>> " You should get client certificate not found."
>> So I didn't try the certutil -M commands.
>>
>> BTW. While setting the date back I have bind (DNS) switched off because it
>> gave a lot of messages in my /var/log/syslog.
>>
> Right, this is for a previous version of IPA on RHEL/Fedora. The RA
> agent cert used to be stored in the Apache NSS database.
That's a problem in general with notes and blogs. When you write
things you don't know what to write about possible validness in
the future.
>
> curl linked aganst gnu_tls? I have no idea, but it isn't this.
>
> This error message originates in curl.
>
> What is the validity of your CA certificate itself? Is it still valid
> back in time?
Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca'
certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA'
certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
Validity:
Not Before: Thu Nov 03 09:45:18 2016
Not After : Mon Nov 03 09:45:18 2036
Ok I'm just flying blind right now, reading curl code, but in the gnutls
code it throws this error if the server cert validation fails.
I don't know where gnutls reads its CA trust from or whether it honors
the system-wide trust, or even if your IPA CA is in the global trust
properly.
When back in time do IPA operations work? ipa user-show admin, ipa
cert-show 1, etc?
rob