On 30-10-18 19:41, Rob Crittenden wrote:
> Kees Bakker wrote:
>> On 29-10-18 19:30, Rob Crittenden wrote:
>>> Kees Bakker via FreeIPA-users wrote:
>>>> On 29-10-18 11:56, Kees Bakker via FreeIPA-users wrote:
>>>>> On 26-10-18 18:20, Florence Blanc-Renaud wrote:
>>>>>> On 10/26/18 6:09 PM, Kees Bakker via FreeIPA-users wrote:
>>>>>>> On 26-10-18 18:00, Timo Aaltonen wrote:
>>>>>>>> On 26.10.2018 18.59, Kees Bakker wrote:
>>>>>>>>> On 26-10-18 14:55, Timo Aaltonen wrote:
>>>>>>>>>> On 26.10.2018 09:59, Kees Bakker via
FreeIPA-users wrote:
>>>>>>>>>>> On 25-10-18 20:46, Timo Aaltonen wrote:
>>>>>>>>>>>> On 25.10.2018 21.44, Rob Crittenden
wrote:
>>>>>>>>>>>>> Kees Bakker wrote:
>>>>>>>>>>>>>> On 25-10-18 16:11, Rob Crittenden
wrote:
>>>>>>>>>>>>>>> Kees Bakker via FreeIPA-users
wrote:
>>>>>>>>>>>>>>>> On 25-10-18 14:18, Rob
Crittenden wrote:
>>>>>>>>>>>>>>>>> Kees Bakker via
FreeIPA-users wrote:
>>>>>>>>>>>>>>>>>> Could it be that
this error already existed since we started? Notice
>>>>>>>>>>>>>>>>>> the Request ID of
2016..., and the expires: 2018-10-24.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> # getcert list -n
ipaCert | sed blabla
>>>>>>>>>>>>>>>>>> Number of
certificates and requests being tracked: 8.
>>>>>>>>>>>>>>>>>> Request ID
'20161103094546':
>>>>>>>>>>>>>>>>>> status:
CA_UNREACHABLE
>>>>>>>>>>>>>>>>>> ca-error:
Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem
with the SSL CA cert (path? access rights?).
>>>>>>>>>>>>>>>>>> stuck: no
>>>>>>>>>>>>>>>>>> key pair
storage:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB',pinfile='/etc/apache2/nssdb/pwdfile.txt'
>>>>>>>>>>>>>>>>>> certificate:
type=NSSDB,location='/etc/apache2/nssdb',nickname='ipaCert',token='NSS
Certificate DB'
>>>>>>>>>>>>>>>>>> CA:
dogtag-ipa-ca-renew-agent
>>>>>>>>>>>>>>>>>> issuer:
CN=Certificate Authority,O=MYDOMAIN
>>>>>>>>>>>>>>>>>> subject:
CN=IPA RA,O=MYDOMAIN
>>>>>>>>>>>>>>>>>> expires:
2018-10-24 08:45:40 UTC
>>>>>>>>>>>>>>>>>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>>>>>>>>>>>>>>>>> eku:
id-kp-serverAuth,id-kp-clientAuth
>>>>>>>>>>>>>>>>>> pre-save
command: /usr/lib/ipa/certmonger/renew_ra_cert_pre
>>>>>>>>>>>>>>>>>> post-save
command: /usr/lib/ipa/certmonger/renew_ra_cert
>>>>>>>>>>>>>>>>>> track: yes
>>>>>>>>>>>>>>>>>> auto-renew:
yes
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In other words,
is this the same issue as
https://pagure.io/freeipa/issue/7422 ?
>>>>>>>>>>>>>>>>> The problem is your
certs expired yesterday so connections won't work
>>>>>>>>>>>>>>>>> (the code and message
don't come from within certmonger).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> certmonger _should_
have renewed them. Try killing ntpd, going back a
>>>>>>>>>>>>>>>>> few days, restart
krb5kdc, dirsrv, httpd and the CA then certmonger and
>>>>>>>>>>>>>>>>> see what happens.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Easy for you to say. You
know what you're doing :-)
>>>>>>>>>>>>>>>> For me it's all
magic.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyway, I'll try it.
I'm just scared to set the clock back, because there may
>>>>>>>>>>>>>>>> be clients in the network
that use this server as a NTP server.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another thing I want to
mention is that the error started showing up two days
>>>>>>>>>>>>>>>> ago, on Oct 22, while the
expiration is today, Oct 24.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It shouldn't take more
than a few minutes to roll back time, restart
>>>>>>>>>>>>>>> services and see what
happens. I think your NTP clients will be able to
>>>>>>>>>>>>>>> recover ok if the server is
not available for a few minutes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> certmonger logs to syslog so
you probably want to look at that to see if
>>>>>>>>>>>>>>> you can find a reason the
certs weren't renewed automatically.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, that didn't help.
>>>>>>>>>>>>>> And in the syslog there was
nothing more than this. (I had to stop the
>>>>>>>>>>>>>> nameserver because it was
spitting out lots of messages.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oct 11 06:00:00 ipasrv
systemd[1]: Time has been changed
>>>>>>>>>>>>>> Oct 11 06:00:00 ipasrv
systemd[52167]: Time has been changed
>>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv
systemd[1]: Stopping Certificate monitoring and PKI enrollment...
>>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv
systemd[1]: Stopped Certificate monitoring and PKI enrollment.
>>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv
systemd[1]: Starting Certificate monitoring and PKI enrollment...
>>>>>>>>>>>>>> Oct 11 06:00:04 ipasrv
systemd[1]: Started Certificate monitoring and PKI enrollment.
>>>>>>>>>>>>>> Oct 11 06:00:05 ipasrv
certmonger[131018]: 2018-10-11 06:00:05 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profile
>>>>>>>>>>>>>> Review: Problem with the SSL CA
cert (path? access rights?).
>>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>>>> Oct 11 06:00:07 ipasrv
certmonger[131018]: 2018-10-11 06:00:07 [131018] Error 77 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert
(path? access rights?).
>>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>>>>>>> Oct 11 06:00:17 ipasrv
certmonger[131018]: 2018-10-11 06:00:17 [131018] Error 77 connecting to
https://ipasrv:8443/ca/agent/ca/profileReview: Problem with the SSL CA cert (path? access
rights?).
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, I think I know what is going on.
This is Ubuntu which AFAIK still
>>>>>>>>>>>>> lacks nss-pem. That is probably why
it can't connect to renew the certs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't know if there is a
workaround. Timo, do you know?
>>>>>>>>>>>> Ubuntu 18.04 and up have libnsspem, and
certmonger depends on it. I've
>>>>>>>>>>>> never tested cert renewal though.
>>>>>>>>>>>>
>>>>>>>>>>> Does that mean, I'm screwed? What options
do I have?
>>>>>>>>>>> Live with it?
>>>>>>>>>>> Migrate to, say Centos?
>>>>>>>>>>> Try to upgrade the server to Ubuntu 18.04
(with uncertainty whether it will work)?
>>>>>>>>>>> Something else?
>>>>>>>>>> Stock 18.04 has other issues, there's an
updated version on
>>>>>>>>>> ppa:freeipa/staging which is backported from
18.10 and should be fine
>>>>>>>>>> and hopefully provided as a stable update on
18.04 later on.
>>>>>>>>>>
>>>>>>>>>> But you could try pulling libnsspem from 18.04,
and *then* roll back time?
>>>>>>>>>>
>>>>>>>>> I installed libnsspem_1.0.3-0ubuntu2_amd64.deb
>>>>>>>>>
>>>>>>>>> Then I stopped ntp (and bind).
>>>>>>>>> Set the time back to Oct 11
>>>>>>>>> Restarted krb5-kdc, dirsrv@MYDOMAIN, apache2,
pki-tomcatd, certmonger
>>>>>>>>> (in that order).
>>>>>>>>>
>>>>>>>>> Oct 11 06:08:03 ipasrv
dogtag-ipa-ca-renew-agent-submit: Forwarding request to dogtag-ipa-renew-agent
>>>>>>>>> Oct 11 06:08:03 ipasrv
dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned 3
>>>>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11
06:08:03 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>>> Oct 11 06:08:12 ipasrv certmonger[168327]: 2018-10-11
06:08:12 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>>>
>>>>>>>>> :-(
>>>>>>>>>
>>>>>>>>> Rob said also to restart CA.
>>>>>>>>> "restart krb5kdc, dirsrv, httpd and the CA
then certmonger"
>>>>>>>>> I don't know which service that is. Does that
matter?
>>>>>>>> systemctl restart ipa?
>>>>>>>>
>>>>>>>>
>>>>>>> I'm a bit scared to restart service ipa, because it also
restarts several other services,
>>>>>>> link bind, and perhaps ntp. The latter is the one that I want
to be absolutely in control
>>>>>>> of not starting.
>>>>>> And you're right! The CA is pki-tomcatd, so you already
restarted it.
>>>>>>
>>>>>>> It's getting too late now, time for weekend. I'll
give it another try on Monday.
>>>>>>> Meanwhile I want to point at the changed message. In case
that rings a bell for
>>>>>>> someone.
>>>>>>>
>>>>>>> Oct 11 06:08:03 ipasrv certmonger[168327]: 2018-10-11
06:08:03 [168327] Error 60 connecting to
https://ipasrv.mydomain:8443/ca/agent/ca/profileReview: Peer certificate cannot be
authenticated with given CA certificates.
>>>>>>>
>>>>>> You can have a look at Rob's blog for additional items to
check:
>>>>>>
https://rcritten.wordpress.com/2017/09/20/peer-certificate-cannot-be-auth...
>>>>> Thanks, I just stumbled on it myself. Interesting read, although I
don't quite
>>>>> understand all details.
>>>>>
>>>>> I really need some guidance what to do next. I tried the date trick,
I installed
>>>>> libnsspem (from Ubunu 18.04). The certmonger error message changed
from
>>>>> Error 77 into Error 60, but the problem remained.
>>>>>
>>>>> Futhermore I noticed that pki-tomcat spits out a warning every 10
seconds
>>>>>
>>>>> Oct 29, 2018 11:47:05 AM org.apache.catalina.core.ContainerBase
backgroundProcess
>>>>> WARNING: Exception processing realm
com.netscape.cms.tomcat.ProxyRealm@5417a64d background process
>>>>> java.lang.NullPointerException
>>>>> at
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:113)
>>>>> at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1357)
>>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1543)
>>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1553)
>>>>> at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1521)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>
>>>>> I could do the date trick again, but then the question is, why
didn't it work last time?
>>>> I tried it once more to set the date back and to restart the renewal
process.
>>>> I keep getting Error 60 with
>>>> Peer certificate cannot be authenticated with given CA certificates.
>>>>
>>>> Also, I had a look at Rob's blog. But I'm lost at what to do
with
>>>> "... the fix was to reset the NSS trust flags in the Apache NSS
database"
>>>> The curl command (with the date set back) seems to connect but it gives
a
>>>> gnutls_handshake() failed: Illegal parameter
>>>>
>>>> And then I don't know what it looks like if
>>>> " You should get client certificate not found."
>>>> So I didn't try the certutil -M commands.
>>>>
>>>> BTW. While setting the date back I have bind (DNS) switched off because
it
>>>> gave a lot of messages in my /var/log/syslog.
>>>>
>>> Right, this is for a previous version of IPA on RHEL/Fedora. The RA
>>> agent cert used to be stored in the Apache NSS database.
>> That's a problem in general with notes and blogs. When you write
>> things you don't know what to write about possible validness in
>> the future.
>>
>>> curl linked aganst gnu_tls? I have no idea, but it isn't this.
>>>
>>> This error message originates in curl.
>>>
>>> What is the validity of your CA certificate itself? Is it still valid
>>> back in time?
>> Yes it is. Again, this is Ubuntu 16.04, I'm looking at ...
>>
>> certutil -L -d /etc/pki/pki-tomcat/alias/ -n 'caSigningCert cert-pki-ca'
>> certutil -L -d /etc/apache2/nssdb/ -n 'GHS.NL IPA CA'
>> certutil -L -d /etc/dirsrv/slapd-GHS-NL -n 'GHS.NL IPA CA'
>>
>> Validity:
>> Not Before: Thu Nov 03 09:45:18 2016
>> Not After : Mon Nov 03 09:45:18 2036
>>
> Ok I'm just flying blind right now, reading curl code, but in the gnutls
> code it throws this error if the server cert validation fails.
>
> I don't know where gnutls reads its CA trust from or whether it honors
> the system-wide trust, or even if your IPA CA is in the global trust
> properly.
>
> When back in time do IPA operations work? ipa user-show admin, ipa
> cert-show 1, etc?
I'll do that maybe later this week. I don't want to mess with the IPA
server while it is still working (sort of).
Meanwhile, here is a new question, a different approach.
Can I manually generate a new certificate?
For example the "Server-Cert cert-pki-ca" of the pki-tomcat server.
And if so, do you have a suggestion about the commands to use?
Since nobody told me
it was a bad idea, I went ahead and created
a new Cert for the pki-tomcat. It was not easy, but now I have a new
cert that is valid. If I check it with openssl.
# openssl s_client -showcerts -connect `hostname`:8443 < /dev/null
CONNECTED(00000003)
depth=1 O = GHS.NL, CN = Certificate Authority
verify return:1
depth=0 O = GHS.NL, CN = rotte.ghs.nl
verify return:1
...
Verify return code: 0 (ok)
The curl test command still fails
# SSL_DIR=/etc/apache2/nssdb/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt
https://`hostname`:8443/ca/agent/ca/profileReview
* Connected to rotte.ghs.nl (172.16.16.75) port 8443 (#0)
* found 1 certificates in /etc/ipa/ca.crt
* found 616 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / RSA_AES_128_CBC_SHA1
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: rotte.ghs.nl (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: O=GHS.NL,CN=rotte.ghs.nl
* start date: Thu, 01 Nov 2018 10:48:12 GMT
* expire date: Mon, 01 Nov 2021 10:48:12 GMT
* issuer: O=GHS.NL,CN=Certificate Authority
* compression: NULL
* ALPN, server did not agree to a protocol
GET /ca/agent/ca/profileReview HTTP/1.1
Host: rotte.ghs.nl:8443
User-Agent: curl/7.47.0
Accept: */*
< HTTP/1.1 500 Internal Server Error
< Server: Apache-Coyote/1.1
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 2161
< Date: Thu, 01 Nov 2018 11:08:53 GMT
< Connection: close
<
{ [2161 bytes data]
100 2161 100 2161 0 0 18051 0 --:--:-- --:--:-- --:--:-- 18159
* Closing connection 0
Certmonger still gives that Error 60 message
Nov 1 13:28:23 ipasrv dogtag-ipa-ca-renew-agent-submit: Forwarding request to
dogtag-ipa-renew-agent
Nov 1 13:28:23 ipasrv dogtag-ipa-ca-renew-agent-submit: dogtag-ipa-renew-agent returned
3
Nov 1 13:28:23 ipasrv certmonger[41919]: 2018-11-01 13:28:23 [41919] Error 60 connecting
to
: Peer certificate cannot be
authenticated with given CA certificates.
Are there log files to be looked at to see what that "Internal Server Error"
is?
--
Kees