Re: pki-tomcatd not starting
by Rob Crittenden
Scott Z. via FreeIPA-users wrote:
> Adding the "NSSEnforceValidCerts off" definitely got me past the HTTPD
> error. It started up and then I ran the systemctl start
> pki-tomcatd@pki-tomcat which seemed to start up without any errors (it
> didn't throw any on the command line), but checking the debug log I see
> I'm still getting the same, original "Peer's Certificate has expired"
> message for "Server-Cert cert-pki-ca". I just can't win 🙂
> It's expired, I know it's expired, why does FreeIPA fight me so hard on
> just trying to renew it?! LOL!
>
> Just for fun I then ran the "getcert renew -i <reqid>" command. But per
> "getcert list", it's still showing as CA_UNREACHABLE and Internal Error.
The CA is a servlet so tomcat can start without the CA starting. I'd
look in the CA logs under /var/log/pki-tomcat/
certmonger logs to syslog so use journalctl to see if it provided any
more details on the failure, but it sounds like an issue with the CA.
rob
> Scott
>
>
> ------------------------------------------------------------------------
> *From:* Rob Crittenden <rcritten(a)redhat.com>
> *Sent:* Tuesday, August 11, 2020 8:07 AM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
> Florence Blanc-Renaud <flo(a)redhat.com>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>
> Scott Z. via FreeIPA-users wrote:
>> Forgot to reply again - ugh!
>> Hmmmm, so my domain is actually "idm.project.its.srv2", so I was
>> literally typing "systemctl start dirsrv(a)idm.project.its.srv2"Â I see
>> what you're saying, I need to put in dashes instead of periods! DOH!Â
>> Done. Moving on...
>> 4) Ran systemctl start krb5kdc
>> 5) Ran systemctl start kadmin
>> 6) Ran systemctl start named-pkcs11
>> 7) Ran systemctl start httpd - got an error here, nothing really
>> useful in the logs or journalctl, it says it's starting the Apache HTTP
>> server, then throws "httpd.service: main process exited, code=exited,
>> status=1/FAILURE", and "Failed to start The Apache HTTP Server".Â
>> Finally there is a mention of 'too much time skew'. I assume the
>> problem is that I'm trying to start HTTPD on a system where the date is
>> almost a year old.Â
>> Although now that I'm looking at /var/log/httpd/error_log, I see mention
>> of "SSL Library Error: -8181 Certificate has expired". CERTIFICATES!!!
>> "Unable to verify certificate 'Server-Cert'. Add "NSSEnfroceValideCerts
>> off" to nss.conf so the server can start until the problem can be
>> resolved", so maybe I'll try that.
>
> That can work, just remember to revert it, but it just bypasses the
> start up check. Clients will still require cert validity.
>
> I don't think it will matter either way as the CA certs renew directly
> against the CA so Apache not running shouldn't be an issue.
>
> rob
>
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Tuesday, August 11, 2020 6:55 AM
>> *To:* Scott Z. <sudz28(a)hotmail.com>; FreeIPA users list
>> <freeipa-users(a)lists.fedorahosted.org>; Rob Crittenden <rcritten(a)redhat.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> Â
>> On 8/11/20 6:39 PM, Scott Z. wrote:
>>> First thing I did when I logged in this morning (I'm on Hawaii Standard
>>> Time) was run "ipactl status". The return was "Directory Services:
>>> STOPPED", and "Directory Service must running in order to obtain status
>>> of other services".
>>> 1) Ran "getcert list", and it shows the 9 certs being tracked (all the
>>> previous 8 plus the 1 expired guy I added yesterday). All look good
>>> except of course my problem child, who's status is CA_UNREACHABLE and
>>> ca-error is Internal error.
>>> 2) Ran "ipa stop", looks like all service stopped successfully.
>>> 2) Changed date back to Sept. 1, 2019.
>>> 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
>>> dirsrv@<domain> failed because a configured resource limit was exceeded."
>>> Â Â Â Â a. when I looked at "journalctl -xe", I just see a couple of
>>> messages that don't tell me much... "Registered Authentication Agent for
>>> unix-process:<blahblah>", followed by "Failed to load environment files:
>>> no such files or directory". Then, "dirsrv@<domain> filed to run
>>> 'start-pre' task: No such files or directory" and finally "Failed to
>>> start 389 Directory Server <domain>".
>>>
>> If your domain is domain.com, you need to run
>> systemctl start dirsrv@DOMAIN-COM
>>
>> I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
>> which would produce the error you're seeing.
>>
>> flo
>>
>>> Not sure now how to proceed at this point.
>>>
>>> BTW, I have decided that once I get through this slog and have a working
>>> server again, I'm going to donate $50 to the Hawaiian Food Bank or the
>>> charity of your choice in appreciation.
>>> Scott
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Monday, August 10, 2020 8:55 PM
>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Rob
>>> Crittenden <rcritten(a)redhat.com>
>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
>>>> I stopped the ntp service with the command "timedatectl set_ntp 0"
>>>> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
>>>> 2019-09-01"
>>>> I waiting a minute and then checked with the "date" command; the problem
>>>> server believes it is Sept. 1st, 2019.
>>>>
>>>> Now when you say 'restart services', I assume you're only referring to
>>>> the ipactl services? In that case I ran "ipactl start
>>>> --ignore-service-failures". Interestingly, when I ran this command it
>>>> not only failed to start pki-tomcatd (which I expected), but actually
>>>> reset the date back to the present/correct time and date. Thus, I
>>>> re-ran the command to set it back to Sept. 1st, 2019.
>>>>
>>> If the server was configured with ntp, "ipactl start" will also restart
>>> ntpd. You need to do the following:
>>> ipactl stop
>>> change date in the past
>>> systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
>>> systemctl start krb5kdc
>>> systemctl start kadmin
>>> systemctl start named-pkcs11 (if IPA is hosting the DNS server)
>>> systemctl start httpd
>>> systemctl start pki-tomcatd@pki-tomcat
>>>
>>> Then try getcert resubmit.
>>>
>>>> I then ran the "getcert resubmit -i <reqID> command. I just now went
>>>> through these steps again, and it's showing "status: CA_UNREACHABLE" and
>>>> "ca-error: Internal Error". Stuck now shows 'no'.
>>>> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
>>>> cert-pki-ca' now yields a new error message, "certutil: could not find
>>>> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
>>>> found"
>>> The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
>>>
>>> HTH,
>>> flo
>>>>
>>>> Many Mahalos for your continued support and patience!
>>>> Scott
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Rob Crittenden <rcritten(a)redhat.com>
>>>> *Sent:* Monday, August 10, 2020 11:36 AM
>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
>>>> Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> Scott Z. via FreeIPA-users wrote:
>>>>> Whoops! Using the additional command to start tracking this paritcular
>>>>> cert that you included in a different message, I got it in the "getcert"
>>>>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
>>>>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
>>>>> <pin>" command).
>>>>>
>>>>> I have the date rolled back to Sept. 1st, 2019. I guess I have 'some'
>>>>> progress now at least, but still have an issue; checking on the cert
>>>>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
>>>>> "stuck: yes".
>>>>
>>>> How did you roll the date back? Did you restart services? What date did
>>>> you pick and does it overlap so that all certs are valid?
>>>>
>>>> rob
>>>>
>>>>>
>>>>> Any additional thoughts or help would be greatly appreciated! And
>>>>> thanks for the help so far.
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
>>>>> *Sent:* Monday, August 10, 2020 10:37 AM
>>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>>>>> Z. <sudz28(a)hotmail.com>
>>>>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>>>>> Â
>>>>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>>>>
>>>>> I backed up the files/directories you mentioned below, then I checked on
>>>>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>>>>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>>>>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug
>>>>> 10 17:20:41 2021 GMT).
>>>>>
>>>>> Based on that information, and knowing that the bad cert is valid from
>>>>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>>>>> since all certs will see that date as valid.
>>>>>
>>>>> The only issue I have now is getting the request ID for the expired
>>>>> cert; it doesn't show up in the list of certs when I do "getcert -list",
>>>>> I can only see it by running "certutil -L -d
>>>>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
>>>>> I run that it does not show any Request ID associated for it?
>>>>> Scott
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Sent:* Monday, August 10, 2020 8:45 AM
>>>>> *To:* Scott Z. <sudz28(a)hotmail.com>
>>>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>> Â
>>>>> Hi,
>>>>>
>>>>> re-adding the mailing list as the conversation could also help others.
>>>>>
>>>>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>>>>> I did notice when I compare it to another IdM server in the environment,
>>>>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>>>>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>>>>> I'm comparing against has a "Signing-Cert" certificate in addition. Is
>>>>>> this because it's the 'Master' or whatever? Should my 'bad' server have
>>>>>> this same Signing-Cert listed?
>>>>>
>>>>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>>>>
>>>>>> Scott
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>>>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>>> /"The interesting part is the list of expired certs on the failing node
>>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>>> instructions are available here:
>>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>>> (Replica IPA Server)"/
>>>>>
>>>>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>>>>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>>>>> the certificates are stored).
>>>>>
>>>>> If the RA cert is valid, you need to find a time window during which the
>>>>> RA cert is already valid (date > notbefore) and the other certs are not
>>>>> expired yet (date < notafter). When you have identified a proper date,
>>>>> stop ntpd (or chronyd, depending on which service is used for time
>>>>> synchronization), move the date back in time to the identified date,
>>>>> start all the services except ntpd, then call "getcert resubmit -i
>>>>> <request id>" for the expired cert(s).
>>>>>
>>>>> Check that the cert has been renewed with "getcert list -i <request
>>>>> id>", the state should display MONITORING. When all the certs are good,
>>>>> you can restart ntpd and the clock will go back to the current date.
>>>>>
>>>>> It's really important to find a date where all the certs are valid
>>>>> because this ensures that the services are able to start and the RA cert
>>>>> allows the authentication that is mandatory for certificate renewal.
>>>>>
>>>>> HTH,
>>>>> flo
>>>>>>
>>>>>> Sadly, after I log in, it's only telling me that it's "Subscriber
>>>>>> Exclusive Content". Not sure what happened with my account, I used to
>>>>>> be able to access these docs with no problem but since I took a RHEL
>>>>>> class a couple of weeks back now it's not working any more. I guess
>>>>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>>>>> Scott
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>>>>> Thanks much for the assistance. Here is where I am with your suggestions:
>>>>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>>>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>>>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>>>>> starts up so it didn't care that it was expired until the server was
>>>>>>> rebooted?)
>>>>>>
>>>>>> certmonger checks the certificate validity periodically (configurable in
>>>>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>>>>> The system probably had an issue that was not detected and the cert
>>>>>> reached its expiration date.
>>>>>>
>>>>>>>
>>>>>>> 2) ran ipactl start --ignore-service-failures
>>>>>>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>>>>>>> 3) ran "kinit admin"
>>>>>>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>>>>>>> 4) Ran "ipa config-show |grep -i master
>>>>>>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>>>>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>>>>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>>>>>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>>>>>> paste the output here because it's on an airgaped environment so while I
>>>>>>> apologize for this and realize it makes things more difficult, perhaps
>>>>>>> if you tell me what I should be looking for or more specifically what
>>>>>>> you're interested in I can pluck that out and manually include it here?
>>>>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>>>>>> certificate on the problem server, and it can theoretically be renew by
>>>>>>> the Master at this time.
>>>>>> The interesting part is the list of expired certs on the failing node
>>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>>> instructions are available here:
>>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>>> (Replica IPA Server)
>>>>>>
>>>>>> flo
>>>>>>
>>>>>>> Many thanks!
>>>>>>> Scott
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>>>>> Not sure I'm sending this to the right place, but here it goes. I
>>>>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>>>>>> access) environment that is running into problems. There are at least 3
>>>>>>>> different IdM servers running in the environment spread out across
>>>>>>>> different geographical areas. One of those areas suffered an unschedule
>>>>>>>> power outage recently, and ever since we brought everything back up, the
>>>>>>>> IdM server for this region is having an issue. Please bear with me as I
>>>>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>>>>
>>>>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>>>>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>>>>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>>>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>>>>>> to start and killing all the other services. I check the log at
>>>>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>>>>>> import or screen capure the logs and put them in this message):
>>>>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>>>>>> And slightly further down in the same log:
>>>>>>>> "/Cannot reset factory: connections not all returned/"
>>>>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>>>>>> LDAP connection factory because some connections are still outstanding/"
>>>>>>>> ... still further down"
>>>>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>>>>>
>>>>>>>> Assuming I have some weird certificate issue with this server in
>>>>>>>> particular, I try to run a few more commands:
>>>>>>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>>>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>>>>>> for it's attributes. Comparing to a second IdM server in this
>>>>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>>>>
>>>>>>> Hi,
>>>>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>>>>>> one is not expired with:
>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>>>>>> | grep 'Not '
>>>>>>>
>>>>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>>>>> need first to find which IPA server is the CA renewal master. On your
>>>>>>> server, force a service start and check the CA renewal master:
>>>>>>> # ipactl start --ignore-service-failures
>>>>>>> # kinit admin
>>>>>>> # ipa config-show | grep "renewal master"
>>>>>>> Â Â IPA CA renewal master: server.domain.com
>>>>>>>
>>>>>>> You need to make sure that all the certificates are valid on the CA
>>>>>>> renewal master:
>>>>>>> (on the CA renewal master)# getcert list | grep -E
>>>>>>> "Request|certificate:|expires:"
>>>>>>>
>>>>>>> - if the CA renewal master is not OK, please post the output of "#
>>>>>>> getcert list" (without the grep) on the CA renewal master. This node
>>>>>>> will have to be repaired first.
>>>>>>> - if the CA renewal master is OK, please post the output of "# getcert
>>>>>>> list" (also without the grep) on the failing node.
>>>>>>>
>>>>>>> We'll be able to help based on this information.
>>>>>>> flo
>>>>>>>
>>>>>>>> I also did a "getcert list", and all certs it has show that they expire
>>>>>>>> in the future (nothing shows as bein currently expired).
>>>>>>>>
>>>>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>>>>>> but how do I track down which 'peer' the log file is talking about that
>>>>>>>> has an expired cert? Meanwhile none of the linux clients that point to
>>>>>>>> this IdM server are allowing people to log in/authenticate.
>>>>>>>> Many thanks for any help!
>>>>>>>> Scott
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>
3 years, 8 months
- 2
- 2
Clarification on CA Cert renewal requested
by Khurrum Maqb
I am running ipa-server 4.6.6 with the same version clients. This IPA server has been around since pre-v1 and has been upgraded till the current version that is shipped with Centos7.
The IPA CA Cert was/is set to expire on Aug 10, 2020.
On the server that is the IPA CA renewal master, I checked the output of `getcert list` and the CA with the certificate "subject: CN=Certificate Authority,O=DOMAIN.COM" and nickname nickname='caSigningCert cert-pki-ca' is shown as renewed till 2040. All other certs that appear in that list are updated without intervention to 2022. It's located at `location='/etc/pki/pki-tomcat/alias'`. So far so good;
BUT I noticed that /etc/ipa/ca.crt on the same server shows as still expiring on August 10:
# openssl x509 -inform pem -enddate -noout -in /etc/ipa/ca.crt
notAfter=Aug 10 21:29:31 2020 GMT
So that means that the caSigningCert cert-pki-ca is set to automatically renew for 20 years But the IPA CA Cert is not.
Next, I saw that there are certs located in /etc/pki/pki-tomcat/alias, /etc/ipa/nssdb/, /etc/httpd/alias/, and /etc/pki/nssdb/.
My questions:
* Is my self-signed IPA CA Cert supposed to be automatically renewed?
* Or is it required that I run `ipa-cacert-manage renew` on the IPA CA renewal master, and then `ipa-certupdate` on all the other server replicas and clients?
* Why do I appear to have duplicate DOMAIN IPA CA certs listed in /etc/ipa/nssdb/, /etc/httpd/alias/? Is one location deprecated?
Thank you for your help!
3 years, 8 months
- 4
- 10
Re: pki-tomcatd not starting
by Rob Crittenden
Scott Z. via FreeIPA-users wrote:
> Forgot to reply again - ugh!
> Hmmmm, so my domain is actually "idm.project.its.srv2", so I was
> literally typing "systemctl start dirsrv(a)idm.project.its.srv2"Â I see
> what you're saying, I need to put in dashes instead of periods! DOH!Â
> Done. Moving on...
> 4) Ran systemctl start krb5kdc
> 5) Ran systemctl start kadmin
> 6) Ran systemctl start named-pkcs11
> 7) Ran systemctl start httpd - got an error here, nothing really
> useful in the logs or journalctl, it says it's starting the Apache HTTP
> server, then throws "httpd.service: main process exited, code=exited,
> status=1/FAILURE", and "Failed to start The Apache HTTP Server".Â
> Finally there is a mention of 'too much time skew'. I assume the
> problem is that I'm trying to start HTTPD on a system where the date is
> almost a year old.Â
> Although now that I'm looking at /var/log/httpd/error_log, I see mention
> of "SSL Library Error: -8181 Certificate has expired". CERTIFICATES!!!
> "Unable to verify certificate 'Server-Cert'. Add "NSSEnfroceValideCerts
> off" to nss.conf so the server can start until the problem can be
> resolved", so maybe I'll try that.
That can work, just remember to revert it, but it just bypasses the
start up check. Clients will still require cert validity.
I don't think it will matter either way as the CA certs renew directly
against the CA so Apache not running shouldn't be an issue.
rob
> Scott
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Tuesday, August 11, 2020 6:55 AM
> *To:* Scott Z. <sudz28(a)hotmail.com>; FreeIPA users list
> <freeipa-users(a)lists.fedorahosted.org>; Rob Crittenden <rcritten(a)redhat.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> Â
> On 8/11/20 6:39 PM, Scott Z. wrote:
>> First thing I did when I logged in this morning (I'm on Hawaii Standard
>> Time) was run "ipactl status". The return was "Directory Services:
>> STOPPED", and "Directory Service must running in order to obtain status
>> of other services".
>> 1) Ran "getcert list", and it shows the 9 certs being tracked (all the
>> previous 8 plus the 1 expired guy I added yesterday). All look good
>> except of course my problem child, who's status is CA_UNREACHABLE and
>> ca-error is Internal error.
>> 2) Ran "ipa stop", looks like all service stopped successfully.
>> 2) Changed date back to Sept. 1, 2019.
>> 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
>> dirsrv@<domain> failed because a configured resource limit was exceeded."
>> Â Â Â Â a. when I looked at "journalctl -xe", I just see a couple of
>> messages that don't tell me much... "Registered Authentication Agent for
>> unix-process:<blahblah>", followed by "Failed to load environment files:
>> no such files or directory". Then, "dirsrv@<domain> filed to run
>> 'start-pre' task: No such files or directory" and finally "Failed to
>> start 389 Directory Server <domain>".
>>
> If your domain is domain.com, you need to run
> systemctl start dirsrv@DOMAIN-COM
>
> I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
> which would produce the error you're seeing.
>
> flo
>
>> Not sure now how to proceed at this point.
>>
>> BTW, I have decided that once I get through this slog and have a working
>> server again, I'm going to donate $50 to the Hawaiian Food Bank or the
>> charity of your choice in appreciation.
>> Scott
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Monday, August 10, 2020 8:55 PM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Rob
>> Crittenden <rcritten(a)redhat.com>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
>>> I stopped the ntp service with the command "timedatectl set_ntp 0"
>>> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
>>> 2019-09-01"
>>> I waiting a minute and then checked with the "date" command; the problem
>>> server believes it is Sept. 1st, 2019.
>>>
>>> Now when you say 'restart services', I assume you're only referring to
>>> the ipactl services? In that case I ran "ipactl start
>>> --ignore-service-failures". Interestingly, when I ran this command it
>>> not only failed to start pki-tomcatd (which I expected), but actually
>>> reset the date back to the present/correct time and date. Thus, I
>>> re-ran the command to set it back to Sept. 1st, 2019.
>>>
>> If the server was configured with ntp, "ipactl start" will also restart
>> ntpd. You need to do the following:
>> ipactl stop
>> change date in the past
>> systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
>> systemctl start krb5kdc
>> systemctl start kadmin
>> systemctl start named-pkcs11 (if IPA is hosting the DNS server)
>> systemctl start httpd
>> systemctl start pki-tomcatd@pki-tomcat
>>
>> Then try getcert resubmit.
>>
>>> I then ran the "getcert resubmit -i <reqID> command. I just now went
>>> through these steps again, and it's showing "status: CA_UNREACHABLE" and
>>> "ca-error: Internal Error". Stuck now shows 'no'.
>>> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
>>> cert-pki-ca' now yields a new error message, "certutil: could not find
>>> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
>>> found"
>> The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
>>
>> HTH,
>> flo
>>>
>>> Many Mahalos for your continued support and patience!
>>> Scott
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Rob Crittenden <rcritten(a)redhat.com>
>>> *Sent:* Monday, August 10, 2020 11:36 AM
>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
>>> Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> Scott Z. via FreeIPA-users wrote:
>>>> Whoops! Using the additional command to start tracking this paritcular
>>>> cert that you included in a different message, I got it in the "getcert"
>>>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
>>>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
>>>> <pin>" command).
>>>>
>>>> I have the date rolled back to Sept. 1st, 2019. I guess I have 'some'
>>>> progress now at least, but still have an issue; checking on the cert
>>>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
>>>> "stuck: yes".
>>>
>>> How did you roll the date back? Did you restart services? What date did
>>> you pick and does it overlap so that all certs are valid?
>>>
>>> rob
>>>
>>>>
>>>> Any additional thoughts or help would be greatly appreciated! And
>>>> thanks for the help so far.
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
>>>> *Sent:* Monday, August 10, 2020 10:37 AM
>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>>>> Z. <sudz28(a)hotmail.com>
>>>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>>>> Â
>>>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>>>
>>>> I backed up the files/directories you mentioned below, then I checked on
>>>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>>>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>>>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug
>>>> 10 17:20:41 2021 GMT).
>>>>
>>>> Based on that information, and knowing that the bad cert is valid from
>>>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>>>> since all certs will see that date as valid.
>>>>
>>>> The only issue I have now is getting the request ID for the expired
>>>> cert; it doesn't show up in the list of certs when I do "getcert -list",
>>>> I can only see it by running "certutil -L -d
>>>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
>>>> I run that it does not show any Request ID associated for it?
>>>> Scott
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Sent:* Monday, August 10, 2020 8:45 AM
>>>> *To:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> Â
>>>> Hi,
>>>>
>>>> re-adding the mailing list as the conversation could also help others.
>>>>
>>>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>>>> I did notice when I compare it to another IdM server in the environment,
>>>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>>>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>>>> I'm comparing against has a "Signing-Cert" certificate in addition. Is
>>>>> this because it's the 'Master' or whatever? Should my 'bad' server have
>>>>> this same Signing-Cert listed?
>>>>
>>>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>>>
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>> /"The interesting part is the list of expired certs on the failing node
>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>> instructions are available here:
>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>> (Replica IPA Server)"/
>>>>
>>>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>>>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>>>> the certificates are stored).
>>>>
>>>> If the RA cert is valid, you need to find a time window during which the
>>>> RA cert is already valid (date > notbefore) and the other certs are not
>>>> expired yet (date < notafter). When you have identified a proper date,
>>>> stop ntpd (or chronyd, depending on which service is used for time
>>>> synchronization), move the date back in time to the identified date,
>>>> start all the services except ntpd, then call "getcert resubmit -i
>>>> <request id>" for the expired cert(s).
>>>>
>>>> Check that the cert has been renewed with "getcert list -i <request
>>>> id>", the state should display MONITORING. When all the certs are good,
>>>> you can restart ntpd and the clock will go back to the current date.
>>>>
>>>> It's really important to find a date where all the certs are valid
>>>> because this ensures that the services are able to start and the RA cert
>>>> allows the authentication that is mandatory for certificate renewal.
>>>>
>>>> HTH,
>>>> flo
>>>>>
>>>>> Sadly, after I log in, it's only telling me that it's "Subscriber
>>>>> Exclusive Content". Not sure what happened with my account, I used to
>>>>> be able to access these docs with no problem but since I took a RHEL
>>>>> class a couple of weeks back now it's not working any more. I guess
>>>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>>>> Thanks much for the assistance. Here is where I am with your suggestions:
>>>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>>>> starts up so it didn't care that it was expired until the server was
>>>>>> rebooted?)
>>>>>
>>>>> certmonger checks the certificate validity periodically (configurable in
>>>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>>>> The system probably had an issue that was not detected and the cert
>>>>> reached its expiration date.
>>>>>
>>>>>>
>>>>>> 2) ran ipactl start --ignore-service-failures
>>>>>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>>>>>> 3) ran "kinit admin"
>>>>>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>>>>>> 4) Ran "ipa config-show |grep -i master
>>>>>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>>>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>>>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>>>>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>>>>> paste the output here because it's on an airgaped environment so while I
>>>>>> apologize for this and realize it makes things more difficult, perhaps
>>>>>> if you tell me what I should be looking for or more specifically what
>>>>>> you're interested in I can pluck that out and manually include it here?
>>>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>>>>> certificate on the problem server, and it can theoretically be renew by
>>>>>> the Master at this time.
>>>>> The interesting part is the list of expired certs on the failing node
>>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>>> instructions are available here:
>>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>>> (Replica IPA Server)
>>>>>
>>>>> flo
>>>>>
>>>>>> Many thanks!
>>>>>> Scott
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>>>> Not sure I'm sending this to the right place, but here it goes. I
>>>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>>>>> access) environment that is running into problems. There are at least 3
>>>>>>> different IdM servers running in the environment spread out across
>>>>>>> different geographical areas. One of those areas suffered an unschedule
>>>>>>> power outage recently, and ever since we brought everything back up, the
>>>>>>> IdM server for this region is having an issue. Please bear with me as I
>>>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>>>
>>>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>>>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>>>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>>>>> to start and killing all the other services. I check the log at
>>>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>>>>> import or screen capure the logs and put them in this message):
>>>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>>>>> And slightly further down in the same log:
>>>>>>> "/Cannot reset factory: connections not all returned/"
>>>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>>>>> LDAP connection factory because some connections are still outstanding/"
>>>>>>> ... still further down"
>>>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>>>>
>>>>>>> Assuming I have some weird certificate issue with this server in
>>>>>>> particular, I try to run a few more commands:
>>>>>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>>>>> for it's attributes. Comparing to a second IdM server in this
>>>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>>>
>>>>>> Hi,
>>>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>>>>> one is not expired with:
>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>>>>> | grep 'Not '
>>>>>>
>>>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>>>> need first to find which IPA server is the CA renewal master. On your
>>>>>> server, force a service start and check the CA renewal master:
>>>>>> # ipactl start --ignore-service-failures
>>>>>> # kinit admin
>>>>>> # ipa config-show | grep "renewal master"
>>>>>> Â Â IPA CA renewal master: server.domain.com
>>>>>>
>>>>>> You need to make sure that all the certificates are valid on the CA
>>>>>> renewal master:
>>>>>> (on the CA renewal master)# getcert list | grep -E
>>>>>> "Request|certificate:|expires:"
>>>>>>
>>>>>> - if the CA renewal master is not OK, please post the output of "#
>>>>>> getcert list" (without the grep) on the CA renewal master. This node
>>>>>> will have to be repaired first.
>>>>>> - if the CA renewal master is OK, please post the output of "# getcert
>>>>>> list" (also without the grep) on the failing node.
>>>>>>
>>>>>> We'll be able to help based on this information.
>>>>>> flo
>>>>>>
>>>>>>> I also did a "getcert list", and all certs it has show that they expire
>>>>>>> in the future (nothing shows as bein currently expired).
>>>>>>>
>>>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>>>>> but how do I track down which 'peer' the log file is talking about that
>>>>>>> has an expired cert? Meanwhile none of the linux clients that point to
>>>>>>> this IdM server are allowing people to log in/authenticate.
>>>>>>> Many thanks for any help!
>>>>>>> Scott
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>
3 years, 8 months
- 2
- 1
Re: pki-tomcatd not starting
by Florence Blanc-Renaud
On 8/11/20 6:39 PM, Scott Z. wrote:
> First thing I did when I logged in this morning (I'm on Hawaii Standard
> Time) was run "ipactl status". The return was "Directory Services:
> STOPPED", and "Directory Service must running in order to obtain status
> of other services".
> 1) Ran "getcert list", and it shows the 9 certs being tracked (all the
> previous 8 plus the 1 expired guy I added yesterday). All look good
> except of course my problem child, who's status is CA_UNREACHABLE and
> ca-error is Internal error.
> 2) Ran "ipa stop", looks like all service stopped successfully.
> 2) Changed date back to Sept. 1, 2019.
> 3) Ran the "systemctl start dirsrv@<domain> and got back "Job for
> dirsrv@<domain> failed because a configured resource limit was exceeded."
> Â Â Â Â a. when I looked at "journalctl -xe", I just see a couple of
> messages that don't tell me much... "Registered Authentication Agent for
> unix-process:<blahblah>", followed by "Failed to load environment files:
> no such files or directory". Then, "dirsrv@<domain> filed to run
> 'start-pre' task: No such files or directory" and finally "Failed to
> start 389 Directory Server <domain>".
>
If your domain is domain.com, you need to run
systemctl start dirsrv@DOMAIN-COM
I suspect that you ran instead systemctl start dirsrv@slapd-DOMAIN-COM
which would produce the error you're seeing.
flo
> Not sure now how to proceed at this point.
>
> BTW, I have decided that once I get through this slog and have a working
> server again, I'm going to donate $50 to the Hawaiian Food Bank or the
> charity of your choice in appreciation.
> Scott
>
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Monday, August 10, 2020 8:55 PM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Rob
> Crittenden <rcritten(a)redhat.com>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
>> I stopped the ntp service with the command "timedatectl set_ntp 0"
>> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
>> 2019-09-01"
>> I waiting a minute and then checked with the "date" command; the problem
>> server believes it is Sept. 1st, 2019.
>>
>> Now when you say 'restart services', I assume you're only referring to
>> the ipactl services? In that case I ran "ipactl start
>> --ignore-service-failures". Interestingly, when I ran this command it
>> not only failed to start pki-tomcatd (which I expected), but actually
>> reset the date back to the present/correct time and date. Thus, I
>> re-ran the command to set it back to Sept. 1st, 2019.
>>
> If the server was configured with ntp, "ipactl start" will also restart
> ntpd. You need to do the following:
> ipactl stop
> change date in the past
> systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
> systemctl start krb5kdc
> systemctl start kadmin
> systemctl start named-pkcs11 (if IPA is hosting the DNS server)
> systemctl start httpd
> systemctl start pki-tomcatd@pki-tomcat
>
> Then try getcert resubmit.
>
>> I then ran the "getcert resubmit -i <reqID> command. I just now went
>> through these steps again, and it's showing "status: CA_UNREACHABLE" and
>> "ca-error: Internal Error". Stuck now shows 'no'.
>> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
>> cert-pki-ca' now yields a new error message, "certutil: could not find
>> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
>> found"
> The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
>
> HTH,
> flo
>>
>> Many Mahalos for your continued support and patience!
>> Scott
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Rob Crittenden <rcritten(a)redhat.com>
>> *Sent:* Monday, August 10, 2020 11:36 AM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
>> Florence Blanc-Renaud <flo(a)redhat.com>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> Scott Z. via FreeIPA-users wrote:
>>> Whoops! Using the additional command to start tracking this paritcular
>>> cert that you included in a different message, I got it in the "getcert"
>>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
>>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
>>> <pin>" command).
>>>
>>> I have the date rolled back to Sept. 1st, 2019. I guess I have 'some'
>>> progress now at least, but still have an issue; checking on the cert
>>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
>>> "stuck: yes".
>>
>> How did you roll the date back? Did you restart services? What date did
>> you pick and does it overlap so that all certs are valid?
>>
>> rob
>>
>>>
>>> Any additional thoughts or help would be greatly appreciated! And
>>> thanks for the help so far.
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
>>> *Sent:* Monday, August 10, 2020 10:37 AM
>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>>> Z. <sudz28(a)hotmail.com>
>>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>>> Â
>>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>>
>>> I backed up the files/directories you mentioned below, then I checked on
>>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug
>>> 10 17:20:41 2021 GMT).
>>>
>>> Based on that information, and knowing that the bad cert is valid from
>>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>>> since all certs will see that date as valid.
>>>
>>> The only issue I have now is getting the request ID for the expired
>>> cert; it doesn't show up in the list of certs when I do "getcert -list",
>>> I can only see it by running "certutil -L -d
>>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
>>> I run that it does not show any Request ID associated for it?
>>> Scott
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Monday, August 10, 2020 8:45 AM
>>> *To:* Scott Z. <sudz28(a)hotmail.com>
>>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> Â
>>> Hi,
>>>
>>> re-adding the mailing list as the conversation could also help others.
>>>
>>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>>> I did notice when I compare it to another IdM server in the environment,
>>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>>> I'm comparing against has a "Signing-Cert" certificate in addition. Is
>>>> this because it's the 'Master' or whatever? Should my 'bad' server have
>>>> this same Signing-Cert listed?
>>>
>>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>>
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> /"The interesting part is the list of expired certs on the failing node
>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>> instructions are available here:
>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>> (Replica IPA Server)"/
>>>
>>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>>> the certificates are stored).
>>>
>>> If the RA cert is valid, you need to find a time window during which the
>>> RA cert is already valid (date > notbefore) and the other certs are not
>>> expired yet (date < notafter). When you have identified a proper date,
>>> stop ntpd (or chronyd, depending on which service is used for time
>>> synchronization), move the date back in time to the identified date,
>>> start all the services except ntpd, then call "getcert resubmit -i
>>> <request id>" for the expired cert(s).
>>>
>>> Check that the cert has been renewed with "getcert list -i <request
>>> id>", the state should display MONITORING. When all the certs are good,
>>> you can restart ntpd and the clock will go back to the current date.
>>>
>>> It's really important to find a date where all the certs are valid
>>> because this ensures that the services are able to start and the RA cert
>>> allows the authentication that is mandatory for certificate renewal.
>>>
>>> HTH,
>>> flo
>>>>
>>>> Sadly, after I log in, it's only telling me that it's "Subscriber
>>>> Exclusive Content". Not sure what happened with my account, I used to
>>>> be able to access these docs with no problem but since I took a RHEL
>>>> class a couple of weeks back now it's not working any more. I guess
>>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>>> Thanks much for the assistance. Here is where I am with your suggestions:
>>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>>> starts up so it didn't care that it was expired until the server was
>>>>> rebooted?)
>>>>
>>>> certmonger checks the certificate validity periodically (configurable in
>>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>>> The system probably had an issue that was not detected and the cert
>>>> reached its expiration date.
>>>>
>>>>>
>>>>> 2) ran ipactl start --ignore-service-failures
>>>>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>>>>> 3) ran "kinit admin"
>>>>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>>>>> 4) Ran "ipa config-show |grep -i master
>>>>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>>>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>>>> paste the output here because it's on an airgaped environment so while I
>>>>> apologize for this and realize it makes things more difficult, perhaps
>>>>> if you tell me what I should be looking for or more specifically what
>>>>> you're interested in I can pluck that out and manually include it here?
>>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>>>> certificate on the problem server, and it can theoretically be renew by
>>>>> the Master at this time.
>>>> The interesting part is the list of expired certs on the failing node
>>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>>> instructions are available here:
>>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>>> (Replica IPA Server)
>>>>
>>>> flo
>>>>
>>>>> Many thanks!
>>>>> Scott
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>>> Not sure I'm sending this to the right place, but here it goes. I
>>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>>>> access) environment that is running into problems. There are at least 3
>>>>>> different IdM servers running in the environment spread out across
>>>>>> different geographical areas. One of those areas suffered an unschedule
>>>>>> power outage recently, and ever since we brought everything back up, the
>>>>>> IdM server for this region is having an issue. Please bear with me as I
>>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>>
>>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>>>> to start and killing all the other services. I check the log at
>>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>>>> import or screen capure the logs and put them in this message):
>>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>>>> And slightly further down in the same log:
>>>>>> "/Cannot reset factory: connections not all returned/"
>>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>>>> LDAP connection factory because some connections are still outstanding/"
>>>>>> ... still further down"
>>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>>>
>>>>>> Assuming I have some weird certificate issue with this server in
>>>>>> particular, I try to run a few more commands:
>>>>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>>>> for it's attributes. Comparing to a second IdM server in this
>>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>>
>>>>> Hi,
>>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>>>> one is not expired with:
>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>>>> | grep 'Not '
>>>>>
>>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>>> need first to find which IPA server is the CA renewal master. On your
>>>>> server, force a service start and check the CA renewal master:
>>>>> # ipactl start --ignore-service-failures
>>>>> # kinit admin
>>>>> # ipa config-show | grep "renewal master"
>>>>> Â Â IPA CA renewal master: server.domain.com
>>>>>
>>>>> You need to make sure that all the certificates are valid on the CA
>>>>> renewal master:
>>>>> (on the CA renewal master)# getcert list | grep -E
>>>>> "Request|certificate:|expires:"
>>>>>
>>>>> - if the CA renewal master is not OK, please post the output of "#
>>>>> getcert list" (without the grep) on the CA renewal master. This node
>>>>> will have to be repaired first.
>>>>> - if the CA renewal master is OK, please post the output of "# getcert
>>>>> list" (also without the grep) on the failing node.
>>>>>
>>>>> We'll be able to help based on this information.
>>>>> flo
>>>>>
>>>>>> I also did a "getcert list", and all certs it has show that they expire
>>>>>> in the future (nothing shows as bein currently expired).
>>>>>>
>>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>>>> but how do I track down which 'peer' the log file is talking about that
>>>>>> has an expired cert? Meanwhile none of the linux clients that point to
>>>>>> this IdM server are allowing people to log in/authenticate.
>>>>>> Many thanks for any help!
>>>>>> Scott
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
3 years, 8 months
- 2
- 1
Re: pki-tomcatd not starting
by Florence Blanc-Renaud
On 8/10/20 11:46 PM, Scott Z. via FreeIPA-users wrote:
> I stopped the ntp service with the command "timedatectl set_ntp 0"
> I set the new date to be Sept. 1st, 2019 with "timedatectl set-time
> 2019-09-01"
> I waiting a minute and then checked with the "date" command; the problem
> server believes it is Sept. 1st, 2019.
>
> Now when you say 'restart services', I assume you're only referring to
> the ipactl services? In that case I ran "ipactl start
> --ignore-service-failures". Interestingly, when I ran this command it
> not only failed to start pki-tomcatd (which I expected), but actually
> reset the date back to the present/correct time and date. Thus, I
> re-ran the command to set it back to Sept. 1st, 2019.
>
If the server was configured with ntp, "ipactl start" will also restart
ntpd. You need to do the following:
ipactl stop
change date in the past
systemctl start dirsrv@DOMAIN-COM (replace with your domain name)
systemctl start krb5kdc
systemctl start kadmin
systemctl start named-pkcs11 (if IPA is hosting the DNS server)
systemctl start httpd
systemctl start pki-tomcatd@pki-tomcat
Then try getcert resubmit.
> I then ran the "getcert resubmit -i <reqID> command. I just now went
> through these steps again, and it's showing "status: CA_UNREACHABLE" and
> "ca-error: Internal Error". Stuck now shows 'no'.
> Re-running "certutil -L -d /etc/pki/pki-tomcat/alias -n 'ServerCert
> cert-pki-ca' now yields a new error message, "certutil: could not find
> cert: ServerCert cert-pki-ca", and ": PR_FILE_NOT_FOUND_ERROR: File not
> found"
The cert nickname should contain a dash: "Server-Cert cert-pki-ca"
HTH,
flo
>
> Many Mahalos for your continued support and patience!
> Scott
>
>
>
>
> ------------------------------------------------------------------------
> *From:* Rob Crittenden <rcritten(a)redhat.com>
> *Sent:* Monday, August 10, 2020 11:36 AM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>;
> Florence Blanc-Renaud <flo(a)redhat.com>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> Scott Z. via FreeIPA-users wrote:
>> Whoops! Using the additional command to start tracking this paritcular
>> cert that you included in a different message, I got it in the "getcert"
>> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
>> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
>> /usr/libexec/ipa/certmonger/stop_pkicad -C
>> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
>> <pin>" command).
>>
>> I have the date rolled back to Sept. 1st, 2019. I guess I have 'some'
>> progress now at least, but still have an issue; checking on the cert
>> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
>> "stuck: yes".
>
> How did you roll the date back? Did you restart services? What date did
> you pick and does it overlap so that all certs are valid?
>
> rob
>
>>
>> Any additional thoughts or help would be greatly appreciated! And
>> thanks for the help so far.
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
>> *Sent:* Monday, August 10, 2020 10:37 AM
>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
>> Z. <sudz28(a)hotmail.com>
>> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
>> Â
>> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>>
>> I backed up the files/directories you mentioned below, then I checked on
>> the ra-agent.pem to see if it was still valid (openssl x509 -in
>> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
>> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug
>> 10 17:20:41 2021 GMT).
>>
>> Based on that information, and knowing that the bad cert is valid from
>> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
>> since all certs will see that date as valid.
>>
>> The only issue I have now is getting the request ID for the expired
>> cert; it doesn't show up in the list of certs when I do "getcert -list",
>> I can only see it by running "certutil -L -d
>> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
>> I run that it does not show any Request ID associated for it?
>> Scott
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Monday, August 10, 2020 8:45 AM
>> *To:* Scott Z. <sudz28(a)hotmail.com>
>> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> Â
>> Hi,
>>
>> re-adding the mailing list as the conversation could also help others.
>>
>> On 8/8/20 12:06 AM, Scott Z. wrote:
>>> I did notice when I compare it to another IdM server in the environment,
>>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>>> I'm comparing against has a "Signing-Cert" certificate in addition. Is
>>> this because it's the 'Master' or whatever? Should my 'bad' server have
>>> this same Signing-Cert listed?
>>
>> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>>
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Scott Z. <sudz28(a)hotmail.com>
>>> *Sent:* Friday, August 7, 2020 10:44 AM
>>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> /"The interesting part is the list of expired certs on the failing node
>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>> instructions are available here:
>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>> (Replica IPA Server)"/
>>
>> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
>> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
>> the certificates are stored).
>>
>> If the RA cert is valid, you need to find a time window during which the
>> RA cert is already valid (date > notbefore) and the other certs are not
>> expired yet (date < notafter). When you have identified a proper date,
>> stop ntpd (or chronyd, depending on which service is used for time
>> synchronization), move the date back in time to the identified date,
>> start all the services except ntpd, then call "getcert resubmit -i
>> <request id>" for the expired cert(s).
>>
>> Check that the cert has been renewed with "getcert list -i <request
>> id>", the state should display MONITORING. When all the certs are good,
>> you can restart ntpd and the clock will go back to the current date.
>>
>> It's really important to find a date where all the certs are valid
>> because this ensures that the services are able to start and the RA cert
>> allows the authentication that is mandatory for certificate renewal.
>>
>> HTH,
>> flo
>>>
>>> Sadly, after I log in, it's only telling me that it's "Subscriber
>>> Exclusive Content". Not sure what happened with my account, I used to
>>> be able to access these docs with no problem but since I took a RHEL
>>> class a couple of weeks back now it's not working any more. I guess
>>> they did something to screw up my account when I took the class. Grrrrr!!!
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Thursday, August 6, 2020 2:46 AM
>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>>> Thanks much for the assistance. Here is where I am with your suggestions:
>>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>>> (almost a year old actually, I assume IPA only checks it when it first
>>>> starts up so it didn't care that it was expired until the server was
>>>> rebooted?)
>>>
>>> certmonger checks the certificate validity periodically (configurable in
>>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>>> The system probably had an issue that was not detected and the cert
>>> reached its expiration date.
>>>
>>>>
>>>> 2) ran ipactl start --ignore-service-failures
>>>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>>>> 3) ran "kinit admin"
>>>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>>>> 4) Ran "ipa config-show |grep -i master
>>>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>>> paste the output here because it's on an airgaped environment so while I
>>>> apologize for this and realize it makes things more difficult, perhaps
>>>> if you tell me what I should be looking for or more specifically what
>>>> you're interested in I can pluck that out and manually include it here?
>>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>>> certificate on the problem server, and it can theoretically be renew by
>>>> the Master at this time.
>>> The interesting part is the list of expired certs on the failing node
>>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>>> instructions are available here:
>>> https://access.redhat.com/solutions/3357331 How do I manually renew
>>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>>> (Replica IPA Server)
>>>
>>> flo
>>>
>>>> Many thanks!
>>>> Scott
>>>>
>>>> ------------------------------------------------------------------------
>>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>>> Not sure I'm sending this to the right place, but here it goes. I
>>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>>> access) environment that is running into problems. There are at least 3
>>>>> different IdM servers running in the environment spread out across
>>>>> different geographical areas. One of those areas suffered an unschedule
>>>>> power outage recently, and ever since we brought everything back up, the
>>>>> IdM server for this region is having an issue. Please bear with me as I
>>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>>
>>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>>> to start and killing all the other services. I check the log at
>>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>>> import or screen capure the logs and put them in this message):
>>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>>> And slightly further down in the same log:
>>>>> "/Cannot reset factory: connections not all returned/"
>>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>>> LDAP connection factory because some connections are still outstanding/"
>>>>> ... still further down"
>>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>>
>>>>> Assuming I have some weird certificate issue with this server in
>>>>> particular, I try to run a few more commands:
>>>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>>> for it's attributes. Comparing to a second IdM server in this
>>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>>
>>>> Hi,
>>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>>> one is not expired with:
>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>>> | grep 'Not '
>>>>
>>>> If the certificate is indeed expired, it will have to be renewed but you
>>>> need first to find which IPA server is the CA renewal master. On your
>>>> server, force a service start and check the CA renewal master:
>>>> # ipactl start --ignore-service-failures
>>>> # kinit admin
>>>> # ipa config-show | grep "renewal master"
>>>> Â Â IPA CA renewal master: server.domain.com
>>>>
>>>> You need to make sure that all the certificates are valid on the CA
>>>> renewal master:
>>>> (on the CA renewal master)# getcert list | grep -E
>>>> "Request|certificate:|expires:"
>>>>
>>>> - if the CA renewal master is not OK, please post the output of "#
>>>> getcert list" (without the grep) on the CA renewal master. This node
>>>> will have to be repaired first.
>>>> - if the CA renewal master is OK, please post the output of "# getcert
>>>> list" (also without the grep) on the failing node.
>>>>
>>>> We'll be able to help based on this information.
>>>> flo
>>>>
>>>>> I also did a "getcert list", and all certs it has show that they expire
>>>>> in the future (nothing shows as bein currently expired).
>>>>>
>>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>>> but how do I track down which 'peer' the log file is talking about that
>>>>> has an expired cert? Meanwhile none of the linux clients that point to
>>>>> this IdM server are allowing people to log in/authenticate.
>>>>> Many thanks for any help!
>>>>> Scott
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>
3 years, 8 months
- 2
- 1
Cannot add user to multimaster when one master down
by louisbohm＠gmail.com
Centos 8.1.1911
IPA Server 4.8.0-13
I created 2 IPA Masters (or I think I did). Called: ipa0.bos1.domain.com and ipa02.bos1.domain.com The created a test client server called: testc8. I setup the client using the command: ipa-client-install --hostname=testc8.bos1.domain.com --mkhomedir --realm=BOS1.DOMAIN.COM --ntp-server=0.ntp.bos1.domain.com --ntp-server=1.ntp.bos1.domain.com
Then to test I shutdown ipa01 (the first master I created).
Logged in to the client with a user I had created before. That failed. I had to update sssd with to include both masters in the ipa_server line under [domain/bos1.domain.com]. Once i did this and restarted sssd I could login with ipa01 down.
Then I tried to create a user from the cli of the client while ipa01 was down. That failed with the following error:
ipa: ERROR: cannot connect to 'https://ipa01.bos1.domain.com/ipa/json': [Errno 113] No route to host
Why the error? Why is it not just adding to ipa02??? Is there a fix?
Thanks.
3 years, 8 months
- 1
- 0
Replication issue with CSN generator
by Morgan Marodin
Hi.
Into my environment I have two IPA server, replicating each other.
They are both 7.6 OS systems, ipa-server RPM version is
4.6.4-10.0.1.el7_6.2.x86_64.
The first server installed was srv01 (many years ago), then I installed the
replica into srv02 (like a year later the 1st node).
When I had a single server I did also a trust with my corporate Active
Directory.
VMs are running in 2 different hypervisor clusters.
Now the replication doesn't works. Into log files I have this error:
*[16/Apr/2020:12:25:36.856632697 +0200] - ERR - csngen_adjust_time -
Adjustment limit exceeded; value - 23221226, limit -
86400[16/Apr/2020:12:25:36.857909222 +0200] - ERR - NSMMReplicationPlugin -
repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com
<http://meTosrv01.ipa.mydomain.com>" (srv01:389): Fatal error - too much
time skew between replicas![16/Apr/2020:12:25:36.862233147 +0200] - ERR -
NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meTosrv01.ipa.mydomain.com
<http://meTosrv01.ipa.mydomain.com>" (srv01:389): Incremental update failed
and requires administrator action*
I tried to force the replica, but the limit exceeded problem doesn't allow
the sync.
I know that the problem is that CSN generator has become grossly skewed.
Using the external script readNsState.py I found that there was as offset
time for about a month, so ... I waited for a month and then the issue
disappeared.
But now the offset is about 9 months ... I can't wait so much time :)
*[root@srv01 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is
BAAAAAAAAACCN/xfAAAAAHbiBAAAAAAABCgAAAAAAAANdQAAAAAAAA==Little EndianFor
replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN
generator state: Replica ID : 4 Sampled Time : 1610364802 Gen
as csn : 5ffc37822996500040000 Time as str : Mon Jan 11 12:33:22
2021 Local Offset : 320118 Remote Offset : 10244 Seq. num :
29965 System time : Tue Apr 21 15:03:45 2020 Diff in sec. :
-22890577 Day:sec diff : -265:5423nsState is
YAAAAAAAAAADLZheAAAAAAAAAAAAAAAAXSgAAAAAAAATAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state:
Replica ID : 96 Sampled Time : 1587031299 Gen as csn :
5e982d03001900960000 Time as str : Thu Apr 16 12:01:39 2020 Local
Offset : 0 Remote Offset : 10333 Seq. num : 19 System time
: Tue Apr 21 15:03:45 2020 Diff in sec. : 442926 Day:sec diff :
5:10926[root@srv02 scripts]# ./readNsState.py
/etc/dirsrv/slapd-IPA-MYDOMAIN-COM/dse.ldifnsState is
AwAAAAAAAABU7p5eAAAAAAAAAAAAAAAAsVNiAQAAAAAAAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=dc\3Dipa\2Cdc\3Dmydomain\2Cdc\3Dcom,cn=mapping
tree,cn=con fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN
generator state: Replica ID : 3 Sampled Time : 1587474004 Gen
as csn : 5e9eee54000000030000 Time as str : Tue Apr 21 15:00:04
2020 Local Offset : 0 Remote Offset : 23221169 Seq. num : 0
System time : Tue Apr 21 15:02:38 2020 Diff in sec. : 154
Day:sec diff : 0:154nsState is
YQAAAAAAAAAuLZheAAAAAEUBAAAAAAAA7SYAAAAAAAASAAAAAAAAAA==Little EndianFor
replica cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
fmtstr=[H6x3QH6x] size=40 len of nsstate is 40 CSN generator state:
Replica ID : 97 Sampled Time : 1587031342 Gen as csn :
5e982d2e001800970000 Time as str : Thu Apr 16 12:02:22 2020 Local
Offset : 325 Remote Offset : 9965 Seq. num : 18 System time
: Tue Apr 21 15:02:38 2020 Diff in sec. : 442816 Day:sec diff :
5:10816*
As you can see in the 1st node the Time as str is Jan 11 of 2021.
With timedatectl command I see that both VMs use the same Time zone and the
clock is correct.
I found this old article to fix my issue:
*https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html
<https://www.redhat.com/archives/freeipa-users/2014-February/msg00007.html>*
But ... I had the same issue in the past, always in the 1st server. So, in
my mind I don't want to try to use that fix.
I have a new hypervisor cluster, so I would prefer to reinstall the 1st
server, using these steps:
1) check if all roles (also the CA) is installed in srv02
You can find here some data about the VMs:
*[root@srv01 ~]# ipa server-show srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com> Server name: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com> Managed suffixes: domain, ca Min domain
level: 0 Max domain level: 1 Enabled server roles: CA server, IPA master,
DNS server, NTP server, AD trust controller[root@srv02 ~]# ipa server-show
srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Server name:
srv02.ipa.mydomain.com <http://srv02.ipa.mydomain.com> Managed suffixes:
domain, ca Min domain level: 0 Max domain level: 1 Enabled server roles:
CA server, IPA master, DNS server, NTP server[root@srv01 ~]# ipa
config-show Maximum username length: 32 Home directory base: /home
Default shell: /bin/bash Default users group: ipausers Default e-mail
domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2
Search size limit: 100 User search fields:
uid,givenname,sn,telephonenumber,ou,title Group search fields:
cn,description Enable migration mode: FALSE Certificate Subject base:
O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration
Notification (days): 4 Password plugin features: AllowNThash SELinux user
map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types:
MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA renewal master:
srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>[root@srv02 ~]# ipa
config-show Maximum username length: 32 Home directory base: /home
Default shell: /bin/bash Default users group: ipausers Default e-mail
domain: ipa.mydomain.com <http://ipa.mydomain.com> Search time limit: 2
Search size limit: 100 User search fields:
uid,givenname,sn,telephonenumber,ou,title Group search fields:
cn,description Enable migration mode: FALSE Certificate Subject base:
O=IPA.MYDOMAIN.COM <http://IPA.MYDOMAIN.COM> Password Expiration
Notification (days): 4 Password plugin features: AllowNThash SELinux user
map order:
guest_u:s0$xguest_u:s0$user_u:s0$staff_u:s0-s0:c0.c1023$unconfined_u:s0-s0:c0.c1023
Default SELinux user: unconfined_u:s0-s0:c0.c1023 Default PAC types:
MS-PAC, nfs:NONE IPA masters: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA NTP servers: srv01.ipa.mydomain.com
<http://srv01.ipa.mydomain.com>, srv02.ipa.mydomain.com
<http://srv02.ipa.mydomain.com> IPA CA renewal master:
srv01.ipa.mydomain.com <http://srv01.ipa.mydomain.com>[root@srv01 ~]#
ipactl statusDirectory Service: RUNNINGkrb5kdc Service: RUNNINGkadmin
Service: RUNNINGnamed Service: RUNNINGhttpd Service: RUNNINGipa-custodia
Service: RUNNINGntpd Service: RUNNINGpki-tomcatd Service: STOPPEDsmb
Service: RUNNINGwinbind Service: RUNNINGipa-otpd Service:
RUNNINGipa-dnskeysyncd Service: RUNNINGipa: INFO: The ipactl command was
successful[root@srv02 ~]# ipactl statusDirectory Service: RUNNINGkrb5kdc
Service: RUNNINGkadmin Service: RUNNINGnamed Service: RUNNINGhttpd Service:
RUNNINGipa-custodia Service: RUNNINGntpd Service: RUNNINGpki-tomcatd
Service: STOPPEDipa-otpd Service: RUNNINGipa-dnskeysyncd Service:
RUNNINGipa: INFO: The ipactl command was successful[root@srv01 ~]# certutil
-L -d /etc/pki/pki-tomcat/aliasCertificate Nickname
Trust Attributes
SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca
u,u,usubsystemCert cert-pki-ca
u,u,ucaSigningCert cert-pki-ca
CTu,Cu,CuocspSigningCert cert-pki-ca
u,u,uauditSigningCert cert-pki-ca
u,u,Pu[root@srv02 ~]# certutil -L -d /etc/pki/pki-tomcat/aliasCertificate
Nickname Trust Attributes
SSL,S/MIME,JAR/XPIServer-Cert cert-pki-ca
u,u,usubsystemCert cert-pki-ca
u,u,ucaSigningCert cert-pki-ca
CTu,u,uocspSigningCert cert-pki-ca
u,u,uauditSigningCert cert-pki-ca u,u,Pu*
It seems that AD trust controller role, IPA CA renewal master, smb and
windbind are only in the 1st server.
And also caSigningCert cert-pki-ca entry is different (CTu,Cu,Cu vs
CTu,u,u).
I can see only in the 1st server these DNS records:
*_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88
srv01_kerberos._tcp.dc._msdcs SRV 0 100 88
srv01_kerberos._udp.Default-First-Site-Name._sites.dc._msdcs SRV 0 100 88
srv01_kerberos._udp.dc._msdcs SRV 0 100 88
srv01_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs 0 100 389
srv01_ldap._tcp.dc._msdcs 0 100 389 srv01*
Srv01 is the first master, I know, but is the server VM that has clock
problems, in both situations.
So I want to keep srv02 and install a new one.
What do I have to do to let the 2nd VM be a single server?
Could I use these URLs?
*https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_Master
<https://www.freeipa.org/page/Backup_and_Restore#One_Server_Loss_-_First_M...>https://www.freeipa.org/page/V4/Server_Roles#Upgrade
<https://www.freeipa.org/page/V4/Server_Roles#Upgrade>*
2) uninstall ipa-server from the 1st server (srv01) and then powering off
it, assuming that all data into the 2nd one are ok (srv02)
3) update freeipa and all other RPM packages into the VM srv02
4) install a new fresh VM, always with 7 release, and create a new replica
Could I use the same old hostname (srv01) and IP address for this new VM?
Or is better to use the same IP but a new name, like srv03?
Do you think this is the right way to solve my issue?
Or do you have any better idea?
Please let me know, thanks.
Bye, Morgan
3 years, 8 months
- 3
- 8
Re: pki-tomcatd not starting
by Rob Crittenden
Scott Z. via FreeIPA-users wrote:
> Whoops! Using the additional command to start tracking this paritcular
> cert that you included in a different message, I got it in the "getcert"
> list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d
> /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B
> /usr/libexec/ipa/certmonger/stop_pkicad -C
> '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P
> <pin>" command).
>
> I have the date rolled back to Sept. 1st, 2019. I guess I have 'some'
> progress now at least, but still have an issue;Â checking on the cert
> with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and
> "stuck: yes".
How did you roll the date back? Did you restart services? What date did
you pick and does it overlap so that all certs are valid?
rob
>
> Any additional thoughts or help would be greatly appreciated! And
> thanks for the help so far.
> Scott
>
> ------------------------------------------------------------------------
> *From:* Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
> *Sent:* Monday, August 10, 2020 10:37 AM
> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott
> Z. <sudz28(a)hotmail.com>
> *Subject:* [Freeipa-users] Re: pki-tomcatd not starting
> Â
> Sorry, I didn't realize I had dropped the mailing list - my mistake!
>
> I backed up the files/directories you mentioned below, then I checked on
> the ra-agent.pem to see if it was still valid (openssl x509 -in
> /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed
> currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After:Â Aug
> 10 17:20:41 2021 GMT).
>
> Based on that information, and knowing that the bad cert is valid from
> Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019
> since all certs will see that date as valid.
>
> The only issue I have now is getting the request ID for the expired
> cert; it doesn't show up in the list of certs when I do "getcert -list",
> I can only see it by running "certutil -L -d
> /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when
> I run that it does not show any Request ID associated for it?
> Scott
>
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Monday, August 10, 2020 8:45 AM
> *To:* Scott Z. <sudz28(a)hotmail.com>
> *Cc:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> Â
> Hi,
>
> re-adding the mailing list as the conversation could also help others.
>
> On 8/8/20 12:06 AM, Scott Z. wrote:
>> I did notice when I compare it to another IdM server in the environment,
>> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
>> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
>> I'm comparing against has a "Signing-Cert" certificate in addition. Is
>> this because it's the 'Master' or whatever? Should my 'bad' server have
>> this same Signing-Cert listed?
>
> /etc/httpd/alias only needs its own Server-Cert + IPA CA.
>
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Scott Z. <sudz28(a)hotmail.com>
>> *Sent:* Friday, August 7, 2020 10:44 AM
>> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> /"The interesting part is the list of expired certs on the failing node
>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>> instructions are available here:
>> https://access.redhat.com/solutions/3357331 How do I manually renew
>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>> (Replica IPA Server)"/
>
> Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
> /etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
> the certificates are stored).
>
> If the RA cert is valid, you need to find a time window during which the
> RA cert is already valid (date > notbefore) and the other certs are not
> expired yet (date < notafter). When you have identified a proper date,
> stop ntpd (or chronyd, depending on which service is used for time
> synchronization), move the date back in time to the identified date,
> start all the services except ntpd, then call "getcert resubmit -i
> <request id>" for the expired cert(s).
>
> Check that the cert has been renewed with "getcert list -i <request
> id>", the state should display MONITORING. When all the certs are good,
> you can restart ntpd and the clock will go back to the current date.
>
> It's really important to find a date where all the certs are valid
> because this ensures that the services are able to start and the RA cert
> allows the authentication that is mandatory for certificate renewal.
>
> HTH,
> flo
>>
>> Sadly, after I log in, it's only telling me that it's "Subscriber
>> Exclusive Content". Not sure what happened with my account, I used to
>> be able to access these docs with no problem but since I took a RHEL
>> class a couple of weeks back now it's not working any more. I guess
>> they did something to screw up my account when I took the class. Grrrrr!!!
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Thursday, August 6, 2020 2:46 AM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
>> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>>> Thanks much for the assistance. Here is where I am with your suggestions:
>>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>>> (almost a year old actually, I assume IPA only checks it when it first
>>> starts up so it didn't care that it was expired until the server was
>>> rebooted?)
>>
>> certmonger checks the certificate validity periodically (configurable in
>> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
>> The system probably had an issue that was not detected and the cert
>> reached its expiration date.
>>
>>>
>>> 2) ran ipactl start --ignore-service-failures
>>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>>> 3) ran "kinit admin"
>>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>>> 4) Ran "ipa config-show |grep -i master
>>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>>> 6) Ran the command "getcert list" on the problem server, but I cannot
>>> paste the output here because it's on an airgaped environment so while I
>>> apologize for this and realize it makes things more difficult, perhaps
>>> if you tell me what I should be looking for or more specifically what
>>> you're interested in I can pluck that out and manually include it here?
>>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>>> certificate on the problem server, and it can theoretically be renew by
>>> the Master at this time.
>> The interesting part is the list of expired certs on the failing node
>> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
>> instructions are available here:
>> https://access.redhat.com/solutions/3357331 How do I manually renew
>> Identity Management (IPA) certificates on RHEL7 after they have expired?
>> (Replica IPA Server)
>>
>> flo
>>
>>> Many thanks!
>>> Scott
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>>> *Sent:* Monday, August 3, 2020 9:34 PM
>>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>>> Not sure I'm sending this to the right place, but here it goes. I
>>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>>> access) environment that is running into problems. There are at least 3
>>>> different IdM servers running in the environment spread out across
>>>> different geographical areas. One of those areas suffered an unschedule
>>>> power outage recently, and ever since we brought everything back up, the
>>>> IdM server for this region is having an issue. Please bear with me as I
>>>> have zero formal experience, training, or real knowledge with IdM.
>>>>
>>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>>> "ipactl restart", and things go fine until it gets to "Starting
>>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>>> to start and killing all the other services. I check the log at
>>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>>> (forgive any mistypings, I have to manually type these in as I can't
>>>> import or screen capure the logs and put them in this message):
>>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>>> And slightly further down in the same log:
>>>> "/Cannot reset factory: connections not all returned/"
>>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>>> LDAP connection factory because some connections are still outstanding/"
>>>> ... still further down"
>>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>>
>>>> Assuming I have some weird certificate issue with this server in
>>>> particular, I try to run a few more commands:
>>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>>> for it's attributes. Comparing to a second IdM server in this
>>>> environment, it seems to be missing a "Signing-Cert"?
>>>>
>>> Hi,
>>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>>> one is not expired with:
>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>>> | grep 'Not '
>>>
>>> If the certificate is indeed expired, it will have to be renewed but you
>>> need first to find which IPA server is the CA renewal master. On your
>>> server, force a service start and check the CA renewal master:
>>> # ipactl start --ignore-service-failures
>>> # kinit admin
>>> # ipa config-show | grep "renewal master"
>>> Â Â IPA CA renewal master: server.domain.com
>>>
>>> You need to make sure that all the certificates are valid on the CA
>>> renewal master:
>>> (on the CA renewal master)# getcert list | grep -E
>>> "Request|certificate:|expires:"
>>>
>>> - if the CA renewal master is not OK, please post the output of "#
>>> getcert list" (without the grep) on the CA renewal master. This node
>>> will have to be repaired first.
>>> - if the CA renewal master is OK, please post the output of "# getcert
>>> list" (also without the grep) on the failing node.
>>>
>>> We'll be able to help based on this information.
>>> flo
>>>
>>>> I also did a "getcert list", and all certs it has show that they expire
>>>> in the future (nothing shows as bein currently expired).
>>>>
>>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>>> but how do I track down which 'peer' the log file is talking about that
>>>> has an expired cert? Meanwhile none of the linux clients that point to
>>>> this IdM server are allowing people to log in/authenticate.
>>>> Many thanks for any help!
>>>> Scott
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>
3 years, 8 months
- 2
- 1
Re: pki-tomcatd not starting
by Scott Z.
Whoops! Using the additional command to start tracking this paritcular cert that you included in a different message, I got it in the "getcert" list (with the "getcert start-tracking -n 'Server-Cert cert-pki-ca' -d /etc/pki/pki-tomcat/alias -c dogtag-ipa-ca-renew-agent -B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -P <pin>" command).
I have the date rolled back to Sept. 1st, 2019. I guess I have 'some' progress now at least, but still have an issue; checking on the cert with "getcert list -i <requestID>", it shows "status: CA_REJECTED", and "stuck: yes".
Any additional thoughts or help would be greatly appreciated! And thanks for the help so far.
Scott
________________________________
From: Scott Z. via FreeIPA-users <freeipa-users(a)lists.fedorahosted.org>
Sent: Monday, August 10, 2020 10:37 AM
To: Florence Blanc-Renaud <flo(a)redhat.com>
Cc: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>; Scott Z. <sudz28(a)hotmail.com>
Subject: [Freeipa-users] Re: pki-tomcatd not starting
Sorry, I didn't realize I had dropped the mailing list - my mistake!
I backed up the files/directories you mentioned below, then I checked on the ra-agent.pem to see if it was still valid (openssl x509 -in /path/to/ra-agent.pem -text -noout), and the ra-agent.pem cert is indeed currently valid (Not before: Aug 21 17:20:41 2019 GMT, Not After: Aug 10 17:20:41 2021 GMT).
Based on that information, and knowing that the bad cert is valid from Oct. 6th 2017 to Sep. 26 2019, I'm going with Sept. 1st of this 2019 since all certs will see that date as valid.
The only issue I have now is getting the request ID for the expired cert; it doesn't show up in the list of certs when I do "getcert -list", I can only see it by running "certutil -L -d /var/lib/pki/pki-tomcat/ca/alias -n 'ServerCert cert-pki-ca'", and when I run that it does not show any Request ID associated for it?
Scott
________________________________
From: Florence Blanc-Renaud <flo(a)redhat.com>
Sent: Monday, August 10, 2020 8:45 AM
To: Scott Z. <sudz28(a)hotmail.com>
Cc: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Subject: Re: [Freeipa-users] Re: pki-tomcatd not starting
Hi,
re-adding the mailing list as the conversation could also help others.
On 8/8/20 12:06 AM, Scott Z. wrote:
> I did notice when I compare it to another IdM server in the environment,
> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
> I'm comparing against has a "Signing-Cert" certificate in addition. Is
> this because it's the 'Master' or whatever? Should my 'bad' server have
> this same Signing-Cert listed?
/etc/httpd/alias only needs its own Server-Cert + IPA CA.
> Scott
>
> ------------------------------------------------------------------------
> *From:* Scott Z. <sudz28(a)hotmail.com>
> *Sent:* Friday, August 7, 2020 10:44 AM
> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> /"The interesting part is the list of expired certs on the failing node
> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
> instructions are available here:
> https://access.redhat.com/solutions/3357331 How do I manually renew
> Identity Management (IPA) certificates on RHEL7 after they have expired?
> (Replica IPA Server)"/
Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
/etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
the certificates are stored).
If the RA cert is valid, you need to find a time window during which the
RA cert is already valid (date > notbefore) and the other certs are not
expired yet (date < notafter). When you have identified a proper date,
stop ntpd (or chronyd, depending on which service is used for time
synchronization), move the date back in time to the identified date,
start all the services except ntpd, then call "getcert resubmit -i
<request id>" for the expired cert(s).
Check that the cert has been renewed with "getcert list -i <request
id>", the state should display MONITORING. When all the certs are good,
you can restart ntpd and the clock will go back to the current date.
It's really important to find a date where all the certs are valid
because this ensures that the services are able to start and the RA cert
allows the authentication that is mandatory for certificate renewal.
HTH,
flo
>
> Sadly, after I log in, it's only telling me that it's "Subscriber
> Exclusive Content". Not sure what happened with my account, I used to
> be able to access these docs with no problem but since I took a RHEL
> class a couple of weeks back now it's not working any more. I guess
> they did something to screw up my account when I took the class. Grrrrr!!!
> Scott
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Thursday, August 6, 2020 2:46 AM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>> Thanks much for the assistance. Here is where I am with your suggestions:
>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>> (almost a year old actually, I assume IPA only checks it when it first
>> starts up so it didn't care that it was expired until the server was
>> rebooted?)
>
> certmonger checks the certificate validity periodically (configurable in
> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
> The system probably had an issue that was not detected and the cert
> reached its expiration date.
>
>>
>> 2) ran ipactl start --ignore-service-failures
>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>> 3) ran "kinit admin"
>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>> 4) Ran "ipa config-show |grep -i master
>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>> 6) Ran the command "getcert list" on the problem server, but I cannot
>> paste the output here because it's on an airgaped environment so while I
>> apologize for this and realize it makes things more difficult, perhaps
>> if you tell me what I should be looking for or more specifically what
>> you're interested in I can pluck that out and manually include it here?
>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>> certificate on the problem server, and it can theoretically be renew by
>> the Master at this time.
> The interesting part is the list of expired certs on the failing node
> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
> instructions are available here:
> https://access.redhat.com/solutions/3357331 How do I manually renew
> Identity Management (IPA) certificates on RHEL7 after they have expired?
> (Replica IPA Server)
>
> flo
>
>> Many thanks!
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Monday, August 3, 2020 9:34 PM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>> Not sure I'm sending this to the right place, but here it goes. I
>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>> access) environment that is running into problems. There are at least 3
>>> different IdM servers running in the environment spread out across
>>> different geographical areas. One of those areas suffered an unschedule
>>> power outage recently, and ever since we brought everything back up, the
>>> IdM server for this region is having an issue. Please bear with me as I
>>> have zero formal experience, training, or real knowledge with IdM.
>>>
>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>> "ipactl restart", and things go fine until it gets to "Starting
>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>> to start and killing all the other services. I check the log at
>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>> (forgive any mistypings, I have to manually type these in as I can't
>>> import or screen capure the logs and put them in this message):
>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>> And slightly further down in the same log:
>>> "/Cannot reset factory: connections not all returned/"
>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>> LDAP connection factory because some connections are still outstanding/"
>>> ... still further down"
>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>
>>> Assuming I have some weird certificate issue with this server in
>>> particular, I try to run a few more commands:
>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>> for it's attributes. Comparing to a second IdM server in this
>>> environment, it seems to be missing a "Signing-Cert"?
>>>
>> Hi,
>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>> one is not expired with:
>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>> | grep 'Not '
>>
>> If the certificate is indeed expired, it will have to be renewed but you
>> need first to find which IPA server is the CA renewal master. On your
>> server, force a service start and check the CA renewal master:
>> # ipactl start --ignore-service-failures
>> # kinit admin
>> # ipa config-show | grep "renewal master"
>> Â Â IPA CA renewal master: server.domain.com
>>
>> You need to make sure that all the certificates are valid on the CA
>> renewal master:
>> (on the CA renewal master)# getcert list | grep -E
>> "Request|certificate:|expires:"
>>
>> - if the CA renewal master is not OK, please post the output of "#
>> getcert list" (without the grep) on the CA renewal master. This node
>> will have to be repaired first.
>> - if the CA renewal master is OK, please post the output of "# getcert
>> list" (also without the grep) on the failing node.
>>
>> We'll be able to help based on this information.
>> flo
>>
>>> I also did a "getcert list", and all certs it has show that they expire
>>> in the future (nothing shows as bein currently expired).
>>>
>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>> but how do I track down which 'peer' the log file is talking about that
>>> has an expired cert? Meanwhile none of the linux clients that point to
>>> this IdM server are allowing people to log in/authenticate.
>>> Many thanks for any help!
>>> Scott
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
3 years, 8 months
- 1
- 0
Re: pki-tomcatd not starting
by Florence Blanc-Renaud
Hi,
re-adding the mailing list as the conversation could also help others.
On 8/8/20 12:06 AM, Scott Z. wrote:
> I did notice when I compare it to another IdM server in the environment,
> if I do a "certutil -L -d /etc.httdp/alias" the non-working server has a
> <DOMAIN> IPA CA certificate and a Server-Cert, but the other one that
> I'm comparing against has a "Signing-Cert" certificate in addition. Is
> this because it's the 'Master' or whatever? Should my 'bad' server have
> this same Signing-Cert listed?
/etc/httpd/alias only needs its own Server-Cert + IPA CA.
> Scott
>
> ------------------------------------------------------------------------
> *From:* Scott Z. <sudz28(a)hotmail.com>
> *Sent:* Friday, August 7, 2020 10:44 AM
> *To:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> /"The interesting part is the list of expired certs on the failing node
> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
> instructions are available here:
> https://access.redhat.com/solutions/3357331 How do I manually renew
> Identity Management (IPA) certificates on RHEL7 after they have expired?
> (Replica IPA Server)"/
Start by making a backup of /etc/dirsrv/slapd-*/*.db, /etc/httpd/alias,
/etc/pki/pki-tomcat/alias and /var/lib/ipa/ra-agent.* (the places where
the certificates are stored).
If the RA cert is valid, you need to find a time window during which the
RA cert is already valid (date > notbefore) and the other certs are not
expired yet (date < notafter). When you have identified a proper date,
stop ntpd (or chronyd, depending on which service is used for time
synchronization), move the date back in time to the identified date,
start all the services except ntpd, then call "getcert resubmit -i
<request id>" for the expired cert(s).
Check that the cert has been renewed with "getcert list -i <request
id>", the state should display MONITORING. When all the certs are good,
you can restart ntpd and the clock will go back to the current date.
It's really important to find a date where all the certs are valid
because this ensures that the services are able to start and the RA cert
allows the authentication that is mandatory for certificate renewal.
HTH,
flo
>
> Sadly, after I log in, it's only telling me that it's "Subscriber
> Exclusive Content". Not sure what happened with my account, I used to
> be able to access these docs with no problem but since I took a RHEL
> class a couple of weeks back now it's not working any more. I guess
> they did something to screw up my account when I took the class. Grrrrr!!!
> Scott
>
> ------------------------------------------------------------------------
> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
> *Sent:* Thursday, August 6, 2020 2:46 AM
> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> *Cc:* Scott Z. <sudz28(a)hotmail.com>
> *Subject:* Re: [Freeipa-users] Re: pki-tomcatd not starting
> On 8/6/20 12:53 AM, Scott Z. via FreeIPA-users wrote:
>> Thanks much for the assistance. Here is where I am with your suggestions:
>> 1) Checked on the cert with "certutil -L -d /etc/pki/pki-tomcat/alias -n
>> 'Server-Cert cert-pki-ca' and I see that the Validity is indeed old
>> (almost a year old actually, I assume IPA only checks it when it first
>> starts up so it didn't care that it was expired until the server was
>> rebooted?)
>
> certmonger checks the certificate validity periodically (configurable in
> certmonger.conf) and tries multiple times to renew soon-to-expire certs.
> The system probably had an issue that was not detected and the cert
> reached its expiration date.
>
>>
>> 2) ran ipactl start --ignore-service-failures
>> Â Â Â Â Â Â a. most services started, obviously pki-tomcatd did not
>> 3) ran "kinit admin"
>> Â Â Â Â Â Â a. was forced to change the password, but otherwise nothing happened
>> 4) Ran "ipa config-show |grep -i master
>> Â Â Â Â Â a. I see that the IPA CA renewal master is a different idm machine.
>> 5) Ran "getcert list | grep -E "Request|certificate:|expires:"
>> Â Â Â Â Â a.I see all certs are currently valid (none expired)
>> 6) Ran the command "getcert list" on the problem server, but I cannot
>> paste the output here because it's on an airgaped environment so while I
>> apologize for this and realize it makes things more difficult, perhaps
>> if you tell me what I should be looking for or more specifically what
>> you're interested in I can pluck that out and manually include it here?
>> So in summary, it is indeed an expired "Server-Cert cert-pki-ca'
>> certificate on the problem server, and it can theoretically be renew by
>> the Master at this time.
> The interesting part is the list of expired certs on the failing node
> (is the RA cert /var/lib/ipa/ra-agent.pem expired?). Detailed
> instructions are available here:
> https://access.redhat.com/solutions/3357331 How do I manually renew
> Identity Management (IPA) certificates on RHEL7 after they have expired?
> (Replica IPA Server)
>
> flo
>
>> Many thanks!
>> Scott
>>
>> ------------------------------------------------------------------------
>> *From:* Florence Blanc-Renaud <flo(a)redhat.com>
>> *Sent:* Monday, August 3, 2020 9:34 PM
>> *To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> *Cc:* Scott Z. <sudz28(a)hotmail.com>
>> *Subject:* Re: [Freeipa-users] pki-tomcatd not starting
>> On 8/3/20 10:14 PM, Scott Z. via FreeIPA-users wrote:
>>> Not sure I'm sending this to the right place, but here it goes. I
>>> inherited a FreeIPA/Identity Manager setup in an enclave (no internet
>>> access) environment that is running into problems. There are at least 3
>>> different IdM servers running in the environment spread out across
>>> different geographical areas. One of those areas suffered an unschedule
>>> power outage recently, and ever since we brought everything back up, the
>>> IdM server for this region is having an issue. Please bear with me as I
>>> have zero formal experience, training, or real knowledge with IdM.
>>>
>>> Logging in to the serverv (it's a VM server, running Centos 7.5), I run
>>> "ipactl status" and it shows "Directory Service: STOPPED". I then run
>>> "ipactl restart", and things go fine until it gets to "Starting
>>> pki-tomcatd Service", where it hangs for quite some time before failing
>>> to start and killing all the other services. I check the log at
>>> /var/log/pki/pki-tomcat/ca/debug and I see various errors such as
>>> (forgive any mistypings, I have to manually type these in as I can't
>>> import or screen capure the logs and put them in this message):
>>> "/java.lang.Exception: Certificate Server-Cert cert-pki-ca is invalid:
>>> Invalid certificate: (-8181) Peer's Certificate has expired/"
>>> And slightly further down in the same log:
>>> "/Cannot reset factory: connections not all returned/"
>>> "/CertificateAuthority.shutdown: failed to reset dbFactory: Cannot reset
>>> LDAP connection factory because some connections are still outstanding/"
>>> ... still further down"
>>> "/returnConn:mNumConns now 3 Invalid class name repositorytop/"
>>>
>>> Assuming I have some weird certificate issue with this server in
>>> particular, I try to run a few more commands:
>>> "certutil -L -d /etc/httpd/alias" --> returns a Server-Cert listing
>>> with u,u,u as it's trust attributes, and <IDM.domain> IPA CA with CT,C,C
>>> for it's attributes. Comparing to a second IdM server in this
>>> environment, it seems to be missing a "Signing-Cert"?
>>>
>> Hi,
>> PKI is using the NSSDB in /etc/pki/pki-tomcat/alias, and its server cert
>> has the nickname 'Server-Cert cert-pki-ca'. You should check that this
>> one is not expired with:
>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'Server-Cert cert-pki-ca'
>> | grep 'Not '
>>
>> If the certificate is indeed expired, it will have to be renewed but you
>> need first to find which IPA server is the CA renewal master. On your
>> server, force a service start and check the CA renewal master:
>> # ipactl start --ignore-service-failures
>> # kinit admin
>> # ipa config-show | grep "renewal master"
>> Â Â IPA CA renewal master: server.domain.com
>>
>> You need to make sure that all the certificates are valid on the CA
>> renewal master:
>> (on the CA renewal master)# getcert list | grep -E
>> "Request|certificate:|expires:"
>>
>> - if the CA renewal master is not OK, please post the output of "#
>> getcert list" (without the grep) on the CA renewal master. This node
>> will have to be repaired first.
>> - if the CA renewal master is OK, please post the output of "# getcert
>> list" (also without the grep) on the failing node.
>>
>> We'll be able to help based on this information.
>> flo
>>
>>> I also did a "getcert list", and all certs it has show that they expire
>>> in the future (nothing shows as bein currently expired).
>>>
>>> I'm confused; it seems to that it is seeing an expired cert *somewhere*,
>>> but how do I track down which 'peer' the log file is talking about that
>>> has an expired cert? Meanwhile none of the linux clients that point to
>>> this IdM server are allowing people to log in/authenticate.
>>> Many thanks for any help!
>>> Scott
>>>
>>>
>>> _______________________________________________
>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>
>
3 years, 8 months
- 2
- 1