Hi!
Forgot kinit:
--
kinit admin
Password for admin@<<REALM>>:
klist
Ticket cache: KEYRING:persistent:0:0
Default principal: admin@<<REALM>>
Valid starting Expires Service principal
06/21/2018 09:55:07 06/22/2018 09:54:54
HTTP/<<ipa1.fqdn>>@<<REALM>>
06/21/2018 09:55:02 06/22/2018 09:54:54
krbtgt/<<REALM>>@<<REALM>>
ipa config-show
ipa: ERROR: No valid Negotiate header in server response
--
Hi,
can you check if the service gssproxy is running:
# systemctl status gssproxy
and if not, restart it.
Flo
Still no luck getting ipa config
Eemeli
-----Original Message-----
From: Florence Blanc-Renaud [mailto:flo@redhat.com]
Sent: torstai 21. kesäkuuta 2018 9.43
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Jokinen Eemeli <Eemeli.Jokinen(a)cinia.fi>
Subject: Re: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade
doesn't complete, pki-tomcatd won't start
On 06/21/2018 07:34 AM, Jokinen Eemeli via FreeIPA-users wrote:
> Hi!
>
> We do have 2 IPA nodes configured but the second node has been down for some time.
Tried to update it to same version as node1:
> - Won't start tells me to use ipa-server-upgrade
> - Ipa-server-upgrade fails at start, doesn't start directory server
> - /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse
> errors but last row tells me
> --
> EMERG - main - Fatal Error---No ports specified. Exiting now.
> --
>
Hi,
in this case let's concentrate on node1.
> On node1 ipa config-show results only
> --
> ipa: ERROR: did not receive Kerberos credentials
> --
You need to run kinit to get kerberos credentials before any ipa * command. If the ipa
stack is stopped because pki-tomcat refused to start, you can force the start up, ignoring
failed services:
$ sudo ipactl start --ignore-service-failures
Then to check who is renewal master:
$ sudo kinit admin
$ sudo ipa config-show
If node1 is not the renewal master, make it the renewal master from now on (follow
instructions from
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/...).
After this step, check the certificates in the NSS databases vs the certificates stored
in LDAP.
HTH,
Flo
>
>
> Eemeli
>
> -----Original Message-----
> From: Florence Blanc-Renaud [mailto:flo@redhat.com]
> Sent: keskiviikko 20. kesäkuuta 2018 19.49
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> Cc: Jokinen Eemeli <Eemeli.Jokinen(a)cinia.fi>
> Subject: Re: [Freeipa-users] Problems after IPA upgrade:
> ipa-server-upgrade doesn't complete, pki-tomcatd won't start
>
> On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
>> Hello all!
>>
>> I have very similiar problem as this one:
>>
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo
>> r
ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
>>
>> ipa-server-upgrade fails as below
>>
>> --
>>
>> Update complete
>>
>> Upgrading IPA services
>>
>> Upgrading the configuration of the IPA services
>>
>> [Verifying that root certificate is published]
>>
>> [Migrate CRL publish directory]
>>
>> CRL tree already moved
>>
>> [Verifying that CA proxy configuration is correct]
>>
>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
>> command ipa-server-upgrade manually.
>>
>> CA did not start in 300.0s
>>
>> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
>> for more information
>>
>> --
>>
>> And the log tells me that CA returns status 500
>>
>> --
>>
>> DEBUG Waiting for CA to start...
>>
>> DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus
>> <http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
>>
>> DEBUG request body ''
>>
>> DEBUG response status 500
>>
>> DEBUG response headers Server: Apache-Coyote/1.1
>>
>> Content-Type: text/html;charset=utf-8
>>
>> Content-Language: en
>>
>> Content-Length: 2208
>>
>> Date: Fri, 15 Jun 2018 10:05:29 GMT
>>
>> Connection: close
>>
>> DEBUG response body '<html><head><title>Apache
Tomcat/7.0.76 - Error
>> report</title><style><!--H1
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5
>> D76;font-size:22px;}
>> H2
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5
>> D76;font-size:16px;}
>> H3
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5
>> D76;font-size:14px;}
>> BODY
>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whi
>> t
>> e;} B
>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#52
>> 5
>> D76;}
>> P
>> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;fon
>> t -size:12px;}A {color : black;}A.name {color : black;}HR {color :
>> #525D76;}--></style> </head><body><h1>HTTP Status 500
- Subsystem
>> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
>> Exception report</p><p><b>message</b> <u>Subsystem
>> unavailable</u></p><p><b>description</b>
<u>The server encountered an
>> internal error that prevented it from fulfilling this
>> request.</u></p><p><b>exception</b>
>> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
>> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstra
>> i
>> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authent
>> i
>> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.v
>> a
>> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
>> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
>> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:44
>> 5
>> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstrac
>> t
>> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abst
>> r
>> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
>> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>> \
>> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut
>> o
>> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Th
>> r
>> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThrea
>> d
>> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thr
>> e ad.java:748)\n</pre></p><p><b>note</b>
>> <u>The full stack trace of the root cause is available in the Apache
>> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
>> Tomcat/7.0.76</h3></body></html>'
>>
>> DEBUG The CA status is: check interrupted due to error: Retrieving CA
>> status failed with status 500
>>
>> DEBUG Waiting for CA to start...
>>
>> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
>> run command ipa-server-upgrade manually.
>>
>> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
>> 172, in execute
>>
>> return_value = self.run()
>>
>> File
>> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrad
>> e
>> .py",
>> line 48, in run
>>
>> raise admintool.ScriptError(str(e))
>>
>> The ipa-server-upgrade command failed, exception: ScriptError: CA did
>> not start in 300.0s
>>
>> ERROR CA did not start in 300.0s
>>
>> --
>>
>> With command "ipactl start --ignore-service-failures" I can start all
>> the services but pki-tomcatd.
>>
>> --
>>
>> Directory Service: RUNNING
>>
>> krb5kdc Service: RUNNING
>>
>> kadmin Service: RUNNING
>>
>> named Service: RUNNING
>>
>> httpd Service: RUNNING
>>
>> pki-tomcatd Service: STOPPED
>>
>> smb Service: RUNNING
>>
>> winbind Service: RUNNING
>>
>> ipa-otpd Service: RUNNING
>>
>> ipa-dnskeysyncd Service: RUNNING
>>
>> ipa: INFO: The ipactl command was successful
>>
>> --
>>
>> Suggested resolution to above problem doesn't help me since the LDAP
>> and NSS DB seem to have same certificates (some difference in
>> wrapping but the string is same if I take out the line breaks) and
>> even the serial number matches.
>>
>> --
>>
>> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca' -a
>>
>> -----BEGIN CERTIFICATE-----
>>
>> MIIDjD.
>>
>> .Prh2G
>>
>> -----END CERTIFICATE-----
>>
>> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
>> |grep Serial
>>
>> Serial Number: 4 (0x4)
>>
>> ldapsearch -LLL -D 'cn=directory manager' -W -b
>> uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
>>
>> Enter LDAP Password:
>>
>> dn: uid=pkidbuser,ou=people,o=ipaca
>>
>> userCertificate:: MIIDjD.
>>
>> .Prh2
>>
>> G
>>
>> description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA
>> Subsystem,
>>
>> O=<<REALM>>
>>
>> seeAlso: CN=CA Subsystem,O=<<REALM>>
>>
>> --
>>
>> And here's where my actual knowledge of things end. I've been trying
>> to figure out all kind of logs (tomcat, Kerberos, directory server,
>> .) but haven't found a solid reason for it. I'm starting to believe
>> this is a certificate issue, because although "getcert list" tells me
>> that the certificate status is "Monitoring" on all certificates the
>> expiry date is already in the past (current date 20.6.2018,
>> certificate expiry
>> 21.03.2018) on 4 certificates and it won't update even if I resubmit
>> it or delete certificate and manually redo it (it got the same date
>> as the "old ones"). The "main certs" ("caSigningCert
cert-pki-ca",
>> "Server-Cert cert-pki-ca" and two directory server certs) are valid
>> for years (until
>> 2020+).
>>
>> --
>>
>> Request ID '20160331084234':
>>
>> status: MONITORING
>>
>> stuck: no
>>
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning
>> C ert cert-pki-ca',token='NSS Certificate DB',pin set
>>
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigning
>> C ert cert-pki-ca',token='NSS Certificate DB'
>>
>> CA: dogtag-ipa-ca-renew-agent
>>
>> issuer: CN=Certificate Authority,O=<<REALM>>
>>
>> subject: CN=OCSP Subsystem,O=<<REALM>>
>>
>> expires: 2018-03-21 09:42:04 UTC
>>
>> key usage:
>> digitalSignature,nonRepudiation,keyCertSign,cRLSign
>>
>> eku: id-kp-OCSPSigning
>>
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>>
>> post-save command:
>> /usr/libexec/ipa/certmonger/renew_ca_cert
>> "ocspSigningCert cert-pki-ca"
>>
>> track: yes
>>
>> auto-renew: yes
>>
>> Request ID '20160331085008':
>>
>> status: MONITORING
>>
>> stuck: no
>>
>> key pair storage:
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='
>> N SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>>
>> certificate:
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='
>> N
>> SS
>> Certificate DB'
>>
>> CA: IPA
>>
>> issuer: CN=Certificate Authority,O=<<REALM>>
>>
>> subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
>>
>> expires: 2020-03-04 09:58:23 UTC
>>
>> principal name:
HTTP/<<ipasrv1.fqdn>>@<<REALM>>
>>
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>>
>> eku: id-kp-serverAuth,id-kp-clientAuth
>>
>> pre-save command:
>>
>> post-save command: /usr/lib64/ipa/certmonger/restart_httpd
>>
>> track: yes
>>
>> auto-renew: yes
>>
>> --
>>
>> Has anyone else bumped into same kind of issues? Any ideas where I
>> should continue looking? I'm starting to run out of ideas.
>>
>> Eemeli Jokinen
>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
>> List Guidelines:
>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>>
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fed
>> o
rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
>>
>
> Hi,
>
> does your topology include multiple CA instances? You need first to find which master
is the CA renewal master:
> ipa config-show | grep "renewal master"
>
> On this host, check that the certificates are still valid and consistent with the
content of the LDAP entries. If it is not the case, you need to repair the CA renewal
master first.
>
> When the CA renewal master is OK, check if the replication is working with the other
CA instances, and repair the other masters.
> HTH,
> Flo
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> List Guidelines:
>
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo
>
rahosted.org/message/PZTT22YGPX2567V7EL7LUFLEO4ZKOH6U/
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...