When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up because pki-tomcatd kept failing. I was able to get it running for now by ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show the failure itself, but not information that helps me fix the problem.
Kristian Petersen via FreeIPA-users wrote:
When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up because pki-tomcatd kept failing. I was able to get it running for now by ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show the failure itself, but not information that helps me fix the problem.
You'll need to share what information you have. I'd start by looking at /var/log/pki/pki-tomcat/ca/debug
rob
That path does not exist.
On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden rcritten@redhat.com wrote:
Kristian Petersen via FreeIPA-users wrote:
When I recently updated one of my IPA servers (it reports 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up because pki-tomcatd kept failing. I was able to get it running for now by ignoring the failure of that one service, but I haven't been able to to determine the cause. The logs are pretty quiet on this one. They show the failure itself, but not information that helps me fix the
problem.
You'll need to share what information you have. I'd start by looking at /var/log/pki/pki-tomcat/ca/debug
rob
Kristian Petersen wrote:
That path does not exist.
Ok, then you need to describe your installation, particularly what services are enabled.
IPA will try to start services based on this search so seeing this output would be useful as well:
$ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn
I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful.
rob
On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Kristian Petersen via FreeIPA-users wrote: > When I recently updated one of my IPA servers (it reports > 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up > because pki-tomcatd kept failing. I was able to get it running for now > by ignoring the failure of that one service, but I haven't been able to > to determine the cause. The logs are pretty quiet on this one. They > show the failure itself, but not information that helps me fix the problem. You'll need to share what information you have. I'd start by looking at /var/log/pki/pki-tomcat/ca/debug rob-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
The installation is a standard RedHat IdM install with DNS, SMB, and CA services installed.
The output of the ldapsearch you mentioned is: -bash-4.2$ ldapsearch -LLL -Y GSSAPI -b cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
SASL/GSSAPI authentication started SASL username: nesretep@CHEM.BYU.EDU SASL SSF: 56 SASL data security layer installed. dn: cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu ipaMaxDomainLevel: 1 ipaReplTopoManagedSuffix: dc=chem,dc=byu,dc=edu ipaReplTopoManagedSuffix: o=ipaca objectClass: top objectClass: nsContainer objectClass: ipaConfigObject objectClass: ipaSupportedDomainLevelConfig objectClass: ipaReplTopoManagedServer cn: ipa1.chem.byu.edu ipaMinDomainLevel: 0
dn: cn=CA,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 50 ipaConfigString: caRenewalMaster cn: CA
dn: cn=KDC,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 10 ipaConfigString: enabledService ipaConfigString: kdcProxyEnabled ipaConfigString: pkinitEnabled cn: KDC
dn: cn=KPASSWD,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc
=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 20 cn: KPASSWD
dn: cn=MEMCACHE,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,d
c=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 39 ipaConfigString: enabledService cn: MEMCACHE
dn: cn=OTPD,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 80 ipaConfigString: enabledService cn: OTPD
dn: cn=HTTP,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 40 ipaConfigString: enabledService cn: HTTP
dn: cn=DNS,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 30 ipaConfigString: enabledService cn: DNS
dn: cn=ADTRUST,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc
=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST
dn: cn=EXTID,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=e
du objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 70 ipaConfigString: enabledService cn: EXTID
dn: cn=DNSKeySync,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu
,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: dnssecVersion 1 ipaConfigString: startOrder 110 ipaConfigString: enabledService cn: DNSKeySync
dn: cn=NTP,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 45 ipaConfigString: enabledService cn: NTP
dn: cn=KEYS,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 41 ipaConfigString: enabledService cn: KEYS
This shows up at the bottom of the ipaupgrade.log file while everything before this looks OK from what I can tell:
2017-09-27T17:18:57Z DEBUG request POST http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus 2017-09-27T17:18:57Z DEBUG request body '' 2017-09-27T17:18:57Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 204, in _httplib_request conn.request(method, uri, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Failed to check CA status: cannot connect to ' http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Ensuring that service pki-tomcatd@pki-tomcat is not running while the next set of commands is being executed. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed
2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z DEBUG Service pki-tomcatd@pki-tomcat is not running, continue. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed
2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z INFO [Migrate CRL publish directory] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2017-09-27T17:18:57Z INFO CRL tree already moved 2017-09-27T17:18:57Z INFO [Verifying that CA proxy configuration is correct] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2017-09-27T17:18:57Z DEBUG Proxy configuration up-to-date 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl start pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=1 2017-09-27T17:18:57Z DEBUG stdout= 2017-09-27T17:18:57Z DEBUG stderr=Job for pki-tomcatd@pki-tomcat.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details.
2017-09-27T17:18:57Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-09-27T17:18:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1652, in upgrade_configuration ca.start('pki-tomcat') File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 401, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2017-09-27T17:18:57Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1 2017-09-27T17:18:57Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details
Any thoughts? Is that URL it is requesting to get the status something that is a valid URL that should be responding? I tried with a simple wget and also get connection refused for the response.
On Tue, Oct 3, 2017 at 8:13 AM, Rob Crittenden rcritten@redhat.com wrote:
Kristian Petersen wrote:
That path does not exist.
Ok, then you need to describe your installation, particularly what services are enabled.
IPA will try to start services based on this search so seeing this output would be useful as well:
$ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn
I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful.
rob
On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Kristian Petersen via FreeIPA-users wrote: > When I recently updated one of my IPA servers (it reports > 4.5.0-21.el7_4.1.2 in yum), the result was that it could startback up
> because pki-tomcatd kept failing. I was able to get it runningfor now
> by ignoring the failure of that one service, but I haven't beenable to
> to determine the cause. The logs are pretty quiet on this one.They
> show the failure itself, but not information that helps me fix theproblem.
You'll need to share what information you have. I'd start by lookingat
/var/log/pki/pki-tomcat/ca/debug rob-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
I'm still struggling with this one and it seems at least partially responsible for the UI misbehaving as we discussed in another thread. Have you had any new insights regarding this?
On Mon, Oct 9, 2017 at 3:54 PM, Kristian Petersen nesretep@chem.byu.edu wrote:
The installation is a standard RedHat IdM install with DNS, SMB, and CA services installed.
The output of the ldapsearch you mentioned is: -bash-4.2$ ldapsearch -LLL -Y GSSAPI -b cn=ipa1.chem.byu.edu,cn= masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu SASL/GSSAPI authentication started SASL username: nesretep@CHEM.BYU.EDU SASL SSF: 56 SASL data security layer installed. dn: cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu ipaMaxDomainLevel: 1 ipaReplTopoManagedSuffix: dc=chem,dc=byu,dc=edu ipaReplTopoManagedSuffix: o=ipaca objectClass: top objectClass: nsContainer objectClass: ipaConfigObject objectClass: ipaSupportedDomainLevelConfig objectClass: ipaReplTopoManagedServer cn: ipa1.chem.byu.edu ipaMinDomainLevel: 0
dn: cn=CA,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 50 ipaConfigString: caRenewalMaster cn: CA
dn: cn=KDC,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 10 ipaConfigString: enabledService ipaConfigString: kdcProxyEnabled ipaConfigString: pkinitEnabled cn: KDC
dn: cn=KPASSWD,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc
=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 20 cn: KPASSWD
dn: cn=MEMCACHE,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,d
c=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 39 ipaConfigString: enabledService cn: MEMCACHE
dn: cn=OTPD,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 80 ipaConfigString: enabledService cn: OTPD
dn: cn=HTTP,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 40 ipaConfigString: enabledService cn: HTTP
dn: cn=DNS,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 30 ipaConfigString: enabledService cn: DNS
dn: cn=ADTRUST,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc
=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST
dn: cn=EXTID,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=e
du objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 70 ipaConfigString: enabledService cn: EXTID
dn: cn=DNSKeySync,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu
,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: dnssecVersion 1 ipaConfigString: startOrder 110 ipaConfigString: enabledService cn: DNSKeySync
dn: cn=NTP,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 45 ipaConfigString: enabledService cn: NTP
dn: cn=KEYS,cn=ipa1.chem.byu.edu,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 41 ipaConfigString: enabledService cn: KEYS
This shows up at the bottom of the ipaupgrade.log file while everything before this looks OK from what I can tell:
2017-09-27T17:18:57Z DEBUG request POST http://ipa1.chem.byu.edu:8080/ ca/admin/ca/getStatus 2017-09-27T17:18:57Z DEBUG request body '' 2017-09-27T17:18:57Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 204, in _httplib_request conn.request(method, uri, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Failed to check CA status: cannot connect to ' http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Ensuring that service pki-tomcatd@pki-tomcat is not running while the next set of commands is being executed. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed
2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z DEBUG Service pki-tomcatd@pki-tomcat is not running, continue. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed
2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z INFO [Migrate CRL publish directory] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/ sysupgrade.state' 2017-09-27T17:18:57Z INFO CRL tree already moved 2017-09-27T17:18:57Z INFO [Verifying that CA proxy configuration is correct] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/ sysrestore.state' 2017-09-27T17:18:57Z DEBUG Proxy configuration up-to-date 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl start pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=1 2017-09-27T17:18:57Z DEBUG stdout= 2017-09-27T17:18:57Z DEBUG stderr=Job for pki-tomcatd@pki-tomcat.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details.
2017-09-27T17:18:57Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-09-27T17:18:57Z DEBUG File "/usr/lib/python2.7/site- packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1652, in upgrade_configuration ca.start('pki-tomcat') File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 401, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2017-09-27T17:18:57Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1 2017-09-27T17:18:57Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details
Any thoughts? Is that URL it is requesting to get the status something that is a valid URL that should be responding? I tried with a simple wget and also get connection refused for the response.
On Tue, Oct 3, 2017 at 8:13 AM, Rob Crittenden rcritten@redhat.com wrote:
Kristian Petersen wrote:
That path does not exist.
Ok, then you need to describe your installation, particularly what services are enabled.
IPA will try to start services based on this search so seeing this output would be useful as well:
$ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn
I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful.
rob
On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Kristian Petersen via FreeIPA-users wrote: > When I recently updated one of my IPA servers (it reports > 4.5.0-21.el7_4.1.2 in yum), the result was that it could startback up
> because pki-tomcatd kept failing. I was able to get it runningfor now
> by ignoring the failure of that one service, but I haven't beenable to
> to determine the cause. The logs are pretty quiet on this one.They
> show the failure itself, but not information that helps me fixthe problem.
You'll need to share what information you have. I'd start bylooking at
/var/log/pki/pki-tomcat/ca/debug rob-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
Kristian Petersen wrote:
I'm still struggling with this one and it seems at least partially responsible for the UI misbehaving as we discussed in another thread. Have you had any new insights regarding this?
I'd start with looking at /var/log/pki/pki-tomcat/ca/debug. You want to find the latest start and work down from there (rather than bottom up).
rob
On Mon, Oct 9, 2017 at 3:54 PM, Kristian Petersen <nesretep@chem.byu.edu mailto:nesretep@chem.byu.edu> wrote:
The installation is a standard RedHat IdM install with DNS, SMB, and CA services installed. The output of the ldapsearch you mentioned is: -bash-4.2$ ldapsearch -LLL -Y GSSAPI -b cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu SASL/GSSAPI authentication started SASL username: nesretep@CHEM.BYU.EDU <mailto:nesretep@CHEM.BYU.EDU> SASL SSF: 56 SASL data security layer installed. dn: cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu ipaMaxDomainLevel: 1 ipaReplTopoManagedSuffix: dc=chem,dc=byu,dc=edu ipaReplTopoManagedSuffix: o=ipaca objectClass: top objectClass: nsContainer objectClass: ipaConfigObject objectClass: ipaSupportedDomainLevelConfig objectClass: ipaReplTopoManagedServer cn: ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> ipaMinDomainLevel: 0 dn: cn=CA,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 50 ipaConfigString: caRenewalMaster cn: CA dn: cn=KDC,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 10 ipaConfigString: enabledService ipaConfigString: kdcProxyEnabled ipaConfigString: pkinitEnabled cn: KDC dn: cn=KPASSWD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 20 cn: KPASSWD dn: cn=MEMCACHE,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,d c=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 39 ipaConfigString: enabledService cn: MEMCACHE dn: cn=OTPD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 80 ipaConfigString: enabledService cn: OTPD dn: cn=HTTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 40 ipaConfigString: enabledService cn: HTTP dn: cn=DNS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 30 ipaConfigString: enabledService cn: DNS dn: cn=ADTRUST,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST dn: cn=EXTID,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=e du objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 70 ipaConfigString: enabledService cn: EXTID dn: cn=DNSKeySync,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu ,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: dnssecVersion 1 ipaConfigString: startOrder 110 ipaConfigString: enabledService cn: DNSKeySync dn: cn=NTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 45 ipaConfigString: enabledService cn: NTP dn: cn=KEYS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 41 ipaConfigString: enabledService cn: KEYS This shows up at the bottom of the ipaupgrade.log file while everything before this looks OK from what I can tell: 2017-09-27T17:18:57Z DEBUG request POST http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus> 2017-09-27T17:18:57Z DEBUG request body '' 2017-09-27T17:18:57Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 204, in _httplib_request conn.request(method, uri, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Failed to check CA status: cannot connect to 'http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus>': [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Ensuring that service pki-tomcatd@pki-tomcat is not running while the next set of commands is being executed. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed 2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z DEBUG Service pki-tomcatd@pki-tomcat is not running, continue. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed 2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z INFO [Migrate CRL publish directory] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2017-09-27T17:18:57Z INFO CRL tree already moved 2017-09-27T17:18:57Z INFO [Verifying that CA proxy configuration is correct] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2017-09-27T17:18:57Z DEBUG Proxy configuration up-to-date 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl start pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=1 2017-09-27T17:18:57Z DEBUG stdout= 2017-09-27T17:18:57Z DEBUG stderr=Job for pki-tomcatd@pki-tomcat.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details. 2017-09-27T17:18:57Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-09-27T17:18:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1652, in upgrade_configuration ca.start('pki-tomcat') File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 401, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run raise CalledProcessError(p.returncode, arg_string, str(output)) 2017-09-27T17:18:57Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1 2017-09-27T17:18:57Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details Any thoughts? Is that URL it is requesting to get the status something that is a valid URL that should be responding? I tried with a simple wget and also get connection refused for the response. On Tue, Oct 3, 2017 at 8:13 AM, Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com>> wrote: Kristian Petersen wrote: > That path does not exist. Ok, then you need to describe your installation, particularly what services are enabled. IPA will try to start services based on this search so seeing this output would be useful as well: $ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful. rob > > On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote: > > Kristian Petersen via FreeIPA-users wrote: > > When I recently updated one of my IPA servers (it reports > > 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up > > because pki-tomcatd kept failing. I was able to get it running for now > > by ignoring the failure of that one service, but I haven't been able to > > to determine the cause. The logs are pretty quiet on this one. They > > show the failure itself, but not information that helps me fix the problem. > > You'll need to share what information you have. I'd start by looking at > /var/log/pki/pki-tomcat/ca/debug > > rob > > > > > -- > Kristian Petersen > System Administrator > Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
You mentioned that once before, but that path doesn't seem to exist on my server for some reason. When I go to /var/log/pki i get: -bash-4.2$ cd /var/log/pki/ -bash-4.2$ ls pki-server-upgrade-10.4.1.log pki-upgrade-10.4.1.log server
In a previous reply, I ran a command you asked me to that showed some information about the setup of our IPA server that you had requested that you may need to look at.
On Thu, Oct 19, 2017 at 1:21 AM, Rob Crittenden rcritten@redhat.com wrote:
Kristian Petersen wrote:
I'm still struggling with this one and it seems at least partially responsible for the UI misbehaving as we discussed in another thread. Have you had any new insights regarding this?
I'd start with looking at /var/log/pki/pki-tomcat/ca/debug. You want to find the latest start and work down from there (rather than bottom up).
rob
On Mon, Oct 9, 2017 at 3:54 PM, Kristian Petersen <nesretep@chem.byu.edu mailto:nesretep@chem.byu.edu> wrote:
The installation is a standard RedHat IdM install with DNS, SMB, and CA services installed. The output of the ldapsearch you mentioned is: -bash-4.2$ ldapsearch -LLL -Y GSSAPI -b cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
SASL/GSSAPI authentication started SASL username: nesretep@CHEM.BYU.EDU <mailto:nesretep@CHEM.BYU.EDU> SASL SSF: 56 SASL data security layer installed. dn: cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
ipaMaxDomainLevel: 1 ipaReplTopoManagedSuffix: dc=chem,dc=byu,dc=edu ipaReplTopoManagedSuffix: o=ipaca objectClass: top objectClass: nsContainer objectClass: ipaConfigObject objectClass: ipaSupportedDomainLevelConfig objectClass: ipaReplTopoManagedServer cn: ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> ipaMinDomainLevel: 0 dn: cn=CA,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 50 ipaConfigString: caRenewalMaster cn: CA dn: cn=KDC,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 10 ipaConfigString: enabledService ipaConfigString: kdcProxyEnabled ipaConfigString: pkinitEnabled cn: KDC dn: cn=KPASSWD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 20 cn: KPASSWD dn: cn=MEMCACHE,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,d c=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 39 ipaConfigString: enabledService cn: MEMCACHE dn: cn=OTPD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 80 ipaConfigString: enabledService cn: OTPD dn: cn=HTTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 40 ipaConfigString: enabledService cn: HTTP dn: cn=DNS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 30 ipaConfigString: enabledService cn: DNS dn: cn=ADTRUST,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST dn: cn=EXTID,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=e du objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 70 ipaConfigString: enabledService cn: EXTID
dn: cn=DNSKeySync,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu ,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: dnssecVersion 1 ipaConfigString: startOrder 110 ipaConfigString: enabledService cn: DNSKeySync dn: cn=NTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu
objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 45 ipaConfigString: enabledService cn: NTP dn: cn=KEYS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed
u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 41 ipaConfigString: enabledService cn: KEYS This shows up at the bottom of the ipaupgrade.log file while everything before this looks OK from what I can tell: 2017-09-27T17:18:57Z DEBUG request POST http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus> 2017-09-27T17:18:57Z DEBUG request body '' 2017-09-27T17:18:57Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 204, in _httplib_request conn.request(method, uri, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, increate_connection raise err error: [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Failed to check CA status: cannot connect to 'http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus': [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Ensuring that service pki-tomcatd@pki-tomcat is not running while the next set of commands is being executed. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed
2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z DEBUG Service pki-tomcatd@pki-tomcat is not running, continue. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed 2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z INFO [Migrate CRL publish directory] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2017-09-27T17:18:57Z INFO CRL tree already moved 2017-09-27T17:18:57Z INFO [Verifying that CA proxy configuration is correct] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2017-09-27T17:18:57Z DEBUG Proxy configuration up-to-date 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl start pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=1 2017-09-27T17:18:57Z DEBUG stdout= 2017-09-27T17:18:57Z DEBUG stderr=Job for pki-tomcatd@pki-tomcat.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details. 2017-09-27T17:18:57Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-09-27T17:18:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/ upgrade.py", line 1913, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/ upgrade.py", line 1652, in upgrade_configuration ca.start('pki-tomcat') File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 401, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run raise CalledProcessError(p.returncode, arg_string, str(output))
2017-09-27T17:18:57Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1 2017-09-27T17:18:57Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details Any thoughts? Is that URL it is requesting to get the status something that is a valid URL that should be responding? I tried with a simple wget and also get connection refused for the response. On Tue, Oct 3, 2017 at 8:13 AM, Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com>> wrote: Kristian Petersen wrote: > That path does not exist. Ok, then you need to describe your installation, particularly what services are enabled. IPA will try to start services based on this search so seeing this output would be useful as well: $ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful. rob > > On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com > <mailto:rcritten@redhat.com mailto:rcritten@redhat.com>> wrote: > > Kristian Petersen via FreeIPA-users wrote: > > When I recently updated one of my IPA servers (it reports > > 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up > > because pki-tomcatd kept failing. I was able to get it running for now > > by ignoring the failure of that one service, but I haven't been able to > > to determine the cause. The logs are pretty quiet on this one. They > > show the failure itself, but not information that helps me fix the problem. > > You'll need to share what information you have. I'd start by looking at > /var/log/pki/pki-tomcat/ca/debug > > rob > > > > > -- > Kristian Petersen > System Administrator > Dept. of Chemistry and Biochemistry
-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
Kristian Petersen via FreeIPA-users wrote:
You mentioned that once before, but that path doesn't seem to exist on my server for some reason. When I go to /var/log/pki i get: -bash-4.2$ cd /var/log/pki/ -bash-4.2$ ls pki-server-upgrade-10.4.1.log pki-upgrade-10.4.1.log server
In a previous reply, I ran a command you asked me to that showed some information about the setup of our IPA server that you had requested that you may need to look at.
Then you don't have a CA installed on this host. This is where the logs would be on a 4.5.0 server. You can try something like find /var/log -name debug in case this was an oft-upgraded server and the path is for an older release.
rob
On Thu, Oct 19, 2017 at 1:21 AM, Rob Crittenden <rcritten@redhat.com mailto:rcritten@redhat.com> wrote:
Kristian Petersen wrote: I'm still struggling with this one and it seems at least partially responsible for the UI misbehaving as we discussed in another thread. Have you had any new insights regarding this? I'd start with looking at /var/log/pki/pki-tomcat/ca/debug. You want to find the latest start and work down from there (rather than bottom up). rob On Mon, Oct 9, 2017 at 3:54 PM, Kristian Petersen <nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu> <mailto:nesretep@chem.byu.edu <mailto:nesretep@chem.byu.edu>>> wrote: The installation is a standard RedHat IdM install with DNS, SMB, and CA services installed. The output of the ldapsearch you mentioned is: -bash-4.2$ ldapsearch -LLL -Y GSSAPI -b cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu SASL/GSSAPI authentication started SASL username: nesretep@CHEM.BYU.EDU <mailto:nesretep@CHEM.BYU.EDU> <mailto:nesretep@CHEM.BYU.EDU <mailto:nesretep@CHEM.BYU.EDU>> SASL SSF: 56 SASL data security layer installed. dn: cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu ipaMaxDomainLevel: 1 ipaReplTopoManagedSuffix: dc=chem,dc=byu,dc=edu ipaReplTopoManagedSuffix: o=ipaca objectClass: top objectClass: nsContainer objectClass: ipaConfigObject objectClass: ipaSupportedDomainLevelConfig objectClass: ipaReplTopoManagedServer cn: ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu> ipaMinDomainLevel: 0 dn: cn=CA,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 50 ipaConfigString: caRenewalMaster cn: CA dn: cn=KDC,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 10 ipaConfigString: enabledService ipaConfigString: kdcProxyEnabled ipaConfigString: pkinitEnabled cn: KDC dn: cn=KPASSWD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: enabledService ipaConfigString: startOrder 20 cn: KPASSWD dn: cn=MEMCACHE,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,d c=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 39 ipaConfigString: enabledService cn: MEMCACHE dn: cn=OTPD,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 80 ipaConfigString: enabledService cn: OTPD dn: cn=HTTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 40 ipaConfigString: enabledService cn: HTTP dn: cn=DNS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 30 ipaConfigString: enabledService cn: DNS dn: cn=ADTRUST,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc =edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 60 ipaConfigString: enabledService cn: ADTRUST dn: cn=EXTID,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=e du objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 70 ipaConfigString: enabledService cn: EXTID dn: cn=DNSKeySync,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu ,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: dnssecVersion 1 ipaConfigString: startOrder 110 ipaConfigString: enabledService cn: DNSKeySync dn: cn=NTP,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=edu objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 45 ipaConfigString: enabledService cn: NTP dn: cn=KEYS,cn=ipa1.chem.byu.edu <http://ipa1.chem.byu.edu> <http://ipa1.chem.byu.edu>,cn=masters,cn=ipa,cn=etc,dc=chem,dc=byu,dc=ed u objectClass: ipaConfigObject objectClass: nsContainer objectClass: top ipaConfigString: startOrder 41 ipaConfigString: enabledService cn: KEYS This shows up at the bottom of the ipaupgrade.log file while everything before this looks OK from what I can tell: 2017-09-27T17:18:57Z DEBUG request POST http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus> <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus>> 2017-09-27T17:18:57Z DEBUG request body '' 2017-09-27T17:18:57Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 204, in _httplib_request conn.request(method, uri, body=request_body, headers=headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Failed to check CA status: cannot connect to 'http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus> <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus <http://ipa1.chem.byu.edu:8080/ca/admin/ca/getStatus>>': [Errno 111] Connection refused 2017-09-27T17:18:57Z DEBUG Ensuring that service pki-tomcatd@pki-tomcat is not running while the next set of commands is being executed. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed 2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z DEBUG Service pki-tomcatd@pki-tomcat is not running, continue. 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl is-active pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=3 2017-09-27T17:18:57Z DEBUG stdout=failed 2017-09-27T17:18:57Z DEBUG stderr= 2017-09-27T17:18:57Z INFO [Migrate CRL publish directory] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysupgrade/sysupgrade.state' 2017-09-27T17:18:57Z INFO CRL tree already moved 2017-09-27T17:18:57Z INFO [Verifying that CA proxy configuration is correct] 2017-09-27T17:18:57Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state' 2017-09-27T17:18:57Z DEBUG Proxy configuration up-to-date 2017-09-27T17:18:57Z DEBUG Starting external process 2017-09-27T17:18:57Z DEBUG args=/bin/systemctl start pki-tomcatd@pki-tomcat.service 2017-09-27T17:18:57Z DEBUG Process finished, return code=1 2017-09-27T17:18:57Z DEBUG stdout= 2017-09-27T17:18:57Z DEBUG stderr=Job for pki-tomcatd@pki-tomcat.service failed because the control process exited with error code. See "systemctl status pki-tomcatd@pki-tomcat.service" and "journalctl -xe" for details. 2017-09-27T17:18:57Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually. 2017-09-27T17:18:57Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1652, in upgrade_configuration ca.start('pki-tomcat') File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 401, in start self.service.start(instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start instance_name, capture_output=capture_output, wait=wait) File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 294, in start skip_output=not capture_output) File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 511, in run raise CalledProcessError(p.returncode, arg_string, str(output)) 2017-09-27T17:18:57Z DEBUG The ipa-server-upgrade command failed, exception: CalledProcessError: Command '/bin/systemctl start pki-tomcatd@pki-tomcat.service' returned non-zero exit status 1 2017-09-27T17:18:57Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details Any thoughts? Is that URL it is requesting to get the status something that is a valid URL that should be responding? I tried with a simple wget and also get connection refused for the response. On Tue, Oct 3, 2017 at 8:13 AM, Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote: Kristian Petersen wrote: > That path does not exist. Ok, then you need to describe your installation, particularly what services are enabled. IPA will try to start services based on this search so seeing this output would be useful as well: $ ldapsearch -LLL -Y GSSAPI -b cn=`hostname`,cn=masters,cn=ipa,cn=etc,dc=example,dc=com cn I'd also suggest you look at /var/log/ipaupgrade.log to see if the upgrade was successful. rob > > On Tue, Oct 3, 2017 at 8:03 AM, Rob Crittenden <rcritten@redhat.com <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>>> wrote: > > Kristian Petersen via FreeIPA-users wrote: > > When I recently updated one of my IPA servers (it reports > > 4.5.0-21.el7_4.1.2 in yum), the result was that it could start back up > > because pki-tomcatd kept failing. I was able to get it running for now > > by ignoring the failure of that one service, but I haven't been able to > > to determine the cause. The logs are pretty quiet on this one. They > > show the failure itself, but not information that helps me fix the problem. > > You'll need to share what information you have. I'd start by looking at > /var/log/pki/pki-tomcat/ca/debug > > rob > > > > > -- > Kristian Petersen > System Administrator > Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry -- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry-- Kristian Petersen System Administrator Dept. of Chemistry and Biochemistry
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
freeipa-users@lists.fedorahosted.org