Hi Team,
We have 2 IPA servers in Mater-Master setup are we facing the below issue on these servers.
Isuue1:
Our httpd certificate has expired because of which our IPA1 UI wasn't working, we are getting “loging failed due to an unknown reason” error while we log in to the UI
1. First, the IPA console was not working as httpd service was stopped, httpd was not starting as HTTP certificate is expired. Added NSSEnforceValidCerts off line in nss.conf to start the service.
2. After the change IPA console was loading we are not able to login to the console as pki-tomcatd service was not running,
[root@ipa1 ca]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: STOPPED
ipa-otpd Service: RUNNING
# systemctl status pki-tomcatd@pki-tomcat.service -l
● pki-tomcatd@pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/lib/systemd/system/pki-tomcatd@.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2019-11-05 10:16:50 GMT; 31min ago
Process: 97068 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited, status=0/SUCCESS)
Main PID: 97233 (java)
CGroup: /system.slice/system-pki\x2dtomcatd.slice/pki-tomcatd@pki-tomcat.service
└─97233 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy org.apache.catalina.startup.Bootstrap start
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: WARNING: Exception processing realm com.netscape.cms.tomcat.ProxyRealm@1896e072 background process
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
Nov 05 10:47:57 ipa1.xxx.xxxxx.com server[97233]: at java.lang.Thread.run(Thread.java:748)
This service wasn’t starting with this error
# less /var/log/pki/pki-tomcat/ca/debug
31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca
[31/Oct/2019:13:24:23][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[31/Oct/2019:13:24:23][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host ipa1.xxx.xxxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Internal Database Error encountered: Could not connect to LDAP server host ipa1.xxx.xxx.com port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:676)
at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
# getcert list
Request ID '20180412150739':
status: SUBMITTING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxx.COM',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='CN=ipa1.xxxx.xxxxx.com,O=xxx.xxxxx.COM',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=xxx.xxxxx.COM
subject: CN=ipa1.xxxx.xxxx.com,O=xxx.xxxxx.COM
expires: 2019-10-25 20:16:38 UTC
principal name: krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Issue2:
On the IPA2 server, we are unable to login with the admin user credentials without OTP, but when an AD user is trying to login with 2FA (i.e, password and OTP) we are getting this error "The password you entered is incorrect."
# [root@ipa2 log]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
ipa-otpd Service: STOPPED
ipa: INFO: The ipactl command was successful
# systemctl status ipa-otpd.socket -l
● ipa-otpd.socket - ipa-otpd socket
Loaded: loaded (/usr/lib/systemd/system/ipa-otpd.socket; disabled; vendor preset: disabled)
Active: failed (Result: resources) since Tue 2019-11-05 08:19:04 GMT; 1h 31min ago
Listen: /var/run/krb5kdc/DEFAULT.socket (Stream)
Accepted: 2; Connected: 0
Nov 05 07:42:53 ipa2.xxxx.xxxx.com systemd[1]: Listening on ipa-otpd socket.
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: ipa-otpd.socket failed to queue service startup job (Maybe the service file is missing or not a template unit?): Resource temporarily unavailable
Nov 05 08:19:04 ipa2.xxxx.xxxx.com systemd[1]: Unit ipa-otpd.socket entered failed state.
# cat /usr/lib/systemd/system/ipa-otpd.socket
[Unit]
Description=ipa-otpd socket
[Socket]
ListenStream=/var/run/krb5kdc/DEFAULT.socket
RemoveOnStop=true
SocketMode=0600
Accept=true
[Install]
WantedBy=krb5kdc.service
We see that data replication is broken between the 2 IPA servers, as the changes made on IPA2 is not reflecting on IPA1
We the below errors as well.
IPA1
Nov 05 10:09:23 ipa1.xxx.xxxx.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948563, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxxx.xxxx.com@xxxx.xxxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
Nov 05 10:14:24 ipa1.corp.endurance.com krb5kdc[28021](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) x.x.x.x: ISSUE: authtime 1572948863, etypes {rep=18 tkt=18 ses=18}, ldap/ipa1.xxxx.xxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM
IPA2
# tailf krb5kdc.log
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: NEEDED_PREAUTH: ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxxx.xxxx.COM@xxxx.xxxx.COM, Additional pre-authentication required
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): AS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM for krbtgt/xxx.xxxx.COM@xxx.xxxx.COM
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) y.y.y.y: ISSUE: authtime 1572947965, etypes {rep=18 tkt=18 ses=18}, ldap/ipa2.xxxx.xxxx.com@xxxx.xxxx.COM for ldap/ipa2.xxxx.xxxx.com@xxx.xxxx.COM
Nov 05 09:59:25 ipa2.xxxx.xxxx.com krb5kdc[2451](info): closing down fd 11
Regards
Nikita S