Hi Rob,
I was able to start my CA via instructions from here:
https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html
I also tried to set the clock back and restart certmonger. Still no luck:
getcert list gives me the following:
Number of certificates and requests being tracked: 16.
Request ID '20171205153653':
status: MONITORING
stuck: no
key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=freeipa.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-11-09 10:39:35 UTC
principal name: krbtgt/CORP.MYDOMAIN.DE(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-pkinit-KPKdc
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
track: yes
auto-renew: yes
Request ID '20180912151607':
status: CA_UNREACHABLE
ca-error: Error 35 connecting to
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
connect error.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=CA Audit,O=CORP.MYDOMAIN.DE
expires: 2019-11-25 15:31:41 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180912151608':
status: CA_UNREACHABLE
ca-error: Error 35 connecting to
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
connect error.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=OCSP Subsystem,O=CORP.MYDOMAIN.DE
expires: 2019-11-25 15:31:40 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180912151609':
status: CA_UNREACHABLE
ca-error: Error 35 connecting to
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
connect error.
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=CA Subsystem,O=CORP.MYDOMAIN.DE
expires: 2019-11-25 15:31:41 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180912151610':
status: NEED_CSR_GEN_PIN
ca-error: Error 35 connecting to
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
connect error.
stuck: yes
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
expires: 2037-12-05 15:31:39 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180912151611':
status: CA_UNREACHABLE
ca-error: Error 35 connecting to
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
connect error.
stuck: no
key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=IPA RA,O=CORP.MYDOMAIN.DE
expires: 2019-11-25 15:32:12 UTC
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20180912151612':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=dmud,O=CORP.MYDOMAIN.DE
expires: 2021-10-29 09:40:17 UTC
email: dmud(a)corp.mydomain.de
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20180912151613':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/dirsrv/slapd-CORP-MYDOMAIN-DE',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/dirsrv/slapd-CORP-MYDOMAIN-DE/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-CORP-MYDOMAIN-DE',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=freeipa.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-11-09 10:39:45 UTC
dns: freeipa.corp.mydomain.de
principal name: ldap/freeipa.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv
CORP-MYDOMAIN-DE
track: yes
auto-renew: yes
Request ID '20180912151615':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=freeipa.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-11-09 10:40:05 UTC
dns: freeipa.corp.mydomain.de
principal name: HTTP/freeipa.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Request ID '20190212162113':
status: MONITORING
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/mail.corp.mydomain.de.ley'
certificate:
type=FILE,location='/etc/pki/tls/certs/mail.corp.mydomain.de.crt'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=mail.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-02-12 16:21:14 UTC
dns: mail.corp.mydomain.de
principal name: SMTP/mail.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191017155747':
status: MONITORING
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/mtls.time-series-analytics-stage.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/mtls.time-series-analytics-stage.corp.mydomain.de.crt'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=mtls.time-series-analytics-stage.corp.mydomain.de,O=
CORP.MYDOMAIN.DE
expires: 2021-10-17 15:57:49 UTC
dns: mtls.time-series-analytics-stage.corp.mydomain.de
principal name: MTLS/
mtls.time-series-analytics-stage.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191026094947':
status: MONITORING
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/nas-smicro.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/nas-smicro.corp.mydomain.de.crt'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=nas-smicro.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-10-26 09:49:48 UTC
dns: nas-smicro.corp.mydomain.de
principal name: HTTPS/nas-smicro.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191026102844':
status: MONITORING
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/pe.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/pe.corp.mydomain.de.crt'
CA: IPA
issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
subject: CN=pe.corp.mydomain.de,O=CORP.MYDOMAIN.DE
expires: 2021-10-26 10:28:45 UTC
dns: pe.corp.mydomain.de
principal name: HTTPS/pe.corp.mydomain.de(a)CORP.MYDOMAIN.DE
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191027134809':
status: CA_UNREACHABLE
ca-error: Server at
https://freeipa.corp.mydomain.de/ipa/xml failed
request, will retry: 907 (RPC failed at server. cannot connect to '
https://freeipa.corp.mydomain.de:443/ca/rest/account/login';: [SSL:
SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822)).
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/lb.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/lb.corp.mydomain.de.crt'
CA: IPA
issuer:
subject:
expires: unknown
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191027135053':
status: CA_REJECTED
ca-error: Server at
https://freeipa.corp.mydomain.de/ipa/xml denied
our request, giving up: 3009 (RPC failed at server. invalid 'csr':
hostname in subject of request '*.lb.corp.mydomain.de' does not match name
or aliases of principal 'HTTP/lb-vmnet.lb.corp.mydomain.de(a)CORP.MYDOMAIN.DE
').
stuck: yes
key pair storage:
type=FILE,location='/etc/pki/tls/private/lb-vmnet.lb.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/lb-vmnet.lb.corp.mydomain.de.crt'
CA: IPA
issuer:
subject:
expires: unknown
pre-save command:
post-save command:
track: yes
auto-renew: yes
Request ID '20191027135738':
status: CA_UNREACHABLE
ca-error: Server at
https://freeipa.corp.mydomain.de/ipa/xml failed
request, will retry: 907 (RPC failed at server. cannot connect to '
https://freeipa.corp.mydomain.de:443/ca/rest/account/login';: [SSL:
SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1822)).
stuck: no
key pair storage:
type=FILE,location='/etc/pki/tls/private/vm-net.lb.corp.mydomain.de.key'
certificate:
type=FILE,location='/etc/pki/tls/certs/vm-net.lb.corp.mydomain.de.crt'
CA: IPA
issuer:
subject:
expires: unknown
pre-save command:
post-save command:
track: yes
auto-renew: yes
/var/log/pki/pki-tomcat/ca/debug:
[14/Feb/2020:15:09:36][http-bio-8080-exec-13]: according to ccMode,
authorization for servlet: caProfileList is LDAP based, not XML {1}, use
default authz mgr: {2}.
[14/Feb/2020:15:09:36][http-bio-8080-exec-13]: according to ccMode,
authorization for servlet: caProfileList is LDAP based, not XML {1}, use
default authz mgr: {2}.
[14/Feb/2020:15:10:12][http-bio-8443-exec-13]: SignedAuditLogger: event
ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:12][http-bio-8443-exec-13]: LogFile: event type not
selected: ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:22][http-bio-8443-exec-14]: SignedAuditLogger: event
ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:22][http-bio-8443-exec-14]: LogFile: event type not
selected: ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:32][http-bio-8443-exec-15]: SignedAuditLogger: event
ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:32][http-bio-8443-exec-15]: LogFile: event type not
selected: ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:42][http-bio-8443-exec-16]: SignedAuditLogger: event
ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:10:42][http-bio-8443-exec-16]: LogFile: event type not
selected: ACCESS_SESSION_ESTABLISH
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: About to start
updateCertStatus
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: Starting updateCertStatus
(entered lock)
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In updateCertStatus()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]:
getInvalidCertificatesByNotBeforeDate filter (certStatus=INVALID)
[14/Feb/2020:15:13:15][CertStatusUpdateTask]:
getInvalidCertificatesByNotBeforeDate: about to call findCertRecordsInList
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 3
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
findCertRecordsInListRawJumpto with Jumpto 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In DBVirtualList filter attrs
startFrom sortKey pageSize filter: (certStatus=INVALID) attrs:
[objectclass, certRecordId, x509cert] pageSize -200 startFrom
20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
getInvalidCertsByNotBeforeDate finally.
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 5
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: searching for
entry 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList.getEntries()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: entries: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: top: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: size: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: index may be empty
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getValidCertsByNotAfterDate
filter (certStatus=VALID)
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 3
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
findCertRecordsInListRawJumpto with Jumpto 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In DBVirtualList filter attrs
startFrom sortKey pageSize filter: (certStatus=VALID) attrs: [objectclass,
certRecordId, x509cert] pageSize -200 startFrom 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 5
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: searching for
entry 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList.getEntries()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: entries: 1
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: top: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: size: 1
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: transidValidCertificates:
list size: 1
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: transitValidCertificates:
ltSize 1
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: Record does not
qualify,notAfter Fri Oct 29 11:40:17 CEST 2021 date Fri Feb 14 15:13:15 CET
2020
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: transitCertList EXPIRED
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]:
getRevokedCertificatesByNotAfterDate filter (certStatus=REVOKED)
[14/Feb/2020:15:13:15][CertStatusUpdateTask]:
getRevokedCertificatesByNotAfterDate: about to call findCertRecordsInList
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: masterConn is connected: true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: getConn: mNumConns now 3
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In
findCertRecordsInListRawJumpto with Jumpto 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: In DBVirtualList filter attrs
startFrom sortKey pageSize filter: (certStatus=REVOKED) attrs:
[objectclass, certRevokedOn, certRecordId, certRevoInfo, notAfter,
x509cert] pageSize -200 startFrom 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 4
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: returnConn: mNumConns now 5
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: searching for
entry 20200214151315Z
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList.getEntries()
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: entries: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: top: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: DBVirtualList: size: 0
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: index may be empty
[14/Feb/2020:15:13:15][CertStatusUpdateTask]: updateCertStatus done
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: About to start
updateSerialNumbers
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Starting
updateSerialNumbers (entered lock)
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: CertificateRepository:
updateCounter mEnableRandomSerialNumbers=false mCounter=-1
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: In
LdapBoundConnFactory::getConn()
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: masterConn is connected:
true
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: getConn: conn is connected
true
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: getConn: mNumConns now 4
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Releasing ldap connection
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: returnConn: mNumConns now 5
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: DBSubsystem:
getEntryAttribute: dn=ou=certificateRepository, ou=ca, o=ipaca
attr=description:;
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: CertificateRepository:
updateCounter mEnableRandomSerialNumbers=false
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: CertificateRepository:
updateCounter CertificateRepositoryMode =
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: CertificateRepository:
updateCounter modeChange=false
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: CertificateRepository:
UpdateCounter mEnableRandomSerialNumbers=false mCounter=-1
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Starting cert checkRanges
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Repository: Server not
completely started. Returning ..
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Starting request checkRanges
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: Repository: Server not
completely started. Returning ..
[14/Feb/2020:15:13:15][SerialNumberUpdateTask]: updateSerialNumbers done
I'm really stuck now.
чт, 13 февр. 2020 г. в 15:58, Dmitri Moudraninets <
dmitry.a.moudraninets(a)gmail.com>:
Hi Rob,
I found this on my second server in /var/log/pki/pki-tomcat/ca/debug:
SSL handshake happened
Could not connect to LDAP server host freeipa-02.corp.mydomain.de port
636 Error netscape.ldap.LDAPException: Authenticatio
n failed (48)
On my primary server I found this:
Internal Database Error encountered: Could not connect to LDAP server host
freeipa-02.corp.mydomain.de port 636 Error netscape.ldap.LDAPException:
Unable to create soc
ket: java.net.UnknownHostException: freeipa-02.corp.mydomain.de: Name or
service not known (-1)
Looks like that it was unable to resolve the name of the second host (why
primary host is connecting to secondary?). I added an entry to hosts file
but still CA does not start.
ср, 12 февр. 2020 г. в 07:58, Dmitri Moudraninets <
dmitry.a.moudraninets(a)gmail.com>:
> Hi Rob,
>
> What cat I do to troubleshoot CA?
>
> On Wed 12. Feb 2020 at 01:00, Rob Crittenden <rcritten(a)redhat.com> wrote:
>
>> Dmitri Moudraninets wrote:
>> > Hi Rob,
>> >
>> >
>> > It seems that it does not help. I found a backup which was made via
>> > ipa-backup this summer. Can I use it somehow for recovery? We did
>> > nothing to certificates since that time. We only added
>> users/groups/servers.
>> >
>> > Current situation:
>> > I can't update certificates. getcert list shows multiple certificates
>> > with CA_UNREACHABLE status:
>> > status: CA_UNREACHABLE
>> > ca-error: Error 35 connecting to
>> >
https://freeipa.corp.mydomain.de:8443/ca/agent/ca/profileReview: SSL
>> > connect error.
>> >
>> >
>> > pki-tomcatd is not starting:
>> > [root@freeipa ipa]# ipactl start --ignore-service-failures
>> > Starting Directory Service
>> > Starting krb5kdc Service
>> > Starting kadmin Service
>> > Starting named Service
>> > Starting httpd Service
>> > Starting ipa-custodia Service
>> > Starting ntpd Service
>> > Starting pki-tomcatd Service
>> > Failed to start pki-tomcatd Service
>> > Forced start, ignoring pki-tomcatd Service, continuing normal operation
>> > Starting smb Service
>> > Starting winbind Service
>> > Starting ipa-otpd Service
>> > Starting ipa-dnskeysyncd Service
>> > ipa: INFO: The ipactl command was successful
>>
>> The CA was working previously, what exactly did you do? Changing the RA
>> cert would in no way affect the startup of the CA. I'd carefully review
>> your shell history to see what you did and check the CA logs to see why
>> it won't start up.
>>
>> Of course the CA is unreachable if it hasn't started, this error is
>> expected. You can't debug a CA not starting up via certmonger as it is
>> just a client (and in some cases uses the previously broken RA cert for
>> communication).
>>
>> So get the CA starting up first, then tackle the RA cert/key.
>>
>> rob
>> >
>> > пн, 25 нояб. 2019 г. в 15:47, Rob Crittenden <rcritten(a)redhat.com
>> > <mailto:rcritten@redhat.com>>:
>> >
>> > Dmitri Moudraninets wrote:
>> > > Hi Rob,
>> > >
>> > > I recovered the key file. Restarted FreeIPA and certmonger. Now
>> issue
>> > > looks different:
>> > > image.png
>> > >
>> > > Subjects disappeared. If I click on a certificate 29 I see this:
>> > > cannot connect to
>> > > '
>>
https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial';:
>> > > [Errno 13] Permission denied
>> >
>> > Set the same ownership/permissions on the key as you did the cert
>> and
>> > run restorecon on it.
>> >
>> > rob
>> >
>> > >
>> > > пн, 25 нояб. 2019 г. в 13:58, Rob Crittenden <
>> rcritten(a)redhat.com
>> > <mailto:rcritten@redhat.com>
>> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>:
>> > >
>> > > Dmitri Moudraninets wrote:
>> > > > Hi Rob,
>> > > >
>> > > >
>> > > >
>> > > > I did the following:
>> > > > I removed original ra-agent.pem and ra-agent key
>> > > > and
>> > > > openssl x509 -in /root/debug.cert -out
>> /var/lib/ipa/ra-agent.pem
>> > > > chown root:ipaapi /var/lib/ipa/ra-agent.pem
>> > > > chmod 0440 /var/lib/ipa/ra-agent.pem
>> > > > restorecon /var/lib/ipa/ra-agent.pem
>> > >
>> > > You removed the key!? I sure hope you have a backup of it.
>> > >
>> > > Put it back and I think that will resolve things.
>> > >
>> > > >
>> > > > Successfully restarted FreeIPA:
>> > > > Directory Service: RUNNING
>> > > > krb5kdc Service: RUNNING
>> > > > kadmin Service: RUNNING
>> > > > named Service: RUNNING
>> > > > httpd Service: RUNNING
>> > > > ipa-custodia Service: RUNNING
>> > > > ntpd Service: RUNNING
>> > > > pki-tomcatd Service: RUNNING
>> > > > smb Service: RUNNING
>> > > > winbind Service: RUNNING
>> > > > ipa-otpd Service: RUNNING
>> > > > ipa-dnskeysyncd Service: RUNNING
>> > > > ipa: INFO: The ipactl command was successful
>> > >
>> > > The agent cert is not required for the CA to operate.
>> > >
>> > > > Now GUI shows different error:
>> > > > cannot connect to
>> > > >
>> > 'https://freeipa.corp.mydomain.de:443/ca/agent/ca/displayBySerial
>> ':
>> > > > [Errno 2] No such file or directory
>> > > >
>> > > >
>> > > > [root@freeipa ~]# getcert list -f
>> /var/lib/ipa/ra-agent.pem
>> > > > Number of certificates and requests being tracked: 16.
>> > > > Request ID '20180912151611':
>> > > > status: NEED_CSR
>> > > > stuck: no
>> > > > key pair storage:
>> type=FILE,location='/var/lib/ipa/ra-agent.key'
>> > > > certificate:
type=FILE,location='/var/lib/ipa/ra-agent.pem'
>> > > > CA: dogtag-ipa-ca-renew-agent
>> > > > issuer: CN=Certificate Authority,O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>
>> > > > subject: CN=IPA RA,O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > expires: 2019-11-25 15:32:12 UTC
>> > > > key usage:
>> > >
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> > > > eku: id-kp-serverAuth,id-kp-clientAuth
>> > > > pre-save command:
>> /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>> > > > post-save command:
>> /usr/libexec/ipa/certmonger/renew_ra_cert
>> > > > track: yes
>> > > > auto-renew: yes
>> > >
>> > > This shows that the certificate has the right subject now
>> > which is good
>> > > but you removed its private key so it won't work.
>> > >
>> > > rob
>> > >
>> > > >
>> > > > сб, 23 нояб. 2019 г. в 20:26, Rob Crittenden
>> > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
>> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>
>> > > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
>> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>>:
>> > > >
>> > > > Dmitri Moudraninets wrote:
>> > > > > Hi Rob,
>> > > > >
>> > > > > ldapsearch -LLL -o ldif-wrap=no -x -D
'cn=directory
>> > manager' -W
>> > > > > -b uid=ipara,ou=People,o=ipaca usercertificate
>> > > > >
>> > > > > shows me the following:
>> > > > >
>> > > > > Issuer: O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>,
>> > > > > CN=Certificate Authority
>> > > > > Validity
>> > > > > Not Before: Dec 5 15:32:12 2017 GMT
>> > > > > Not After : *Nov 25 15:32:12 2019*
GMT
>> > > > >
>> > > > > It's going to expire on Monday. Can it be a
problem?
>> > > >
>> > > > You didn't provide the cert subject so I can't
be sure
>> > this is
>> > > the right
>> > > > cert. If it contains CN = IPA RA then it is.
>> > > >
>> > > > And yes, it expires in two days. What you'd need
to do
>> is
>> > > restore it per
>> > > > my previous instruction into
/var/lib/ipa/ra-agent.pem
>> > on the
>> > > renewal
>> > > > master (ipa config-show to see which one it is).
>> > > >
>> > > > Then run:
>> > > >
>> > > > # getcert resubmit -f /var/lib/ipa/ra-agent.pem
>> > > >
>> > > > That should renew the cert.
>> > > >
>> > > > On the other masters I'd run the same command and
that
>> > may fix
>> > > things
>> > > > there as well.
>> > > >
>> > > > rob
>> > > >
>> > > > > I tried this command:
>> > > > > openssl x509 -text -in /var/lib/ipa/ra-agent.pem
>> > > > >
>> > > > > and it shows the following:
>> > > > > Certificate:
>> > > > > Data:
>> > > > > Version: 3 (0x2)
>> > > > > Serial Number: 28 (0x1c)
>> > > > > Signature Algorithm: sha256WithRSAEncryption
>> > > > > Issuer: O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE> <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>,
>> > > > > CN=Certificate Authority
>> > > > > Validity
>> > > > > Not Before: Oct 29 10:39:47 2019 GMT
>> > > > > Not After : Oct 29 09:39:47 2021 GMT
>> > > > > Subject: O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
<
http://CORP.MYDOMAIN.DE>, CN=dmud
>> > > > > Subject Public Key Info:
>> > > > > Public Key Algorithm: rsaEncryption
>> > > > > Public-Key: (2048 bit)
>> > > > > Modulus:
>> > > > >
>> > > 00:ba:09:81:99:9b:17:99:07:5a:10:28:c8:7a:03:
>> > > > > ...
>> > > > >
>> > > 18:db:02:ce:b4:66:ce:5a:e9:12:af:d3:da:bf:f7:
>> > > > > 66:5f
>> > > > > Exponent: 65537 (0x10001)
>> > > > > X509v3 extensions:
>> > > > > X509v3 Authority Key Identifier:
>> > > > > keyid:D2:...70:BF
>> > > > >
>> > > > > X509v3 Subject Key Identifier:
>> > > > > DE:...:51:0A
>> > > > > X509v3 Subject Alternative Name:
>> > > > > email:dmud@corp.mydomain.de
>> > <mailto:email%3Admud@corp.mydomain.de>
>> > > <mailto:email%3Admud@corp.mydomain.de
>> > <mailto:email%253Admud@corp.mydomain.de>>
>> > > > <mailto:email%3Admud@corp.mydomain.de
>> > <mailto:email%253Admud@corp.mydomain.de>
>> > > <mailto:email%253Admud@corp.mydomain.de
>> > <mailto:email%25253Admud@corp.mydomain.de>>>
>> > > > > <mailto:email%3Admud@corp.mydomain.de
>> > <mailto:email%253Admud@corp.mydomain.de>
>> > > <mailto:email%253Admud@corp.mydomain.de
>> > <mailto:email%25253Admud@corp.mydomain.de>>
>> > > > <mailto:email%253Admud@corp.mydomain.de
>> > <mailto:email%25253Admud@corp.mydomain.de>
>> > > <mailto:email%25253Admud@corp.mydomain.de
>> > <mailto:email%2525253Admud@corp.mydomain.de>>>>
>> > > > > Authority Information Access:
>> > > > > OCSP -
>> > > URI:http://ipa-ca.corp.mydomain.de/ca/ocsp
>> > > > >
>> > > > >
>> > > > > I did nothing to /var/lib/ipa/ra-agent.pem yet.
>> > > > >
>> > > > >
>> > > > > чт, 21 нояб. 2019 г. в 16:54, Rob Crittenden
>> > > <rcritten(a)redhat.com <mailto:rcritten@redhat.com>
>> > <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>
>> > > > <mailto:rcritten@redhat.com <mailto:
>> rcritten(a)redhat.com>
>> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>
>> > > > > <mailto:rcritten@redhat.com
>> > <mailto:rcritten@redhat.com> <mailto:rcritten@redhat.com
>> > <mailto:rcritten@redhat.com>>
>> > > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>
>> > <mailto:rcritten@redhat.com
<mailto:rcritten@redhat.com>>>>>:
>> > > > >
>> > > > > Dmitri Moudraninets wrote:
>> > > > > > Hi Rob,
>> > > > > >
>> > > > > > Yes both masters are failing the same
way.
>> Output
>> > > of openssl
>> > > > x509
>> > > > > -noout
>> > > > > > -modulus -in /var/lib/ipa/ra-agent.pem
is the
>> > same on both
>> > > > masters.
>> > > > > > Output of openssl rsa -noout -modulus
-in
>> > > > /var/lib/ipa/ra-agent.key is
>> > > > > > also the same on both masters. But the
output
>> of
>> > the first
>> > > > command is
>> > > > > > not the same as the output of the
second
>> command.
>> > > > > >
>> > > > > > I can't remember that I troubleshoot
any other
>> > > problems but we
>> > > > > tried to
>> > > > > > generate some personal certificates for
some
>> users.
>> > > Also we
>> > > > tried to
>> > > > > > generate certificates with key files for
some
>> of our
>> > > internal
>> > > > > services.
>> > > > > > We did that for the first time and it
worked
>> at the
>> > > end. Also I
>> > > > > changed
>> > > > > > the admin password not so long ago.
>> > > > > >
>> > > > > >
>> > > > > > Below you can find the output of the
requested
>> > commands:
>> > > > > >
>> > > > > >
>> > > > > > [root@second_master ~]# getcert list -f
>> > > > /var/lib/ipa/ra-agent.pem
>> > > > > > Number of certificates and requests
being
>> > tracked: 9.
>> > > > > > Request ID '20180912151730':
>> > > > > > status: MONITORING
>> > > > > > stuck: no
>> > > > > > key pair storage:
>> > > type=FILE,location='/var/lib/ipa/ra-agent.key'
>> > > > > > certificate:
>> > > type=FILE,location='/var/lib/ipa/ra-agent.pem'
>> > > > > > CA: dogtag-ipa-ca-renew-agent
>> > > > > > issuer: CN=Certificate
>> > Authority,O=CORP.MYDOMAIN.DE <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>
>> > > > > <
http://CORP.MYDOMAIN.DE>
>> > > > > > <
http://CORP.MYDOMAIN.DE>
>> > > > > > subject: CN=dmud,O=CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>
<
http://CORP.MYDOMAIN.DE>
>> > > > > <
http://CORP.MYDOMAIN.DE>
>> > > > > >
*<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
I see a
>> > username here.
>> > > > Does it have
>> > > > > > to be like that?*
>> > > > > > expires: 2021-10-29 09:39:47 UTC
>> > > > > > email: dmud(a)corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de
>> >>
>> > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de
>> >>>
>> > > > <mailto:dmud@corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de>>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>
>> > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>>>
>> > > > > <mailto:dmud@corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de
>> >>
>> > <mailto:dmud@corp.mydomain.de <mailto:dmud@corp.mydomain.de>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de
>> >>>
>> > > > <mailto:dmud@corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de> <mailto:dmud@corp.mydomain.de
>> > <mailto:dmud@corp.mydomain.de>>
>> > > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>
>> > <mailto:dmud@corp.mydomain.de
<mailto:dmud@corp.mydomain.de>>>>>
>> > > > > > key usage:
>> > > > >
>> > >
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> > > > > > eku: id-kp-serverAuth,id-kp-clientAuth
>> > > > > > pre-save command:
>> > > /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>> > > > > > post-save command:
>> > > /usr/libexec/ipa/certmonger/renew_ra_cert
>> > > > > > track: yes
>> > > > > > auto-renew: yes
>> > > > >
>> > > > > Right, someone overwrote the RA agent
>> certificate.
>> > > > >
>> > > > > Look to see if the user entry in the CA has
the
>> > right cert:
>> > > > >
>> > > > > $ ldapsearch -LLL -o ldif-wrap=no -x -D
>> 'cn=directory
>> > > manager'
>> > > > -W -b
>> > > > > uid=ipara,ou=People,o=ipaca usercertificate
>> > > > >
>> > > > > Put the base64 value of the usercertificate
>> > attribute into a
>> > > > file and
>> > > > > add a prefix/suffix around it:
>> > > > >
>> > > > > -----BEGIN CERTIFICATE-----
>> > > > > MII....blah=
>> > > > > -----END CERTIFICATE-----
>> > > > >
>> > > > > $ openssl x509 -text -in /path/to/file
>> > > > >
>> > > > > If the Subject is O = CORP.MYDOMAIN.DE
>> > <
http://CORP.MYDOMAIN.DE>
>> > > <
http://CORP.MYDOMAIN.DE>
>> > > > <
http://CORP.MYDOMAIN.DE>
<
http://CORP.MYDOMAIN.DE>,
>> CN
>> > > > > = IPA RA then that's a good
>> > > > > start. Also look at the expires date to be
sure
>> it is
>> > > still valid.
>> > > > >
>> > > > > Assuming that is ok then re-run the openssl
>> > modulus commands
>> > > > to ensure
>> > > > > they are the same.
>> > > > >
>> > > > > Assuming that too is ok then you have the
proper,
>> > valid RA
>> > > > agent cert.
>> > > > > In that case I'd move the current file
out of the
>> > way, who
>> > > > knows what it
>> > > > > is, then run:
>> > > > >
>> > > > > # openssl x509 -in /path/to/file -out
>> > > > /var/lib/ipa/ra-agent.pem (just to
>> > > > > properly format the agent cert)
>> > > > > # chown root:ipaapi
/var/lib/ipa/ra-agent.pem
>> > > > > # chmod 0440 /var/lib/ipa/ra-agent.pem
>> > > > > # restorecon /var/lib/ipa/ra-agent.pem
>> > > > >
>> > > > > Then try something like: ipa cert-show 1
>> > > > >
>> > > > > This will exercise the RA agent cert and as
long
>> > as you
>> > > don't
>> > > > get an
>> > > > > error back things are working again.
>> > > > >
>> > > > > The cert is common among all masters so you
can
>> > copy the
>> > > file
>> > > > to your
>> > > > > other master(s), ensuring proper ownership,
>> > permissions and
>> > > > SELinux
>> > > > > context.
>> > > > >
>> > > > > rob
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > WBR
>> > > > > Dmitry
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > With best regards/Mit freundlichen Grüßen
>> > > >
>> > > > Moudraninets Dmitry, RHCSA
>> > > >
http://www.linkedin.com/in/moudraninets
>> > > >
http://www.xing.com/profile/Dmitry_Mudraninets
>> > >
>> > >
>> > >
>> > > --
>> > > With best regards/Mit freundlichen Grüßen
>> > >
>> > > Moudraninets Dmitry, RHCSA
>> > >
http://www.linkedin.com/in/moudraninets
>> > >
http://www.xing.com/profile/Dmitry_Mudraninets
>> >
>> >
>> >
>> > --
>> > With best regards/Mit freundlichen Grüßen
>> >
>> > Moudraninets Dmitry, RHCSA
>> >
http://www.linkedin.com/in/moudraninets
>> >
http://www.xing.com/profile/Dmitry_Mudraninets
>>
>> --
> With best regards/Mit freundlichen Grüßen
>
> Moudraninets Dmitry, RHCSA
>
http://www.linkedin.com/in/moudraninets
>
http://www.xing.com/profile/Dmitry_Mudraninets
>
--
With best regards/Mit freundlichen Grüßen
Moudraninets Dmitry, RHCSA
http://www.linkedin.com/in/moudraninets
http://www.xing.com/profile/Dmitry_Mudraninets
--
With best regards/Mit freundlichen Grüßen
Moudraninets Dmitry, RHCSA
http://www.linkedin.com/in/moudraninets
http://www.xing.com/profile/Dmitry_Mudraninets