Last FreeIPA master is failing

[Ricardo Mendes]

Hi all,

I'm having serious issues with our FreeIPA setup and I need some direction.

Our FreeIPA setup had two master-replicas. Late last month one of the hypervisors at OVH
died, they replaced hardware but the server is having issues so hasn't come up yet. So
for all matters, one master-replica is dead.
The original master was configured with letsencrypt-freeipa which failed to renew
certificates.

There are around 10 clients connected to it, and several services authenticate against it.
One for example is Gitlab, but I am still able to login to Gitlab. Another example we have
a number of pfSense routers that also use LDAP auth and that always fails we had to
fallback to the local admin user.
One of the most critical services is the DNS. When DNS goes down, everything goes down,
including email. This is currently one of the most critical services.

ipactl always fails. I have to manually start the services using systemctl, like
`systemctl start
{named-pkcs11,httpd,ipa-custodia,ipa-dnskeysyncd,ipa-ods-exporter,ods-enforcerd,krb5kdc,kadmin}`

getcert list returns 7 certificates, all MONITORING, none expired.

# getcert list -d /etc/httpd/alias -n ipaCert
No request found that matched arguments.

I can run ldap commands on the cli.

ALL ipa commands fail:
# ipa userlist
ipa: ERROR: cannot connect to 'any of the configured servers':
https://main.domain.io/ipa/json, https://secondary.domain.io/ipa/json

# certutil -L -d /etc/httpd/alias

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

DSTRootCAX3                                                  C,,  
CN=main.domain.io                                      u,u,u
letsencryptx3                                                C,,  
letsencryptx3                                                C,,  
ISRGRootCAX1                                                 C,,  
DOMAIN.IO IPA CA                                               CT,C,

the ipa-cert-fix command with increased verbosity:

```
ipapython.ipautil: DEBUG: stderr=
ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n
transportCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: transportCert
cert-pki-kra
: PR_FILE_NOT_FOUND_ERROR: File not found

ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n
storageCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: storageCert cert-pki-kra
: PR_FILE_NOT_FOUND_ERROR: File not found

ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/pki/pki-tomcat/alias -L -n
auditSigningCert cert-pki-kra -a -f /etc/pki/pki-tomcat/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: auditSigningCert
cert-pki-kra
: PR_FILE_NOT_FOUND_ERROR: File not found

ipapython.ipautil: DEBUG: Starting external process
ipapython.ipautil: DEBUG: args=/usr/bin/certutil -d dbm:/etc/httpd/alias -L -n Server-Cert
-a -f /etc/httpd/alias/pwdfile.txt
ipapython.ipautil: DEBUG: Process finished, return code=255
ipapython.ipautil: DEBUG: stdout=
ipapython.ipautil: DEBUG: stderr=certutil: Could not find cert: Server-Cert
: PR_FILE_NOT_FOUND_ERROR: File not found

ipapython.admintool: DEBUG:   File
"/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py",
line 100, in run
    certs, extra_certs = expired_certs(now)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py",
line 142, in expired_certs
    return expired_dogtag_certs(now), expired_ipa_certs(now)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_cert_fix.py",
line 191, in expired_ipa_certs
    cert = db.get_cert('Server-Cert')
  File "/usr/lib/python2.7/site-packages/ipapython/certdb.py", line 744, in
get_cert
    raise RuntimeError("Failed to get %s" % nickname)

ipapython.admintool: DEBUG: The ipa-cert-fix command failed, exception: RuntimeError:
Failed to get Server-Cert
ipapython.admintool: ERROR: Failed to get Server-Cert
ipapython.admintool: ERROR: The ipa-cert-fix command failed.
```

I thought this command was to fix the certificates, so I don't get it why it fails if
one certificate is missing.
But anyway, can someone PLEASE give me some help I'm not great with certificates and
I'm not being able to fix this.

If there's a way of creating a new master from start and then importing the data would
be nice, but looking at ipa-backup/restore it clearly says it has to be the same server.