6:51 a.m.
As an update - just in case somebody comes across this thread in the future
I copied the environment to a test rig and performed the surgery as
proposed. And it worked. I was able to promote a new replica.
For those really interested in the details, here's the series of steps I
performed - some steps are slightly edited, so the working directory for
each step might not be quite right, but it's not far off.
mkdir work
cd work
mkdir ds2
cd ds2
tar -zxvf ~/ds2-alias.tgz
cd ..
mkdir new
certutil -N -d new
cp -a /etc/pki/pki-tomcat/alias/pwdfile.txt new
mkdir stage
cd stage
for i in 'caSigningCert cert-pki-ca' 'Server-Cert cert-pki-ca'
'ocspSigningCert cert-pki-ca' 'subsystemCert cert-pki-ca' ; do echo
"$i" ;
pk12util -o "$i" -d /etc/pki/pki-tomcat/alias/ -n "$i" -k
/etc/pki/pki-tomcat/alias/pwdfile.txt -w
/etc/pki/pki-tomcat/alias/pwdfile.txt ; done
pk12util -d ../ds2/etc/pki/pki-tomcat/alias/ -n 'auditSigningCert
cert-pki-ca' -k ../ds2/etc/pki/pki-tomcat/alias/pwdfile.txt -w
/etc/pki/pki-tomcat/alias/pwdfile.txt -o 'auditSigningCert cert-pki-ca'
for i in * ; do pk12util -i "$i" -w ../new/pwdfile.txt -k
../new/pwdfile.txt -d ../new/ -n "$i" ; done
cd ../new
certutil -L -d .
certutil -L -d /etc/pki/pki-tomcat/alias/
certutil -d . -n 'caSigningCert cert-pki-ca' -M -t 'CTu,Cu,Cu'
certutil -d . -n 'auditSigningCert cert-pki-ca' -M -t 'u,u,uP'
certutil -L -d .
cd ..
ipactl stop
systemctl stop certmonger
cd /etc/pki/pki-tomcat/
mv alias alias-old
mv ~/work/new alias
chown -R pkiuser:pkiuser alias
restorecon -R alias
chcon -R -u system_u alias
systemctl start certmonger
ipactl start
I'll perform the same-ish series of steps in production in a maintenance
window in the not too distant future.
I'm still wondering how this might have happened, whether some cosmic event
has corrupted the NSSDB, or ... /shrug
Anyway, I think it's basically fixed.
Regards
A
On 10 July 2018 at 21:54, Andy Stubbs <andy.stubbs(a)treatwell.com> wrote:
Hi
So, I have what I think seems to be a slightly odd problem. And I think
I've worked out what the solution might be - but not the root cause. In any
case, I wanted to run it by you all and see whether you agree or have any
insight into it.
The background
running 6 directory servers 4.5.0-21 on CentOS 7.4.1708, 3 of which have
the CA role. I've been running the directory blissfully uneventfully for
7ish months now. We have experimented a little bit with the CA features,
but nothing that can't be done trivially with the web interface (on
reflection I'm sure it probably is trivial to revoke your primary
certificate authority with the web interface, but you know what I mean).
The problem
In the past few days I've had the occasion to try to create a new replica
but on each attempt, the process fails around this time:
[4/4]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
The ipa-replica-install command failed, exception: HTTPError: 404 Client
Error: Not Found
404 Client Error: Not Found
The ipa-replica-install command failed. See /var/log/ipareplica-install.log
for more information
Now, I've learned a fair amount over the past few days digging into this,
like what ipa-custodia is, and how to poke it.
It seems that at this point, the process is still actually actively doing
things - it appears to be generating some kind of NSS certificate/key
store. And that process is failing, because apparently it can't find the
key for the entry "auditSigningCert cert-pki-ca" - specifically in
custodiainstance.__get_keys the call to cli.fetch_key is failing for this
nickname (but no others).
So, more digging, and I find that yes indeed, the private key appears to
be missing from the cert database on one of the directory servers
(specifically the "first" directory server).
I haven't quite joined the dots on how custodia is working here, but using
the following command:
sudo certutil -L -d /etc/pki/pki-tomcat/alias
I can determine that on the first directory server, the trust attributes
for this cert are ",,P" whereas on the other two CA directory servers, the
trust attributes are "u,u,uP", and that indeed the key is missing from the
first directory server in this database.
I also note that the cert databases seem to be divergent in other ways
between the CA servers. Which I find interesting.
But anyway, so my next action is to copy the cert databases to another
machine and to try to import the cert/key from a "good" CA db to the
"bad"
CA db using pk12util.
This gives me a segmentation fault.
So, I try with a new DB. I export all the cert/key pairs from the "bad" CA
individually and import them into a new DB, replicating the trust
attributes. So far so good. I also export the missing cert/key from a
"good" CA and import that into the same new DB. Also apparently good.
The solution?
So, at this point, I feel relatively confident that I have constructed a
good DB and I should be able to perform some surgery to remove the old
"bad" DB and replace it with this "good" DB.
My questions are:
1. Does this approach seem reasonable or am I oversimplifying?
2. If this is a reasonable approach: what's my best method for performing
the surgery? ipactl stop, move bad db directory out of way, move "good" db
in, don't forget the selinux stuff, then ipactl start again?
3. How could this even happen in the first place? Is it a known issue?
4. Shouldn't the CA databases basically all look the same between servers
created at the same time? Why might they diverge?
5. Do you have any other comments or questions which you feel might be
pertinent?
Thanks in advance for any input or insights shared.
Best Regards
Andy
--
<https://www.treatwell.com/>
Andrew Stubbs, PhD
Head of Technical Operations
treatwell.co.uk
--
<https://www.treatwell.com/>
Andrew Stubbs, PhD
Head of Technical Operations
+44 203 770 4582
+44 7711 002930
treatwell.co.uk