Building production machines out-of-place, regenerating certs when a machine's identity changes, etc.

[Philip Prindeville]

I recently rebuilt a failing mail server (sendmail and cyrus-imapd), replacing the hardware and building the replacement machine offline (leaving the current server in place while I did so).

This would seem normal enough to do, but had some unintended pitfalls that really should be more addressable.

Because the new machine was built offline while the old machine was still in-place, all of the packages that generate certificates--and there are a fair number of them:

sendmail
cyrus-imapd
openssl
openssh
httpd/mod_ssl

did so incorrectly.  That is, they ended up using localhost.localdomain as the identity of the machine... and once I had the machine in place, I brought it up to find that various clients were now complaining about invalid certs.

So I had to dig through what had happened, why, and what needed to be done (or redone) to fix this.

Here are some observations and their accompanying conclusions.

* A lot of packages generate their certs silently as part of the %post scripting

This is certainly true of mod_ssl and cyrus-imapd.

* A lot of packages also provide default answers to the certificate fields:

Running:

# rpm -q --scripts mod_ssl

will get you:

...
at<<  EOF | /usr/bin/openssl req -new -key /etc/pki/tls/private/localhost.key \
          -x509 -days 365 -set_serial $RANDOM \
          -out /etc/pki/tls/certs/localhost.crt 2>/dev/null
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
${FQDN}
root@${FQDN}
EOF
...


or

# rpm -q --scripts cyrus-imapd
...
/bin/cat<<  EOF | make cyrus-imapd.pem
--
SomeState
SomeCity
SomeOrganization
SomeOrganizationalUnit
localhost.localdomain
root at localhost.localdomain
EOF
...

etc.  It seems to be fairly common.

Openssl provides the script  /etc/pki/tls/certs/make-dummy-cert, which contains the function answers():


answers() {
	echo --
	echo SomeState
	echo SomeCity
	echo SomeOrganization
	echo SomeOrganizationalUnit
	echo localhost.localdomain
	echo root at localhost.localdomain
}

which is used to pipe form data to "openssl req" when generating a new certificate request.


This seems to be duplicated in several places, and indeed some of the information could easily be captured statically at install time, and other data gleaned from the running system (such as the last two fields).

* The certificate fields should be consistent

The easiest way to do this is to have a standard file containing the first 5 fields above (C, ST, L, O, OU).  This file could be populated by Anaconda/Kickstart during installation.

It's presence would then be a requirement for other packages that dynamically create certs at install time.

* A permanent or 'real' identity should be an RPM requirement

Packages should have the ability to have the system configured as its ultimate identity before being installed.

This could be accomplished by an RPM pseudo-requirement that some package generates (via Provides:) as part of its installation.

This could be the same package that provides the certificate "seed" information, but would also check to make sure that hostname, the domain, etc. were proper values.

* If a machine gets relocated, it should be able to re-seed its certificates

This last one is perhaps the most obscure change.

If I build a machine "offline", test it with a temporary identity, and decide its "ready for prime-time", I should be able to reboot it with its new identity, and manually re-run the RPM scripts (or some idempotent portion of them) that regenerates all of the identity-derived information... typically this would be based on hostname, domain, IP address, and the certificate seed information (C, ST, L, O, and OU).

We'd need a %post subsection that could be run idempotently to generate new certificates, for instance.

In the examples of the above packages (openssl, mod_ssl, cyrus-imapd, etc) new .crt, .key, or .pem files would be generated... the serial number incremented (where necessary), and the service restarted.

* This affects a lot more than just the packages that generate certs and keys

This also would affect rpm (and possibly yum as well), in addition to the Anaconda/Kickstart.

It might even be necessary to have a new package that runs early at boot time that checks packages (services) for "refreshed" certificates, and disables (chkconfig service off) any services that have out-of-date certificates/keys.


So how far into left field am I on this, anyway?

Thanks,

-Philip