Building production machines out-of-place, regenerating certs when a machine's identity changes, etc.

[nodata]

On 26/11/10 23:47, Philip Prindeville wrote:
> I recently rebuilt a failing mail server (sendmail and cyrus-imapd), replacing the hardware and building the replacement machine offline (leaving the current server in place while I did so).
>
> This would seem normal enough to do, but had some unintended pitfalls that really should be more addressable.
>
> Because the new machine was built offline while the old machine was still in-place, all of the packages that generate certificates--and there are a fair number of them:
>
> sendmail
> cyrus-imapd
> openssl
> openssh
> httpd/mod_ssl
>
> did so incorrectly.  That is, they ended up using localhost.localdomain as the identity of the machine... and once I had the machine in place, I brought it up to find that various clients were now complaining about invalid certs.
>
> So I had to dig through what had happened, why, and what needed to be done (or redone) to fix this.
>
> Here are some observations and their accompanying conclusions.
>
> * A lot of packages generate their certs silently as part of the %post scripting
>
> This is certainly true of mod_ssl and cyrus-imapd.
>
> * A lot of packages also provide default answers to the certificate fields:
>
> Running:
>
> # rpm -q --scripts mod_ssl
>
> will get you:
>
> ...
> at<<   EOF | /usr/bin/openssl req -new -key /etc/pki/tls/private/localhost.key \
>            -x509 -days 365 -set_serial $RANDOM \
>            -out /etc/pki/tls/certs/localhost.crt 2>/dev/null
> --
> SomeState
> SomeCity
> SomeOrganization
> SomeOrganizationalUnit
> ${FQDN}
> root@${FQDN}
> EOF
> ...
>
>
> or
>
> # rpm -q --scripts cyrus-imapd
> ...
> /bin/cat<<   EOF | make cyrus-imapd.pem
> --
> SomeState
> SomeCity
> SomeOrganization
> SomeOrganizationalUnit
> localhost.localdomain
> root at localhost.localdomain
> EOF
> ...
>
> etc.  It seems to be fairly common.
>
> Openssl provides the script  /etc/pki/tls/certs/make-dummy-cert, which contains the function answers():
>
>
> answers() {
> 	echo --
> 	echo SomeState
> 	echo SomeCity
> 	echo SomeOrganization
> 	echo SomeOrganizationalUnit
> 	echo localhost.localdomain
> 	echo root at localhost.localdomain
> }
>
> which is used to pipe form data to "openssl req" when generating a new certificate request.
>
>
> This seems to be duplicated in several places, and indeed some of the information could easily be captured statically at install time, and other data gleaned from the running system (such as the last two fields).
>
> * The certificate fields should be consistent
>
> The easiest way to do this is to have a standard file containing the first 5 fields above (C, ST, L, O, OU).  This file could be populated by Anaconda/Kickstart during installation.
>
> It's presence would then be a requirement for other packages that dynamically create certs at install time.
>
> * A permanent or 'real' identity should be an RPM requirement
>
> Packages should have the ability to have the system configured as its ultimate identity before being installed.
>
> This could be accomplished by an RPM pseudo-requirement that some package generates (via Provides:) as part of its installation.
>
> This could be the same package that provides the certificate "seed" information, but would also check to make sure that hostname, the domain, etc. were proper values.
>
> * If a machine gets relocated, it should be able to re-seed its certificates
>
> This last one is perhaps the most obscure change.
>
> If I build a machine "offline", test it with a temporary identity, and decide its "ready for prime-time", I should be able to reboot it with its new identity, and manually re-run the RPM scripts (or some idempotent portion of them) that regenerates all of the identity-derived information... typically this would be based on hostname, domain, IP address, and the certificate seed information (C, ST, L, O, and OU).
>
> We'd need a %post subsection that could be run idempotently to generate new certificates, for instance.
>
> In the examples of the above packages (openssl, mod_ssl, cyrus-imapd, etc) new .crt, .key, or .pem files would be generated... the serial number incremented (where necessary), and the service restarted.
>
> * This affects a lot more than just the packages that generate certs and keys
>
> This also would affect rpm (and possibly yum as well), in addition to the Anaconda/Kickstart.
>
> It might even be necessary to have a new package that runs early at boot time that checks packages (services) for "refreshed" certificates, and disables (chkconfig service off) any services that have out-of-date certificates/keys.
>
>
> So how far into left field am I on this, anyway?
>
> Thanks,
>
> -Philip
>
>

I don't agree. If you are replacing a production machine, you take the 
keys from the old machine and use them. If you don't want to do that, 
you buy new, probably stronger, certificates that are also valid. I 
think your case only covers self-signed certificates.