Making migrations common (and not scary)

[David Timothy Strauss]

There are two primary parts to any service: configuration and state.
Configuration is, more or less, straightforward with the limitations
you've mentioned.

State is the complex one. I've worked with all the following
relationships of state between servers:

 (a) Replication (master/slave and ring-based)
 (b) Serialization/archive for backup purposes
 (c) Serialization/archive for server migration
 (d) Serialization/archive for replica-bootstrapping purposes (often a
special case of b or c)

This is based on work with Cassandra, Redis, MariaDB, Solr,
PostgreSQL, and I'm sure a few others. It would be really neat for our
featured roles to come pre-packaged with consistent access to
facilities like these.

I see this potentially happening in several stages:

 (1) Dump and load for configuration and state. Support using the
archive for backup or server-migration purposes.
 (2) Support replica creation, including whatever's required to
bootstrap replicas and having a secure relationship between the
servers.
 (3) Use replicas as the server migration facility. At my company, we
call this "mutiny." It's where a replica gets created, caught up, and
then takes over the master role (however necessary).
 (4) Support HA tools using replicas for fail-over.

Some of these facilities may be free with container tools like Docker.