Freeze break: puppet update

[Toshio Kuratomi]

On Mon, Apr 16, 2012 at 10:27:52AM -0600, Kevin Fenzi wrote:
> Greetings
> 
> There's a new puppet release that has a number of security fixes to it. 
> 
> I'd like to do the following: 
> 
> - Update puppet in our stg machines and wait an hour for it to run
>   twice ok on those machines. This may fail due to the older master, we
>   will need to watch the output here. 
> 
> - Update on lockbox01 (our puppetmaster) and wait for a cycle to make
>   sure all machines process ok.
> 
Is the new puppet server supposed to work with old puppet clients and vice
versa?  By doing this one piece at a time, like this, we're going to
experience the following scenarios:

* Old server, new client (stg at step 1)
* New server, old client (prod at step 2)
* New server, new client (stg at step 2, prod at the end phase)

So we're going to go through a more combinations than we're worried
about servicing in the end and we're not actually going to be testing our
intermediate production step at all....

Maybe our steps should be:

1 Update puppet on our stg machines.  Make sure that runs twice ok
2 Update puppet on our production client machines.  Make sure those run
  twice okay.
3 Update the puppet server with the option to revert.

That way, step 1 and 2 are tested.  Only step 3 is untested by us (which
should be the most tested step by upstream/other sites).  And if we have to
revert step 3 we'll only be reverting the puppetmaster on lockbox.  (It
kinda sucks that we won't be testing the final configuration in this
scenario.  But it avoids the stage where, for the whole time we're testing
the final configuration in stg we're running an untested configuration in
production).

> - Finally, update puppet on all the other boxes. 
> 
> We should be able to back out to the current version at any of these
> points if it fails. 
> 
Do we know that the new puppet on lockbox won't create new data structures
that the old puppet won't understand?  If we don't, perhaps we want to take
an lvm snapshot before we update to the new puppet on the server (and tell
people not to make puppet git commits until we know we're not going to
back out).

After looking at the security issues this solves, I think we do want to go
ahead with the update.  But we should: 1) make sure we understand what it
takes to back out of this.  2) Alert the other groups that we think this is
important enough that we have to do it now but if we have difficulties it
could impact getting the beta release out on time.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20120416/a8410dc9/attachment.sig>