Freeze break: puppet update

[Kevin Fenzi]

On Mon, 16 Apr 2012 09:56:25 -0700
Toshio Kuratomi <a.badger at gmail.com> wrote:

> Is the new puppet server supposed to work with old puppet clients and
> vice versa?  By doing this one piece at a time, like this, we're
> going to experience the following scenarios:
> 
> * Old server, new client (stg at step 1)
> * New server, old client (prod at step 2)
> * New server, new client (stg at step 2, prod at the end phase)
> 
> So we're going to go through a more combinations than we're worried
> about servicing in the end and we're not actually going to be testing
> our intermediate production step at all....

True. Yes, I think either client/server should work. 

> Maybe our steps should be:
> 
> 1 Update puppet on our stg machines.  Make sure that runs twice ok
> 2 Update puppet on our production client machines.  Make sure those
> run twice okay.
> 3 Update the puppet server with the option to revert.
> 
> That way, step 1 and 2 are tested.  Only step 3 is untested by us
> (which should be the most tested step by upstream/other sites).  And
> if we have to revert step 3 we'll only be reverting the puppetmaster
> on lockbox.  (It kinda sucks that we won't be testing the final
> configuration in this scenario.  But it avoids the stage where, for
> the whole time we're testing the final configuration in stg we're
> running an untested configuration in production).

Yeah, I like that better... easier/faster to revert. 

> > - Finally, update puppet on all the other boxes. 
> > 
> > We should be able to back out to the current version at any of these
> > points if it fails. 
> > 
> Do we know that the new puppet on lockbox won't create new data
> structures that the old puppet won't understand?  If we don't,
> perhaps we want to take an lvm snapshot before we update to the new
> puppet on the server (and tell people not to make puppet git commits
> until we know we're not going to back out).

It's not supposed to. 

I can make a backup of the puppet files tho just to be sure. 

> After looking at the security issues this solves, I think we do want
> to go ahead with the update.  But we should: 1) make sure we
> understand what it takes to back out of this.  2) Alert the other
> groups that we think this is important enough that we have to do it
> now but if we have difficulties it could impact getting the beta
> release out on time.

i can also lock commits by changing perms on the git repo while we are
updating. 

kevin


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20120416/dec4a779/attachment.sig>