Mass reboots/updates improvements

[Kevin Fenzi]

Greetings. 

I think we have some lessons learned and things we could improve based
on the issues we ran into yesterday on the mass reboot/updates. ;) 

Issues/Observations: 

* We can't seem to complete everything in a 2 hour window. We should
  block out more time, and/or have things more organized. 

* The build system /mnt/koji issues are due to a guest that was moved
  from one machine to a new one, and then somehow started on both after
  reboots. ;( 

* There were a few cases of too many cooks doing things at once to a
  machine. 

* Some physical machines were poorly or not at all labeled in things
  like pdu's and serial consoles. 

* We need to be better about retiring machines. Sometimes it's hard to
  see what shouldn't be up or should be. 

Ideas/improvements: 

* I'd like to look at splitting all our hosts into 3 groups (based on
  who we need to notify about reboots or outages): 

a) End users will see/notice an outage if this machine is down/not
working.

b) Fedora package maintainers or contributors will notice if this
machine/service is not working/down. 

c) Everything else. Including things that if they were single instances
would fit in the above, but are spread out, so they can be
rebooted/updated one at a time (ie, app servers, etc). 

I've made a tenative list with all our hosts in these groups: 
~kevin/mass-reboot-list on puppet01. Please look and see if you see
anything that looks wrong or needs adjusting. 

With this split out, we can do any machines in "c" as we like as long
as we are careful, we can do 'b' machines if we announce to
devel-announce and schedule a window and 'a' machines if we announce to
the main fedora announce list and schedule a window. All the windows
should be shorter than what we saw yesterday. 

* We might look at having a updates miester (czar?:) who would be the
  only one allowed to touch machines in a read/write way. By default
  everyone else is hands off unless the updates miester asks them to
  work on something. This would allow us to not interfere with each
  other or duplicate effort. 

* Seth is working on tooling to tell us anytime we have a virtual
  machine thats set to start on boot, but not started now, or not set
  to start on boot but started now. 

* We need to go and label things in all the pdu's etc. I can look at
  doing that and writing up a file somewhere with all the places a
  particular machine is. Then, the ones we can't find, we will fill in
  when smooge and I are out at phx2. 

* I have started a SOP for retiring machines. It needs a lot of work: 
https://fedoraproject.org/wiki/Infrastructure_retire_machine_SOP
please modify and clean up. The goal should be making it very clear
when a machine has been retired so we don't confuse it with anything
active. 

There is a new rhel5 kernel out (yes, right after we applied the last
one yesterday.), so I would suggest we look at implementing some or all
of these that make sense for those updates. ;) 

Thoughts? Rants? more suggestions?

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20110601/5af0d264/attachment.bin