Outage notes

[Stephen John Smoogen]

On Wed, Jan 12, 2011 at 08:06, seth vidal <skvidal at fedoraproject.org> wrote:
> Hi Everyone,
>  I took some notes while we were rebooting boxes I wanted to share them
> with everyone for future outages.
>
> Ordering of the bounces:
> 1. xen14: puppet is on there and if that is back up first we have a
> place to stand for pushing out any changes (dns changes for example via
> puppet) - xen14 takes about 4 minutes to restart/POST

Most of the new IBM hardware can take 4-6 minutes to reboot. I don't
know if there is some flags I should have put in it, but it is deadly
slow.


> Overall things to think about for the future:
> 1. dumping a complete virsh list - including how much memory is actually
> being used per vm per server before we start reboots
> 2. checking what disks need fscks because of mounted time and doing
> those earlier or separately.
> 3. verifying that all running vms are:
>   a. intended to be running
>   b. have a config file
>   c. are set to autostart
> 4. verifying that all NOT running vms are:
>   a. intended to be off
>   b. are NOT set to autostart

looks good. I thought koji2 was running before the reboots but it may
have been a ghost vm.

> thoughts welcome.
> -sv
>
>
>
>
> _______________________________________________
> infrastructure mailing list
> infrastructure at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/infrastructure
>



-- 
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren