Freeze break: db-koji01 and bvirthost09 reboot

[Pierre-Yves Chibon]

On Fri, Apr 10, 2015 at 11:04:55AM -0600, Kevin Fenzi wrote:
> I was going to wait until after freeze for this, but with us slipping a
> week I think it might be worth doing now. 
> 
> For the last few weeks we have been having issues with db-koji01. 
> The problem started when I moved it's backend storage from one iscsi/pv
> to another iscsi/pv. The load has been high since then and it's not as
> performant as it was. 
> 
> Effects: 
> 
> * koji alerts in nagios make us need to restart httpd on koji01 (which
>   we can do without outage, but means a human has to wake up and go do
>   it). 
> 
> * If koji01 httpd isn't restarted, kojira sometimes will timeout and
>   not launch newrepos. (We worked around this by increasing the
>   timeout, but it's only a matter of time before it hits this again). 
> 
> * Pages on koji that need lots of db access are slower than they
>   were/need to be. 
> 
> Cause: 
> 
> Not entirely sure what the base cause is. lvdisplay shows the guest is
> on the right iscsi volume, there's no iscsi errors or the like. The
> host did have stale lvm data due to lvmetad running, but that shouldn't
> have affected the running guest(s). I can only think there's something
> still trying to hit the old no longer used iscsi volume and causing
> extra load. 
> 
> What I would like to do: 
> 
> * Stop postgres on db-koji01. This will cause the hub to show db down
>   to anyone looking. 
> 
> * rsync /var/lib/pgsql off to backup03. This should take less than
>   10min. 
> 
> * shutdown db-koji01 and dhcp01. 
> 
> * Reboot bvirthost09 
> 
> * See if the issue clears up. If something happens and db-koji01
>   doesn't come back up right, we can make a new one and
>   sync /var/lib/pgsql back to it and be back up pretty quickly.
>   Hopefully it won't come to that. 
> 
> I'd like to schedule this possibly over the weekend off hours when koji
> isn't all that busy. 
> 
> Thoughts? +1s?

+1 for me and fingers crossed :)


Pierre