backup02 io scheduler change

[Stephen John Smoogen]

On Wed, May 18, 2011 at 13:57, Kevin Fenzi <kevin at scrye.com> wrote:
> On Wed, 18 May 2011 15:51:08 -0400
> "Adam M. Dutko" <dutko.adam at gmail.com> wrote:
>
>> > I think this is a good test to see what is the problem. The
>> > deadlocks and OOM's seem to happen at 0400 when other virtual
>> > systems are
>>
>> Hrm... so all of these are xen instances and they're doing backups at
>> the same time. If the rsync processes are going into a D state I'd
>> think it's an I/O exhaustion problem. Would it be possible to alter
>> the backup schedule and stagger them if the scheduler change doesn't
>> work?
>
> backup02 itself is only doing 1 rsync at a time. ;)
>
> I'm not sure what other virtual systems are being backed up that would
> affect it. None of the other guests on ibiblio01 are backing up to
> backup01. I guess they do run daily jobs at the same time, so that
> could put stress on the host... but it's pretty weird.
>
> I made the deadline change, we will see if that helps any.

The other fix is to go to the 5.4 or 5.3 kernel. It magically does not
happen then and does with the late 5.5/5.6 ones. I think I ran into
that back in November when me and Seth tried to debug it. I think we
had it stick to that kernel on reboot.. but its not there anymore.

-- 
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren