Builder monitoring

[Jeremy Katz]

On Wed, 2007-08-15 at 23:03 -0500, Dennis Gilmore wrote:
> Once upon a time Wednesday 15 August 2007, Mike McGrath wrote:
> > We've recently started taking a closer look at the builders and as a
> > result we've been getting many notifications.  As a baseline, the
> > builders need some work.  One thing we've been getting alerted of is
> > memory usage, in particular less than 10% swap free.
> >
> > We have a couple of options.
> >
> > 1) stop monitoring swap on the builders
> > 2) reduce resolution on the builders (check less often and require more
> > failures in a row before notification, right now is 3 failures in a row
> > results in a notification)
> > 3) increase swap (presently about 1G / core so between 2G and 8G of swap
> > on the boxes)
[snip]
> Some builders could really do with some more ram.  but in the mean time 
> perhaps up swap to 1.5g/core 

I'd say a combination of 2 and 3 is going to get the best results of
improving reliability and reducing false positives

Jeremy