Builder monitoring
We've recently started taking a closer look at the builders and as a
result we've been getting many notifications. As a baseline, the
builders need some work. One thing we've been getting alerted of is
memory usage, in particular less than 10% swap free.
We have a couple of options.
1) stop monitoring swap on the builders
2) reduce resolution on the builders (check less often and require more
failures in a row before notification, right now is 3 failures in a row
results in a notification)
3) increase swap (presently about 1G / core so between 2G and 8G of swap
on the boxes)
Build guys! What do you think?
-Mike