[fedora-arm] builder io isue

[Gordan Bobic]

On 12/25/2011 05:25 AM, Dennis Gilmore wrote:
> When we were testing build were happening really fast, once we loaded
> up the build jobs things have become really slow
>
> http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=238672 started
> at 1:19 utc and at 5:18 utc  four hours later the buildrequires are
> still being installed. australia is completely io bound. i think we
> need to see how we can spread the io load. ~30 hosts reading and
> writing to the 4 spindles just saturates the disk io.

Is your RAID aligned properly? I have found that making sure that your 
RAID and FS are aligned properly makes a big difference to performance, 
at least on my workloads. Have a look here:

http://www.altechnative.net/?p=96

Just taking the default options when setting up the RAID and FS 
typically hammer the IOPS performance down to that of a single disk.

> i guess options
> are find a way to add more spindles. move /var/lib/mock to sdcard, see
> if we can get some kind of san that has alot of smaller fast disks.

That will make it an order of magnitude worse. A typical class 10 SD 
card only manages about 20 random write IOPS.

> get
> some 2.5" usb drives one for each builder. some other idea?  is there
> anyway we could add 4-8 more disks to australia. the size of the
> matters little. gaining more iops by adding more spindles would help.

You could just replace the disks with some decent SSDs and be done with 
it (aligning the RAID and FS as explained on the above linked page still 
applies just the same). Especially since 15K rpm disks are now highly 
uncompetitively priced against SSDs.

Gordan