Disk IO issues

[Matt Domsch]

On Wed, Dec 31, 2008 at 02:42:27PM -0600, Mike McGrath wrote:
> Lets pool some knowledge together because at this point, I'm missing
> something.
> 
> I've been doing all measurements with sar as bonnie, etc, causes builds to
> timeout.
> 
> Problem: We're seeing slower then normal disk IO.  At least I think we
> are.  This is a PERC5/E and MD1000 array.
> 
> When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> around 4-6MBytes/s

That's sucky.
 
> When I do a cp of a large file "cp /mnt/koji/out /tmp/" I get
> 30-40MBytes/s.
> 
> If I "cat /dev/sde > /dev/null" I get between 225-300MBytes/s read.

That's about what I would expect for straight block reads.

> The above tests are pretty consistent.  /dev/sde is a raid5 array,
> hardware raid.

Remember, RAID 5's worst performance is for writes.  In your 14-drive
array, it has to calculate parity across all the drives, then write
the data across all the drives.  As long as it's pure writes (e.g. not
read/modify/write) it's not so bad, but still slower than you might
think.

What ext3 journaling options are enabled (e.g. what does 'mount' say)?
If it's data=ordered (the default), that's OK.  If it's data=journal,
then all the data gets written twice (first to the journal, then the
journal to the disk), which is really really slow, and the size of the
journal would really make a difference too.

RAID controllers also tend to benefit from using the noop scheduler,
which effectively defers the scheduling to the RAID controller.

Note that cp doesn't fdatasync(), so the I/Os will be scheduled, but
not necessarily completed, when cp returns.  Which might make your
numbers even more optimistic than they really are. :-(

-- 
Matt Domsch
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & www.dell.com/linux