Disk IO issues

[Stephen John Smoogen]

On Wed, Dec 31, 2008 at 1:42 PM, Mike McGrath <mmcgrath at redhat.com> wrote:
> Lets pool some knowledge together because at this point, I'm missing
> something.
>
> I've been doing all measurements with sar as bonnie, etc, causes builds to
> timeout.
>
> Problem: We're seeing slower then normal disk IO.  At least I think we
> are.  This is a PERC5/E and MD1000 array.
>
> When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> around 4-6MBytes/s
>
> When I do a cp of a large file "cp /mnt/koji/out /tmp/" I get
> 30-40MBytes/s.
>
> Then I "dd if=/dev/sde of=/dev/null" I get around 60-70 MBytes/s read.
>
> If I "cat /dev/sde > /dev/null" I get between 225-300MBytes/s read.

I thought the /dev/null numbers were not good indicators  (I remember
Stephen Tweedie or someone in the RH Kernel team lecturing that
numbers while consistent would not show real world issues and could be
much higher than what really happens.) The lesson was always send it
to a real file that its going to open/close/deal with.. even if the
file is in a ram disk.

I do know that dd defaults to 512 block size which makes it different
speeds for copies (whoops Ricky confirms this ). Also stuff that will
fit inside of the PERC Cache and how the journal is going to be
written/committed are going to make differences...

The next difference is how a system sees inside of a disk and how the
disk sees itself. The /dev/xxx are always going to be much higher
because there is no filesystem interaction and the controller is going
to be just pulling from hardware.. it might even optimize doing that
(raw partition style) so that you get insane speeds but as soon as you
put in a filesystem poof.





-- 
Stephen J Smoogen. -- BSD/GNU/Linux
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"