Disk IO issues

[Mike McGrath]

On Wed, 31 Dec 2008, Mike McGrath wrote:

> Lets pool some knowledge together because at this point, I'm missing
> something.
>
> I've been doing all measurements with sar as bonnie, etc, causes builds to
> timeout.
>
> Problem: We're seeing slower then normal disk IO.  At least I think we
> are.  This is a PERC5/E and MD1000 array.
>
> When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> around 4-6MBytes/s
>
> When I do a cp of a large file "cp /mnt/koji/out /tmp/" I get
> 30-40MBytes/s.
>
> Then I "dd if=/dev/sde of=/dev/null" I get around 60-70 MBytes/s read.
>
> If I "cat /dev/sde > /dev/null" I get between 225-300MBytes/s read.
>
> The above tests are pretty consistent.  /dev/sde is a raid5 array,
> hardware raid.
>
> So my question here is, wtf?  I've been working to do a backup which I
> would think would either cause network utilization to max out, or disk io
> to max out.  I'm not seeing either.  Sar says the disks are 100% utilized
> but I can cause major increases in actual disk reads and writes by just
> running additional commands.  Also, if the disks were 100% utilized I'd
> expect we would see lots more iowait.  We're not though, iowait on the box
> is only %0.06 today.
>
> So, long story short, we're seeing much better performance when just
> reading or writing lots of data (though dd is many times slower then cat).
> But with our real-world traffic, we're just seeing crappy crappy IO.
>
> Thoughts, theories or opinions?  Some of the sysadmin noc guys have access
> to run diagnostic commands, if you want more info about a setting, let me
> know.
>
> I should also mention there's lots going on with this box, for example its
> hardware raid, lvm and I've got xen running on it (though the tests above
> were not in a xen guest).
>

Also for the curious:

dumpe2fs 1.39 (29-May-2006)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1342177280
Block count:              2684354560
Reserved block count:     134217728
Free blocks:              1407579323
Free inodes:              1336866363
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      384
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Thu Jan 17 14:52:03 2008
Last mount time:          Fri Dec  5 18:51:44 2008
Last write time:          Fri Dec  5 18:51:44 2008
Mount count:              17
Maximum mount count:      24
Last checked:             Sat May 24 03:14:41 2008
Check interval:           15552000 (6 months)
Next check after:         Thu Nov 20 03:14:41 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      1b6393b1-472c-4005-ae87-9603eea9f45b
Journal backup:           inode blocks
Journal size:             128M