Disk IO issues

[Kostas Georgiou]

On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:

> On Thu, 1 Jan 2009, James Antill wrote:
> 
> > On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
> > > Lets pool some knowledge together because at this point, I'm missing
> > > something.
> > >
> > > I've been doing all measurements with sar as bonnie, etc, causes builds to
> > > timeout.
> > >
> > > Problem: We're seeing slower then normal disk IO.  At least I think we
> > > are.  This is a PERC5/E and MD1000 array.
> > >
> > > When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> > > around 4-6MBytes/s
> >
> >  This _might_ not be "IO" in a normal sense, -a to cp means:
> >
> >  file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
> >
> > ...esp. given that you aren't getting large IOWait times, you might want
> > to strace -T the cp and do some perl/whatever on the result to see what
> > is eating up the time.
> 
> Even with non cp type things (like a bacula backup) it just doesn't seem
> as fast as I would expect it to be.  I've never actually done trending at
> this level / scale on a filesystem / drive before.  So I really don't have
> a good baseline except that it just seems slow to me.
> 
> Other then the much faster direct block access and the large file reads, I
> don't have much else to go on that makes me think its slow.

Do writes show the same pattern? If you use selinux/ACLs/xattrs the default
inode size of 128 can cause slowdowns (#205161 for example).  

Can you run blktrace+seekwatcher (both in EPEL) to get an idea on
what is going on? An iostat -x -k /dev/sde 1 output will also be
helpfull.

Kostas