Advanced Format drives + RAID5 + LVM + EXT4 = migraine

[Jake Peavy]

Hey all,

I'm going a little off my rocker here trying to understand how to best
configure a 3-drive RAID5 array using the new Western Digital 4k sector
drives.

I've got 3xWD15EARS (1.5T 4k sector advanced format drives).

If I was using the drive as-is, my understanding is I should create a
partition starting at sector 64.  This would solve all my problems.

But I want to put them into a RAID5 array, so currently I didn't partition
at all and just passed mdadm the whole drive.  Not sure if this is the best
approach.  I used 512kB as chunk size.

Then I want to create LVM on the array.  I didn't pass any special parms to
{pv,vg,lv}create (should I?)

Lastly, I want to create an EXT4 filesystem on the array.  I used the stride
calculator at http://busybox.net/~aldot/mkfs_stride.htmlhttp://busybox.net/%7Ealdot/mkfs_stride.html
with
raid level = 5,  devices = 3, chunk size = 512 and filesystem blocks = 4 (?)
which suggested I use mkfs.ext4 -b 4096 -E stride=128,stripe-width=256
(which I did).

After doing all this, I get writes to my filesystem of only 38MB/s.  Reads
of 119MB/s.

[root@morbo opt]# hdparm  -Tt /dev/mapper/array_vg-lv_opt

/dev/mapper/array_vg-lv_opt:
 Timing cached reads:   1574 MB in  2.00 seconds = 786.85 MB/sec
 Timing buffered disk reads:  332 MB in  3.00 seconds = 110.57 MB/sec
[root@morbo opt]# time (dd if=/dev/zero of=zerofile bs=1M count=8192; sync)
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 226.646 s, 37.9 MB/s

real    3m51.062s
user    0m0.072s
sys    0m39.794s
[root@morbo opt]# dd if=zerofile of=/dev/null bs=1M
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB) copied, 72.4094 s, 119 MB/s

I have a feeling my array should perform much better than this and it's
probably because I haven't taken the 4k sector size into account and I'm
bleeding performance.

Can anyone comment?  If I need to take the 4k sector size into account,
where should I do so?  With partitions?  In pvcreate?  Some other place?  Or
multiple places?

TIA,

[Robert Arkiletian]

On Sun, Aug 29, 2010 at 10:08 AM, Jake Peavy  wrote:
...
> I have a feeling my array should perform much better than this and it's
> probably because I haven't taken the 4k sector size into account and I'm
> bleeding performance.
>
> Can anyone comment?  If I need to take the 4k sector size into account,
> where should I do so?  With partitions?  In pvcreate?  Some other place?  Or
> multiple places?

http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/

[Jake Peavy]

On Sun, Aug 29, 2010 at 3:06 PM, Robert Arkiletian  wrote:
> On Sun, Aug 29, 2010 at 10:08 AM, Jake Peavy  wrote:
> ...
> > I have a feeling my array should perform much better than this and it's
> > probably because I haven't taken the 4k sector size into account and I'm
> > bleeding performance.
> >
> > Can anyone comment?  If I need to take the 4k sector size into account,
> > where should I do so?  With partitions?  In pvcreate?  Some other place?
>
> Or
> > multiple places?
>
> http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/

Yeah, I had read that prior to posting, but let me know if I followed it
correctly.  It sounds to me (from the sidebar) that I don't need to do
anything because I'm using software RAID and it already operates with 4kB
multiples.  Yes?

If so, and I have everything configured correctly, the question then
becomes: How does one know if one is getting the most out of their
hardware?  I was expecting more like 70-100MB/s writes, and 100-200MB/s
reads.

TIA