3:20 a.m.
I was going back over the archives and saw a lot of people complaining
about how slow RAID5 was, and did some quick research into this.
Yes, it's true that it can be slowed down if you're rewriting
fragmentary data in place, since this takes a read-modify-write operation.
But that's true of most filesystems.
If you're *not* a database weenie, and you're doing usual manly things
with your filesystem (like lots of compiles, for instance), you're
typically not going to be modifying files in place at all.
You're going to either be writing out new files, or else opening
existing files for writing, doing a truncate(fd, (off_t) 0) on them, and
then writing them out, etc.
Or else you're going to be opening lots and lots of files for reading...
Both of which RAID5 does as well as the next thing (which would be RAID0).
Why? Because if you're writing out files, you don't need to
read-modify-write, you can just write (and write the stripe parity data
at the same time).
"Ok, sure, you say, but what about the 16K (or whatever) stripe size
versus writing data out in smaller chunks?"
Yes? What about it? You might be writing blocks out via stdio
buffering in 4K chunks or whatever, but unless you're doing a sync()
after every single write (buffer flush), the filesystem (ext3 or
whatever) is free to accumulate consecutive blocks into handy (read:
more efficient) chunks (such as multiples of the stripe size) and write
them out all at once... again eliminating the need to do a
read-modify-write (since you only need to do a read-modify-write if
you're writing incomplete blocks).
So is it just the database-heads that are maligning RAID5, or are there
other performance issues I don't know about?
Because my empirical experience has always been that when writing large
files, RAID5 performs on par with RAID0.
Ditto with reading lots of files.
-Philip
5:04 p.m.
Gordon Messmer wrote:
Philip A. Prindeville wrote:
>
> If you're *not* a database weenie, and you're doing usual manly things
> with your filesystem (like lots of compiles, for instance), you're
> typically not going to be modifying files in place at all.
That's not quite it. RAID 5 performance suffers because every write
requires that the entire block that's being written be read from every
drive in the array, parity calculated, and then the data and parity
written out. For each block written, the array has to do N reads plus
two writes.
No. Even in the worst case it would read N-2 blocks (you are writing a new data
block and calculating new parity), and two writes. But normally, writing
sequential data, you can wait until you have enough data for an entire stripe at
once, read nothing, and write once to each drive. You should be able to do this
in parallel, but unlike RAID0 I've never measured it happening. Tuning the
"stripe_cache_size" and creating a filesystem using the "stride="
option will help.
It doesn't matter whether you're writing new files or
modifying existing
files, because all of this happens at the block level. It's especially
bad on journalled filesystems, where writing to a file will update the
files blocks, plus the filesystem's journal's blocks, and finally the
filesystem's blocks.
No again. You read the parity block and the old data block, XOR first the old
then the new data with the parity block, and write the new data and parity.
> So is it just the database-heads that are maligning RAID5, or are
> there other performance issues I don't know about?
Most of your comments don't reflect the way RAID 5 actually functions in
any way.
> Because my empirical experience has always been that when writing
> large files, RAID5 performs on par with RAID0.
The system on which you were testing was probably limited by other
factors, if that was the case. A RAID 0 disk array will be much faster
than a RAID 5 array.
RAID 5 tends to be most appropriate when you're trying to get as much
disk space as you can with the lowest cost, you won't be running
multiple simultaneous jobs on the same disk array, and when you'll be
collecting data at a rate that's relatively low. Usually, that's
backups. Your network is probably slower than your disk array (unless
the array is very large -- array speed decreases with array size), so
streaming data in over the network to your disk array won't bog it down.
Virtually any interactive workload will benefit from a better disk
configuration.
--
Bill Davidsen <davidsen(a)tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
2:06 a.m.
Bill Davidsen wrote:
Gordon Messmer wrote:
...
No. Even in the worst case it would read N-2 blocks (you are writing
a
new data block and calculating new parity), and two writes.
Let's just say that I've seen controllers behave in ways that I don't
understand, and that I agree, the cost should not be as great as I
previously estimated.
> It doesn't matter whether you're writing new files or
modifying
> existing files, because all of this happens at the block level. It's
> especially bad on journalled filesystems, where writing to a file will
> update the files blocks, plus the filesystem's journal's blocks, and
> finally the filesystem's blocks.
>
No again. You read the parity block and the old data block, XOR first
the old then the new data with the parity block, and write the new data
and parity.
Yes, I understand what you're saying, but that in no way contradicts
what I wrote there. Regardless of whether you create a new file or
modify an existing file, there will be changes made to the filesystem to
reflect the fact that changes have been made. If you modify a file, the
inode's mtime is updated. If you create a new file, then a new inode is
written, and the directory entry is modified. In both cases, the blocks
which hold the file's data are written, the journal is written before
the filesystem is updated, the filesystem is updated with the changes in
the journal, and then the journal is modified again to mark it complete.
We can argue about how much overhead RAID5 has, but I don't think you
can argue either that there is *no* overhead or that the filesystem is
not a database. Any given write to the disk will involve updating the
journal twice and the filesystem once, which more or less creates the
"small random writes" that RAID5 is so poor at performing.
12:17 a.m.
Bill Davidsen wrote:
Gordon Messmer wrote:
> Your assertion ignores the fact that filesystems themselves are, in
> fact, databases. Real-world experience with many production systems
> and many workloads has convinced me to use RAID 5 as rarely as
> possible. Even when I'm forced to use it, I generally choose a RAID
> 5+0 configuration as I get much better performance.
>
Or you might want to read the man pages for md and mdadm. RAID10 is
faster (assuming you use the "far 2" config). No, RAID10 is not another
name for RAID1+0...
When I read the man page for "md", I see:
RAID10
RAID10 provides a combination of RAID1 and RAID0, and is
sometimes known as RAID1+0.
...so I'm not sure what man page you've been reading.
I know that RAID 10 is faster than RAID 5+0. I meant that in the rare
circumstance when I'm trying to build a very large volume of disks (say,
20), I'll usually create four RAID5 arrays with five disks each, and
then stripe them. The resulting array (RAID 5+0) will be more resilient
to failure and perform much better than a single RAID5 array containing
all 20 disks.
5:07 p.m.
Gordon Messmer wrote:
Chris Tyler wrote:
> On Tue, 2008-12-30 at 01:02 -0800, Gordon Messmer wrote:
>> That's not quite it. RAID 5 performance suffers because every write
>> requires that the entire block that's being written be read from
>> every drive in the array, parity calculated, and then the data and
>> parity written out. For each block written, the array has to do N
>> reads plus two writes.
>
> You don't have to read all of the drives -- just the block you're
> updating and the parity block. XOR the old data you're about to
> overwrite with the parity block and the new data and you'll have the new
> parity block. Total activity: two reads plus two writes.
I've understood that to be the case, but while watching the drive
activity lights on RAID5 arrays, it seems like I always see the entire
set flash at the same time. I guess I'll have to investigate that
further to find out why. Thanks.
>> RAID 5 tends to be most appropriate when you're trying to get as much
>> disk space as you can with the lowest cost, you won't be running
>> multiple simultaneous jobs on the same disk array, and when you'll be
>> collecting data at a rate that's relatively low.
>
> I'd say the other way around -- RAID 5 is poor at small writes (hence
> the OP's comments about database updates), but very nearly approaches
> RAID-0 speeds when reading or writing large quantities of sequential
> data.
Your assertion ignores the fact that filesystems themselves are, in
fact, databases. Real-world experience with many production systems and
many workloads has convinced me to use RAID 5 as rarely as possible.
Even when I'm forced to use it, I generally choose a RAID 5+0
configuration as I get much better performance.
Or you might want to read the man pages for md and mdadm. RAID10 is faster
(assuming you use the "far 2" config). No, RAID10 is not another name for
RAID1+0...
--
Bill Davidsen <davidsen(a)tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
2:33 a.m.
Chris Tyler wrote:
On Tue, 2008-12-30 at 01:02 -0800, Gordon Messmer wrote:
> That's not quite it. RAID 5 performance suffers because every write
> requires that the entire block that's being written be read from every
> drive in the array, parity calculated, and then the data and parity
> written out. For each block written, the array has to do N reads plus
> two writes.
You don't have to read all of the drives -- just the block you're
updating and the parity block. XOR the old data you're about to
overwrite with the parity block and the new data and you'll have the new
parity block. Total activity: two reads plus two writes.
I've understood that to be the case, but while watching the drive
activity lights on RAID5 arrays, it seems like I always see the entire
set flash at the same time. I guess I'll have to investigate that
further to find out why. Thanks.
> RAID 5 tends to be most appropriate when you're trying to get
as much
> disk space as you can with the lowest cost, you won't be running
> multiple simultaneous jobs on the same disk array, and when you'll be
> collecting data at a rate that's relatively low.
I'd say the other way around -- RAID 5 is poor at small writes (hence
the OP's comments about database updates), but very nearly approaches
RAID-0 speeds when reading or writing large quantities of sequential
data.
Your assertion ignores the fact that filesystems themselves are, in
fact, databases. Real-world experience with many production systems and
many workloads has convinced me to use RAID 5 as rarely as possible.
Even when I'm forced to use it, I generally choose a RAID 5+0
configuration as I get much better performance.
11:51 a.m.
On Tue, 2008-12-30 at 01:02 -0800, Gordon Messmer wrote:
Philip A. Prindeville wrote:
>
> If you're *not* a database weenie, and you're doing usual manly things
> with your filesystem (like lots of compiles, for instance), you're
> typically not going to be modifying files in place at all.
That's not quite it. RAID 5 performance suffers because every write
requires that the entire block that's being written be read from every
drive in the array, parity calculated, and then the data and parity
written out. For each block written, the array has to do N reads plus
two writes.
You don't have to read all of the drives -- just the block you're
updating and the parity block. XOR the old data you're about to
overwrite with the parity block and the new data and you'll have the new
parity block. Total activity: two reads plus two writes.
But that's the worst case. The OP's point is that you don't need to do
any reads if you're writing enough sequential data to fill N-1 blocks --
e.g., in a 5-drive array, with a 4k block size, writing 16k of data
(four 4k blocks in the same parity stripe) can be done without any
reading at all -- just 5 writes. If each drive is on a separate
controller, those writes can be done in parallel. For multi-megabyte
writes, this scenario is very common. (For a 32k block size on a 5-drive
array, you'd need to write 128k for a readless update).
> Because my empirical experience has always been that when
writing large
> files, RAID5 performs on par with RAID0.
Exactly.
RAID 5 tends to be most appropriate when you're trying to get as
much
disk space as you can with the lowest cost, you won't be running
multiple simultaneous jobs on the same disk array, and when you'll be
collecting data at a rate that's relatively low.
I'd say the other way around -- RAID 5 is poor at small writes (hence
the OP's comments about database updates), but very nearly approaches
RAID-0 speeds when reading or writing large quantities of sequential
data.
-Chris
3:02 a.m.
Philip A. Prindeville wrote:
If you're *not* a database weenie, and you're doing usual manly things
with your filesystem (like lots of compiles, for instance), you're
typically not going to be modifying files in place at all.
That's not quite it. RAID 5 performance suffers because every write
requires that the entire block that's being written be read from every
drive in the array, parity calculated, and then the data and parity
written out. For each block written, the array has to do N reads plus
two writes.
It doesn't matter whether you're writing new files or modifying existing
files, because all of this happens at the block level. It's especially
bad on journalled filesystems, where writing to a file will update the
files blocks, plus the filesystem's journal's blocks, and finally the
filesystem's blocks.
So is it just the database-heads that are maligning RAID5, or are
there
other performance issues I don't know about?
Most of your comments don't reflect the way RAID 5 actually functions in
any way.
Because my empirical experience has always been that when writing
large
files, RAID5 performs on par with RAID0.
The system on which you were testing was probably limited by other
factors, if that was the case. A RAID 0 disk array will be much faster
than a RAID 5 array.
RAID 5 tends to be most appropriate when you're trying to get as much
disk space as you can with the lowest cost, you won't be running
multiple simultaneous jobs on the same disk array, and when you'll be
collecting data at a rate that's relatively low. Usually, that's
backups. Your network is probably slower than your disk array (unless
the array is very large -- array speed decreases with array size), so
streaming data in over the network to your disk array won't bog it down.
Virtually any interactive workload will benefit from a better disk
configuration.
9:55 a.m.
Philip A. Prindeville wrote:
I was going back over the archives and saw a lot of people
complaining
about how slow RAID5 was, and did some quick research into this.
Yes, it's true that it can be slowed down if you're rewriting
fragmentary data in place, since this takes a read-modify-write operation.
But that's true of most filesystems.
If you're *not* a database weenie, and you're doing usual manly things
with your filesystem (like lots of compiles, for instance), you're
typically not going to be modifying files in place at all.
You're going to either be writing out new files, or else opening
existing files for writing, doing a truncate(fd, (off_t) 0) on them, and
then writing them out, etc.
Or else you're going to be opening lots and lots of files for reading...
Both of which RAID5 does as well as the next thing (which would be RAID0).
Why? Because if you're writing out files, you don't need to
read-modify-write, you can just write (and write the stripe parity data
at the same time).
"Ok, sure, you say, but what about the 16K (or whatever) stripe size
versus writing data out in smaller chunks?"
Yes? What about it? You might be writing blocks out via stdio
buffering in 4K chunks or whatever, but unless you're doing a sync()
after every single write (buffer flush), the filesystem (ext3 or
whatever) is free to accumulate consecutive blocks into handy (read:
more efficient) chunks (such as multiples of the stripe size) and write
them out all at once... again eliminating the need to do a
read-modify-write (since you only need to do a read-modify-write if
you're writing incomplete blocks).
So is it just the database-heads that are maligning RAID5, or are there
other performance issues I don't know about?
You should really read the linux-raid mailing list for a while and look at the
benchmarking done and reported there by many people who have been doing this for
a while. Clearly there are many performance issues you don't know about.
Because my empirical experience has always been that when writing
large
files, RAID5 performs on par with RAID0.
Based on benchmarks done by many people your results don't match ours. RAID5
write is slow because of the way it's done in the md code, and unless you are
testing on a minimal size array (2-3 drives) this should be obvious. Both RAID0
and RAID10 outperform significantly.
Ditto with reading lots of files.
Sustained read should be N-1 times the speed of a raw disk.
The advantage of RAID5 is that it's the cheapest way (in disk count) to have
redundancy.
--
Bill Davidsen <davidsen(a)tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
12:12 p.m.
Gordon Messmer wrote:
Bill Davidsen wrote:
> Gordon Messmer wrote:
>> Your assertion ignores the fact that filesystems themselves are, in
>> fact, databases. Real-world experience with many production systems
>> and many workloads has convinced me to use RAID 5 as rarely as
>> possible. Even when I'm forced to use it, I generally choose a RAID
>> 5+0 configuration as I get much better performance.
>>
> Or you might want to read the man pages for md and mdadm. RAID10 is
> faster (assuming you use the "far 2" config). No, RAID10 is not
> another name for RAID1+0...
When I read the man page for "md", I see:
RAID10
RAID10 provides a combination of RAID1 and RAID0, and is
sometimes known as RAID1+0.
...so I'm not sure what man page you've been reading.
The rest of that section... the md implementation is not a stripe over mirrors,
but overlays the mirroring over all devices. The first paragraph is confusing if
you stop there (and maybe if you don't). But there is no array of RAID0 striped
over multiple arrays of RAIDx (x =1,5,6), it's all one array, and significantly
faster than 1+0 in the case where the layout is far=2.
I know that RAID 10 is faster than RAID 5+0. I meant that in the
rare
circumstance when I'm trying to build a very large volume of disks (say,
20), I'll usually create four RAID5 arrays with five disks each, and
then stripe them. The resulting array (RAID 5+0) will be more resilient
to failure and perform much better than a single RAID5 array containing
all 20 disks.
--
Bill Davidsen <davidsen(a)tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
5590
days inactive
5599
days old
9 comments
4 participants
participants (4)
-
Bill Davidsen -
Chris Tyler -
Gordon Messmer -
Philip Prindeville