Hi, I have a fedora20 system acting as a backup server, and I've exceeded its capacity. I'd like to build a bigger one, probably using fedora21.
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
I'd like to have 6TB of usable space using 2TB disks.
Is ext4 still best for this?
Some RAID variant or is there something better?
Are there any NAS projects that may be beneficial?
Thanks for any ideas. Alex
On 03/05/2015 08:48 AM, Alex Regan wrote:
Hi, I have a fedora20 system acting as a backup server, and I've exceeded its capacity. I'd like to build a bigger one, probably using fedora21.
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
Five 1TB disks in a RAID5 should give you about 4TB usable storage. Are you sure you're not using RAID6 (two parity drives)?
I'd like to have 6TB of usable space using 2TB disks.
Four 2TB drives in a RAID5 or five 2TB drives in a RAID6 would give you this. I'd vote for the RAID6.
Is ext4 still best for this?
BTRFS or (gulp!) XFS might be better, although ext4 would work.
Some RAID variant or is there something better?
The bigger the partition (LUN, PV, LV, whatever), the longer the recovery times are in case of a disk failure. I run a number of very large storage platforms (>500TB) and as soon as any LUN hits the 1TB mark, I immediately go to RAID6, simply because there is a possibility that a second drive may go bad while the first one is rebuilding. RAID6 gives me that cushion.
There are a couple of things I do:
1. I prefer using hardware RAID over software RAID. More expensive, but I feel it's more reliable.
2. I like using hot-swappable drive arrays so drive replacement is easy.
3. I like having my drives from different manufacturing batches because (and this is just based on experience--I can't prove it) when one drive from a batch dies, another from that same batch with the same number of running hours on it will likely die soon.
Are there any NAS projects that may be beneficial?
The underlying technology of the drive arrays will be the same in a NAS as a SAN. It's only the access method that's different and the fact that some attributes (permissions, ACLs, etc.) may not be translatable between the native system and a NAS. Generally they are translatable on a SAN (and I include raw SAN LUNs shared via iSCSI in this) simply because it is a directly coupled system and uses the host's native filesystems.
Your mileage may vary and I'm sure others on the list have equally strong opinions. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - To err is human. To forgive, a large sum of money is needed. - ----------------------------------------------------------------------
Hi,
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
Five 1TB disks in a RAID5 should give you about 4TB usable storage. Are you sure you're not using RAID6 (two parity drives)?
Yes, that's what I also thought. It's been so long since it was built that after just checking, I see it's actually only four disks, with two other 1TB drives in the system set up as a mirror.
I'd like to have 6TB of usable space using 2TB disks.
Four 2TB drives in a RAID5 or five 2TB drives in a RAID6 would give you this. I'd vote for the RAID6.
Is ext4 still best for this?
BTRFS or (gulp!) XFS might be better, although ext4 would work.
Is btrfs used in production? I wasn't sure that it was fully stable yet.
Some RAID variant or is there something better?
The bigger the partition (LUN, PV, LV, whatever), the longer the recovery times are in case of a disk failure. I run a number of very large storage platforms (>500TB) and as soon as any LUN hits the 1TB mark, I immediately go to RAID6, simply because there is a possibility that a second drive may go bad while the first one is rebuilding. RAID6 gives me that cushion.
Yes, cool, I'd definitely use RAID6 then. There is that relatively larger portion of unusable data, however. It's not so bad on a 500TB array, but with just 5TB or 6TB, it's more expensive. It's always about tradeoffs, though.
- I prefer using hardware RAID over software RAID. More expensive, but
I feel it's more reliable.
I like using hot-swappable drive arrays so drive replacement is easy.
I like having my drives from different manufacturing batches because
(and this is just based on experience--I can't prove it) when one drive from a batch dies, another from that same batch with the same number of running hours on it will likely die soon.
Good tips. I've got a few Adaptecs in production, but have always been worried about 1) having to be physically at the machine to fix the serious (any?) problems 2) increased sense of general lack of control and more of a feeling of one wrong move destroys the whole array, 3) lack of status reporting without being in the GUI.
I believe Adaptec has some remote logging capabilities, but historically it's been limited and perhaps even now only on certain models.
Are there any NAS projects that may be beneficial?
The underlying technology of the drive arrays will be the same in a NAS as a SAN. It's only the access method that's different and the fact that some attributes (permissions, ACLs, etc.) may not be translatable between the native system and a NAS. Generally they are translatable on a SAN (and I include raw SAN LUNs shared via iSCSI in this) simply because it is a directly coupled system and uses the host's native filesystems.
I thought SANs were generally attached via iSCSI, correct? There are cases where they are not? Maybe over dedicated ethernet?
That's a much bigger proposition, but interesting idea. This creates the ability for a separate enclosure dedicated to storing the array's disks, correct?
How about backup applications?
I'm currently using rsync with the --hard-links option and a shell script built years ago. It was built before Amanda had support for spanning tapes/disks.
Is there something that's robust and easier to setup than Amanda?
Thanks, Alex
On 03/05/2015 10:53 AM, Alex Regan wrote:
Hi,
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
Five 1TB disks in a RAID5 should give you about 4TB usable storage. Are you sure you're not using RAID6 (two parity drives)?
Yes, that's what I also thought. It's been so long since it was built that after just checking, I see it's actually only four disks, with two other 1TB drives in the system set up as a mirror.
I'd like to have 6TB of usable space using 2TB disks.
Four 2TB drives in a RAID5 or five 2TB drives in a RAID6 would give you this. I'd vote for the RAID6.
Is ext4 still best for this?
BTRFS or (gulp!) XFS might be better, although ext4 would work.
Is btrfs used in production? I wasn't sure that it was fully stable yet.
btrfs is in production now. F21 fully supports it and (I think) it's the default for F21.
Some RAID variant or is there something better?
The bigger the partition (LUN, PV, LV, whatever), the longer the recovery times are in case of a disk failure. I run a number of very large storage platforms (>500TB) and as soon as any LUN hits the 1TB mark, I immediately go to RAID6, simply because there is a possibility that a second drive may go bad while the first one is rebuilding. RAID6 gives me that cushion.
Yes, cool, I'd definitely use RAID6 then. There is that relatively larger portion of unusable data, however. It's not so bad on a 500TB array, but with just 5TB or 6TB, it's more expensive. It's always about tradeoffs, though.
Yes, but again, with a rebuild of a failed drive taking a LONG time, I'm worried about a second drive going "poof!" during that period. Another 2TB drive seems cheap compared to a degraded RAID5 losing a second drive during rebuild.
- I prefer using hardware RAID over software RAID. More expensive, but
I feel it's more reliable.
I like using hot-swappable drive arrays so drive replacement is easy.
I like having my drives from different manufacturing batches because
(and this is just based on experience--I can't prove it) when one drive from a batch dies, another from that same batch with the same number of running hours on it will likely die soon.
Good tips. I've got a few Adaptecs in production, but have always been worried about 1) having to be physically at the machine to fix the serious (any?) problems 2) increased sense of general lack of control and more of a feeling of one wrong move destroys the whole array, 3) lack of status reporting without being in the GUI.
Adaptec is good. MegaRAID also is good and there are monitoring tools available for OpsView and Nagios to watch it. We use it a lot.
I believe Adaptec has some remote logging capabilities, but historically it's been limited and perhaps even now only on certain models.
Are there any NAS projects that may be beneficial?
The underlying technology of the drive arrays will be the same in a NAS as a SAN. It's only the access method that's different and the fact that some attributes (permissions, ACLs, etc.) may not be translatable between the native system and a NAS. Generally they are translatable on a SAN (and I include raw SAN LUNs shared via iSCSI in this) simply because it is a directly coupled system and uses the host's native filesystems.
I thought SANs were generally attached via iSCSI, correct? There are cases where they are not? Maybe over dedicated ethernet?
The classic case is SAN attachment via fiberchannel HBAs.
That's a much bigger proposition, but interesting idea. This creates the ability for a separate enclosure dedicated to storing the array's disks, correct?
Yup. Our big stuff is in entirely separate racks from the servers.
How about backup applications?
Anything will work. Amanda, Bacula, you name it. We've done it all. It's more about how reliable your hardware is and what utility you feel comfortable with.
I'm currently using rsync with the --hard-links option and a shell script built years ago. It was built before Amanda had support for spanning tapes/disks.
Is there something that's robust and easier to setup than Amanda?
Look at bacula. It works pretty well.
---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks@alldigital.com - - AIM/Skype: therps2 ICQ: 22643734 Yahoo: origrps2 - - - - Do not taunt the sysadmins, for they are subtle and quick to anger - ----------------------------------------------------------------------
On 2015-03-05 09:48, Alex Regan wrote:
Hi, I have a fedora20 system acting as a backup server, and I've exceeded its capacity. I'd like to build a bigger one, probably using fedora21.
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
I'd like to have 6TB of usable space using 2TB disks.
Is ext4 still best for this?
Some RAID variant or is there something better?
Are there any NAS projects that may be beneficial?
Thanks for any ideas. Alex
A few years ago I read and article about the exact problem you are talking about. Long rebuild times with larger HDDs.
Two things they pointed out is that with the amount of data on the drives, there is also an issue with the write errors being within the amount of data during the rebuild process.
Of course the cloud providers want you to use their services and you can see that with all the recent articles about the death of backups due to clouds.
I got rid of my raid arrays after I lost a drive and lost part of the data on a rebuild. Still needed to use a different backup for recovery.
Some articles I have read are pointing out that people are looking for reliable file systems with error correction built in and then using normal backup techniques.
This is the same option that I am looking at for the future.
Robin
On Thu, Mar 5, 2015 at 9:48 AM, Alex Regan mysqlstudent@gmail.com wrote:
Hi, I have a fedora20 system acting as a backup server, and I've exceeded its capacity. I'd like to build a bigger one, probably using fedora21.
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long, so I'd like to consider another method of providing redundancy, and would like suggestions.
I'd like to have 6TB of usable space using 2TB disks.
If you're concerned about rebuild times, go with more small drives and move to raid6.
Another option is considering two 6TB drives in a raid1, per GB the 6TB Red drives are lower priced than the 2TB Reds. Of course, raid1 makes it more expensive in any case, but at reduced complexity.
And yet another option is GlusterFS either with multiple bricks on a single node, or multiple nodes. These can even be done with ARM boards. So you could have a volume that's for things you wan "3 copies" or, and have it automatically keep two local copies with one async offsite copy; another volume that's more disposable, e.g. maybe just one copy for VM images, and those would never get copied offsite.
CentOS 7's installer has Backup Client and Backup Server options as Add-Ons, I haven't looked into what that is, but might be an idea worth exploring.
Is ext4 still best for this?
Sure, it's fine, no complaints. I'd probably opt for XFS due to its long standing optimizations for both hardware and software RAID, parallel IO, etc. And piles of NAS, SANs, and even DVRs using XFS. It has a huge install base for this use case.
Some RAID variant or is there something better?
Are there any NAS projects that may be beneficial?
lwn.net has recent reviews of Rockstor, EasyNAS, OpenMediaVault, FreeNAS (BSD ZFS based).
On Thu, Mar 5, 2015 at 9:48 AM, Alex Regan mysqlstudent@gmail.com wrote:
I currently have a 3TB backup system using five 1TB disks in RAID5. Restore times in case of disk failure are already exceedingly long,
Oh yeah, several things you need to check if you're using mdadm created RAID.
md/stripe_cache_size in sysfs defaults to 256, it needs to be higher, probably 1024 but do some reading to get more specific advice. Low values cause slow performance in general, but in particular rebuilds.
These too may be too low by default, in particular max. /proc/sys/dev/raid/speed_limit_max /proc/sys/dev/raid/speed_limit_min
A pernicious problem that's totally non-obvious and comes up on linux-raid@ list all the time. Mismatching SCT ERC and SCSI command timer. The former needs to be shorter than the latter. Both are per drive, not per array.
smartctl -l scterc <dev> cat /sys/block/sdX/device/timeout
And don't forget period scrubs: echo check > /sys/block/mdX/md/sync_action cat /sys/block/mdX/mismatch_cnt
On Thu, Mar 5, 2015 at 12:30 PM, Rick Stevens ricks@alldigital.com wrote:
On 03/05/2015 10:53 AM, Alex Regan wrote:
Is btrfs used in production? I wasn't sure that it was fully stable yet.
btrfs is in production now. F21 fully supports it and (I think) it's the default for F21.
Btrfs isn't the default in Fedora 21 and isn't expected to be until Fedora 23 or 24.
The Btrfs raid56 code has been in the kernel for a while, but only since kernel-3.19 is btrfs replace, scrub, and self-heal possible. I'd say it's now a good deal closer to stable than experimental, but raid56 is still a gray area. I can't suggest enough having more than one backup, but that's emphasized more if the primary one is on Btrfs.
-
Alex -
Chris Murphy -
Rick Stevens -
Robin Laing