Hi,
I've built an 11TB RAID5 array using fedora23 (soon to be fedora24) to be used for backup for a bunch of mail and web servers, and thought I'd inquire about the best filesystem to use.
I suppose I'm most familiar with ext4, although I understand xfs is the default filesystem during install now.
I believe most files will be maildir style email files, as well as maillogs, and some tar.bz2 files, so the file size ranges will be varied.
We're using a homegrown backup script that uses rsync with the --link-dest option to preserve space. I'd like to move to bacula, but don't have the time/resources right now.
I'm concerned about recovery time for failed disks, but that's probably a conversation for another post.
I thought the info below would be helpful - /proc/mdstat and hdparm output.
md125 : active raid5 sdh1[4] sde1[0] sdf1[1] sdg1[2] 11720656896 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU] bitmap: 0/30 pages [0KB], 65536KB chunk
# hdparm -i /dev/sdh
/dev/sdh:
Model=WDC WD4000FYYZ-01UL1B3, FwRev=01.01K04, SerialNo=WD-WMC130F0NJV0 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=7814037168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
Thanks, Alex
On Mon, 2016-10-10 at 11:25 -0400, Alex wrote:
I suppose I'm most familiar with ext4, although I understand xfs is the default filesystem during install now.
Where did you read that? AFAIK the default continues to be ext4.
I believe most files will be maildir style email files, as well as maillogs, and some tar.bz2 files, so the file size ranges will be varied.
We're using a homegrown backup script that uses rsync with the --link-dest option to preserve space. I'd like to move to bacula, but don't have the time/resources right now.
You might want to look at rsnapshot (www.rsnapshot.org), which is based on rsync.
poc
On 10/10/2016 09:17 AM, Patrick O'Callaghan wrote:
On Mon, 2016-10-10 at 11:25 -0400, Alex wrote:
I suppose I'm most familiar with ext4, although I understand xfs is the default filesystem during install now.
Where did you read that? AFAIK the default continues to be ext4.
Alex is thinking of RHEL/CentOS 7, whose default is XFS. ext4 is probably a better option for a filesystem with a large number of small files. XFS continues to be slower for metadata operations.
On 10/12/2016 08:53 AM, Heinz Diehl wrote:
On 10.10.2016, Gordon Messmer wrote:
Ext4 is probably a better option for a filesystem with a large number of small files. XFS continues to be slower for metadata operations.
It was, some years ago. This is no longer the case.
Given those, would you go so far as to recommend xfs in lieu of ext4 for general usage?
On 12Oct2016 09:40, Mike Wright nobody@nospam.hostisimo.com wrote:
On 10/12/2016 08:53 AM, Heinz Diehl wrote:
On 10.10.2016, Gordon Messmer wrote:
Ext4 is probably a better option for a filesystem with a large number of small files. XFS continues to be slower for metadata operations.
It was, some years ago. This is no longer the case.
Given those, would you go so far as to recommend xfs in lieu of ext4 for general usage?
Yes.
Cameron Simpson cs@zip.com.au
On 10/12/2016 08:53 AM, Heinz Diehl wrote:
It was, some years ago. This is no longer the case.
I suggest that users evaluate their options under their own workload. When I ran tests last year on CentOS 7 for rsnapshot storing maildirs (the exact workload in question here), ext4 was significantly faster on the hardware I was using.
On Wed, Oct 12, 2016 at 2:03 PM, Gordon Messmer gordon.messmer@gmail.com wrote:
On 10/12/2016 08:53 AM, Heinz Diehl wrote:
It was, some years ago. This is no longer the case.
I suggest that users evaluate their options under their own workload. When I ran tests last year on CentOS 7 for rsnapshot storing maildirs (the exact workload in question here), ext4 was significantly faster on the hardware I was using.
There are some other considerations. XFS has a good track record for large numerical calculations, video production, and remote sensing. These are applications where the data are lost due to downtime recovering from a crash are much larger than the data that was "in flight" when the crash occurred. XFS puts priority on metadata consistency, so filesystems are up and running very quickly after a crash.
There are applications where losing a few bytes of data should "never" happen, even it results in substantial downtime.
users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-leave@lists.fedoraproject.org
Hi,
On Mon, Oct 10, 2016 at 12:17 PM, Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Mon, 2016-10-10 at 11:25 -0400, Alex wrote:
I suppose I'm most familiar with ext4, although I understand xfs is the default filesystem during install now.
Where did you read that? AFAIK the default continues to be ext4.
I thought I recalled it being the default during install the last time I performed one.
We're using a homegrown backup script that uses rsync with the --link-dest option to preserve space. I'd like to move to bacula, but don't have the time/resources right now.
You might want to look at rsnapshot (www.rsnapshot.org), which is based on rsync.
Thanks, I'll check it out.
On 10/10/2016 03:50 PM, Alex wrote:
Where did you read that? AFAIK the default continues to be ext4.
I thought I recalled it being the default during install the last time I performed one.
The Fedora Server installation defaults to XFS. Workstation defaults to ext4.
For your use case either filesystem will work. Use what you are comfortable with. Both are stable, supported, and will give you full performance from your setup.
On 10Oct2016 15:59, Michael Cronenworth mike@cchtml.com wrote:
On 10/10/2016 03:50 PM, Alex wrote:
Where did you read that? AFAIK the default continues to be ext4.
I thought I recalled it being the default during install the last time I performed one.
The Fedora Server installation defaults to XFS. Workstation defaults to ext4.
For your use case either filesystem will work. Use what you are comfortable with. Both are stable, supported, and will give you full performance from your setup.
Use XFS. It is stable; never needs fsck. If ext4 needs to repair it will take days/weeks on a filesystem that size, and need insane amounts of RAM (if the NAS is hosting this, it may not have much RAM).
Both will work until you need to fsck (eg power outage or other instant/unclean shutdown). After that, you will wish you were using XFS.
Cheers, Cameron Simpson cs@zip.com.au
Hi,
On Mon, Oct 10, 2016 at 5:58 PM, cs@zip.com.au wrote:
On 10Oct2016 15:59, Michael Cronenworth mike@cchtml.com wrote:
On 10/10/2016 03:50 PM, Alex wrote:
Where did you read that? AFAIK the default continues to be ext4.
I thought I recalled it being the default during install the last time I performed one.
The Fedora Server installation defaults to XFS. Workstation defaults to ext4.
For your use case either filesystem will work. Use what you are comfortable with. Both are stable, supported, and will give you full performance from your setup.
Use XFS. It is stable; never needs fsck. If ext4 needs to repair it will take days/weeks on a filesystem that size, and need insane amounts of RAM (if the NAS is hosting this, it may not have much RAM).
Both will work until you need to fsck (eg power outage or other instant/unclean shutdown). After that, you will wish you were using XFS.
Awesome, thanks guys.
On 10/10/2016 04:58 PM, cs@zip.com.au wrote:
Use XFS. It is stable; never needs fsck. If ext4 needs to repair it will take days/weeks on a filesystem that size, and need insane amounts of RAM (if the NAS is hosting this, it may not have much RAM).
Both will work until you need to fsck (eg power outage or other instant/unclean shutdown). After that, you will wish you were using XFS.
"Never needs fsck"? What crazy alternate reality do you live in?
Outrageous claims like this just don't have any place on this list. If you want to have a rant you're welcome to start a blog.
On 10/12/2016 06:40 AM, Michael Cronenworth wrote:
"Never needs fsck"? What crazy alternate reality do you live in?
It's only slightly exaggerated. XFS has online fsck, which means that the kernel can fix some errors as it encounters them. Others... well, I *have* seen XFS require an offline fsck on a Linux NAS appliance, so I know "never" isn't literally true.
On 12Oct2016 08:40, Gordon Messmer gordon.messmer@gmail.com wrote:
On 10/12/2016 06:40 AM, Michael Cronenworth wrote:
"Never needs fsck"? What crazy alternate reality do you live in?
It's only slightly exaggerated. XFS has online fsck, which means that the kernel can fix some errors as it encounters them. Others... well, I *have* seen XFS require an offline fsck on a Linux NAS appliance, so I know "never" isn't literally true.
Except it the wildest scenarios, XFS fsks at mount, almost immediately. Go and cat (yes, cat) the fsck.xfs command. Its data processes are reliable and well behaved, and it is extensively tested.
"Never" is indeed not _literally_ true, but it is effectively true, far far far far more than is so with ext4. Ext4 really needs fsck after an unclean unmount, and it is not cheap for large filesystems.
The two are like night and day in the recovery scenario (== xfs pretty much never needs manual recovery, and recover is very fast). Even mkfs is light night and day with xfs vs ext4.
Cheers, Cameron Simpson cs@zip.com.au
On 10/12/2016 03:57 PM, cs@zip.com.au wrote:
Except it the wildest scenarios, XFS fsks at mount, almost immediately.
Is that different from fsck.ext4 replaying the journal?
Go and cat (yes, cat) the fsck.xfs command.
OK. I'm not sure what you think I'll learn by doing so. fsck.xfs exists because Linux's "fsck" supports multiple filesystems and Irix's did not. On Irix, fsck was exclusively for EFS. XFS used xfs_check and xfs_repair. When XFS was ported to Linux, the tools were not renamed; instead fsck.xfs was put in place to direct users to the correct tool when they ran "fsck -t xfs".
None of that makes XFS immune to corruption, nor reduce the time or the memory required to fix an XFS filesystem if it's damaged.
"Never" is indeed not _literally_ true, but it is effectively true, far far far far more than is so with ext4. Ext4 really needs fsck after an unclean unmount, and it is not cheap for large filesystems.
With journaling and write barriers in place, I don't know any reason that ext4 would be any more affected by a power loss than XFS would be, and having had to perform recoveries on both, I don't find one to be significantly better than the other.
Thus, my advice remains that users should test both because they do differ in performance in different workloads. If one will be faster under normal operation, 99.95% of the time, and require slightly more down time to recover in the other .05% of the time, then selecting the option which is superior 99.95% of the time is a perfectly rational thing to do.
That's especially true of backup systems where down time does not carry the same impact as down time in a production system does.
The two are like night and day in the recovery scenario (== xfs pretty much never needs manual recovery, and recover is very fast).
That seems subjective. I've personally had to recover more XFS filesystems than I have ext4 filesystems, and I use ext4 filesystems more often. My subjective experience is quite different than yours.
On 13Oct2016 14:29, Gordon Messmer gordon.messmer@gmail.com wrote:
On 10/12/2016 03:57 PM, cs@zip.com.au wrote:
Except it the wildest scenarios, XFS fsks at mount, almost immediately.
Is that different from fsck.ext4 replaying the journal?
Not fundamentally.
Go and cat (yes, cat) the fsck.xfs command.
OK. I'm not sure what you think I'll learn by doing so.
Only that it is a shell script that handles the typically used boot options, but doesn't do anything else.
fsck.xfs exists because Linux's "fsck" supports multiple filesystems and Irix's did not. On Irix, fsck was exclusively for EFS. XFS used xfs_check and xfs_repair. When XFS was ported to Linux, the tools were not renamed; instead fsck.xfs was put in place to direct users to the correct tool when they ran "fsck -t xfs".
And when they really need a repair that the XFS mount process doesn't accomplish.
None of that makes XFS immune to corruption, nor reduce the time or the memory required to fix an XFS filesystem if it's damaged.
Of course not, though I have the impression that the repair tool is less memory intensive than the ext4 equivalent.
"Never" is indeed not _literally_ true, but it is effectively true, far far far far more than is so with ext4. Ext4 really needs fsck after an unclean unmount, and it is not cheap for large filesystems.
With journaling and write barriers in place, I don't know any reason that ext4 would be any more affected by a power loss than XFS would be, and having had to perform recoveries on both, I don't find one to be significantly better than the other.
Fair enough. We may be into the realm where a question on an XFS list may be more illuminating about the modern state of play.
Thus, my advice remains that users should test both because they do differ in performance in different workloads. If one will be faster under normal operation, 99.95% of the time, and require slightly more down time to recover in the other .05% of the time, then selecting the option which is superior 99.95% of the time is a perfectly rational thing to do.
That's especially true of backup systems where down time does not carry the same impact as down time in a production system does.
The two are like night and day in the recovery scenario (== xfs pretty much never needs manual recovery, and recover is very fast).
That seems subjective. I've personally had to recover more XFS filesystems than I have ext4 filesystems, and I use ext4 filesystems more often. My subjective experience is quite different than yours.
Apparently so. I've had very good experiences with XFS, and plan to stick with it at present.
Out of interest, how recent was your need to recover XFS, and what were the circumstances that caused it to be necessary?
Cheers, Cameron Simpson cs@zip.com.au
On 10/13/2016 11:49 PM, cs@zip.com.au wrote:
Out of interest, how recent was your need to recover XFS, and what were the circumstances that caused it to be necessary?
I don't remember the date, specifically, and it's not clear what the circumstances are. One of the consequences of on-line fs checking is that errors tend to be detected long after they actually occur.
On Sat, Oct 15, 2016 at 6:25 PM, Gordon Messmer gordon.messmer@gmail.com wrote:
On 10/13/2016 11:49 PM, cs@zip.com.au wrote:
Out of interest, how recent was your need to recover XFS, and what were the circumstances that caused it to be necessary?
I don't remember the date, specifically, and it's not clear what the circumstances are. One of the consequences of on-line fs checking is that errors tend to be detected long after they actually occur.
Not sure what you mean -- certainly with most linux distros fsck gets run at intervals on clean reboot and does on rare occasions find problems to repair.
I've never seen a problem with XFS filesystems on Scientific Linux 7, but I have a lot more system hours experience with xfs on SGI Irix, where the first indication that a disk was failing would come from errors running the nightly fsr. When XFS was first ported to linux I used it on 32-bit linux to recover data disks removed from dead SGI workstations. The XFS utilities were extremely slow in that environment. One reason given for the poor performance was that XFS was written with deeply nested function calls with long argument lists that were very inefficient on 32-bit Intel hardware at that time.
On 10/16/2016 05:42 AM, George N. White III wrote:
I don't remember the date, specifically, and it's not clear what the circumstances are. One of the consequences of on-line fs checking is that errors tend to be detected long after they actually occur.Not sure what you mean -- certainly with most linux distros fsck gets run at intervals on clean reboot and does on rare occasions find problems to repair.
Yes, but as we've discussed, fsck.xfs (and hence, fsck -t xfs) is a no-op. XFS is checked and repaired on-line, as it's used, in the case of simple errors. If corruption is introduced, it may not be found until some time later, in which case xfs_repair will be needed.
On 10/10/2016 05:25 PM, Alex wrote:
Hi,
I've built an 11TB RAID5 array using fedora23 (soon to be fedora24) to be used for backup for a bunch of mail and web servers, and thought I'd inquire about the best filesystem to use.
I suppose I'm most familiar with ext4, although I understand xfs is the default filesystem during install now.
I've been doing this for more than ten years: massive backups with rsync and --link-dest.
I started with reiserfs and it went on for more than ten years without any problem. Then I rebuilt the machines and decided to move to ext4. Ironically enough, I got into a serious bug in ext4 online expansion that caused data corruption. It is now solved, of course. Today, I would go for ext4. I have just a couple of system still on reiserfs, with very old (read: proven and trusted) software versions.
Regards.
-
Alex -
Cameron Simpson -
George N. White III -
Gordon Messmer -
Heinz Diehl -
Michael Cronenworth -
Mike Wright -
Patrick O'Callaghan -
Roberto Ragusa