<font face="tahoma,sans-serif"><br></font><br><div class="gmail_quote">2011/7/13 Josef Bacik <span dir="ltr"><<a href="mailto:josef@toxicpanda.com">josef@toxicpanda.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div><div></div><div class="h5">On Wed, Jul 13, 2011 at 4:53 PM, Manuel Escudero <<a href="mailto:Jmlevick@gmail.com">Jmlevick@gmail.com</a>> wrote:<br>
> Today I'll be switching from BTRFS to Ext4 again because of the troubles<br>
> I've been having with<br>
> the New Linux Filesystem. As BTRFS is going to be the Default in F16 I<br>
> wanted the developers to<br>
> know what kind of troubles I've been experiencing with this FS in F15 so<br>
> they can take a look<br>
> at them in order to have a better F16 release:<br>
> The Good:<br>
> Since BTRFS arrived into my computer (Everything in the HDD is formated with<br>
> BTRFS excluding "/boot")<br>
> I've seen a performance improvement in the data transfer part from and to<br>
> the computer (copying files seem to<br>
> be faster than before) But that's all about the good things I noticed...<br>
> The Bad:<br>
> BTRFS has reduced system's overall performance, at this point, sometimes it<br>
> is OK, sometimes it is<br>
> VERY BAD, I've noticed "Performance Peaks" in F15 with BTRFS and the Boot<br>
> times are not nice: I mean,<br>
> they are not the slowest ones, but they're not as good as Before in F14 with<br>
> Ext4 instead of BTRFS.<br>
> The performance Running/Launching apps has been afected too and now the PC<br>
> freezes sometimes (that never<br>
> happened in F14 unless I forced it a lot with 4 VM's to suck the 4GB of RAM<br>
> I have); And Now it freezes<br>
> very often when it wants without a lot of effort.<br>
> The Ugly:<br>
> Running VM's when having their virtual HDD's stored in a BTRFS partition is<br>
> DEATH!<br>
> They're very slow, sometimes they open, sometimes they not, usually they<br>
> freeze, You can't<br>
> work with them. Same thing about Gnome Shell working over a BTRFS partition:<br>
> it is really slow,<br>
> sometimes it reacts but most of the time is pretty unresponsive.<br>
> Reading in the Web, I found that some users think that the BTRFS poor<br>
> performance is caused by some<br>
> special kind of fragmentation it suffers, others think it's because of it's<br>
> CopyonWrite attributes and some<br>
> others blame other stuff, God Knows! the only thing I know is that BTRFS is<br>
> not ready for being<br>
> used in normal production machines (as I tought) and it needs to be fixed<br>
> before the release of F16, because it's<br>
> performance is really far from good...<br>
> Other Stuff I noticed is that with Kernel 2.6.38.8-35 the system seems to<br>
> work better that with the previous one,<br>
> just a little, but is some kind of improvement.<br>
> Here you have all the info I found on the net about BTRFS Performance<br>
> issues noticed by users:<br>
> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=689127" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=689127</a><br>
> <a href="http://arosenfeld.wordpress.com/2010/12/27/back-to-ext4-from-btrfs/" target="_blank">http://arosenfeld.wordpress.com/2010/12/27/back-to-ext4-from-btrfs/</a><br>
> <a href="http://www.vyatta4people.org/btrfs-is-a-bad-choice-when-running-kvm/" target="_blank">http://www.vyatta4people.org/btrfs-is-a-bad-choice-when-running-kvm/</a><br>
> <a href="http://lkml.org/lkml/2010/7/13/475" target="_blank">http://lkml.org/lkml/2010/7/13/475</a><br>
> <a href="http://blog.patshead.com/2011/03/btrfs---six-months-later.html" target="_blank">http://blog.patshead.com/2011/03/btrfs---six-months-later.html</a><br>
> I only have a question:<br>
> Why Any Kind of VM is Sooo Slow when being stored on a BTRFS<br>
> partition? Any Way to Solve this? or at least have a BTRFS performance<br>
> improvement?<br>
<br>
</div></div>Yeah VMs are a particular problem with Btrfs. There are a ton of<br>
reasons for this, for example by default we use fsync. Fsync _sucks_<br>
for btrfs currently, and it has historically not been a well optimized<br>
piece of code. I'm working on fixing this, but it requires VFS level<br>
changes that are currently sitting in Al's queue. I suspect they will<br>
go into 3.1 and so we can move ahead with our work, but for now, it<br>
sucks. You can use cache=none you get better performance, but still<br>
not that great. And this is all because of one major thing<br>
<br>
Btrfs has threads for _everything_. This works out fantastically when<br>
you have big chunks of reads or writes you want done. This _sucks_<br>
when you are doing little piddly io's. The reason for all of this is<br>
because we don't want you to get bottlenecked on us<br>
calculating/verifying checksums, so we farm all IO and endio out to<br>
different threads, which as I said works out great if you are trying<br>
to cram gigs of data down your drives throat.<br>
<br>
But with VMs you are doing small scattered IO's, so the IO comes down,<br>
we prepare it, and farm it off to a thread and wait for that thread to<br>
wake up and submit the io. Then the io is completed and that is<br>
farmed off to another thread and we wait on that. This switching<br>
around and waiting for things to wake up is hugely painful when all<br>
you want to do is write a few bytes. If you were to do<br>
<br>
dd if=/dev/zero of=/mnt/btrfs/file bs=4k count=100 oflag=direct<br>
<br>
on a btrfs fs and then do it on an ext4 fs, you would see about a 20%<br>
difference between the 2. But if you do say bs=20M, the gap closes<br>
quite a bit.<br>
<br>
I fixed part of this problem for O_DIRECT (which is cache=none with<br>
qemu), if the IO's are small we don't send it off to a thread but<br>
submit it within our threads context, which is what got us with 20% of<br>
ext4 as opposed to 50%. The other half is doing the completion in the<br>
submitters context, which is going to take some extra work. I'm<br>
fixing this in the fsync case as well, but as I said we need a VFS<br>
patch to do it properly so that will be a little later coming. After<br>
that I can do the endio part of it and hopefully get us within<br>
spitting distance of ext4.<br>
<br>
So there's my long ass explanation of why VMs on Btrfs suck. I'm<br>
sorry, I'm aware of the problem and I'm trying to fix it, but it's a<br>
slow going process.<br>
<br>
As for your other spikes, can you test an upstream kernel? I've done<br>
various other performance things to try and get rid of those problems<br>
and would like to know how it helps. If the spikes last long enough a<br>
sysrq+w would be very helpful in seeing what is going on so we can try<br>
and address the problems you are seeing. Thanks,<br>
<br>
Josef<br>
<font color="#888888">--<br>
devel mailing list<br>
<a href="mailto:devel@lists.fedoraproject.org">devel@lists.fedoraproject.org</a><br>
<a href="https://admin.fedoraproject.org/mailman/listinfo/devel" target="_blank">https://admin.fedoraproject.org/mailman/listinfo/devel</a><br>
</font></blockquote></div><br><br><div>@Josef: Obviously you have more technical knowledge than I do,</div><div>but I understood the threads/few bytes problem. Somewhere near there</div><div>was my guessing about the VM's problem...</div>
<div><br></div><div>I'll be glad of bulding a vanilla kernel and then "mash it up" into my F15 install but:</div><div><br></div><div>A) This is a Very important production machine</div><div><br></div><div>and</div>
<div><br></div><div>B) I have a lot of work to do and not so much time... At this moment I'm making a Quick Backup </div><div>because I'm going to reinstall F15 with Ext4 as FileSystem :( part of my job is working</div>
<div>hand by hand with VM's, sorry..</div><div><br></div><div>Hope this thing get's solved before the F16 launch, thanks for the explanation!<br clear="all"><br>-- <br>Manuel Escudero<div><div>Linux User #509052<br>
Twitter: <a href="http://twitter.com/Jmlevick" target="_blank">@Jmlevick</a><br>Blogger: <a href="http://xenodesystems.blogspot.com/" target="_blank">Blog Xenode</a></div><div>PGP/GnuPG: E2F5 12FA E1C3 FA58 CF15 8481 B77B 00CA C1E1 0FA7</div>
</div>Xenode Systems - <a href="http://www.xenodesystems.com/" target="_blank">xenodesystems.com</a> - "Conéctate a Tu Mundo"<br>
</div>