8 p.m.
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
So, there was recently a Thing where btrfs installs were broken, and
this got accepted as a release blocker:
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
Summary: This bug was introduced and discovered in linux-next, it
started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
resulted in a somewhat transient deadlock which caused installs to
hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly
how much faster should it have been fixed in order to avoid worrying
about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@
7/25 22:33 utc bug was first reported in Fedora bugzilla
7/26 19:20 utc I confirmed upstream's patch related to this bug with
upstream and updated the Fedora bug
7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option
in the installer and if there are bugs they Beta blocking, how could
or should this have been fixed sooner? What about the handling should
have been different?
I note here that ext2 and ext3 are offered as file systems in
Custom/Advanced partitioning and in this sense have parity with Btrfs.
If this same bug occurred in ext2 or ext3 would or should that cause
discussion to drop them from the installer, even if the bug were fixed
within 24 hours of discovery and patch? What about vfat? That's
literally the only truly required filesystem that must work, for the
most commonly supported hardware so it can't be dropped, we'd just be
stuck until it got fixed. That work would have to be done upstream,
yes?
--
Chris Murphy
7:41 a.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Fri, Aug 23, 2019 at 8:00 PM Chris Murphy <lists(a)colorremedies.com> wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
> So, there was recently a Thing where btrfs installs were broken, and
> this got accepted as a release blocker:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1733388
Summary: This bug was introduced and discovered in linux-next, it
started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
resulted in a somewhat transient deadlock which caused installs to
hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly
how much faster should it have been fixed in order to avoid worrying
about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@
7/25 22:33 utc bug was first reported in Fedora bugzilla
7/26 19:20 utc I confirmed upstream's patch related to this bug with
upstream and updated the Fedora bug
7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option
in the installer and if there are bugs they Beta blocking, how could
or should this have been fixed sooner? What about the handling should
have been different?
I note here that ext2 and ext3 are offered as file systems in
Custom/Advanced partitioning and in this sense have parity with Btrfs.
If this same bug occurred in ext2 or ext3 would or should that cause
discussion to drop them from the installer, even if the bug were fixed
within 24 hours of discovery and patch? What about vfat? That's
literally the only truly required filesystem that must work, for the
most commonly supported hardware so it can't be dropped, we'd just be
stuck until it got fixed. That work would have to be done upstream,
yes?
From my standpoint, ext4 and xfs are the primary supported root
filesystems. I don't think that anything else should be release
blocking. As for whether the installer exposes other options, it is
really more of an installer and QA question. We are certainly not even
discussing turning them off in the kernel at this point, and I don't
think that we should.
> --
> Chris Murphy
> _______________________________________________
> kernel mailing list -- kernel(a)lists.fedoraproject.org
> To unsubscribe send an email to kernel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
10:16 a.m.
New subject: Discussion: what would not blocking on btrfs look like?
On 8/23/19 9:00 PM, Chris Murphy wrote:
On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
<adamwill(a)fedoraproject.org> wrote:
> So, there was recently a Thing where btrfs installs were broken, and
> this got accepted as a release blocker:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1733388
Summary: This bug was introduced and discovered in linux-next, it
started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
resulted in a somewhat transient deadlock which caused installs to
hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
deletions (1/2 the insertions are comments).
How remarkable or interesting is this bug? And in particular, exactly
how much faster should it have been fixed in order to avoid worrying
about it being a blocker bug?
7/25 14:27 utc bug patch was submitted to linux-btrfs@
7/25 22:33 utc bug was first reported in Fedora bugzilla
7/26 19:20 utc I confirmed upstream's patch related to this bug with
upstream and updated the Fedora bug
7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
So in the context of status quo, where Btrfs is presented as an option
in the installer and if there are bugs they Beta blocking, how could
or should this have been fixed sooner? What about the handling should
have been different?
That's a fair question. This bug actually represents how this _should_
work. The concern is that in the past we haven't seen a lot engagement
in the past. Maybe today that has changed as demonstrated by this thread.
I'm still concerned about having this be a blocker vs. just keeping it
as an option, simply because a blocker stops the entire release and it
can be a last minute scramble to get things fixed. This was the ideal
case for a blocker bugs and I'm skeptical about all bugs going this well.
If we had a few more people who were willing to be on the btrfs alias and
do the work for blocker bugs it would be a much stronger case.
I note here that ext2 and ext3 are offered as file systems in
Custom/Advanced partitioning and in this sense have parity with Btrfs.
If this same bug occurred in ext2 or ext3 would or should that cause
discussion to drop them from the installer, even if the bug were fixed
within 24 hours of discovery and patch? What about vfat? That's
literally the only truly required filesystem that must work, for the
most commonly supported hardware so it can't be dropped, we'd just be
stuck until it got fixed. That work would have to be done upstream,
yes?
I don't think that's really a fair comparison. Just because options
are presented doesn't mean all of them are equal. ext2/ext3 and vfat
have been in development for much longer than btrfs and length of development
is something that's particularly important for file system stability
from talking with file system developers. It's not impossible for there
to be bugs in ext4 for example (we've certainly seen them before) but
btrfs is only now gaining overall stability and we're still more likely to see
bugs, especially with custom setups where people are likely to find
edge cases.
Thanks,
Laura
10:39 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott <labbott(a)redhat.com> wrote:
On 8/23/19 9:00 PM, Chris Murphy wrote:
> On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
> <adamwill(a)fedoraproject.org> wrote:
>
>> So, there was recently a Thing where btrfs installs were broken, and
>> this got accepted as a release blocker:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1733388
>
> Summary: This bug was introduced and discovered in linux-next, it
> started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
> appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
> resulted in a somewhat transient deadlock which caused installs to
> hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
> deletions (1/2 the insertions are comments).
>
> How remarkable or interesting is this bug? And in particular, exactly
> how much faster should it have been fixed in order to avoid worrying
> about it being a blocker bug?
>
> 7/25 14:27 utc bug patch was submitted to linux-btrfs@
> 7/25 22:33 utc bug was first reported in Fedora bugzilla
> 7/26 19:20 utc I confirmed upstream's patch related to this bug with
> upstream and updated the Fedora bug
> 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
>
> So in the context of status quo, where Btrfs is presented as an option
> in the installer and if there are bugs they Beta blocking, how could
> or should this have been fixed sooner? What about the handling should
> have been different?
>
That's a fair question. This bug actually represents how this _should_
work. The concern is that in the past we haven't seen a lot engagement
in the past. Maybe today that has changed as demonstrated by this thread.
I'm still concerned about having this be a blocker vs. just keeping it
as an option, simply because a blocker stops the entire release and it
can be a last minute scramble to get things fixed. This was the ideal
case for a blocker bugs and I'm skeptical about all bugs going this well.
If we had a few more people who were willing to be on the btrfs alias and
do the work for blocker bugs it would be a much stronger case.
Out of curiosity, how many such issues have we had in the past 2
years? I personally can't recall any monumental occasions where people
were scrambling over *Btrfs* in Fedora. If anything, we continue to
inherit the work that SUSE and Facebook are doing upstream as part of
us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached
out to btrfs folks in Fedora? Chris and myself are the current ones,
but there have been others in the past. Both of us are subscribed to
the linux-btrfs mailing list, and Chris has a decent rapport with most
of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't
have any for the majority of filesystems Fedora supports, only XFS. Is
there some kind of problem with communicating with the upstream kernel
developers about Fedora bugs that I'm not aware of?
> I note here that ext2 and ext3 are offered as file systems in
> Custom/Advanced partitioning and in this sense have parity with Btrfs.
> If this same bug occurred in ext2 or ext3 would or should that cause
> discussion to drop them from the installer, even if the bug were fixed
> within 24 hours of discovery and patch? What about vfat? That's
> literally the only truly required filesystem that must work, for the
> most commonly supported hardware so it can't be dropped, we'd just be
> stuck until it got fixed. That work would have to be done upstream,
> yes?
>
I don't think that's really a fair comparison. Just because options
are presented doesn't mean all of them are equal. ext2/ext3 and vfat
have been in development for much longer than btrfs and length of development
is something that's particularly important for file system stability
from talking with file system developers. It's not impossible for there
to be bugs in ext4 for example (we've certainly seen them before) but
btrfs is only now gaining overall stability and we're still more likely to see
bugs, especially with custom setups where people are likely to find
edge cases.
Nope. We can totally use this because LVM has not existed as long (we
use LVM + filesystem by default, not plain partitions), and we still
encounter quirks with things like thinp LVM combined with these
filesystems. OverlayFS is mostly hot garbage (kernel people know it,
container people know it, filesystem people know it, etc.), and yet we
continue to try to use it in more places. Stratis is in an odd state
of limbo now, since its main developer and advocate left Red Hat.
There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here,
but in this realm, it seems like it is...
--
真実はいつも一つ！/ Always, there's only one truth!
6:53 a.m.
New subject: Discussion: what would not blocking on btrfs look like?
On 8/26/19 11:39 PM, Neal Gompa wrote:
On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott
<labbott(a)redhat.com> wrote:
>
> On 8/23/19 9:00 PM, Chris Murphy wrote:
>> On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
>> <adamwill(a)fedoraproject.org> wrote:
>>
>>> So, there was recently a Thing where btrfs installs were broken, and
>>> this got accepted as a release blocker:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1733388
>>
>> Summary: This bug was introduced and discovered in linux-next, it
>> started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
>> appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
>> resulted in a somewhat transient deadlock which caused installs to
>> hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
>> deletions (1/2 the insertions are comments).
>>
>> How remarkable or interesting is this bug? And in particular, exactly
>> how much faster should it have been fixed in order to avoid worrying
>> about it being a blocker bug?
>>
>> 7/25 14:27 utc bug patch was submitted to linux-btrfs@
>> 7/25 22:33 utc bug was first reported in Fedora bugzilla
>> 7/26 19:20 utc I confirmed upstream's patch related to this bug with
>> upstream and updated the Fedora bug
>> 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora bug
>>
>> So in the context of status quo, where Btrfs is presented as an option
>> in the installer and if there are bugs they Beta blocking, how could
>> or should this have been fixed sooner? What about the handling should
>> have been different?
>>
>
> That's a fair question. This bug actually represents how this _should_
> work. The concern is that in the past we haven't seen a lot engagement
> in the past. Maybe today that has changed as demonstrated by this thread.
> I'm still concerned about having this be a blocker vs. just keeping it
> as an option, simply because a blocker stops the entire release and it
> can be a last minute scramble to get things fixed. This was the ideal
> case for a blocker bugs and I'm skeptical about all bugs going this well.
> If we had a few more people who were willing to be on the btrfs alias and
> do the work for blocker bugs it would be a much stronger case.
>
Out of curiosity, how many such issues have we had in the past 2
years? I personally can't recall any monumental occasions where people
were scrambling over *Btrfs* in Fedora. If anything, we continue to
inherit the work that SUSE and Facebook are doing upstream as part of
us continually updating our kernels, which I'm grateful for.
And in the instances where we've had such issues, has anyone reached
out to btrfs folks in Fedora? Chris and myself are the current ones,
but there have been others in the past. Both of us are subscribed to
the linux-btrfs mailing list, and Chris has a decent rapport with most
of the btrfs developers.
What more do you want? Actual btrfs developers in Fedora? We don't
have any for the majority of filesystems Fedora supports, only XFS. Is
there some kind of problem with communicating with the upstream kernel
developers about Fedora bugs that I'm not aware of?
Again, it's about length of overall development. ext and XFS have
a much longer history in general which is something that's important
for file system stability in general. It's also a bit of a catch-22
where the rate of btrfs use in Fedora is so low we don't actually
see issues.
>> I note here that ext2 and ext3 are offered as file systems
in
>> Custom/Advanced partitioning and in this sense have parity with Btrfs.
>> If this same bug occurred in ext2 or ext3 would or should that cause
>> discussion to drop them from the installer, even if the bug were fixed
>> within 24 hours of discovery and patch? What about vfat? That's
>> literally the only truly required filesystem that must work, for the
>> most commonly supported hardware so it can't be dropped, we'd just be
>> stuck until it got fixed. That work would have to be done upstream,
>> yes?
>>
>
> I don't think that's really a fair comparison. Just because options
> are presented doesn't mean all of them are equal. ext2/ext3 and vfat
> have been in development for much longer than btrfs and length of development
> is something that's particularly important for file system stability
> from talking with file system developers. It's not impossible for there
> to be bugs in ext4 for example (we've certainly seen them before) but
> btrfs is only now gaining overall stability and we're still more likely to see
> bugs, especially with custom setups where people are likely to find
> edge cases.
>
Nope. We can totally use this because LVM has not existed as long (we
use LVM + filesystem by default, not plain partitions), and we still
encounter quirks with things like thinp LVM combined with these
filesystems. OverlayFS is mostly hot garbage (kernel people know it,
container people know it, filesystem people know it, etc.), and yet we
continue to try to use it in more places. Stratis is in an odd state
of limbo now, since its main developer and advocate left Red Hat.
> There are plenty of examples of Red Hat doing crazy/experimental
things... I'd like to think Red Hat isn't supposed to be special here,
but in this realm, it seems like it is...
btrfs still doesn't give me the warm fuzzies and I also think this
is a bigger issue than other features simply because user data is at
stake. We do need to consider that the failure case is not "I can't do X"
but "my precious data which I have been trying to snapshot is now
inaccessible" in a way that's even worse than say rpm database
corruption. Even if it is in the advanced partitioning or not the
default, we can still end up with people clicking in because they
read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and
btrfs as release criteria. I think you are correct that there's
enough community support to justify keeping btrfs around at least
in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident
about that if we could have a file system developer on the btrfs
alias. I'm glad to hear the btrfs upstream community has been
receptive to bugs but it's still much easier to make things
happen if we have contributors who are active in the Fedora
community, especially if we want the advanced features that
btrfs has (which is why people want it anyway). So, who would
you suggest to work with us in Fedora?
Thanks,
Laura
12:58 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
On 8/26/19 11:39 PM, Neal Gompa wrote:
> On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott <labbott(a)redhat.com> wrote:
> >
> > On 8/23/19 9:00 PM, Chris Murphy wrote:
> > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
> > > <adamwill(a)fedoraproject.org> wrote:
> > >
> > > > So, there was recently a Thing where btrfs installs were broken, and
> > > > this got accepted as a release blocker:
> > > >
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388
> > >
> > > Summary: This bug was introduced and discovered in linux-next, it
> > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
> > > appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
> > > resulted in a somewhat transient deadlock which caused installs to
> > > hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
> > > deletions (1/2 the insertions are comments).
> > >
> > > How remarkable or interesting is this bug? And in particular, exactly
> > > how much faster should it have been fixed in order to avoid worrying
> > > about it being a blocker bug?
> > >
> > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@
> > > 7/25 22:33 utc bug was first reported in Fedora bugzilla
> > > 7/26 19:20 utc I confirmed upstream's patch related to this bug with
> > > upstream and updated the Fedora bug
> > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora
bug
> > >
> > > So in the context of status quo, where Btrfs is presented as an option
> > > in the installer and if there are bugs they Beta blocking, how could
> > > or should this have been fixed sooner? What about the handling should
> > > have been different?
> > >
> >
> > That's a fair question. This bug actually represents how this _should_
> > work. The concern is that in the past we haven't seen a lot engagement
> > in the past. Maybe today that has changed as demonstrated by this thread.
> > I'm still concerned about having this be a blocker vs. just keeping it
> > as an option, simply because a blocker stops the entire release and it
> > can be a last minute scramble to get things fixed. This was the ideal
> > case for a blocker bugs and I'm skeptical about all bugs going this well.
> > If we had a few more people who were willing to be on the btrfs alias and
> > do the work for blocker bugs it would be a much stronger case.
> >
>
> Out of curiosity, how many such issues have we had in the past 2
> years? I personally can't recall any monumental occasions where people
> were scrambling over *Btrfs* in Fedora. If anything, we continue to
> inherit the work that SUSE and Facebook are doing upstream as part of
> us continually updating our kernels, which I'm grateful for.
>
> And in the instances where we've had such issues, has anyone reached
> out to btrfs folks in Fedora? Chris and myself are the current ones,
> but there have been others in the past. Both of us are subscribed to
> the linux-btrfs mailing list, and Chris has a decent rapport with most
> of the btrfs developers.
>
> What more do you want? Actual btrfs developers in Fedora? We don't
> have any for the majority of filesystems Fedora supports, only XFS. Is
> there some kind of problem with communicating with the upstream kernel
> developers about Fedora bugs that I'm not aware of?
>
Again, it's about length of overall development. ext and XFS have
a much longer history in general which is something that's important
for file system stability in general. It's also a bit of a catch-22
where the rate of btrfs use in Fedora is so low we don't actually
see issues.
> > > I note here that ext2 and ext3 are offered as file systems in
> > > Custom/Advanced partitioning and in this sense have parity with Btrfs.
> > > If this same bug occurred in ext2 or ext3 would or should that cause
> > > discussion to drop them from the installer, even if the bug were fixed
> > > within 24 hours of discovery and patch? What about vfat? That's
> > > literally the only truly required filesystem that must work, for the
> > > most commonly supported hardware so it can't be dropped, we'd just
be
> > > stuck until it got fixed. That work would have to be done upstream,
> > > yes?
> > >
> >
> > I don't think that's really a fair comparison. Just because options
> > are presented doesn't mean all of them are equal. ext2/ext3 and vfat
> > have been in development for much longer than btrfs and length of development
> > is something that's particularly important for file system stability
> > from talking with file system developers. It's not impossible for there
> > to be bugs in ext4 for example (we've certainly seen them before) but
> > btrfs is only now gaining overall stability and we're still more likely to
see
> > bugs, especially with custom setups where people are likely to find
> > edge cases.
> >
>
> Nope. We can totally use this because LVM has not existed as long (we
> use LVM + filesystem by default, not plain partitions), and we still
> encounter quirks with things like thinp LVM combined with these
> filesystems. OverlayFS is mostly hot garbage (kernel people know it,
> container people know it, filesystem people know it, etc.), and yet we
> continue to try to use it in more places. Stratis is in an odd state
> of limbo now, since its main developer and advocate left Red Hat.
> > There are plenty of examples of Red Hat doing crazy/experimental
> things... I'd like to think Red Hat isn't supposed to be special here,
> but in this realm, it seems like it is...
>
>
btrfs still doesn't give me the warm fuzzies and I also think this
is a bigger issue than other features simply because user data is at
stake. We do need to consider that the failure case is not "I can't do X"
but "my precious data which I have been trying to snapshot is now
inaccessible" in a way that's even worse than say rpm database
corruption. Even if it is in the advanced partitioning or not the
default, we can still end up with people clicking in because they
read an article about how btrfs was the hot new thing.
There are two parts to this here: killing off btrfs entirely and
btrfs as release criteria. I think you are correct that there's
enough community support to justify keeping btrfs around at least
in the kernel (I can't speak for anaconda here)
As for btrfs as release criteria, I'd feel much more confident
about that if we could have a file system developer on the btrfs
alias. I'm glad to hear the btrfs upstream community has been
receptive to bugs but it's still much easier to make things
happen if we have contributors who are active in the Fedora
community, especially if we want the advanced features that
btrfs has (which is why people want it anyway). So, who would
you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize
from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty
happy with how it works stability wise. That being said we're slightly more
fault tolerant than most users. If you guys are hitting problems chances are
we'll hit them eventually as well, so it makes sense for us to be on top of
them.
I agree it would be better if somebody inside Fedora was able to help out, but
again I'm only an email away. Thanks,
Josef
1:35 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On 8/28/19 1:58 PM, Josef Bacik wrote:
On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
> On 8/26/19 11:39 PM, Neal Gompa wrote:
>> On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott <labbott(a)redhat.com> wrote:
>>>
>>> On 8/23/19 9:00 PM, Chris Murphy wrote:
>>>> On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
>>>> <adamwill(a)fedoraproject.org> wrote:
>>>>
>>>>> So, there was recently a Thing where btrfs installs were broken, and
>>>>> this got accepted as a release blocker:
>>>>>
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1733388
>>>>
>>>> Summary: This bug was introduced and discovered in linux-next, it
>>>> started to affect Fedora 5.3.0-rc0 kernels in openqa tests, patch
>>>> appeared during rc1, and the patch was merged into 5.3.0-rc2. The bug
>>>> resulted in a somewhat transient deadlock which caused installs to
>>>> hang, but no corruption. The fix, 2 files changed, 12 insertions, 8
>>>> deletions (1/2 the insertions are comments).
>>>>
>>>> How remarkable or interesting is this bug? And in particular, exactly
>>>> how much faster should it have been fixed in order to avoid worrying
>>>> about it being a blocker bug?
>>>>
>>>> 7/25 14:27 utc bug patch was submitted to linux-btrfs@
>>>> 7/25 22:33 utc bug was first reported in Fedora bugzilla
>>>> 7/26 19:20 utc I confirmed upstream's patch related to this bug with
>>>> upstream and updated the Fedora bug
>>>> 7/26 22:50 utc I confirmed it was merged into rc2, and updated the Fedora
bug
>>>>
>>>> So in the context of status quo, where Btrfs is presented as an option
>>>> in the installer and if there are bugs they Beta blocking, how could
>>>> or should this have been fixed sooner? What about the handling should
>>>> have been different?
>>>>
>>>
>>> That's a fair question. This bug actually represents how this _should_
>>> work. The concern is that in the past we haven't seen a lot engagement
>>> in the past. Maybe today that has changed as demonstrated by this thread.
>>> I'm still concerned about having this be a blocker vs. just keeping it
>>> as an option, simply because a blocker stops the entire release and it
>>> can be a last minute scramble to get things fixed. This was the ideal
>>> case for a blocker bugs and I'm skeptical about all bugs going this
well.
>>> If we had a few more people who were willing to be on the btrfs alias and
>>> do the work for blocker bugs it would be a much stronger case.
>>>
>>
>> Out of curiosity, how many such issues have we had in the past 2
>> years? I personally can't recall any monumental occasions where people
>> were scrambling over *Btrfs* in Fedora. If anything, we continue to
>> inherit the work that SUSE and Facebook are doing upstream as part of
>> us continually updating our kernels, which I'm grateful for.
>>
>> And in the instances where we've had such issues, has anyone reached
>> out to btrfs folks in Fedora? Chris and myself are the current ones,
>> but there have been others in the past. Both of us are subscribed to
>> the linux-btrfs mailing list, and Chris has a decent rapport with most
>> of the btrfs developers.
>>
>> What more do you want? Actual btrfs developers in Fedora? We don't
>> have any for the majority of filesystems Fedora supports, only XFS. Is
>> there some kind of problem with communicating with the upstream kernel
>> developers about Fedora bugs that I'm not aware of?
>>
>
> Again, it's about length of overall development. ext and XFS have
> a much longer history in general which is something that's important
> for file system stability in general. It's also a bit of a catch-22
> where the rate of btrfs use in Fedora is so low we don't actually
> see issues.
>
>>>> I note here that ext2 and ext3 are offered as file systems in
>>>> Custom/Advanced partitioning and in this sense have parity with Btrfs.
>>>> If this same bug occurred in ext2 or ext3 would or should that cause
>>>> discussion to drop them from the installer, even if the bug were fixed
>>>> within 24 hours of discovery and patch? What about vfat? That's
>>>> literally the only truly required filesystem that must work, for the
>>>> most commonly supported hardware so it can't be dropped, we'd
just be
>>>> stuck until it got fixed. That work would have to be done upstream,
>>>> yes?
>>>>
>>>
>>> I don't think that's really a fair comparison. Just because options
>>> are presented doesn't mean all of them are equal. ext2/ext3 and vfat
>>> have been in development for much longer than btrfs and length of
development
>>> is something that's particularly important for file system stability
>>> from talking with file system developers. It's not impossible for there
>>> to be bugs in ext4 for example (we've certainly seen them before) but
>>> btrfs is only now gaining overall stability and we're still more likely
to see
>>> bugs, especially with custom setups where people are likely to find
>>> edge cases.
>>>
>>
>> Nope. We can totally use this because LVM has not existed as long (we
>> use LVM + filesystem by default, not plain partitions), and we still
>> encounter quirks with things like thinp LVM combined with these
>> filesystems. OverlayFS is mostly hot garbage (kernel people know it,
>> container people know it, filesystem people know it, etc.), and yet we
>> continue to try to use it in more places. Stratis is in an odd state
>> of limbo now, since its main developer and advocate left Red Hat.
>>> There are plenty of examples of Red Hat doing crazy/experimental
>> things... I'd like to think Red Hat isn't supposed to be special here,
>> but in this realm, it seems like it is...
>>
>>
>
> btrfs still doesn't give me the warm fuzzies and I also think this
> is a bigger issue than other features simply because user data is at
> stake. We do need to consider that the failure case is not "I can't do
X"
> but "my precious data which I have been trying to snapshot is now
> inaccessible" in a way that's even worse than say rpm database
> corruption. Even if it is in the advanced partitioning or not the
> default, we can still end up with people clicking in because they
> read an article about how btrfs was the hot new thing.
>
> There are two parts to this here: killing off btrfs entirely and
> btrfs as release criteria. I think you are correct that there's
> enough community support to justify keeping btrfs around at least
> in the kernel (I can't speak for anaconda here)
>
> As for btrfs as release criteria, I'd feel much more confident
> about that if we could have a file system developer on the btrfs
> alias. I'm glad to hear the btrfs upstream community has been
> receptive to bugs but it's still much easier to make things
> happen if we have contributors who are active in the Fedora
> community, especially if we want the advanced features that
> btrfs has (which is why people want it anyway). So, who would
> you suggest to work with us in Fedora?
You can always CC me, if I get an email from you or anybody else I recognize
from the fedora kernel team I'm going to pay attention to it.
Facebook runs more btrfs file systems than Fedora has installs, so we're pretty
happy with how it works stability wise. That being said we're slightly more
fault tolerant than most users. If you guys are hitting problems chances are
we'll hit them eventually as well, so it makes sense for us to be on top of
them.
I agree it would be better if somebody inside Fedora was able to help out, but
again I'm only an email away. Thanks,
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla(a)colorremedies.com
This technically meets the requirements if you are willing to stay on this
alias and (continue) to help with requests as needed. I would feel more
confident if we had a few more people involved as well. Even better
would be proactively going through the bugzillas to help find the
btrfs ones.
Fedora is ultimately a community project and as far as the kernel
goes, there does seem to be enough interest from the community if
Josef et. al. are willing to be involved. I hope to see this
continue.
Thanks,
Laura
1:39 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
On 8/28/19 1:58 PM, Josef Bacik wrote:
> On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
> > On 8/26/19 11:39 PM, Neal Gompa wrote:
> > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott <labbott(a)redhat.com>
wrote:
> > > >
> > > > On 8/23/19 9:00 PM, Chris Murphy wrote:
> > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
> > > > > <adamwill(a)fedoraproject.org> wrote:
> > > > >
> > > > > > So, there was recently a Thing where btrfs installs were
broken, and
> > > > > > this got accepted as a release blocker:
> > > > > >
> > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388
> > > > >
> > > > > Summary: This bug was introduced and discovered in linux-next,
it
> > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests,
patch
> > > > > appeared during rc1, and the patch was merged into 5.3.0-rc2.
The bug
> > > > > resulted in a somewhat transient deadlock which caused installs
to
> > > > > hang, but no corruption. The fix, 2 files changed, 12
insertions, 8
> > > > > deletions (1/2 the insertions are comments).
> > > > >
> > > > > How remarkable or interesting is this bug? And in particular,
exactly
> > > > > how much faster should it have been fixed in order to avoid
worrying
> > > > > about it being a blocker bug?
> > > > >
> > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@
> > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla
> > > > > 7/26 19:20 utc I confirmed upstream's patch related to this
bug with
> > > > > upstream and updated the Fedora bug
> > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and updated
the Fedora bug
> > > > >
> > > > > So in the context of status quo, where Btrfs is presented as an
option
> > > > > in the installer and if there are bugs they Beta blocking, how
could
> > > > > or should this have been fixed sooner? What about the handling
should
> > > > > have been different?
> > > > >
> > > >
> > > > That's a fair question. This bug actually represents how this
_should_
> > > > work. The concern is that in the past we haven't seen a lot
engagement
> > > > in the past. Maybe today that has changed as demonstrated by this
thread.
> > > > I'm still concerned about having this be a blocker vs. just
keeping it
> > > > as an option, simply because a blocker stops the entire release and
it
> > > > can be a last minute scramble to get things fixed. This was the
ideal
> > > > case for a blocker bugs and I'm skeptical about all bugs going
this well.
> > > > If we had a few more people who were willing to be on the btrfs alias
and
> > > > do the work for blocker bugs it would be a much stronger case.
> > > >
> > >
> > > Out of curiosity, how many such issues have we had in the past 2
> > > years? I personally can't recall any monumental occasions where
people
> > > were scrambling over *Btrfs* in Fedora. If anything, we continue to
> > > inherit the work that SUSE and Facebook are doing upstream as part of
> > > us continually updating our kernels, which I'm grateful for.
> > >
> > > And in the instances where we've had such issues, has anyone reached
> > > out to btrfs folks in Fedora? Chris and myself are the current ones,
> > > but there have been others in the past. Both of us are subscribed to
> > > the linux-btrfs mailing list, and Chris has a decent rapport with most
> > > of the btrfs developers.
> > >
> > > What more do you want? Actual btrfs developers in Fedora? We don't
> > > have any for the majority of filesystems Fedora supports, only XFS. Is
> > > there some kind of problem with communicating with the upstream kernel
> > > developers about Fedora bugs that I'm not aware of?
> > >
> >
> > Again, it's about length of overall development. ext and XFS have
> > a much longer history in general which is something that's important
> > for file system stability in general. It's also a bit of a catch-22
> > where the rate of btrfs use in Fedora is so low we don't actually
> > see issues.
> >
> > > > > I note here that ext2 and ext3 are offered as file systems in
> > > > > Custom/Advanced partitioning and in this sense have parity with
Btrfs.
> > > > > If this same bug occurred in ext2 or ext3 would or should that
cause
> > > > > discussion to drop them from the installer, even if the bug were
fixed
> > > > > within 24 hours of discovery and patch? What about vfat?
That's
> > > > > literally the only truly required filesystem that must work, for
the
> > > > > most commonly supported hardware so it can't be dropped,
we'd just be
> > > > > stuck until it got fixed. That work would have to be done
upstream,
> > > > > yes?
> > > > >
> > > >
> > > > I don't think that's really a fair comparison. Just because
options
> > > > are presented doesn't mean all of them are equal. ext2/ext3 and
vfat
> > > > have been in development for much longer than btrfs and length of
development
> > > > is something that's particularly important for file system
stability
> > > > from talking with file system developers. It's not impossible for
there
> > > > to be bugs in ext4 for example (we've certainly seen them before)
but
> > > > btrfs is only now gaining overall stability and we're still more
likely to see
> > > > bugs, especially with custom setups where people are likely to find
> > > > edge cases.
> > > >
> > >
> > > Nope. We can totally use this because LVM has not existed as long (we
> > > use LVM + filesystem by default, not plain partitions), and we still
> > > encounter quirks with things like thinp LVM combined with these
> > > filesystems. OverlayFS is mostly hot garbage (kernel people know it,
> > > container people know it, filesystem people know it, etc.), and yet we
> > > continue to try to use it in more places. Stratis is in an odd state
> > > of limbo now, since its main developer and advocate left Red Hat.
> > > > There are plenty of examples of Red Hat doing crazy/experimental
> > > things... I'd like to think Red Hat isn't supposed to be special
here,
> > > but in this realm, it seems like it is...
> > >
> > >
> >
> > btrfs still doesn't give me the warm fuzzies and I also think this
> > is a bigger issue than other features simply because user data is at
> > stake. We do need to consider that the failure case is not "I can't do
X"
> > but "my precious data which I have been trying to snapshot is now
> > inaccessible" in a way that's even worse than say rpm database
> > corruption. Even if it is in the advanced partitioning or not the
> > default, we can still end up with people clicking in because they
> > read an article about how btrfs was the hot new thing.
> >
> > There are two parts to this here: killing off btrfs entirely and
> > btrfs as release criteria. I think you are correct that there's
> > enough community support to justify keeping btrfs around at least
> > in the kernel (I can't speak for anaconda here)
> >
> > As for btrfs as release criteria, I'd feel much more confident
> > about that if we could have a file system developer on the btrfs
> > alias. I'm glad to hear the btrfs upstream community has been
> > receptive to bugs but it's still much easier to make things
> > happen if we have contributors who are active in the Fedora
> > community, especially if we want the advanced features that
> > btrfs has (which is why people want it anyway). So, who would
> > you suggest to work with us in Fedora?
>
> You can always CC me, if I get an email from you or anybody else I recognize
> from the fedora kernel team I'm going to pay attention to it.
>
> Facebook runs more btrfs file systems than Fedora has installs, so we're pretty
> happy with how it works stability wise. That being said we're slightly more
> fault tolerant than most users. If you guys are hitting problems chances are
> we'll hit them eventually as well, so it makes sense for us to be on top of
> them.
>
> I agree it would be better if somebody inside Fedora was able to help out, but
> again I'm only an email away. Thanks,
>
So it appears you are on the btrfs alias already:
fedora-kernel-btrfs: fs-maint@redhat.com,josef@toxicpanda.com,bugzilla(a)colorremedies.com
This technically meets the requirements if you are willing to stay on this
alias and (continue) to help with requests as needed. I would feel more
confident if we had a few more people involved as well. Even better
would be proactively going through the bugzillas to help find the
btrfs ones.
Yeah that goes into a bucket that basically is ignored. The only time I'll peek
in there is if somebody specifically pokes me, because generally speaking we hit
the problems and fix them welllllll before Fedora users start to notice them.
If it'll make people feel more comfortable I'll start sticking my head in there
every week or so and see if something needs my attention. Thanks,
Josef
2:01 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik <josef(a)toxicpanda.com> wrote:
On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
> On 8/28/19 1:58 PM, Josef Bacik wrote:
> > On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
> > > On 8/26/19 11:39 PM, Neal Gompa wrote:
> > > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott
<labbott(a)redhat.com> wrote:
> > > > >
> > > > > On 8/23/19 9:00 PM, Chris Murphy wrote:
> > > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
> > > > > > <adamwill(a)fedoraproject.org> wrote:
> > > > > >
> > > > > > > So, there was recently a Thing where btrfs installs
were broken, and
> > > > > > > this got accepted as a release blocker:
> > > > > > >
> > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1733388
> > > > > >
> > > > > > Summary: This bug was introduced and discovered in
linux-next, it
> > > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa tests,
patch
> > > > > > appeared during rc1, and the patch was merged into
5.3.0-rc2. The bug
> > > > > > resulted in a somewhat transient deadlock which caused
installs to
> > > > > > hang, but no corruption. The fix, 2 files changed, 12
insertions, 8
> > > > > > deletions (1/2 the insertions are comments).
> > > > > >
> > > > > > How remarkable or interesting is this bug? And in
particular, exactly
> > > > > > how much faster should it have been fixed in order to avoid
worrying
> > > > > > about it being a blocker bug?
> > > > > >
> > > > > > 7/25 14:27 utc bug patch was submitted to linux-btrfs@
> > > > > > 7/25 22:33 utc bug was first reported in Fedora bugzilla
> > > > > > 7/26 19:20 utc I confirmed upstream's patch related to
this bug with
> > > > > > upstream and updated the Fedora bug
> > > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and
updated the Fedora bug
> > > > > >
> > > > > > So in the context of status quo, where Btrfs is presented
as an option
> > > > > > in the installer and if there are bugs they Beta blocking,
how could
> > > > > > or should this have been fixed sooner? What about the
handling should
> > > > > > have been different?
> > > > > >
> > > > >
> > > > > That's a fair question. This bug actually represents how
this _should_
> > > > > work. The concern is that in the past we haven't seen a lot
engagement
> > > > > in the past. Maybe today that has changed as demonstrated by
this thread.
> > > > > I'm still concerned about having this be a blocker vs. just
keeping it
> > > > > as an option, simply because a blocker stops the entire release
and it
> > > > > can be a last minute scramble to get things fixed. This was the
ideal
> > > > > case for a blocker bugs and I'm skeptical about all bugs
going this well.
> > > > > If we had a few more people who were willing to be on the btrfs
alias and
> > > > > do the work for blocker bugs it would be a much stronger case.
> > > > >
> > > >
> > > > Out of curiosity, how many such issues have we had in the past 2
> > > > years? I personally can't recall any monumental occasions where
people
> > > > were scrambling over *Btrfs* in Fedora. If anything, we continue to
> > > > inherit the work that SUSE and Facebook are doing upstream as part
of
> > > > us continually updating our kernels, which I'm grateful for.
> > > >
> > > > And in the instances where we've had such issues, has anyone
reached
> > > > out to btrfs folks in Fedora? Chris and myself are the current ones,
> > > > but there have been others in the past. Both of us are subscribed to
> > > > the linux-btrfs mailing list, and Chris has a decent rapport with
most
> > > > of the btrfs developers.
> > > >
> > > > What more do you want? Actual btrfs developers in Fedora? We
don't
> > > > have any for the majority of filesystems Fedora supports, only XFS.
Is
> > > > there some kind of problem with communicating with the upstream
kernel
> > > > developers about Fedora bugs that I'm not aware of?
> > > >
> > >
> > > Again, it's about length of overall development. ext and XFS have
> > > a much longer history in general which is something that's important
> > > for file system stability in general. It's also a bit of a catch-22
> > > where the rate of btrfs use in Fedora is so low we don't actually
> > > see issues.
> > >
> > > > > > I note here that ext2 and ext3 are offered as file systems
in
> > > > > > Custom/Advanced partitioning and in this sense have parity
with Btrfs.
> > > > > > If this same bug occurred in ext2 or ext3 would or should
that cause
> > > > > > discussion to drop them from the installer, even if the bug
were fixed
> > > > > > within 24 hours of discovery and patch? What about vfat?
That's
> > > > > > literally the only truly required filesystem that must
work, for the
> > > > > > most commonly supported hardware so it can't be
dropped, we'd just be
> > > > > > stuck until it got fixed. That work would have to be done
upstream,
> > > > > > yes?
> > > > > >
> > > > >
> > > > > I don't think that's really a fair comparison. Just
because options
> > > > > are presented doesn't mean all of them are equal. ext2/ext3
and vfat
> > > > > have been in development for much longer than btrfs and length
of development
> > > > > is something that's particularly important for file system
stability
> > > > > from talking with file system developers. It's not
impossible for there
> > > > > to be bugs in ext4 for example (we've certainly seen them
before) but
> > > > > btrfs is only now gaining overall stability and we're still
more likely to see
> > > > > bugs, especially with custom setups where people are likely to
find
> > > > > edge cases.
> > > > >
> > > >
> > > > Nope. We can totally use this because LVM has not existed as long
(we
> > > > use LVM + filesystem by default, not plain partitions), and we still
> > > > encounter quirks with things like thinp LVM combined with these
> > > > filesystems. OverlayFS is mostly hot garbage (kernel people know it,
> > > > container people know it, filesystem people know it, etc.), and yet
we
> > > > continue to try to use it in more places. Stratis is in an odd state
> > > > of limbo now, since its main developer and advocate left Red Hat.
> > > > > There are plenty of examples of Red Hat doing
crazy/experimental
> > > > things... I'd like to think Red Hat isn't supposed to be
special here,
> > > > but in this realm, it seems like it is...
> > > >
> > > >
> > >
> > > btrfs still doesn't give me the warm fuzzies and I also think this
> > > is a bigger issue than other features simply because user data is at
> > > stake. We do need to consider that the failure case is not "I
can't do X"
> > > but "my precious data which I have been trying to snapshot is now
> > > inaccessible" in a way that's even worse than say rpm database
> > > corruption. Even if it is in the advanced partitioning or not the
> > > default, we can still end up with people clicking in because they
> > > read an article about how btrfs was the hot new thing.
> > >
> > > There are two parts to this here: killing off btrfs entirely and
> > > btrfs as release criteria. I think you are correct that there's
> > > enough community support to justify keeping btrfs around at least
> > > in the kernel (I can't speak for anaconda here)
> > >
> > > As for btrfs as release criteria, I'd feel much more confident
> > > about that if we could have a file system developer on the btrfs
> > > alias. I'm glad to hear the btrfs upstream community has been
> > > receptive to bugs but it's still much easier to make things
> > > happen if we have contributors who are active in the Fedora
> > > community, especially if we want the advanced features that
> > > btrfs has (which is why people want it anyway). So, who would
> > > you suggest to work with us in Fedora?
> >
> > You can always CC me, if I get an email from you or anybody else I recognize
> > from the fedora kernel team I'm going to pay attention to it.
> >
> > Facebook runs more btrfs file systems than Fedora has installs, so we're
pretty
> > happy with how it works stability wise. That being said we're slightly
more
> > fault tolerant than most users. If you guys are hitting problems chances are
> > we'll hit them eventually as well, so it makes sense for us to be on top
of
> > them.
> >
> > I agree it would be better if somebody inside Fedora was able to help out, but
> > again I'm only an email away. Thanks,
> >
>
> So it appears you are on the btrfs alias already:
>
> fedora-kernel-btrfs:
fs-maint@redhat.com,josef@toxicpanda.com,bugzilla(a)colorremedies.com
>
> This technically meets the requirements if you are willing to stay on this
> alias and (continue) to help with requests as needed. I would feel more
> confident if we had a few more people involved as well. Even better
> would be proactively going through the bugzillas to help find the
> btrfs ones.
Yeah that goes into a bucket that basically is ignored. The only time I'll peek
in there is if somebody specifically pokes me, because generally speaking we hit
the problems and fix them welllllll before Fedora users start to notice them.
Fedora chugs along at the rate of daily upstream Linus snapshots. If
you're hitting and fixing issues before Fedora users see them, I'm
curious why Fedora users would ever see them.
Where does the lag come from? Are the fixes queued internally?
Staged in an upstream subsystem tree? Is there a way for interested
btrfs people to proactively just get those fixed in Fedora before
users hit them?
josh
2:07 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote:
On Wed, Aug 28, 2019 at 2:40 PM Josef Bacik
<josef(a)toxicpanda.com> wrote:
>
> On Wed, Aug 28, 2019 at 02:35:39PM -0400, Laura Abbott wrote:
> > On 8/28/19 1:58 PM, Josef Bacik wrote:
> > > On Tue, Aug 27, 2019 at 07:53:20AM -0400, Laura Abbott wrote:
> > > > On 8/26/19 11:39 PM, Neal Gompa wrote:
> > > > > On Mon, Aug 26, 2019 at 11:16 AM Laura Abbott
<labbott(a)redhat.com> wrote:
> > > > > >
> > > > > > On 8/23/19 9:00 PM, Chris Murphy wrote:
> > > > > > > On Fri, Aug 23, 2019 at 1:17 PM Adam Williamson
> > > > > > > <adamwill(a)fedoraproject.org> wrote:
> > > > > > >
> > > > > > > > So, there was recently a Thing where btrfs
installs were broken, and
> > > > > > > > this got accepted as a release blocker:
> > > > > > > >
> > > > > > > >
https://bugzilla.redhat.com/show_bug.cgi?id=1733388
> > > > > > >
> > > > > > > Summary: This bug was introduced and discovered in
linux-next, it
> > > > > > > started to affect Fedora 5.3.0-rc0 kernels in openqa
tests, patch
> > > > > > > appeared during rc1, and the patch was merged into
5.3.0-rc2. The bug
> > > > > > > resulted in a somewhat transient deadlock which caused
installs to
> > > > > > > hang, but no corruption. The fix, 2 files changed, 12
insertions, 8
> > > > > > > deletions (1/2 the insertions are comments).
> > > > > > >
> > > > > > > How remarkable or interesting is this bug? And in
particular, exactly
> > > > > > > how much faster should it have been fixed in order to
avoid worrying
> > > > > > > about it being a blocker bug?
> > > > > > >
> > > > > > > 7/25 14:27 utc bug patch was submitted to
linux-btrfs@
> > > > > > > 7/25 22:33 utc bug was first reported in Fedora
bugzilla
> > > > > > > 7/26 19:20 utc I confirmed upstream's patch
related to this bug with
> > > > > > > upstream and updated the Fedora bug
> > > > > > > 7/26 22:50 utc I confirmed it was merged into rc2, and
updated the Fedora bug
> > > > > > >
> > > > > > > So in the context of status quo, where Btrfs is
presented as an option
> > > > > > > in the installer and if there are bugs they Beta
blocking, how could
> > > > > > > or should this have been fixed sooner? What about the
handling should
> > > > > > > have been different?
> > > > > > >
> > > > > >
> > > > > > That's a fair question. This bug actually represents
how this _should_
> > > > > > work. The concern is that in the past we haven't seen a
lot engagement
> > > > > > in the past. Maybe today that has changed as demonstrated
by this thread.
> > > > > > I'm still concerned about having this be a blocker vs.
just keeping it
> > > > > > as an option, simply because a blocker stops the entire
release and it
> > > > > > can be a last minute scramble to get things fixed. This was
the ideal
> > > > > > case for a blocker bugs and I'm skeptical about all
bugs going this well.
> > > > > > If we had a few more people who were willing to be on the
btrfs alias and
> > > > > > do the work for blocker bugs it would be a much stronger
case.
> > > > > >
> > > > >
> > > > > Out of curiosity, how many such issues have we had in the past
2
> > > > > years? I personally can't recall any monumental occasions
where people
> > > > > were scrambling over *Btrfs* in Fedora. If anything, we continue
to
> > > > > inherit the work that SUSE and Facebook are doing upstream as
part of
> > > > > us continually updating our kernels, which I'm grateful
for.
> > > > >
> > > > > And in the instances where we've had such issues, has anyone
reached
> > > > > out to btrfs folks in Fedora? Chris and myself are the current
ones,
> > > > > but there have been others in the past. Both of us are
subscribed to
> > > > > the linux-btrfs mailing list, and Chris has a decent rapport
with most
> > > > > of the btrfs developers.
> > > > >
> > > > > What more do you want? Actual btrfs developers in Fedora? We
don't
> > > > > have any for the majority of filesystems Fedora supports, only
XFS. Is
> > > > > there some kind of problem with communicating with the upstream
kernel
> > > > > developers about Fedora bugs that I'm not aware of?
> > > > >
> > > >
> > > > Again, it's about length of overall development. ext and XFS
have
> > > > a much longer history in general which is something that's
important
> > > > for file system stability in general. It's also a bit of a
catch-22
> > > > where the rate of btrfs use in Fedora is so low we don't
actually
> > > > see issues.
> > > >
> > > > > > > I note here that ext2 and ext3 are offered as file
systems in
> > > > > > > Custom/Advanced partitioning and in this sense have
parity with Btrfs.
> > > > > > > If this same bug occurred in ext2 or ext3 would or
should that cause
> > > > > > > discussion to drop them from the installer, even if
the bug were fixed
> > > > > > > within 24 hours of discovery and patch? What about
vfat? That's
> > > > > > > literally the only truly required filesystem that must
work, for the
> > > > > > > most commonly supported hardware so it can't be
dropped, we'd just be
> > > > > > > stuck until it got fixed. That work would have to be
done upstream,
> > > > > > > yes?
> > > > > > >
> > > > > >
> > > > > > I don't think that's really a fair comparison. Just
because options
> > > > > > are presented doesn't mean all of them are equal.
ext2/ext3 and vfat
> > > > > > have been in development for much longer than btrfs and
length of development
> > > > > > is something that's particularly important for file
system stability
> > > > > > from talking with file system developers. It's not
impossible for there
> > > > > > to be bugs in ext4 for example (we've certainly seen
them before) but
> > > > > > btrfs is only now gaining overall stability and we're
still more likely to see
> > > > > > bugs, especially with custom setups where people are likely
to find
> > > > > > edge cases.
> > > > > >
> > > > >
> > > > > Nope. We can totally use this because LVM has not existed as
long (we
> > > > > use LVM + filesystem by default, not plain partitions), and we
still
> > > > > encounter quirks with things like thinp LVM combined with these
> > > > > filesystems. OverlayFS is mostly hot garbage (kernel people know
it,
> > > > > container people know it, filesystem people know it, etc.), and
yet we
> > > > > continue to try to use it in more places. Stratis is in an odd
state
> > > > > of limbo now, since its main developer and advocate left Red
Hat.
> > > > > > There are plenty of examples of Red Hat doing
crazy/experimental
> > > > > things... I'd like to think Red Hat isn't supposed to be
special here,
> > > > > but in this realm, it seems like it is...
> > > > >
> > > > >
> > > >
> > > > btrfs still doesn't give me the warm fuzzies and I also think
this
> > > > is a bigger issue than other features simply because user data is at
> > > > stake. We do need to consider that the failure case is not "I
can't do X"
> > > > but "my precious data which I have been trying to snapshot is
now
> > > > inaccessible" in a way that's even worse than say rpm
database
> > > > corruption. Even if it is in the advanced partitioning or not the
> > > > default, we can still end up with people clicking in because they
> > > > read an article about how btrfs was the hot new thing.
> > > >
> > > > There are two parts to this here: killing off btrfs entirely and
> > > > btrfs as release criteria. I think you are correct that there's
> > > > enough community support to justify keeping btrfs around at least
> > > > in the kernel (I can't speak for anaconda here)
> > > >
> > > > As for btrfs as release criteria, I'd feel much more confident
> > > > about that if we could have a file system developer on the btrfs
> > > > alias. I'm glad to hear the btrfs upstream community has been
> > > > receptive to bugs but it's still much easier to make things
> > > > happen if we have contributors who are active in the Fedora
> > > > community, especially if we want the advanced features that
> > > > btrfs has (which is why people want it anyway). So, who would
> > > > you suggest to work with us in Fedora?
> > >
> > > You can always CC me, if I get an email from you or anybody else I
recognize
> > > from the fedora kernel team I'm going to pay attention to it.
> > >
> > > Facebook runs more btrfs file systems than Fedora has installs, so
we're pretty
> > > happy with how it works stability wise. That being said we're
slightly more
> > > fault tolerant than most users. If you guys are hitting problems chances
are
> > > we'll hit them eventually as well, so it makes sense for us to be on
top of
> > > them.
> > >
> > > I agree it would be better if somebody inside Fedora was able to help out,
but
> > > again I'm only an email away. Thanks,
> > >
> >
> > So it appears you are on the btrfs alias already:
> >
> > fedora-kernel-btrfs:
fs-maint@redhat.com,josef@toxicpanda.com,bugzilla(a)colorremedies.com
> >
> > This technically meets the requirements if you are willing to stay on this
> > alias and (continue) to help with requests as needed. I would feel more
> > confident if we had a few more people involved as well. Even better
> > would be proactively going through the bugzillas to help find the
> > btrfs ones.
>
> Yeah that goes into a bucket that basically is ignored. The only time I'll
peek
> in there is if somebody specifically pokes me, because generally speaking we hit
> the problems and fix them welllllll before Fedora users start to notice them.
Fedora chugs along at the rate of daily upstream Linus snapshots. If
you're hitting and fixing issues before Fedora users see them, I'm
curious why Fedora users would ever see them.
Where does the lag come from? Are the fixes queued internally?
Staged in an upstream subsystem tree? Is there a way for interested
btrfs people to proactively just get those fixed in Fedora before
users hit them?
For this particular example we saw the problem in testing and had a patch on the
mailinglist before you hit the problem. It was in a tree and sent to Linus, and
was merged the day after the bugzilla was reported. So yes before users see
them, unless they are subscribed to the daily snapshots, which I assume is just
for testing right? Or were you guys going to ship 5.3-rc0?
On one hand I understand all of the consternation around making btrfs bugs
blockers for Fedora, but on the other hand it seems a bit silly to be having
this conversation at all based on hitting a bug that went into the merge window
and then was fixed before rc1 was even cut. Thanks,
Josef
4:59 p.m.
New subject: Discussion: what would not blocking on btrfs look like?
On Wed, Aug 28, 2019 at 1:07 PM Josef Bacik <josef(a)toxicpanda.com> wrote:
On Wed, Aug 28, 2019 at 03:01:16PM -0400, Josh Boyer wrote:
> Fedora chugs along at the rate of daily upstream Linus snapshots. If
> you're hitting and fixing issues before Fedora users see them, I'm
> curious why Fedora users would ever see them.
>
> Where does the lag come from? Are the fixes queued internally?
> Staged in an upstream subsystem tree? Is there a way for interested
> btrfs people to proactively just get those fixed in Fedora before
> users hit them?
For this particular example we saw the problem in testing and had a patch on the
mailinglist before you hit the problem. It was in a tree and sent to Linus, and
was merged the day after the bugzilla was reported. So yes before users see
them, unless they are subscribed to the daily snapshots, which I assume is just
for testing right? Or were you guys going to ship 5.3-rc0?
On one hand I understand all of the consternation around making btrfs bugs
blockers for Fedora, but on the other hand it seems a bit silly to be having
this conversation at all based on hitting a bug that went into the merge window
and then was fixed before rc1 was even cut. Thanks,
It is silly, if it's really safe to say that Btrfs won't be the
default file system in any release blocking deliverable. Having
blocker status was always a means to that end. But right now it's
maybe three (?) automated tests that depend on Btrfs working. If
Fedora Workstation defaulted to Btrfs, that's dozens or even hundreds
of tests? Adam?
Bug fix was merged in rc2. The patch on linux-btrfs@
25 Jul 2019 11:27:29 +0300 which is just before
kernel-5.3.0-0.rc1.git3.1.fc31, 2019-07-25 21:01:20
Plausibly it was in all rc0 and rc1 kernels. What if this deadlock
were happening in ext4 for all rc0 and rc1 kernels? What questions get
asked? How did the bug not get caught by xfstests before it got into
linux-next? Does anyone pop a gasket on lkml? Is this just a weird
sometimes it happens rarely kind of thing?
I don't know the exact nature of the bug, which goes to the kernel
team's point that someone who does know needs to tell them the autopsy
summary so they can really assess the default fs question.
And another question for QA. If it were Btrfs by default for
Workstation, would you just convert all the tests that rely only on
ext4 now to Btrfs? Or duplicate those tests so you can run them in
parallel? How much more testing is that and what's the impact on time
and resources?
--
Chris Murphy
1701
days inactive
1705
days old
kernel@lists.fedoraproject.org
10 comments
6 participants
participants (6)
-
Chris Murphy -
Josef Bacik -
Josh Boyer -
Justin Forbes -
Laura Abbott -
Neal Gompa