Re: Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

I was an early adopter and used BTRFS for many years, singing its praises.
I was particularly interested in the RAID capabilities.  Then in 2016 the
bomb was dropped that:

"It turns out the RAID5 and RAID6 code for the Btrfs file-system's built-in
RAID support is faulty and users should not be making use of it if you care
about your data.

There has been this mailing list thread
<https://www.mail-archive.com/[e-mail]/msg55161.html>
since the end of July about Btrfs scrub recalculating the wrong parity in
RAID5. The wrong parity and unrecoverable errors has been confirmed by
multiple parties. The Btrfs RAID 5/6 code has been called as much as fatally
flawed
<https://www.mail-archive.com/[e-mail]/msg55179.html> --
"more or less fatally flawed, and a full scrap and rewrite to an entirely
different raid56 mode on-disk format may be necessary to fix it. And what's
even clearer is that people /really/ shouldn't be using raid56 mode for
anything but testing with throw-away data, at this point. Anything else is
simply irresponsible."

The current situation as I understand it is the problem is "mostly fixed" -
whatever that means.

So, BTRFS is great, ready for prime time... many people are using it, etc.
etc. etc. until something goes wrong and then you get... well, it's
experimental and not intended for production.  Sucks to be you.

At some point you have to fish or cut bait.  I was under the impression
that Redhat had done exactly that with the announcement and direction of
Stratis.

Why are we not concentrating on Stratis and XFS?  Seems to me after waiting
for almost a decade for the promise of BTRFS to be fulfilled and then
having so many
people be burned, we should be turning the page rather than continuing to
rehash the same old arguments.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Igor Raits]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Sat, 2020-06-27 at 12:37 -0700, Gerald B. Cox wrote:
>  I was an early adopter and used BTRFS for many years, singing its
>  praises.
>  I was particularly interested in the RAID capabilities.  Then in 2016
>  the
>  bomb was dropped that:
>  
>  "It turns out the RAID5 and RAID6 code for the Btrfs file-system's
>  built-in
>  RAID support is faulty and users should not be making use of it if
>  you care
>  about your data. 
Well, the RAID5 and RAID6 in Btrfs is still not really working fine and
there are some threads in linux-btrfs@ at kernel ML. Workstation users
are not using RAID (even if they do, I am not even sure how it is
configured today with Anaconda). Even for those that do, RAID0/1/10
seems to be working just fine with Btrfs. RAID5/6 is definitely not
something that people run on their laptops.

>  There has been this mailing list thread
>  < 
>  https://www.mail-archive.com/[e-mail]/msg55161.html
>  >
>  since the end of July about Btrfs scrub recalculating the wrong
>  parity in
>  RAID5. The wrong parity and unrecoverable errors has been confirmed
>  by
>  multiple parties. The Btrfs RAID 5/6 code has been called as much as
>  fatally
>  flawed
>  < 
>  https://www.mail-archive.com/[e-mail]/msg55179.html
>  > --
>  "more or less fatally flawed, and a full scrap and rewrite to an
>  entirely
>  different raid56 mode on-disk format may be necessary to fix it. And
>  what's
>  even clearer is that people /really/ shouldn't be using raid56 mode
>  for
>  anything but testing with throw-away data, at this point. Anything
>  else is
>  simply irresponsible."
>  
>  The current situation as I understand it is the problem is "mostly
>  fixed" -
>  whatever that means.
>  
>  So, BTRFS is great, ready for prime time... many people are using it,
>  etc.
>  etc. etc. until something goes wrong and then you get... well, it's
>  experimental and not intended for production.  Sucks to be you.
>  
>  At some point you have to fish or cut bait.  I was under the
>  impression
>  that Redhat had done exactly that with the announcement and direction
>  of
>  Stratis.
>  
>  Why are we not concentrating on Stratis and XFS?  Seems to me after
>  waiting
>  for almost a decade for the promise of BTRFS to be fulfilled and then
>  having so many
>  people be burned, we should be turning the page rather than
>  continuing to
>  rehash the same old arguments. 
Well, because those have different use-cases? One of the problems with
XFS is that you can't shrink it. With systemd-homed on the horizon, it
will become a problem. It also does not like much when power goes down
(even with battery on laptops it is not uncommon), two my colleagues
got XFS destroyed (for one of them, twice) when they had to turn off
their laptops using power button in last ~2years. Then they have
switched to ext4 and nothing like that has happened again. But that one
does not have features that are very useful on laptops.

Stratis uses XFS under the hood, so it has same problem, but also it
still can't be used as a root filesystem. And generally it is designed
for different use-case than a workstation.

>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:  
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines:  
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:  
>  https://lists.fedoraproject.org/archives/list/[e-mail] 
- -- 
Igor Raits <ignatenkobrain(a)fedoraproject.org&gt;
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl73qagACgkQEV1auJxc
> Hh7sthAAg+omUP71jyuAEXoy1/+B3m+OSikBEAxRS055en4JaKROLQ2UVJOSCzs+
> KSJkg413ex8ORxH3l8vhnIlWCVXwV8de2duAt85DpknGe0XHpnLTtC45/xejUiln
> tjpz8RqyEYTXMEE7KhJpc3KCLgN1VHyVtr+6Y0P1WUOBQWj1iF5kfrB5J+VhYpVO
> qygMcjlSiFtJW4gfs8siukSebn1t3ILGX9eOvxjhNGgrsfZ4bboq5N4rhFb2+NWF
> sjBmjE9qIFJK21xgp1XPJHiF4SsfL8CQZqgvBGYcnfmcGb7wiXPZEvzVJ5VVSNh2
> 7l4VWrix+YDXKmhD7WccCLZOjEGJg7CSOHndm6FPQt6++OGMsWUdmZNAsB9Iw6Ia
> Y5oDYuVxRqH5VLPKwKfb4FIQJDr/RESJU7Xd9W0+RPwjLb73k6sPyn3eWBMbiWaa
> 5e8G/GlWJtT1GGQT3fBnUtKhRBrCJQ/tH/oDmWQ7BLgUWisiU2eOaMB3MCWDq13F
> uaKB63CU8EnvdDJGCbjy0U+D/e2ZRD8tKNcJAn1dl8+sBD28/vd5cO2aTv4AYSjG
> Jf9e7RHTy+giMx4jvKbPaQb6187Im92C/7J8rBuk7PhLhtZQU7RM5CPj9aiqYkWg
> 7OoNVWQSRbljS0V3J0qsUmvwXf5PngLjCKqBot2vbZIV7FM0sls=
> =Q/6i
> -----END PGP SIGNATURE-----

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 1:38 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
>  I was an early adopter and used BTRFS for many years, singing its praises.  I was
> particularly interested in the RAID capabilities.  Then in 2016 the bomb was dropped
> that:
>
>  "It turns out the RAID5 and RAID6 code for the Btrfs file-system's built-in RAID
> support is faulty and users should not be making use of it if you care about your
> data." 
The more recent threads that are relevant:
https://lore.kernel.org/linux-btrfs/[e-mail]...
https://lore.kernel.org/linux-btrfs/[e-mail]...

The proposal has nothing to do with raid56, let alone by default. The
installer doesn't offer it as an option. And it's not relevant to the
desktop. We're talking about single device btrfs file systems.

I'd say questions about the multiple device aspects of btrfs are
relevant and on-topic but not as a criticism of the proposal. If some
folks want to take that particular toy out of the toy box, they are
advised that btrfs raid is different than what they're used to, and
they shouldn't make assumptions about it.


>  So, BTRFS is great, ready for prime time... many people are using it,
> etc. etc. etc. until something goes wrong and then you get... well, it's experimental
> and not intended for production.  Sucks to be you. 
The raid56 criticism is relevant to raid56 use cases. What you're
doing above is an association fallacy.

>  At some point you have to fish or cut bait.  I was under the
> impression that Redhat had done exactly that with the announcement and direction of
> Stratis.
>
>  Why are we not concentrating on Stratis and XFS? 
This is partly addressed in the device-mapper section of the proposal,
as it depends on dm-thin. Integration with the desktop represents
non-trivial work to get both CLI and GUI apps to report thin pool
values, which is necessary because only it knows the truth of free and
used space. There is no compression, integrity, or cgroup2-IO
isolation support either, all of which are features we think are
useful to users now and in the near future.

Also, Stratis specifically it's not supported as system root, and has
no installer support.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

On Sat, Jun 27, 2020 at 1:23 PM Chris Murphy <lists(a)colorremedies.com&gt;
wrote:

>  The proposal has nothing to do with raid56, let alone by default. The
>  installer doesn't offer it as an option. And it's not relevant to the
>  desktop. We're talking about single device btrfs file systems.
>
>  
Isn't the proposal talking about BTRFS as a default for workstations?  Are
you saying that Anaconda is just going to check to see if a PC has only one
hard drive and then install BTRFS there, but if it has two devices use
something else?  Why would we be installing something by default that has
widely known broken functionality?  I would think it would be more
appropriate to have people who specifically want to use BTRFS functionality
and are aware and knowledgeable of the risks to seek it out rather than
have it be some sort of selective default.  The target audience you're
aiming at by making it the default doesn't know FAT from NTFS from EXT4
from XFS from BTRFS or do they care.   Neither are they aware or even care
about purported benefits.


>  > So, BTRFS is great, ready for prime time... many people are using it,
>  etc. etc. etc. until something goes wrong and then you get... well, it's
>  experimental and not intended for production.  Sucks to be you.
>
>  The raid56 criticism is relevant to raid56 use cases. What you're
>  doing above is an association fallacy.
>  
Actually no, this doesn't apply to just raid56.  I haven't seen where BTRFS
has been declared production ready.  It's always been this or that feature
is OK, another feature is mostly OK, or don't use this feature.  Then when
something goes wrong, the response is it's still experimental and not
intended for production workloads.  People then mention Facebook uses it...
but my understanding is that Facebook is "testing it in production".  They
don't RELY on it in production.  Why do we want to push something out like
that as a default?

>  Why are we not concentrating on Stratis and XFS?
>
>  This is partly addressed in the device-mapper section of the proposal,
>  as it depends on dm-thin. Integration with the desktop represents
>  non-trivial work to get both CLI and GUI apps to report thin pool
>  values, which is necessary because only it knows the truth of free and
>  used space. There is no compression, integrity, or cgroup2-IO
>  isolation support either, all of which are features we think are
>  useful to users now and in the near future.
>
>  Also, Stratis specifically it's not supported as system root, and has
>  no installer support.
>  
Ok, then why aren't we working on that?

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Josef Bacik]

On 6/27/20 4:53 PM, Gerald B. Cox wrote:
>  
>  
>  On Sat, Jun 27, 2020 at 1:23 PM Chris Murphy <lists(a)colorremedies.com 
>  <mailto:[e-mail]>> wrote:
>  
>  
>      The proposal has nothing to do with raid56, let alone by default. The
>      installer doesn't offer it as an option. And it's not relevant to the
>      desktop. We're talking about single device btrfs file systems.
>  
>  Isn't the proposal talking about BTRFS as a default for workstations?  Are you 
>  saying that Anaconda is just going to check to see if a PC has only one hard 
>  drive and then install BTRFS there, but if it has two devices use something 
>  else?  Why would we be installing something by default that has widely known 
>  broken functionality?  I would think it would be more appropriate to have people 
>  who specifically want to use BTRFS functionality and are aware and knowledgeable 
>  of the risks to seek it out rather than have it be some sort of selective 
>  default.  The target audience you're aiming at by making it the default doesn't 
>  know FAT from NTFS from EXT4 from XFS from BTRFS or do they care.   Neither are 
>  they aware or even care about purported benefits.
>  
>  
>       > So, BTRFS is great, ready for prime time... many people are using it,
>      etc. etc. etc. until something goes wrong and then you get... well, it's
>      experimental and not intended for production.  Sucks to be you.
>  
>      The raid56 criticism is relevant to raid56 use cases. What you're
>      doing above is an association fallacy.
>  
>  
>  Actually no, this doesn't apply to just raid56.  I haven't seen where BTRFS has 
>  been declared production ready.  It's always been this or that feature is OK, 
>  another feature is mostly OK, or don't use this feature.  Then when something 
>  goes wrong, the response is it's still experimental and not intended for 
>  production workloads.  People then mention Facebook uses it... but my 
>  understanding is that Facebook is "testing it in production".  They don't
> RELY 
>  on it in production.  Why do we want to push something out like that as a default?
Huh?  We absolutely rely on it in production, our whole container system is 
built around it.  It's literally the only thing that works with IO isolation 
with cgroups.  IDK where you're getting your information, but its wrong.  It's 
deployed on millions of machines, most of our infrastructure is built around 
using multiple core features.  Thanks,

Josef

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

On Sat, Jun 27, 2020 at 2:01 PM Josef Bacik <josef(a)toxicpanda.com&gt; wrote:

>  On 6/27/20 4:53 PM, Gerald B. Cox wrote:
>  >
>  >
>  > On Sat, Jun 27, 2020 at 1:23 PM Chris Murphy <lists(a)colorremedies.com
>  > <mailto:[e-mail]>> wrote:
>  >
>  >
>  >     The proposal has nothing to do with raid56, let alone by default. The
>  >     installer doesn't offer it as an option. And it's not relevant to the
>  >     desktop. We're talking about single device btrfs file systems.
>  >
>  > Isn't the proposal talking about BTRFS as a default for workstations?
>  Are you
>  > saying that Anaconda is just going to check to see if a PC has only one
>  hard
>  > drive and then install BTRFS there, but if it has two devices use
>  something
>  > else?  Why would we be installing something by default that has widely
>  known
>  > broken functionality?  I would think it would be more appropriate to
>  have people
>  > who specifically want to use BTRFS functionality and are aware and
>  knowledgeable
>  > of the risks to seek it out rather than have it be some sort of
>  selective
>  > default.  The target audience you're aiming at by making it the default
>  doesn't
>  > know FAT from NTFS from EXT4 from XFS from BTRFS or do they care.
>  Neither are
>  > they aware or even care about purported benefits.
>  >
>  >
>  >      > So, BTRFS is great, ready for prime time... many people are using
>  it,
>  >     etc. etc. etc. until something goes wrong and then you get... well,
>  it's
>  >     experimental and not intended for production.  Sucks to be you.
>  >
>  >     The raid56 criticism is relevant to raid56 use cases. What you're
>  >     doing above is an association fallacy.
>  >
>  >
>  > Actually no, this doesn't apply to just raid56.  I haven't seen where
>  BTRFS has
>  > been declared production ready.  It's always been this or that feature
>  is OK,
>  > another feature is mostly OK, or don't use this feature.  Then when
>  something
>  > goes wrong, the response is it's still experimental and not intended for
>  > production workloads.  People then mention Facebook uses it... but my
>  > understanding is that Facebook is "testing it in production".  They
>  don't RELY
>  > on it in production.  Why do we want to push something out like that as
>  a default?
>
>  Huh?  We absolutely rely on it in production, our whole container system
>  is
>  built around it.  It's literally the only thing that works with IO
>  isolation
>  with cgroups.  IDK where you're getting your information, but its wrong.
>  It's
>  deployed on millions of machines, most of our infrastructure is built
>  around
>  using multiple core features.  Thanks,
>
>  
Actually, I got my information here:
https://btrfs.wiki.kernel.org/index.php/Production_Users

Since you're responding here now, I'm assuming you're speaking officially
for the BTRFS project.  Could you
please either give the up-to-date official status or point me to some
project information that has been kept current?

So you're saying now that BTRFS is now officially production ready?

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Michel Alexandre Salim]

On 6/27/20 2:08 PM, Gerald B. Cox wrote:
>  
>  
>  On Sat, Jun 27, 2020 at 2:01 PM Josef Bacik <josef(a)toxicpanda.com 
>  <mailto:[e-mail]>> wrote:
>  
>      On 6/27/20 4:53 PM, Gerald B. Cox wrote:
>       > 
...
>       > production workloads.  People then mention Facebook uses
> it...
>      but my
>       > understanding is that Facebook is "testing it in production". 
>      They don't RELY
>       > on it in production.  Why do we want to push something out like
>      that as a default?
>  
>      Huh?  We absolutely rely on it in production, our whole container
>      system is
>      built around it.  It's literally the only thing that works with IO
>      isolation
>      with cgroups.  IDK where you're getting your information, but its
>      wrong.  It's
>      deployed on millions of machines, most of our infrastructure is
>      built around
>      using multiple core features.  Thanks,
>  
>  
>  Actually, I got my information here: 
>  https://btrfs.wiki.kernel.org/index.php/Production_Users
>  
>  Since you're responding here now, I'm assuming you're speaking 
>  officially for the BTRFS project.  Could you
>  please either give the up-to-date official status or point me to some 
>  project information that has been kept current?
>  
>  So you're saying now that BTRFS is now officially production ready?
>   
That wiki page you linked to has stated that it's deployed on millions 
of servers - since the edit on 10 November 2018:

https://btrfs.wiki.kernel.org/index.php?title=Production_Users&diff=p...

Regards,

-- 
Michel Alexandre Salim
profile: https://keybase.io/michel_slm
chat via email: https://delta.chat/
GPG key: 96A7 A6ED FB4D 2113 4056 3257 CAF9 AD10 ACB1 BEF2

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 2:53 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:

>  Isn't the proposal talking about BTRFS as a default for
> workstations?  Are you saying that Anaconda is just going to check to see if a PC has only
> one hard drive and then install BTRFS there, but if it has two devices use something else?
This is a really good point. The installer does let you choose
multiple drives and apply default partitioning scheme.  The last time
I tested this ages ago, the result is a simple linear/concat of the
drives, using LVM.

For btrfs, it is either 'single' or 'raid0' profile for data, but
'raid1' for metadata (the file system itself).

I need to test it or maybe someone beats me to it by looking at the
code. But either way it's equal to or better than the current default.


>  Why would we be installing something by default that has widely known
> broken functionality? 
Because the default configuration we're using isn't broken and is
better than the alternatives being evaluated.


>   I would think it would be more appropriate to have people who
> specifically want to use BTRFS functionality and are aware and knowledgeable of the risks
> to seek it out rather than have it be some sort of selective default.  The target audience
> you're aiming at by making it the default doesn't know FAT from NTFS from EXT4
> from XFS from BTRFS or do they care.   Neither are they aware or even care about purported
> benefits. 
And they're going to get into trouble with raid56 how? Are you going
to tell them they should convert to raid56? How is it even relevant?


>  Actually no, this doesn't apply to just raid56.  I haven't
> seen where BTRFS has been declared production ready.  It's always been this or that
> feature is OK, another feature is mostly OK, or don't use this feature.  Then when
> something goes wrong, the response is it's still experimental and not intended for
> production workloads.  People then mention Facebook uses it... but my understanding is
> that Facebook is "testing it in production".  They don't RELY on it in
> production.  Why do we want to push something out like that as a default? 
Literally none of this is correct.


> > > Why are we not concentrating on Stratis and XFS?
> >
> > This is partly addressed in the device-mapper section of the proposal,
> > as it depends on dm-thin. Integration with the desktop represents
> > non-trivial work to get both CLI and GUI apps to report thin pool
> > values, which is necessary because only it knows the truth of free and
> > used space. There is no compression, integrity, or cgroup2-IO
> > isolation support either, all of which are features we think are
> > useful to users now and in the near future.
> >
> > Also, Stratis specifically it's not supported as system root, and has
> > no installer support.
>
>
>  Ok, then why aren't we working on that? 
I don't know. Why aren't you working on that? By all means put
together a competing proposal.

I'm working on what I'm working on.

--
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 3:29 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  For btrfs, it is either 'single' or 'raid0' profile for data, but
>  'raid1' for metadata (the file system itself).
>
>  I need to test it or maybe someone beats me to it by looking at the
>  code. But either way it's equal to or better than the current default. 
For btrfs, it's raid0 data, raid1 metadata.
For current default (lvm+ext4) it's linear/concat.

It's straightforward to do data 'single' which is closer to the
current default. But no matter what, you lose a drive and you can't
boot. As a UI/UX it might need to be reevaluated what the user expects
to happen when they pick two devices and automatic partitioning. I
would argue it's ambiguous and it'd be better to only allow one device
to be selected for auto/default path. But the current UI sorta makes
this difficult, and hard to convey. You're encouraged to pick the
device(s) you want first rather than a path (auto, custom, advanced),
where it's legit to pick multiple devices for custom.

I'm not sure anyone does multiple device auto partition installs though.

-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tom Seewald]

>  For btrfs, it's raid0 data, raid1 metadata. 
Surely this is
considered a serious installer bug? Users who choose an option called "raid1"
with btrfs would, and should, expect to have data redundancy.

Even if this bug has existed for a long time, it doesn't make it any less dangerous.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sun, Jun 28, 2020 at 3:11 PM Tom Seewald <tseewald(a)gmail.com&gt; wrote:
>  > For btrfs, it's raid0 data, raid1 metadata.
>  Surely this is considered a serious installer bug? Users who choose an option called
> "raid1" with btrfs would, and should, expect to have data redundancy. 
The context of that is: the default when the user does not specify. If
the user chooses 'raid1' in the installer, they get 'raid1' for both
data and metadata.

When you merely choose 2+ devices and use custom partitioning's
"autopartition" option for the Btrfs preset. (a) the UI doesn't show
the profile (single, dup, raid0, raid1, raid10) being used. (b)
mkfs.btrfs default for the multiple device case,  which is raid0 data,
raid1 metadata. If you go look for this, you can certainly set it to
raid1 and then you'll get raid1.

This further relates to the automatic partitioning behavior because
(a) it accepts multiple devices (b) there is a default (c) you can't
see what it is and you can't change it. I'm pretty sure it's rare for
people to pick 2+ devices and then do automatic partitioning. But it
is handled, and what you get depends on the installer default (file
system):

LVM = LVM linear across all devices selected, with ext4 on /home and /
LV's. i.e. linear/concat
Btrfs = raid0 data, raid1 metadata across all devices selected.

The confusing part for 100% of new users to Btrfs is the split between
data (file contents) and metadata (the file system itself). In these
two examples, they both lack meaningful redundancy. What does it mean
for btrfs to have raid1 metadata, but raid0 data in the face of device
failure? Well, it means the file system is still working normally, but
hunks of your files are missing in 64KiB sized strips. Some will read
some won't. But raid0 is faster than linear (called 'single' profile
on btrfs) so it's probably fine leaving it as is.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tom Seewald]

>  The context of that is: the default when the user does not specify.
> If
>  the user chooses 'raid1' in the installer, they get 'raid1' for both
>  data and metadata. 
This does not seem to be the case, and from what I can tell
Garry experienced this problem as well.

I tested this in a VM with two disks, I manually selected the "raid1" profile
with btrfs in the advanced custom partitioning screen. After the installation I ran
"btrfs filesystem df /", which output:

Data, RAID0: total=2.00GiB, used=1.54GiB
System, RAID1: total=8.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=46.28MiB
GlobalReserve, single: total=4.39MiB, used=0.00B

I interpret "Data, RAID0" to mean that the data is striped rather than mirrored,
and thus my data will be lost if/when a single drive fails. I hope that I made some
obvious mistake, as this appears to be a pretty serious problem otherwise.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sun, Jun 28, 2020 at 3:50 PM Tom Seewald <tseewald(a)gmail.com&gt; wrote:
>  > The context of that is: the default when the user does not specify. If
>  > the user chooses 'raid1' in the installer, they get 'raid1' for
> both
>  > data and metadata.
>  This does not seem to be the case, and from what I can tell Garry experienced this
> problem as well.
>
>  I tested this in a VM with two disks, I manually selected the "raid1" profile
> with btrfs in the advanced custom partitioning screen. 
There are three paths/UIs: Automatic, Custom, and Advanced Custom (blivet-gui).

It works as expected with Custom.

It does not work with Advanced Custom: even though raid1 is selected,
it's plainly wrong in the overview where it shows a value consistent
with 'single' or 'raid0' profile, rather than half the quantity of
storage I'd expect. When I go to 'display info about selected device'
it shows data and metadata as 'none' which is not a valid profile for
btrfs.

https://bugzilla.redhat.com/show_bug.cgi?id=1851779

>  I interpret "Data, RAID0" to mean that the data is striped
> rather than mirrored, and thus my data will be lost if/when a single drive fails. I hope
> that I made some obvious mistake, as this appears to be a pretty serious problem
> otherwise. 
You didn't make a mistake. Pretty sure it's a blocker bug too so I've
proposed it as such.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tom Seewald]

>  You didn't make a mistake. Pretty sure it's a blocker bug too
> so I've
>  proposed it as such. 
Thank you for doing that, I appreciate it.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

On Sat, Jun 27, 2020 at 2:30 PM Chris Murphy <lists(a)colorremedies.com&gt;
wrote:

>  On Sat, Jun 27, 2020 at 2:53 PM Gerald B. Cox <gbcox(a)bzb.us&gt;
> wrote:
>
>  > Why would we be installing something by default that has widely known
>  broken functionality?
>
>  Because the default configuration we're using isn't broken and is
>  better than the alternatives being evaluated.
>  
That raises the question of why RHEL deprecated BTRFS and why Fedora now
apparently believes that BTRFS is better than XFS?


>  >  I would think it would be more appropriate to have people who
>  specifically want to use BTRFS functionality and are aware and
>  knowledgeable of the risks to seek it out rather than have it be some sort
>  of selective default.  The target audience you're aiming at by making it
>  the default doesn't know FAT from NTFS from EXT4 from XFS from BTRFS or do
>  they care.   Neither are they aware or even care about purported benefits.
>
>  And they're going to get into trouble with raid56 how? Are you going
>  to tell them they should convert to raid56? How is it even relevant?
>  
That's not what I said ... in fact, no where in the paragraph did I mention
raid56.  Re-read what I wrote.


>  Literally none of this is correct.
>  
Then BTRFS needs to clarify/correct their documentation.


>  >> > Why are we not concentrating on Stratis and XFS?
>  >>
>  ...
>  >
>  > Ok, then why aren't we working on that?
>
>  I don't know. Why aren't you working on that? By all means put
>  together a competing proposal.
>
>  I'm working on what I'm working on.
>  
<Sigh> That's an obvious dodge, Chris.  Stratis and XFS are Redhat
projects.  Many times Fedora demos technologies for RHEL.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 4:16 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
>  On Sat, Jun 27, 2020 at 2:30 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
> >
> > On Sat, Jun 27, 2020 at 2:53 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
> >
> > > Why would we be installing something by default that has widely known broken
> functionality?
> >
> > Because the default configuration we're using isn't broken and is
> > better than the alternatives being evaluated.
>
>
>  That raises the question of why RHEL deprecated BTRFS and why Fedora now apparently
> believes that BTRFS is better than XFS? 
I have no knowledge of Red Hat past, present or future decisions. I'm
speaking as one of the proposals' owners. The proposal attempts to
provide persuasive reasoning why Btrfs solves certain problems on the
desktop better than alternatives. If you don't think it's persuasive,
that's fine. And if Fedora disagrees, it's not going to become the
default.

But if you can state clearly why it isn't persuasive in a way anyone
could possibly answer, I'm sure someone will try. And it would help
improve the proposal.


> > >  I would think it would be more appropriate to have people
> who specifically want to use BTRFS functionality and are aware and knowledgeable of the
> risks to seek it out rather than have it be some sort of selective default.  The target
> audience you're aiming at by making it the default doesn't know FAT from NTFS from
> EXT4 from XFS from BTRFS or do they care.   Neither are they aware or even care about
> purported benefits.
> >
> > And they're going to get into trouble with raid56 how? Are you going
> > to tell them they should convert to raid56? How is it even relevant?
>
>
>  That's not what I said ... in fact, no where in the paragraph did I mention raid56. 
> Re-read what I wrote. 
It doesn't make much sense without the prior context of raid56 that
you started out with.

I don't think most folks care beyond wanting it to be reliable, and it
is. And they like the idea of being warned if their data has been
corrupted, and it does. And like the idea of saving space and wear and
tear due to less writes, and it does that too, through transparent
compression. Why do they need to be aware of extraneous, esoteric
details? They can still benefit from some things. And people who care
about more things can go explore more.

> > I'm working on what I'm working on.
>
>  <Sigh> That's an obvious dodge, Chris.  Stratis and XFS are Redhat projects. 
> Many times Fedora demos technologies for RHEL. 
I have literally nothing to do with Red Hat at all whatsoever. I don't
consider my contributions in Fedora as having anything to do with Red
Hat.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

On Sat, Jun 27, 2020 at 5:04 PM Chris Murphy <lists(a)colorremedies.com&gt;
wrote:

>  But if you can state clearly why it isn't persuasive in a way anyone
>  could possibly answer, I'm sure someone will try. And it would help
>  improve the proposal.
>  
Making something the default is a high bar to clear.  There needs to be a
compelling reason why?  The things listed in the proposal may be nice for
some people, but the uninformed masses don't care.  Further complicating
the matter is that Redhat deprecated BTRFS.  That to me raises a big red
flag that needs to be addressed.

===> You need to clearly identify what gave Redhat heartburn and identify
what has changed to make you believe those issues have been addressed.

Another thing that is particularly troubling is I can't find where it is
stated that there is a production release of BTRFS.  I've seen statements
that it is "testing in production" - whatever that means... and that it has
been deployed on "millions of servers" - but the only statement as to code
stability just says that "The Btrfs code base is under heavy development."
I can't find an official statement from the project that there has ever
been a production release.  This is concerning to me because when I
reported problems in the past I was told basically, "silly you... BTRFS
should only be used in non-critical systems - if you're concerned about
stability you shouldn't be running it."

===> If we are considering BTRFS as a default, at a bare minimum there
should be an official production release from the project.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Michael Catanzaro]

On Sat, Jun 27, 2020 at 6:25 pm, Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
>  Making something the default is a high bar to clear.  There needs to
>
>  be a compelling reason why?  The things listed in the proposal may be 
>  nice for some people, but the uninformed masses don't care. 
There is a large list of benefits, listed at 
https://fedoraproject.org/wiki/Changes/BtrfsByDefault#Benefit_to_Fedora. 
In particular, improved detection of data corruption is definitely 
something that users care about, probably reason enough on its own to 
make btrfs default. Then Workstation#154 in particular is a very 
serious user experience problem that we have been trying to solve for 
about a year now. Under the optional optimizations section, there are 
the disk lifetime benefits from using zstd compression on select 
directories.

>  Another thing that is particularly troubling is I can't find
> where it 
>  is stated that there is a production release of BTRFS. I've seen 
>  statements that it is "testing in production" - whatever that 
>  means... and that it has been deployed on "millions of servers" - but 
>  the only statement as to code stability just says that "The Btrfs 
>  code base is under heavy development." I can't find an official 
>  statement from the project that there has ever been a production 
>  release. This is concerning to me because when I reported problems in 
>  the past I was told basically, "silly you... BTRFS should only be 
>  used in non-critical systems - if you're concerned about stability 
>  you shouldn't be running it." 
Josef has provided numerous statements in this thread regarding general 
stability. See https://btrfs.wiki.kernel.org/index.php/Status for a 
detailed breakdown.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Justin Forbes]

On Sat, Jun 27, 2020 at 5:17 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
>  On Sat, Jun 27, 2020 at 2:30 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
> >
> > On Sat, Jun 27, 2020 at 2:53 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
> >
> > > Why would we be installing something by default that has widely known broken
> functionality?
> >
> > Because the default configuration we're using isn't broken and is
> > better than the alternatives being evaluated.
>
>
>  That raises the question of why RHEL deprecated BTRFS and why Fedora now apparently
> believes that BTRFS is better than XFS?
>  
Please note the Workstation SIG is not all of Fedora, this is not
being proposed for the server spin.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gerald B. Cox]

Thanks for the reply Justin, but that doesn't answer my two concerns which
I reposted earlier.  I don't believe the questions I asked were
unreasonable for something we're making a distribution default, regardless
of spin, and they shouldn't be hard questions to answer.  Everyone knows
that if BTRFS was in production status the headlines would be everywhere,
i.e.:  "After a Decade in Development BTRFS reaches Production Milestone"
but that's not the case because we all know it isn't.  Not to mention the
Redhat stance in all this.

On Sun, Jun 28, 2020 at 3:00 PM Justin Forbes <jmforbes(a)linuxtx.org&gt; wrote:

>  On Sat, Jun 27, 2020 at 5:17 PM Gerald B. Cox <gbcox(a)bzb.us&gt;
> wrote:
>  >
>  >
>  >
>  > On Sat, Jun 27, 2020 at 2:30 PM Chris Murphy <lists(a)colorremedies.com&gt;
>  wrote:
>  >>
>  >> On Sat, Jun 27, 2020 at 2:53 PM Gerald B. Cox <gbcox(a)bzb.us&gt; wrote:
>  >>
>  >> > Why would we be installing something by default that has widely known
>  broken functionality?
>  >>
>  >> Because the default configuration we're using isn't broken and is
>  >> better than the alternatives being evaluated.
>  >
>  >
>  > That raises the question of why RHEL deprecated BTRFS and why Fedora now
>  apparently believes that BTRFS is better than XFS?
>  >
>
>  Please note the Workstation SIG is not all of Fedora, this is not
>  being proposed for the server spin.
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Garry T. Williams]

On Saturday, 27 June 2020 17:29:23 EDT Chris Murphy wrote:
>  For btrfs, it is either 'single' or 'raid0' profile
> for data, but
>  'raid1' for metadata (the file system itself).
>  
>  I need to test it or maybe someone beats me to it by looking at the
>  code. But either way it's equal to or better than the current default. 
I just did that install (KDE) and it was raid0 for data (raid1 for
metatdata).

I switched to raid1 for data as soon as I noticed what had happened.

-- 
Garry T. Williams

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams <gtwilliams(a)gmail.com&gt; wrote:
>  On Saturday, 27 June 2020 17:29:23 EDT Chris Murphy wrote:
>  > For btrfs, it is either 'single' or 'raid0' profile for data, but
>  > 'raid1' for metadata (the file system itself).
>  >
>  > I need to test it or maybe someone beats me to it by looking at the
>  > code. But either way it's equal to or better than the current default.
>
>  I just did that install (KDE) and it was raid0 for data (raid1 for
>  metatdata).
>
>  I switched to raid1 for data as soon as I noticed what had happened. 

Just a PSA: btrfs raid1 does not have a concept of automatic degraded
mount in the face of a device failure. By default systemd will not
even attempt to mount it if devices are missing. And it's not advised
to use 'degraded' mount option in fstab. If you do need to mount
degraded and later the missing device is found (?) you need to scrub
to catch up the formerly missing device to the current state.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gabriel Ramirez]

On 6/27/20 9:06 PM, Chris Murphy wrote:
>  On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams
> <gtwilliams(a)gmail.com&gt; wrote:
> > On Saturday, 27 June 2020 17:29:23 EDT Chris Murphy wrote:
> >> For btrfs, it is either 'single' or 'raid0' profile for data,
> but
> >> 'raid1' for metadata (the file system itself).
> >>
> >> I need to test it or maybe someone beats me to it by looking at the
> >> code. But either way it's equal to or better than the current default.
> > I just did that install (KDE) and it was raid0 for data (raid1 for
> > metatdata).
> >
> > I switched to raid1 for data as soon as I noticed what had happened.
>
>  Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  mount in the face of a device failure. By default systemd will not
>  even attempt to mount it if devices are missing. And it's not advised
>  to use 'degraded' mount option in fstab. If you do need to mount
>  degraded and later the missing device is found (?) you need to scrub
>  to catch up the formerly missing device to the current state.
>  
That seems a step back, in my current scenario almost all my machines 
have 2 disks (not hotswap) in raid 1 with mdadm and the partitions are 
ext4, so when a disk failed:

the machine keeps working until I do a shutdown

replace the disk

start the machine with all services running

create the disk partitions, if needed reboot again, and add the 
partitions to the raid devices

So seems with btrfs raid in my situation more downtime will be required, 
but I always do custom partitioning, so this change doesn't impact me, 
but if more people will be use btrfs it need more documentation.

and yeah sometime a disk was missing and after a reboot was found so I 
has to add it to the raid devices again, but the machine was online too


Gabriel

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 9:25 PM Gabriel Ramirez <gabrielo(a)aekert.com&gt; wrote:
>  On 6/27/20 9:06 PM, Chris Murphy wrote:
>  > On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams <gtwilliams(a)gmail.com&gt;
> wrote:
>  >> On Saturday, 27 June 2020 17:29:23 EDT Chris Murphy wrote:
>  >>> For btrfs, it is either 'single' or 'raid0' profile for
> data, but
>  >>> 'raid1' for metadata (the file system itself).
>  >>>
>  >>> I need to test it or maybe someone beats me to it by looking at the
>  >>> code. But either way it's equal to or better than the current default.
>  >> I just did that install (KDE) and it was raid0 for data (raid1 for
>  >> metatdata).
>  >>
>  >> I switched to raid1 for data as soon as I noticed what had happened.
>  >
>  > Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  > mount in the face of a device failure. By default systemd will not
>  > even attempt to mount it if devices are missing. And it's not advised
>  > to use 'degraded' mount option in fstab. If you do need to mount
>  > degraded and later the missing device is found (?) you need to scrub
>  > to catch up the formerly missing device to the current state.
>  >
>  That seems a step back, 
Yes. You do gain self-healing and unambiguous scrubs, which apply only
to the used portion of the drives. Three steps forward half step back?
The priority would be to replace the failing/failed drive before a
reboot. Yes, the use case where the drive dies on startup and
unattended boot is needed is weaker.

>  in my current scenario almost all my machines
>  have 2 disks (not hotswap) in raid 1 with mdadm and the partitions are
>  ext4, so when a disk failed:
>
>  the machine keeps working until I do a shutdown' 
Same on btrfs raid1. The gotcha is with the reboot while the device is
still failed and not yet replaced.

It is also valid to do btrfs on mdadm raid1, in which case you get
integrity checking, but lose the self-healing of btrfs native raid1.

>  replace the disk 
'btrfs replace' -

There will be a "decoder ring" guide to adapt to various commands.  A
super rough draft, more of a quick and dirty concept, is here:
https://fedoraproject.org/wiki/User:Chrismurphy/lvm2btrfs

And hopefully this will expand into "how do I?" step by step use cases.

Of course the single drive case folks aren't expected to know btrfs
commands. If they want more detailed info however, they're there.

>  start the machine with all services running
>
>  create the disk partitions, if needed reboot again, and add the
>  partitions to the raid devices 
Off hand I'm not thinking a reboot is needed. You can partition the
replacement drive, and then do 'btrfs replace' - whether it's
pre-emptive or for a missing device. This command combines: mkfs /
device add, and replication of raid1 block groups using a variation on
scrub.


>  So seems with btrfs raid in my situation more downtime will be
> required,
>  but I always do custom partitioning, so this change doesn't impact me,
>  but if more people will be use btrfs it need more documentation. 
Likely.


-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Gabriel Ramirez]

On 6/27/20 11:09 PM, Chris Murphy wrote:
>  On Sat, Jun 27, 2020 at 9:25 PM Gabriel Ramirez
> <gabrielo(a)aekert.com&gt; wrote:
> > On 6/27/20 9:06 PM, Chris Murphy wrote:
> >>
> >> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
> >> mount in the face of a device failure. By default systemd will not
> >> even attempt to mount it if devices are missing. And it's not advised
> >> to use 'degraded' mount option in fstab. If you do need to mount
> >> degraded and later the missing device is found (?) you need to scrub
> >> to catch up the formerly missing device to the current state.
> >>
> > That seems a step back,
>  Yes. You do gain self-healing and unambiguous scrubs, which apply only
>  to the used portion of the drives. Three steps forward half step back?
>  The priority would be to replace the failing/failed drive before a
>  reboot. Yes, the use case where the drive dies on startup and
>  unattended boot is needed is weaker. 
yeah, but coming from mdadm, I (will be)/(was) expecting the same or 
better from a new filesystem, or documentation to resolve the situation 
"shutdown, replace the disk, boot with another media etc..."

because when the problem happens, googling sometimes fails too

>  'btrfs replace' -
>
>  There will be a "decoder ring" guide to adapt to various commands.  A
>  super rough draft, more of a quick and dirty concept, is here:
>  https://fedoraproject.org/wiki/User:Chrismurphy/lvm2btrfs
>
>  And hopefully this will expand into "how do I?" step by step use cases.
that will be appreciated because btrfs seems to have many options
>  Of course the single drive case folks aren't expected to know
> btrfs
>  commands. If they want more detailed info however, they're there.
>
> > start the machine with all services running
> >
> > create the disk partitions, if needed reboot again, and add the
> > partitions to the raid devices
>  Off hand I'm not thinking a reboot is needed. You can partition the
>  replacement drive, and then do 'btrfs replace' - whether it's
>  pre-emptive or for a missing device. This command combines: mkfs /
>  device add, and replication of raid1 block groups using a variation on
>  scrub. 
at least one shutdown, start is needed (I have to open the computer 
case, replace the hard disk)

another reboot if after partition the drive the kernel complains about 
"the current partition will be used until reboot"

Gabriel

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sun, Jun 28, 2020 at 12:39 AM Gabriel Ramirez <gabrielo(a)aekert.com&gt; wrote:
>  On 6/27/20 11:09 PM, Chris Murphy wrote:
>  > On Sat, Jun 27, 2020 at 9:25 PM Gabriel Ramirez <gabrielo(a)aekert.com&gt; wrote:
>  >> On 6/27/20 9:06 PM, Chris Murphy wrote:
>  >>>
>  >>> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  >>> mount in the face of a device failure. By default systemd will not
>  >>> even attempt to mount it if devices are missing. And it's not advised
>  >>> to use 'degraded' mount option in fstab. If you do need to mount
>  >>> degraded and later the missing device is found (?) you need to scrub
>  >>> to catch up the formerly missing device to the current state.
>  >>>
>  >> That seems a step back,
>  > Yes. You do gain self-healing and unambiguous scrubs, which apply only
>  > to the used portion of the drives. Three steps forward half step back?
>  > The priority would be to replace the failing/failed drive before a
>  > reboot. Yes, the use case where the drive dies on startup and
>  > unattended boot is needed is weaker.
>
>  yeah, but coming from mdadm, I (will be)/(was) expecting the same or
>  better from a new filesystem, or documentation to resolve the situation
>  "shutdown, replace the disk, boot with another media etc..."
>
>  because when the problem happens, googling sometimes fails too 
Some distros have /usr/lib/udev/rules.d/64-btrfs.rules and others don't.

The behavior now, with that rule, udev waits for all btrfs devices to
be present. And I think it's an indefinite wait. Is it a bug in the
rule or is it a feature request? I don't know. But, I think at the
least we'd want a timer to get to a dracut shell eventually.

At that shell, you can e.g.
mount -o degraded /dev/vda1 /sysroot
exit

And the startup will proceed normally the rest of the way and you can
do a device replacement.

A better udev rule might be one that waits for all devices, and then
if there are still missing devices at say 90s is able to add
'degraded' to the initial mount attempt. That might mostly fix this.
But I don't know if sd-udev can conditionally add (insert) mount
options or if it could learn to do so. There is still some very small
window of risk for a "split brain" type problem - if different devices
get mount degraded on subsequent boots - but fixing/hardening against
that requires kernel work.


>  at least one shutdown, start is needed (I have to open the computer
>  case, replace the hard disk) 
Yep, if you can't hotswap you do need one.


>  another reboot if after partition the drive the kernel complains
> about
>  "the current partition will be used until reboot" 
This shouldn't be necessary. The drive is not in use, so it's possible
to apply the partition map, and the kernel should just update. If not
then partprobe should do it. (Otherwise we'd have to reboot to do
software installs, but we don't.) The limitation you're talking about
is when repartitioning a drive that is in use already - and the old
partition map is basically pinned in memory. And yes it's annoying.



--
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tomasz Torcz]

On Sun, Jun 28, 2020 at 11:15:01AM -0600, Chris Murphy wrote:
>  On Sun, Jun 28, 2020 at 12:39 AM Gabriel Ramirez
> <gabrielo(a)aekert.com&gt; wrote:
>  >
>  > On 6/27/20 11:09 PM, Chris Murphy wrote:
>  > > On Sat, Jun 27, 2020 at 9:25 PM Gabriel Ramirez <gabrielo(a)aekert.com&gt;
> wrote:
>  > >> On 6/27/20 9:06 PM, Chris Murphy wrote:
>  > >>>
>  > >>> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  > >>> mount in the face of a device failure. By default systemd will not
>  > >>> even attempt to mount it if devices are missing. And it's not
> advised
>  > >>> to use 'degraded' mount option in fstab. If you do need to
> mount
>  > >>> degraded and later the missing device is found (?) you need to scrub
>  > >>> to catch up the formerly missing device to the current state.
>  > >>>
>  > >> That seems a step back,
>  > > Yes. You do gain self-healing and unambiguous scrubs, which apply only
>  > > to the used portion of the drives. Three steps forward half step back?
>  > > The priority would be to replace the failing/failed drive before a
>  > > reboot. Yes, the use case where the drive dies on startup and
>  > > unattended boot is needed is weaker.
>  >
>  > yeah, but coming from mdadm, I (will be)/(was) expecting the same or
>  > better from a new filesystem, or documentation to resolve the situation
>  > "shutdown, replace the disk, boot with another media etc..."
>  >
>  > because when the problem happens, googling sometimes fails too
>  
>  Some distros have /usr/lib/udev/rules.d/64-btrfs.rules and others don't.
>  
>  The behavior now, with that rule, udev waits for all btrfs devices to
>  be present. And I think it's an indefinite wait. Is it a bug in the
>  rule or is it a feature request? I don't know. But, I think at the
>  least we'd want a timer to get to a dracut shell eventually.
>  
>  At that shell, you can e.g.
>  mount -o degraded /dev/vda1 /sysroot
>  exit
>  
>  And the startup will proceed normally the rest of the way and you can
>  do a device replacement.
>  
>  A better udev rule might be one that waits for all devices, and then
>  if there are still missing devices at say 90s is able to add
>  'degraded' to the initial mount attempt. That might mostly fix this.
>  But I don't know if sd-udev can conditionally add (insert) mount
>  options or if it could learn to do so. There is still some very small
>  window of risk for a "split brain" type problem - if different devices
>  get mount degraded on subsequent boots - but fixing/hardening against
>  that requires kernel work. 
  Nb. this is how mdadm works - assembly starts a timer from udev rule.
If the timer passes but not all devices are present, the array is
started in degraded mode:
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/tree/udev-md-raid-as...
https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/tree/systemd

  This is a bit hackish, as there is a hardcoded amount of seconds system
will wait for all devices.
  Also, this scheme was created by mdadm maintainer. He can be trusted
to be knowledgable enough about md. Btrfs assembly rules, on the other
hand, are maintained in systemd.  Systemd developers are not btrfs
experts, so they are cautious and quite conservative.
degraded-assembly-after-timeout should be shipped in btrfs-progs, and
maintained by people who are btrfs experts.


-- 
Tomasz Torcz                                                       72->|   80->|
tomek(a)pipebreaker.pl                                               72->|   80->|

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Lennart Poettering]

On So, 28.06.20 11:15, Chris Murphy (lists(a)colorremedies.com) wrote:

>  On Sun, Jun 28, 2020 at 12:39 AM Gabriel Ramirez
> <gabrielo(a)aekert.com&gt; wrote:
>  >
>  > On 6/27/20 11:09 PM, Chris Murphy wrote:
>  > > On Sat, Jun 27, 2020 at 9:25 PM Gabriel Ramirez <gabrielo(a)aekert.com&gt;
> wrote:
>  > >> On 6/27/20 9:06 PM, Chris Murphy wrote:
>  > >>>
>  > >>> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  > >>> mount in the face of a device failure. By default systemd will not
>  > >>> even attempt to mount it if devices are missing. And it's not
> advised
>  > >>> to use 'degraded' mount option in fstab. If you do need to
> mount
>  > >>> degraded and later the missing device is found (?) you need to scrub
>  > >>> to catch up the formerly missing device to the current state.
>  > >>>
>  > >> That seems a step back,
>  > > Yes. You do gain self-healing and unambiguous scrubs, which apply only
>  > > to the used portion of the drives. Three steps forward half step back?
>  > > The priority would be to replace the failing/failed drive before a
>  > > reboot. Yes, the use case where the drive dies on startup and
>  > > unattended boot is needed is weaker.
>  >
>  > yeah, but coming from mdadm, I (will be)/(was) expecting the same or
>  > better from a new filesystem, or documentation to resolve the situation
>  > "shutdown, replace the disk, boot with another media etc..."
>  >
>  > because when the problem happens, googling sometimes fails too
>
>  Some distros have /usr/lib/udev/rules.d/64-btrfs.rules and others don't.
>
>  The behavior now, with that rule, udev waits for all btrfs devices to
>  be present. And I think it's an indefinite wait. Is it a bug in the
>  rule or is it a feature request? I don't know. But, I think at the
>  least we'd want a timer to get to a dracut shell eventually. 
systemd is not in the business of being a RAID manager, i.e. it has no
understanding of a policy regarding how long to wait for how many
devices before continuing in degraded mode. This is something the
btrfs people have to come up with.

At the moment we support an absolute minimum mechanism: we wait until
all members of the btrfs RAID array are there, and the last device
popping up of it then marks the btrfs fs to be ready to mount. That's
all. That's what 64-btrfs.rules does, nothing more.

Now, if you put a time-out on waiting for a file system (which is the
default, though most installers turn it off when LUKS is involved,
since required interactivity — i.e. entering the pw — can take any
time in the world) then you will enter emergency mode if RAID is not
complete, and you can figure out yourself if you want to continue in
degraded mode, systemd won't help you and this is not going to change.

If there's demand for a RAID policy mgr for btrfs, it needs to be done
by btrfs people, and maintained by them, I see no point in maintaining
anything but the most minimum support for btrfs activation in systemd.

Lennart

--
Lennart Poettering, Berlin

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tom Seewald]

>  On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams
> <gtwilliams(a)gmail.com&gt; wrote:
>  
>  
>  Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  mount in the face of a device failure. By default systemd will not
>  even attempt to mount it if devices are missing. 
Is this hopefully seen by upstream as a bug that will be fixed? This removes the system
availability benefits of raid, and I've never heard of another system that would
behave like this, whether that's zfs, md, or hardware raid.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sat, Jun 27, 2020 at 9:40 PM Tom Seewald <tseewald(a)gmail.com&gt; wrote:
>  > On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams <gtwilliams(a)gmail.com&gt;
> wrote:
>  >
>  >
>  > Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  > mount in the face of a device failure. By default systemd will not
>  > even attempt to mount it if devices are missing.
>
>  Is this hopefully seen by upstream as a bug that will be fixed? This removes the system
> availability benefits of raid, and I've never heard of another system that would
> behave like this, whether that's zfs, md, or hardware raid. 
I'm not sure where it is in the priority list.

If you're doing a preemptive replace, there's no degraded state. Even
if there's a crash during this replace, all devices are present, so
it'll boot normally. The difficulty is if a drive has died, and
there's a reboot before a replace has started.

There might be a way to add a delay timeout to the systemd-udev rule
to make sure a degraded mount is intentional, rather than due to some
transient delay of one drive - something like this already exists with
mdadm assembly in dracut, maybe 90 seconds? Anyway, that doesn't
eliminate all the risk but it might be OK for some setups.

Other work is needed in this area, for example installations on UEFI
don't automatically create two EFI system partitions, so there aren't
two bootloaders or NVRAM entries. There's a hack/workaround that
upstream linux-raid@ doesn't like, where md raid1 is used for the ESP,
etc. /boot/efi should just go away entirely. There is no good reason
for bootloader volumes being persistently mounted, they aren't user
domain anyway, users shouldn't have to know about such things or
repair them or sync them. I don't think it's in the initial mandate
but something like this might be possible with the new bootupd
project.
https://github.com/coreos/bootupd



-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Neal Gompa]

On Sun, Jun 28, 2020 at 12:47 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Other work is needed in this area, for example installations on UEFI
>  don't automatically create two EFI system partitions, so there aren't
>  two bootloaders or NVRAM entries. There's a hack/workaround that
>  upstream linux-raid@ doesn't like, where md raid1 is used for the ESP,
>  etc. /boot/efi should just go away entirely. There is no good reason
>  for bootloader volumes being persistently mounted, they aren't user
>  domain anyway, users shouldn't have to know about such things or
>  repair them or sync them. I don't think it's in the initial mandate
>  but something like this might be possible with the new bootupd
>  project.
>  https://github.com/coreos/bootupd
>  
We'd have to stop installing files into /boot directly with RPM for
that sort of thing. While that would be beneficial for a whole host of
reasons, I am unsure how easy it would be to make that change.



-- 
真実はいつも一つ！/ Always, there's only one truth!

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Tom Seewald]

>  I'm not sure where it is in the priority list.
>  
>  If you're doing a preemptive replace, there's no degraded state. Even
>  if there's a crash during this replace, all devices are present, so
>  it'll boot normally. The difficulty is if a drive has died, and
>  there's a reboot before a replace has started. 
The main problem is that drive failures can occur when a system crashes, like a sudden
power loss. In such a case all other raid 1 arrays would be able to boot normally and
continue to be available with one failed drive. Given this honestly bizarre behavior of
btrfs in the face of drive failure, I don't think its raid implementation can be
considered production ready yet.

I understand that raid setups are not the common case, but if it is an option in the
installer I think it should at least come with a warning.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Sun, Jun 28, 2020 at 12:04 AM Tom Seewald <tseewald(a)gmail.com&gt; wrote:
>  > I'm not sure where it is in the priority list.
>  >
>  > If you're doing a preemptive replace, there's no degraded state. Even
>  > if there's a crash during this replace, all devices are present, so
>  > it'll boot normally. The difficulty is if a drive has died, and
>  > there's a reboot before a replace has started.
>
>  The main problem is that drive failures can occur when a system crashes, like a sudden
> power loss. In such a case all other raid 1 arrays would be able to boot normally and
> continue to be available with one failed drive. Given this honestly bizarre behavior of
> btrfs in the face of drive failure, I don't think its raid implementation can be
> considered production ready yet. 
I am not discounting the validity of your use case, or your
requirement. I'm just explaining how it works and why there's this
limitation. And folks do use it in production nevertheless, so they
are accepting this weakness in favor of other strengths. It's not a
right or wrong, it's a question of having to make choices, and picking
your consequences. Sometimes that's hard. It would be nice if this
were easy and we could just say categorically across the board Btrfs
is better for all use cases so just use that. :D

>  I understand that raid setups are not the common case, but if it is
> an option in the installer I think it should at least come with a warning. 
It is an option in custom partitioning for 8 years.

-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Przemek Klosowski]

On 6/27/20 11:40 PM, Tom Seewald wrote:
> > On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams
> <gtwilliams(a)gmail.com&gt; wrote:
> >
> >
> > Just a PSA: btrfs raid1 does not have a concept of automatic degraded
> > mount in the face of a device failure. By default systemd will not
> > even attempt to mount it if devices are missing.
>  Is this hopefully seen by upstream as a bug that will be fixed? This removes the system
> availability benefits of raid, and I've never heard of another system that would
> behave like this, whether that's zfs, md, or hardware raid. 
I agree that it's useful and common for data but booting from degraded 
RAID is not universally supported (I think it depends on the boot 
manager). Having said that, the flip side of it is that automatic 
degraded mount results in a non-redundant system, requiring manual 
intervention to restore proper redundancy, anyway.

In other words, we still need a mechanism to notify and ensure that the 
raid is rebuilt---too much automation could lead people into false sense 
of security.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Przemek Klosowski]

On 6/29/20 12:38 PM, Przemek Klosowski via devel wrote:
>  On 6/27/20 11:40 PM, Tom Seewald wrote:
> >> On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams 
> >> <gtwilliams(a)gmail.com&gt; wrote:
> >>
> >>
> >> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
> >> mount in the face of a device failure. By default systemd will not
> >> even attempt to mount it if devices are missing.
> > Is this hopefully seen by upstream as a bug that will be fixed? This 
> > removes the system availability benefits of raid, and I've never 
> > heard of another system that would behave like this, whether that's 
> > zfs, md, or hardware raid.
>
>  I agree that it's useful and common for data but booting from degraded 
>  RAID is not universally supported (I think it depends on the boot 
>  manager). Having said that, the flip side of it is that automatic 
>  degraded mount results in a non-redundant system, requiring manual 
>  intervention to restore proper redundancy, anyway. 
Oh, and it occurred to me that people probably use raid for two slightly 
different reasons: data safety and availability. The first group may 
actually prefer (or at least not mind) having to fix the raid before 
proceeding.

The second group presumably needs not just automatic degraded mode, but 
a system that maintains hot spares and automatically deploys them.

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chris Murphy]

On Mon, Jun 29, 2020 at 10:38 AM Przemek Klosowski via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On 6/27/20 11:40 PM, Tom Seewald wrote:
>  >> On Sat, Jun 27, 2020 at 7:32 PM Garry T. Williams
> <gtwilliams(a)gmail.com&gt; wrote:
>  >>
>  >>
>  >> Just a PSA: btrfs raid1 does not have a concept of automatic degraded
>  >> mount in the face of a device failure. By default systemd will not
>  >> even attempt to mount it if devices are missing.
>  > Is this hopefully seen by upstream as a bug that will be fixed? This removes the
> system availability benefits of raid, and I've never heard of another system that
> would behave like this, whether that's zfs, md, or hardware raid.
>
>  I agree that it's useful and common for data but booting from degraded
>  RAID is not universally supported (I think it depends on the boot
>  manager). Having said that, the flip side of it is that automatic
>  degraded mount results in a non-redundant system, requiring manual
>  intervention to restore proper redundancy, anyway. 
It's a really good point.

>  In other words, we still need a mechanism to notify and ensure that the
>  raid is rebuilt---too much automation could lead people into false sense
>  of security. 
I think there is a notification in GNOME Shell in case of mdadm
degraded state. And it's appropriate to expect parity for the Btrfs
case.

-- 
Chris Murphy

Fedora 33 System-Wide Change proposal: Make btrfs the default file system for desktop variants

[Chuck Anderson]

On Sat, Jun 27, 2020 at 08:06:26PM -0600, Chris Murphy wrote:
>  Just a PSA: btrfs raid1 does not have a concept of automatic
> degraded
>  mount in the face of a device failure. By default systemd will not
>  even attempt to mount it if devices are missing. And it's not advised
>  to use 'degraded' mount option in fstab. If you do need to mount
>  degraded and later the missing device is found (?) you need to scrub
>  to catch up the formerly missing device to the current state. 
An alternative to raid1 might be lsyncd (Live Syncing Daemon).  I use
it to sync my single SSD to a HDD in near-real-time, because doing
raid1 with devices that have different performance characteristics
does not work well (the slower device slows down all writes, even when
using write-behind modes of mdraid).

Or maybe even btrfs send/receive between two separate btrfs
filesystems on the two disks?  I have no experience with that (or
btrfs at all really).

I'm cautiously optimistic and would like to see btrfs adopted as the
default due to the huge benefit to having data integrity features (I
run FreeNAS at home for that reason), but I'm honestly a bit scared
off by some of the recent comments in this thread.