BTRFS partition corrupted after deleting files in /home

[Sreyan Chakravarty]

Hi,

I don't know what happened, but I just logged into my system and
deleted some unused files in my home directory. Just some directories.

Suddenly everything on my system became read only. While rebooting I
think I saw messages like:

"BTRFS Error"

I booted into a live environment to restore my snapshots, then while
mounting the BTRFS root partition I got:

mount: /mnt: wrong fs type, bad option, bad superblock on
/dev/mapper/dm_crypt, missing codepage or helper program, or other
error.

I don't know what to do, my system won't boot.

Why is BTRFS so unstable ?

Please don't tell me to erase everything and start again. that is not feasible.

Is there any other way?
-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Sun, Jan 3, 2021 at 9:40 PM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  Hi,
>
>  I don't know what happened, but I just logged into my system and
>  deleted some unused files in my home directory. Just some directories.
>
>  Suddenly everything on my system became read only. While rebooting I
>  think I saw messages like:
>
>  "BTRFS Error"
>
>  I booted into a live environment to restore my snapshots, then while
>  mounting the BTRFS root partition I got:
>
>  mount: /mnt: wrong fs type, bad option, bad superblock on
>  /dev/mapper/dm_crypt, missing codepage or helper program, or other
>  error.
>
>  I don't know what to do, my system won't boot.
>
>  Why is BTRFS so unstable ?
>
>  Please don't tell me to erase everything and start again. that is not feasible.
>
>  Is there any other way? 
Ok, now I am scared:

https://www.suse.com/support/kb/doc/?id=000018769

The above post says that in case of fatal issues the filesystem goes read-only.

What do I do now?

-- 
Regards,
Sreyan Chakravarty

[Andrej Podzimek]

>  I don't know what happened, but I just logged into my system and
>  deleted some unused files in my home directory. Just some directories.
>  
>  Suddenly everything on my system became read only. While rebooting I
>  think I saw messages like:
>  
>  "BTRFS Error"
>  
>  I booted into a live environment to restore my snapshots, then while
>  mounting the BTRFS root partition I got:
>  
>  mount: /mnt: wrong fs type, bad option, bad superblock on
>  /dev/mapper/dm_crypt, missing codepage or helper program, or other
>  error. 
Are you sure you are opening the right LUKS device in the live environment? Is the LUKS
device readable (e.g. just using "cat /dev/mapper/dm_crypt > /dev/null")?
(Does its size look right, e.g. in "lsblk -p"?) Do you get any read errors in
dmesg (for NVME / SAS / SATA)? If you pipe your direct partition read through "pv
-arb" ("pv -arb /dev/mapper/dm_crypt > /dev/null") (or another cat-like
tool that shows the data rate), does it look reasonable?



Saving a binary image of your device would be a good first step — if the device is still
readable.


>  I don't know what to do, my system won't boot.
>  
>  Why is BTRFS so unstable ? 
What makes you so sure that this is a Btrfs problem, as opposed to a SSD or hard drive
failure or a RAM failure causing data corruption?
  (Were there no other errors before the Btrfs errors in "dmesg"?)


While data loss of any kind is (understandably) frustrating, claiming that Btrfs is
“unstable” is plain wrong and unhelpful and it is unlikely to motivate Btrfs experts to
chime in and help.
  :-/


(For example, I have been using Btrfs on all my systems since ~2010 and Btrfs RAID5/6
setups since ~2016. All I can say is that it has been perfectly stable in my case, saving
my data many times over as my poor choice of SMR disks kept backfiring on me.)


>  Please don't tell me to erase everything and start again. that is
> not feasible.
>  
>  Is there any other way? 
A few suggestions:
0. Take a binary backup of your Btrfs device, if it’s still readable.

1. Check your RAM. Does the machine have ECC? You may want to give it a few hours of
memtest, no matter what.

2. Check your SSD / disk whether it’s reading at a reasonable pace and showing nothing
suspicious in "smartctl -A" and "dmesg".

3. Then there are a few tools (see man btrfs-check, man btrfs-rescue, man btrfs-restore)
you might want to try, depending on the situation. Some of them require help from Btrfs
experts (at which point you may want to ask on their kernel mailing lists).


No filesystem can cope with failing hardware or human error (such as unintentional
low-level writes into a partition). It’s always good to rule out these cases first, before
suspecting the filesystem. (If there was a stability issue, large production setups
(https://lwn.net/Articles/824855/) would most likely spot it first.)


Andrej

[Sreyan Chakravarty]

On Sun, Jan 3, 2021 at 11:06 PM Andrej Podzimek via users
<users(a)lists.fedoraproject.org&gt; wrote:
>  Are you sure you are opening the right LUKS device in the live environment? Is the LUKS
> device readable (e.g. just using "cat /dev/mapper/dm_crypt > /dev/null")?
> (Does its size look right, e.g. in "lsblk -p"?) Do you get any read errors in
> dmesg (for NVME / SAS / SATA)? If you pipe your direct partition read through "pv
> -arb" ("pv -arb /dev/mapper/dm_crypt > /dev/null") (or another cat-like
> tool that shows the data rate), does it look reasonable? 
Yes it is fully readable.

I just got a full ddrescue image that had 0 bad-sectors. So nothing is
wrong with my disk.

This is the ddrescue output:

GNU ddrescue 1.25
Press Ctrl-C to interrupt
Initial status (read from mapfile)
rescued: 998575 MB, tried: 0 B, bad-sector: 0 B, bad areas: 0

Current status
    ipos:        0 B, non-trimmed:        0 B,  current rate:       0 B/s
    opos:        0 B, non-scraped:        0 B,  average rate:       0 B/s
non-tried:        0 B,  bad-sector:        0 B,    error rate:       0 B/s
 rescued:  998575 MB,   bad areas:        0,        run time:          0s
pct rescued:  100.00%, read errors:        0,  remaining time:         n/a
                             time since last successful read:         n/a
Finished

As you can see there are no bad sectors.

$ pv -arb /dev/mapper/dm_crypt > /dev/null
452GiB [92.0MiB/s] [97.5MiB/s]

The data rate is also reasonable.

>  Saving a binary image of your device would be a good first step — if the device is still
> readable.
>  
Yes, I did that that's why you are getting a late reply.

>  What makes you so sure that this is a Btrfs problem, as opposed to a
> SSD or hard drive failure or a RAM failure causing data corruption?
>    (Were there no other errors before the Btrfs errors in "dmesg"?) 
I think it is BTRFS because I recently had to do a lot of snapshot
creation and restoration.

Also, I don't think my RAM is to blame since I have never had a
problem with it, even now I have been on my live system for about 14
hrs, since I had to get all my work done from there.

>  While data loss of any kind is (understandably) frustrating, claiming that Btrfs is
> “unstable” is plain wrong and unhelpful and it is unlikely to motivate Btrfs experts to
> chime in and help.
>    :-/
>  
I believe it's better to call this out, rather than worry about
hurting peoples feelings.

>  A few suggestions:
>  0. Take a binary backup of your Btrfs device, if it’s still readable. 
Done.

>  1. Check your RAM. Does the machine have ECC? You may want to give it
> a few hours of memtest, no matter what.
>  
I don't think my RAM is at fault. What is an ECC ?
I will give it a memtest irregardless and get back to you, but I think
it will be a waste of time.

>  2. Check your SSD / disk whether it’s reading at a reasonable pace
> and showing nothing suspicious in "smartctl -A" and "dmesg".
>  
SmartCTL output:
https://pastebin.com/raw/B6AdLZXt

I ran the smartctl test a month ago, since I though there was
something wrong with my HDD but the guys on the mailing list told me I
did not have to worry.

https://listi.jpberlin.de/pipermail/smartmontools-support/2020-November/0...

>  3. Then there are a few tools (see man btrfs-check, man btrfs-rescue,
> man btrfs-restore) you might want to try, depending on the situation. Some of them require
> help from Btrfs experts (at which point you may want to ask on their kernel mailing
> lists).
>  
Yeah that's the only option I have left.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Sun, Jan 3, 2021, 9:11 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:

>  Hi,
>
>  I don't know what happened, but I just logged into my system and
>  deleted some unused files in my home directory. Just some directories.
>
>  Suddenly everything on my system became read only. While rebooting I
>  think I saw messages like:
>
>  "BTRFS Error"
>
>  I booted into a live environment to restore my snapshots, then while
>  mounting the BTRFS root partition I got:
>
>  mount: /mnt: wrong fs type, bad option, bad superblock on
>  /dev/mapper/dm_crypt, missing codepage or helper program, or other
>  error.
>  
Try to mount normally, then:

dmesg
btrfs check --readonly

Post the results.

You can then try:

mount -o ro,usebackuproot

--
Chris Murphy

[Chris Murphy]

By the way, fpaste is installed by default in Fedora, and it has a new
feature: --btrfsinfo

fpaste --btrfsinfo

Then paste the URL here. It'll likely format way better than whatever
MUA you're using.

---
Chris Murphy

[Sreyan Chakravarty]

On Mon, Jan 4, 2021 at 11:45 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  By the way, fpaste is installed by default in Fedora, and it has a new
>  feature: --btrfsinfo
>
>  fpaste --btrfsinfo
>
>  Then paste the URL here. It'll likely format way better than whatever
>  MUA you're using.
>  
I ran this command from the live environment since my system can no longer boot:

https://paste.centos.org/view/672bf2b0

Hope it contains vital information.


--
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Mon, Jan 4, 2021 at 1:16 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Try to mount normally, then: 
I am unable to mount normally :

# mount -t btrfs /dev/mapper/dm_crypt /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on
/dev/mapper/dm_crypt, missing codepage or helper program, or other
error.

>  dmesg 
This is what I get in dmesg:

[29867.234062] BTRFS info (device dm-4): disk space caching is enabled
[29867.234067] BTRFS info (device dm-4): has skinny extents
[29867.317955] BTRFS error (device dm-4): parent transid verify failed
on 55640064 wanted 44146 found 44438
[29867.326701] BTRFS error (device dm-4): parent transid verify failed
on 55640064 wanted 44146 found 44438
[29867.326727] BTRFS warning (device dm-4): failed to read root (objectid=9): -5
[29867.333668] BTRFS error (device dm-4): open_ctree failed

>  btrfs check --readonly 
A lot of errors, could not even upload to pastebin.

This is in my Google Drive:
https://drive.google.com/file/d/1dpW7aftB3FuD8i1J7d4nRrzZHaGF4vuN/view?us...

Let me know if you are not able to download. It's compressed via gzip.

>  mount -o ro,usebackuproot
>  
mount -o ro,usebackuproot /dev/mapper/dm_crypt /mnt/
mount: /mnt: wrong fs type, bad option, bad superblock on
/dev/mapper/dm_crypt, missing codepage or helper program, or other
error.


Something is horribly wrong.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 4, 2021 at 6:59 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Mon, Jan 4, 2021 at 1:16 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  > Try to mount normally, then:
>
>  I am unable to mount normally :
>
>  # mount -t btrfs /dev/mapper/dm_crypt /mnt/
>  mount: /mnt: wrong fs type, bad option, bad superblock on
>  /dev/mapper/dm_crypt, missing codepage or helper program, or other
>  error.
>
>  >
>  > dmesg
>
>  This is what I get in dmesg:
>
>  [29867.234062] BTRFS info (device dm-4): disk space caching is enabled
>  [29867.234067] BTRFS info (device dm-4): has skinny extents
>  [29867.317955] BTRFS error (device dm-4): parent transid verify failed
>  on 55640064 wanted 44146 found 44438
>  [29867.326701] BTRFS error (device dm-4): parent transid verify failed
>  on 55640064 wanted 44146 found 44438
>  [29867.326727] BTRFS warning (device dm-4): failed to read root (objectid=9): -5
>  [29867.333668] BTRFS error (device dm-4): open_ctree failed 
transid errors like this indicate out of order writes due to drive
firmware not honoring file system write ordering and then getting a
badly timed crash/powerfail/shutdown. However...

You report that the file system went read only while using it. This
suggests a dropped write and the file system went read-only to limit
the damage. Ideally we'd get the log, if it made it to disk, to see
what lead up to this so we can determine what the problem is and get
it fixed. What I can tell you is this is not user error but that's not
much comfort.


>  > btrfs check --readonly
>
>  A lot of errors, could not even upload to pastebin.
>
>  This is in my Google Drive:
>  https://drive.google.com/file/d/1dpW7aftB3FuD8i1J7d4nRrzZHaGF4vuN/view?us...
Yeah that's bad. I think it's fixable. We need to get a metadata dump
of the file system to see if fsck will fix it.

btrfs-image -c9 -t4 /dev/sdXY /mnt/path/to/file

That will include filenames but not any data. If you need to mask
filenames, add -ss option to the above. (-s won't help here). And the
path to file if you're on a live USB stick can just be something like
~/sreyan-btrfs.img and then put it up on the google drive. By the way
I'm on irc.freenode.net as cmurf that's usually the easier way to get
help, on #fedora channel.

Also, have you ever done a balance on this file system? (That is not a
suggestion that you should or shouldn't have. Just a yes or no
question to try and piece together some other data points.)


-- 
Chris Murphy

[Sreyan Chakravarty]

On Mon, Jan 4, 2021 at 10:14 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  transid errors like this indicate out of order writes due to drive
>  firmware not honoring file system write ordering and then getting a
>  badly timed crash/powerfail/shutdown. 
First of all thanks for your quick response.

So would I be correct assuming that the problem is in my firmware ? Or
is it too early to say anything like that ?

Is my firmware so outdated that it can't handle BTRFS ?

>  You report that the file system went read only while using it. This
>  suggests a dropped write and the file system went read-only to limit
>  the damage. Ideally we'd get the log, if it made it to disk, to see
>  what lead up to this so we can determine what the problem is and get
>  it fixed. What I can tell you is this is not user error but that's not
>  much comfort.
>  
Well it doesn't provide comfort but at least I can say that it wasn't
me who messed up my filesystem.

>  Yeah that's bad. I think it's fixable. We need to get a metadata dump
>  of the file system to see if fsck will fix it.
>
>  btrfs-image -c9 -t4 /dev/sdXY /mnt/path/to/file
>
>  That will include filenames but not any data. If you need to mask
>  filenames, add -ss option to the above. (-s won't help here). And the
>  path to file if you're on a live USB stick can just be something like
>  ~/sreyan-btrfs.img and then put it up on the google drive. 
I don't think there is any hope for my data, as I can't even create
the meta-data image:

# btrfs-image -c9 -t4 /dev/mapper/dm_crypt /run/media/liveuser/Backup\
Plus/btrfs_meta.img

parent transid verify failed on 55640064 wanted 44146 found 44438
parent transid verify failed on 55640064 wanted 44146 found 44438
parent transid verify failed on 55640064 wanted 44146 found 44438
Ignoring transid failure
parent transid verify failed on 55902208 wanted 44170 found 44438
Ignoring transid failure
parent transid verify failed on 56410112 wanted 44170 found 44439
Ignoring transid failure
parent transid verify failed on 58621952 wanted 44170 found 44439
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=178081497088 item=246 parent
level=1 child level=2
ERROR: cannot go to next leaf -5
ERROR: create failed: -5

What do I do now ?

>  I'm on irc.freenode.net as cmurf that's usually the easier
> way to get
>  help, on #fedora channel.
>  
Do I need to have a bouncer ? I am in India, and I believe you are in
the US, so when you are active, I am usually sleeping.

>  Also, have you ever done a balance on this file system? (That is not
> a
>  suggestion that you should or shouldn't have. Just a yes or no
>  question to try and piece together some other data points.)
>  
No never did anything like that.


--
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 4, 2021 at 11:32 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Mon, Jan 4, 2021 at 10:14 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  > transid errors like this indicate out of order writes due to drive
>  > firmware not honoring file system write ordering and then getting a
>  > badly timed crash/powerfail/shutdown.
>
>  First of all thanks for your quick response.
>
>  So would I be correct assuming that the problem is in my firmware ? Or
>  is it too early to say anything like that ? 
Too early. The usual case of transid errors is drive firmware bugs
*and* ill timed shutdown. Since you don't have an ill timed shutdown,
it's less likely this is a drive firmware bug, but can't be ruled out.
i.e. I'm proposing there might be a software bug here and we just need
to figure it out. Bad memory usually shows up as bit flips and doesn't
result in damage like this - but it has to be considered whether a
bitflip can affect code.

It can also be a kernel bug - the storage stack has many layers, not
just Btrfs and dm-crypt. But no one wants to go blaming other people's
work without understanding the problem.

>  Is my firmware so outdated that it can't handle BTRFS ? 
No. It's a bit complicated.

Buggy drive firmware is common. But normally it doesn't matter mainly
due to good luck. More than one thing has to go wrong to cause a
problem like (a) firmware bug exists (b) firmware bug is triggered (c)
crash/powerfail. If one of those is not true, then it's not a problem.
There is also the transient hardware defect problem that can act like
a bug but it's just rotting the metadata or data. It's not obvious but
it is possible to piece together what's happened when we have enough
information.



>  # btrfs-image -c9 -t4 /dev/mapper/dm_crypt
> /run/media/liveuser/Backup\
>  Plus/btrfs_meta.img
>
>  parent transid verify failed on 55640064 wanted 44146 found 44438
>  parent transid verify failed on 55640064 wanted 44146 found 44438
>  parent transid verify failed on 55640064 wanted 44146 found 44438
>  Ignoring transid failure
>  parent transid verify failed on 55902208 wanted 44170 found 44438
>  Ignoring transid failure
>  parent transid verify failed on 56410112 wanted 44170 found 44439
>  Ignoring transid failure
>  parent transid verify failed on 58621952 wanted 44170 found 44439
>  Ignoring transid failure
>  ERROR: child eb corrupted: parent bytenr=178081497088 item=246 parent
>  level=1 child level=2
>  ERROR: cannot go to next leaf -5
>  ERROR: create failed: -5
>
>  What do I do now ? 
Rats. Can you retry by adding -w option? In the meantime I'll report
back to upstream and see what they recommend next.



>  > I'm on irc.freenode.net as cmurf that's usually the
> easier way to get
>  > help, on #fedora channel.
>  >
>
>  Do I need to have a bouncer ? I am in India, and I believe you are in
>  the US, so when you are active, I am usually sleeping. 
An alternative is matrix. We have a matrix-irc bridge in #fedora and
pretty soon I think the plan is to switch mainly to matrix. So if you
know about matrix then you can  join #fedora - but I don't know how to
explain it very well since I don't use matrix yet. I think it keeps
the history for you, unlike IRC (I use a bouncer so I will see your
messages later). I keep weird hours so it might overlap at some point.




--
Chris Murphy

[Chris Murphy]

On Mon, Jan 4, 2021 at 1:56 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  An alternative is matrix. We have a matrix-irc bridge in #fedora and
>  pretty soon I think the plan is to switch mainly to matrix. So if you
>  know about matrix then you can  join #fedora - but I don't know how to
>  explain it very well since I don't use matrix yet. I think it keeps
>  the history for you, unlike IRC (I use a bouncer so I will see your
>  messages later). I keep weird hours so it might overlap at some point. 
OK one way to do this is get an account at element.io and use their
webapp (or android or ios). I still don't know how you find #fedora
channel once you're on matrix but maybe that's obvious - probably
something like find irc.freenode.net and then find #fedora or
#fedora-qa (I'm on both channels).

-- 
Chris Murphy

[Matthew Miller]

On Mon, Jan 04, 2021 at 02:01:50PM -0700, Chris Murphy wrote:
>  OK one way to do this is get an account at element.io and use their
>  webapp (or android or ios). I still don't know how you find #fedora
>  channel once you're on matrix but maybe that's obvious - probably
>  something like find irc.freenode.net and then find #fedora or
>  #fedora-qa (I'm on both channels). 
It's gonna all change soon once we have the official Fedora server up and
going. So find it however you can at this point, and then expect it to
change. :)


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Chris Murphy]

On Mon, Jan 4, 2021 at 1:56 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Mon, Jan 4, 2021 at 11:32 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  > What do I do now ?
>
>  Rats. Can you retry by adding -w option? In the meantime I'll report
>  back to upstream and see what they recommend next. 
Ignore the above. New plan.

Can you clone and build this? And then do 'btrfs-image -c9 -t4 -w
/dev/sdXY /mnt/pathtoimagefile'

https://github.com/josefbacik/btrfs-progs/tree/more-robust-image

The Fedora btrfs-progs spec file says you'll need:
BuildRequires: gcc, autoconf, automake
BuildRequires: e2fsprogs-devel, libuuid-devel, zlib-devel, libzstd-devel
BuildRequires: libacl-devel, libblkid-devel, lzo-devel
BuildRequires: asciidoc, xmlto
BuildRequires: systemd
BuildRequires: python3-devel >= 3.4
BuildRequires: make

I'm not sure if it's easier to build the whole thing or if you can
just build btrfs-image using the stub makefile in image/ directory.
When I build it doesn't take long anyway, the longest is docs :P If
you can't get it to build, lemme know.

If it does build and now you get an image file; if it's bigger than
20M it won't attach to the bug I filed; so you'll have to upload it to
e.g. google drive and the post the URL in the bug or ping me on
#fedora. Then we'll test if current fsck can fix it, and if not then
enhance it. So we're maybe half way there. Obviously the first goal is
fix the fs, but then a second goal is to see if the systemd journal on
this file system has any hints about what went wrong because that's
the *actual* important thing to get to, is the finish line. That way
it hopefully doesn't happen again to you or anyone else.

It's bad luck and tedious but sometimes the way it goes.


-- 
Chris Murphy

[Chris Murphy]

On Mon, Jan 4, 2021 at 3:00 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Ignore the above. New plan.
>
>  Can you clone and build this? And then do 'btrfs-image -c9 -t4 -w
>  /dev/sdXY /mnt/pathtoimagefile' 
New new plan, ngompa built it for us in Fedora copr.

sudo dnf install
https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...

That will replace your btrfs-progs with josef's special repo with the
better -w. Later, you can revert back to the original:

sudo dnf install
https://kojipkgs.fedoraproject.org//packages/btrfs-progs/5.9/1.fc33/x86_6...

But there's no urgency to revert, it's the same thing just with this
btrfs-image enhancement.



-- 
Chris Murphy

[Sreyan Chakravarty]

On Tue, Jan 5, 2021 at 5:27 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  New new plan, ngompa built it for us in Fedora copr.
>
>  sudo dnf install
>
> https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...
>
>  That will replace your btrfs-progs with josef's special repo with the
>  better -w. Later, you can revert back to the original:
>
>  sudo dnf install
>
> https://kojipkgs.fedoraproject.org//packages/btrfs-progs/5.9/1.fc33/x86_6...
>
>  But there's no urgency to revert, it's the same thing just with this
>  btrfs-image enhancement.
>  
It worked. Installing the COPR build worked without any problems.

But I did get this on the console:

parent transid verify failed on 55640064 wanted 44146 found 44438
parent transid verify failed on 55640064 wanted 44146 found 44438
parent transid verify failed on 55640064 wanted 44146 found 44438
Ignoring transid failure

You can find the metadata image over here:
https://drive.google.com/file/d/1MIwwtKvt8zQxrMomhvtBZXx_0au2-Pw6/view?us...

The file is over 300 MB, and compressed with GZIP. Let me know if you
are unable to download it.

No file names are obfuscated, I did not use the -s option, so
hopefully that should help you.

Let me know what's next.

PS: I have messaged you on IRC #fedora.
-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Tue, Jan 5, 2021 at 5:27 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Mon, Jan 4, 2021 at 3:00 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  > Ignore the above. New plan.
>  >
>  > Can you clone and build this? And then do 'btrfs-image -c9 -t4 -w
>  > /dev/sdXY /mnt/pathtoimagefile'
>
>  New new plan, ngompa built it for us in Fedora copr.
>
>  sudo dnf install
>
> https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...
>
>  That will replace your btrfs-progs with josef's special repo with the
>  better -w. Later, you can revert back to the original:
>
>  sudo dnf install
>
> https://kojipkgs.fedoraproject.org//packages/btrfs-progs/5.9/1.fc33/x86_6...
>
>  But there's no urgency to revert, it's the same thing just with this
>  btrfs-image enhancement.
>  
Can you tell me how you are planning to use the metadata image ?

Are you going to mount it with losetup and then run btrfs-check on it ?

I am asking because I can perform the checks from my side.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Thu, Jan 7, 2021 at 1:40 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  Can you tell me how you are planning to use the metadata image ?
>
>  Are you going to mount it with losetup and then run btrfs-check on it ?
>
>  I am asking because I can perform the checks from my side. 
Sorry I missed you on IRC.

The Btrfs dev has been doing this with a devel branch, it's not in the
released version of btrfs-progs yet. And it needed further enhancement
for this case, because a whole bunch of writes in a commit just didn't
get to stable media. This fsck enhancement is still being tested, but
should be ready soon, and when it is I'll get it built in copr.

In the meantime, about your btrfs restore command:

# btrfs restore -sxmS -D /dev/mapper/dm_crypt
/run/media/liveuser/Backup\ Plus/restore/

Try adding -i and remove -D. The -D means files aren't restored. And
-m and -x require files to be restored first or you get additional
errors.

However, note that the -s option restores the contents of all
snapshots, and puts them into directories as independent (not deduped)
files in the restore destination. So if you have a file in 50
snapshots, that file gets replicated 50 times. So while you get all
your files out of the snapshots, it's not really putting Humpty Dumpty
back together again in terms of the original organization.

You might prefer just getting all the files out of a particular
snapshot. That's the -r option. But *sigh* it only takes subvolids.
Since you can't mount, you can't use 'btrfs sub list' to get a listing
of names to ids. There is another way to do it:

sudo btrfs inspect-internal dump-tree -t 1 /dev/sdXY | grep -A1 'ROOT_REF'

That'll produce something like this, maybe many lines if you have many
snapshots.

    item 4 key (FS_TREE ROOT_REF 268) itemoff 14921 itemsize 28
        root ref key dirid 256 sequence 5 name home
    item 5 key (FS_TREE ROOT_REF 525) itemoff 14899 itemsize 22
        root ref key dirid 256 sequence 33 name root

Above, the first item is the subvolume "home" and its ID is 268. The
second is the subvolume "root" and its ID is 525. Snapshots appear in
this same list with the same scheme. If I want to pull out just the
files in "home" I do this:

# btrfs restore -v -i -r 268 /dev/sdXY /path/to/restore/to

You could add -x -m to this if you want.

There is more here, in case I'm not around again, but hopefully we'll
figure out an overlap time on IRC.
https://btrfs.wiki.kernel.org/index.php/Restore



-- 
Chris Murphy

[Sreyan Chakravarty]

On Fri, Jan 8, 2021 at 9:59 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Sorry I missed you on IRC. 
No problem.

I apologize for hounding you on both IRC and this mailing list.

But I don't have any other option, no one knows what to do here.

>  The Btrfs dev has been doing this with a devel branch, it's not in the
>  released version of btrfs-progs yet. And it needed further enhancement
>  for this case, because a whole bunch of writes in a commit just didn't
>  get to stable media. This fsck enhancement is still being tested, but
>  should be ready soon, and when it is I'll get it built in copr.
>  
Ok, I am confused.

Is there an fsck fix available for my problem ?

Can my filesystem be fixed? Or do I have to erase everything and do a
reinstall ?

Also is the fix for my particular problem being worked on by the devs
at the particular moment ?

Are we living that close to the edge ?

>  Try adding -i and remove -D. The -D means files aren't restored. And
>  -m and -x require files to be restored first or you get additional
>  errors.
>  
Yeah I just wanted to see what happens in the dry run.

>  However, note that the -s option restores the contents of all
>  snapshots, and puts them into directories as independent (not deduped)
>  files in the restore destination. So if you have a file in 50
>  snapshots, that file gets replicated 50 times. So while you get all
>  your files out of the snapshots, it's not really putting Humpty Dumpty
>  back together again in terms of the original organization. 
I did not know that.

I guess that makes sense since snapshot subvolumes don't make much
sense outside the context of BTRFS.

>  You might prefer just getting all the files out of a particular
>  snapshot. That's the -r option. But *sigh* it only takes subvolids.
>  Since you can't mount, you can't use 'btrfs sub list' to get a listing
>  of names to ids. There is another way to do it:
>
>  sudo btrfs inspect-internal dump-tree -t 1 /dev/sdXY | grep -A1 'ROOT_REF'
>
>  That'll produce something like this, maybe many lines if you have many
>  snapshots.
>
>      item 4 key (FS_TREE ROOT_REF 268) itemoff 14921 itemsize 28
>          root ref key dirid 256 sequence 5 name home
>      item 5 key (FS_TREE ROOT_REF 525) itemoff 14899 itemsize 22
>          root ref key dirid 256 sequence 33 name root
>
>  Above, the first item is the subvolume "home" and its ID is 268. The
>  second is the subvolume "root" and its ID is 525. Snapshots appear in
>  this same list with the same scheme. If I want to pull out just the
>  files in "home" I do this:
>
>  # btrfs restore -v -i -r 268 /dev/sdXY /path/to/restore/to
>  
I will try this out.

>  You could add -x -m to this if you want.
>  
Wait didn't you just say -x and -m will give errors. I don't
understand what you mean by "-m and -x require files to be restored
first or you get additional errors."

>  There is more here, in case I'm not around again, but hopefully
> we'll
>  figure out an overlap time on IRC.
>  https://btrfs.wiki.kernel.org/index.php/Restore
>  
If you want I am available on US time but only if you let me know in advance.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Fri, Jan 8, 2021 at 4:42 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  Is there an fsck fix available for my problem ? 
Yes it just got done.

>  Can my filesystem be fixed? Or do I have to erase everything and do a
>  reinstall ? 
Strong likelihood that it can be fixed. But if the user data is
important we should do a btrfs restore first. That's a safe read-only
operation, does not change the file system like fsck which always has
risk.


>  Also is the fix for my particular problem being worked on by the
> devs
>  at the particular moment ? 
Yes.


>  Are we living that close to the edge ? 
It's actually an unusual failure, so it is an edge case. I'm happy to
discuss this part later. Priority is to get your data backed up, and
fix the file system.


>  > You could add -x -m to this if you want.
>  >
>
>  Wait didn't you just say -x and -m will give errors. I don't
>  understand what you mean by "-m and -x require files to be restored
>  first or you get additional errors." 
-m and -x only get applied after files are restored. The files are
extracted and saved. And then the metadata/xattr is applied to the
files.

-D prevents the files from being restored. It's a dry run. Since
there's no files, -m -x results in errors.


>  If you want I am available on US time but only if you let me know in
> advance. 
I saw your message today on IRC and replied about 15 seconds later but
didn't get a reply. Anyway, next email I'll write up a procedure.

-- 
Chris Murphy

[Sreyan Chakravarty]

On Sat, Jan 9, 2021 at 7:34 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Strong likelihood that it can be fixed. But if the user data is
>  important we should do a btrfs restore first. That's a safe read-only
>  operation, does not change the file system like fsck which always has
>  risk.
>  
Doesn't it make more sense to use ddrescue and make a filesystem image ?

I mean if the fsck makes things worse then I can restore with ddrescue.

Rather than taking a btrfs restore it's better I take an image right ?

>  > Also is the fix for my particular problem being worked on by the devs
>  > at the particular moment ?
>
>  Yes.
>  
That's pretty cool. But what happens when BTRFS crashes on me again ?

I mean, this is the reason I keep complaining that BTRFS is not stable enough.

I can't keep complaining to you or the devs each time the filesystem
crashes on me, can I.

>  It's actually an unusual failure, so it is an edge case. I'm happy to
>  discuss this part later. Priority is to get your data backed up, and
>  fix the file system.
>  
So is this a problem of my machine/firmware or a problem in BTRFS itself ?

I am either extremely lucky or extremely unlucky if I have spotted an
error in the filesystem itself.

>  I saw your message today on IRC and replied about 15 seconds later but
>  didn't get a reply. Anyway, next email I'll write up a procedure. 
Apologies, I must have missed it. I think you message me around 1 in
the morning in Indian Standard Time.

-- 
Regards,
Sreyan Chakravarty

[Gordon Messmer]

On 1/9/21 9:09 AM, Sreyan Chakravarty wrote:
>  I mean, this is the reason I keep complaining that BTRFS is not
> stable enough.
>  ...
>  So is this a problem of my machine/firmware or a problem in BTRFS itself ? 

Since I haven't seen it asked yet: Does this system have ECC RAM?

I don't want to shortcut the debugging process or suggest that I think 
there are no bugs in btrfs, but one of the things we expected to happen 
in the move to checksummed filesystems is that bits that flipped in 
memory and then got written to disk were going to be discovered, where 
older filesystems had no way to discover that.  In those cases, bailing 
out and informing you that corruption was found isn't a sign that the 
system is "not stable enough", it's the desired outcome of actually 
checking the results.

[Sreyan Chakravarty]

On Sun, Jan 10, 2021 at 12:22 AM Gordon Messmer
<gordon.messmer(a)gmail.com&gt; wrote:
>  On 1/9/21 9:09 AM, Sreyan Chakravarty wrote:
>  > I mean, this is the reason I keep complaining that BTRFS is not stable enough.
>  > ...
>  > So is this a problem of my machine/firmware or a problem in BTRFS itself ?
>
>
>  Since I haven't seen it asked yet: Does this system have ECC RAM?
>
>  I don't want to shortcut the debugging process or suggest that I think
>  there are no bugs in btrfs, but one of the things we expected to happen
>  in the move to checksummed filesystems is that bits that flipped in
>  memory and then got written to disk were going to be discovered, where
>  older filesystems had no way to discover that.  In those cases, bailing
>  out and informing you that corruption was found isn't a sign that the
>  system is "not stable enough", it's the desired outcome of actually
>  checking the results. 
I don't know.

How can I check from a live environment ?

-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Sun, Jan 10, 2021 at 12:22 AM Gordon Messmer
<gordon.messmer(a)gmail.com&gt; wrote:
>  On 1/9/21 9:09 AM, Sreyan Chakravarty wrote:
>  > I mean, this is the reason I keep complaining that BTRFS is not stable enough.
>  > ...
>  > So is this a problem of my machine/firmware or a problem in BTRFS itself ?
>
>
>  Since I haven't seen it asked yet: Does this system have ECC RAM?
>
>  I don't want to shortcut the debugging process or suggest that I think
>  there are no bugs in btrfs, but one of the things we expected to happen
>  in the move to checksummed filesystems is that bits that flipped in
>  memory and then got written to disk were going to be discovered, where
>  older filesystems had no way to discover that.  In those cases, bailing
>  out and informing you that corruption was found isn't a sign that the
>  system is "not stable enough", it's the desired outcome of actually
>  checking the results. 
https://askubuntu.com/a/391178/628460

Will the above work in telling me if I have ECC RAM ?

-- 
Regards,
Sreyan Chakravarty

[Gordon Messmer]

On 1/9/21 11:49 AM, Sreyan Chakravarty wrote:
>  https://askubuntu.com/a/391178/628460
>  Will the above work in telling me if I have ECC RAM ? 
It should:

# lshw -C memory
...
   *-memory
        description: System Memory
        physical id: 27
        slot: System board or motherboard
        size: 32GiB
        capabilities: ecc
        configuration: errordetection=multi-bit-ecc

[Sreyan Chakravarty]

On Sun, Jan 10, 2021 at 1:43 AM Gordon Messmer <gordon.messmer(a)gmail.com&gt; wrote:
>  On 1/9/21 11:49 AM, Sreyan Chakravarty wrote:
>  > https://askubuntu.com/a/391178/628460
>  > Will the above work in telling me if I have ECC RAM ?
>
>  It should:
>
>  # lshw -C memory
>  ...
>     *-memory
>          description: System Memory
>          physical id: 27
>          slot: System board or motherboard
>          size: 32GiB
>          capabilities: ecc
>          configuration: errordetection=multi-bit-ecc
>  
Nope. I don't have ECC RAM.

Here is the output from `lshw -C memory`
https://pastebin.com/raw/CUDp9dv9

And here is the output from `dmidecode --type memory`
https://pastebin.com/raw/mWxaLUiA

I see no ECC in any of them.

If I don't have ECC does that mean I shouldn't use BTRFS ?
-- 
Regards,
Sreyan Chakravarty

[Matthew Miller]

On Sun, Jan 10, 2021 at 02:01:56AM +0530, Sreyan Chakravarty wrote:
>  If I don't have ECC does that mean I shouldn't use BTRFS ?
The chance of data corruption due to memory errors is rare, but it does
happen. If it happened and you were using ext4, the result wouldn't be that
everything is fine, it's that you'd have corrupted data -- hopefully not
important, but... you won't know.

Now, I don't think btrfs's current state of "now your system won't boot
and
you need to be an expert to figure out what's going on" is ideal either, but
it's a rare situation and recovery tools will improve.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Sreyan Chakravarty]

On Sun, Jan 10, 2021 at 2:11 AM Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Sun, Jan 10, 2021 at 02:01:56AM +0530, Sreyan Chakravarty wrote:
>  > If I don't have ECC does that mean I shouldn't use BTRFS ?
>
>  The chance of data corruption due to memory errors is rare, but it does
>  happen. If it happened and you were using ext4, the result wouldn't be that
>  everything is fine, it's that you'd have corrupted data -- hopefully not
>  important, but... you won't know.
>
>  Now, I don't think btrfs's current state of "now your system won't boot
> and
>  you need to be an expert to figure out what's going on" is ideal either, but
>  it's a rare situation and recovery tools will improve.
>  
With the amount of time that Chris and everyone else put into this, I
just hope this bug is fixed for good.


-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 11, 2021, 12:22 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt;
wrote:

>  On Sun, Jan 10, 2021 at 2:11 AM Matthew Miller
> <mattdm(a)fedoraproject.org&gt;
>  wrote:
>  >
>  > On Sun, Jan 10, 2021 at 02:01:56AM +0530, Sreyan Chakravarty wrote:
>  > > If I don't have ECC does that mean I shouldn't use BTRFS ?
>  >
>  > The chance of data corruption due to memory errors is rare, but it does
>  > happen. If it happened and you were using ext4, the result wouldn't be
>  that
>  > everything is fine, it's that you'd have corrupted data -- hopefully not
>  > important, but... you won't know.
>  >
>  > Now, I don't think btrfs's current state of "now your system won't
> boot
>  and
>  > you need to be an expert to figure out what's going on" is ideal either,
>  but
>  > it's a rare situation and recovery tools will improve.
>  >
>
>  With the amount of time that Chris and everyone else put into this, I
>  just hope this bug is fixed for good.
>  

No bug has been identified.

Have you gone through the numbered list and tried to repair? It is
important you install specifically the btrfs-progs in that email. It has an
enhancement for this particular kind of damage.

https://lists.fedoraproject.org/archives/list/[e-mail].o...

--
Chris Murphy

[Sreyan Chakravarty]

On Mon, 11 Jan 2021, 3:29 pm Chris Murphy, <lists(a)colorremedies.com&gt; wrote:

>  On Mon, Jan 11, 2021, 12:22 AM Sreyan Chakravarty <sreyan32(a)gmail.com
> >  
>  wrote:
>
> > On Sun, Jan 10, 2021 at 2:11 AM Matthew Miller <mattdm(a)fedoraproject.org
> >  
> > wrote:
> >
> >  
> > > On Sun, Jan 10, 2021 at 02:01:56AM +0530, Sreyan
> Chakravarty wrote:
> > > > If I don't have ECC does that mean I shouldn't use BTRFS ?
> >
> >  
> > > The chance of data corruption due to memory errors is
> rare, but it does
> > > happen. If it happened and you were using ext4, the result wouldn't be
> > that
> > > everything is fine, it's that you'd have corrupted data -- hopefully
> not
> > > important, but... you won't know.
> >
> >  
> > > Now, I don't think btrfs's current state of
> "now your system won't boot
> > and
> > > you need to be an expert to figure out what's going on" is ideal
> > either, but
> > > it's a rare situation and recovery tools will improve.
> >
> >  
> >  
> > With the amount of time that
> Chris and everyone else put into this, I
> > just hope this bug is fixed for good.
> >  
>
>  No bug has been identified.
>
>  Have you gone through the numbered list and tried to repair? It is
>  important you install specifically the btrfs-progs in that email. It has an
>  enhancement for this particular kind of damage.
>
>
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  
Yes I have installed the specific version of btrfs-progs that you have
shared.

But the repair has been running for over 3 hours now.

I do have a 930 GiB partition.

>

[Chris Murphy]

On Sat, Jan 9, 2021 at 1:32 PM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  If I don't have ECC does that mean I shouldn't use BTRFS ? 
No. And also, this problem is not related to a memory error. In this
case multiple writes from one commit are missing. They just didn't
make it to stable media. There isn't enough information to know why.
We might get lucky and find some clues in the systemd journal, before
the file system went read only.

What I can tell you is Btrfs doesn't have any known or suspected
defects related to this kind of problem. It writes a superblock only
once it has reason to believe the prior metadata writes were on stable
media. There is also a write time tree checker designed to complain
and go read only the moment there's an inconsistency when writing.
This file system has a longer period of time of writes following the
missing writes, which is also why the 'usebackup' root option didn't
work. And the dropped writes also affect both copies of metadata. It's
very unusual.

Sreyan, is the drive that this file system is on the same drive as in
this reported thread?
https://lists.fedoraproject.org/archives/list/[e-mail].o...



-- 
Chris Murphy

[Sreyan Chakravarty]

On Sun, Jan 10, 2021 at 3:10 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  No. And also, this problem is not related to a memory error. In this
>  case multiple writes from one commit are missing. They just didn't
>  make it to stable media. There isn't enough information to know why.
>  We might get lucky and find some clues in the systemd journal, before
>  the file system went read only.
>
>  What I can tell you is Btrfs doesn't have any known or suspected
>  defects related to this kind of problem. It writes a superblock only
>  once it has reason to believe the prior metadata writes were on stable
>  media. There is also a write time tree checker designed to complain
>  and go read only the moment there's an inconsistency when writing.
>  This file system has a longer period of time of writes following the
>  missing writes, which is also why the 'usebackup' root option didn't
>  work. And the dropped writes also affect both copies of metadata. It's
>  very unusual.
>
>  Sreyan, is the drive that this file system is on the same drive as in
>  this reported thread?
>
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>
>  
Yes it is the same drive.

Does that mean something is wrong with the drive ?

-- 
Regards,
Sreyan Chakravarty

[Matthew Miller]

On Mon, Jan 11, 2021 at 12:54:07PM +0530, Sreyan Chakravarty wrote:
>  > Sreyan, is the drive that this file system is on the same drive
> as in
>  > this reported thread?
>  >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  >
>  
>  Yes it is the same drive.
>  Does that mean something is wrong with the drive ? 
It seems very likely -- either with the drive, or a cable, or the
controller, or some other aspect of your system. You're getting serious
errors regardless of what you're actually putting on the drive.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Sreyan Chakravarty]

On Mon, Jan 11, 2021 at 10:55 PM Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Mon, Jan 11, 2021 at 12:54:07PM +0530, Sreyan Chakravarty wrote:
>  > > Sreyan, is the drive that this file system is on the same drive as in
>  > > this reported thread?
>  > >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > >
>  >
>  > Yes it is the same drive.
>  > Does that mean something is wrong with the drive ?
>
>  It seems very likely -- either with the drive, or a cable, or the
>  controller, or some other aspect of your system. You're getting serious
>  errors regardless of what you're actually putting on the drive.
>  
I beg to differ.

I never faced problems like this when I was on "thick/older" LVM with EXT4.

It is only when I switched to thin-pools and then to BTRFS that I
started getting these problems.
-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 11, 2021 at 11:30 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Mon, Jan 11, 2021 at 10:55 PM Matthew Miller
>  <mattdm(a)fedoraproject.org&gt; wrote:
>  >
>  > On Mon, Jan 11, 2021 at 12:54:07PM +0530, Sreyan Chakravarty wrote:
>  > > > Sreyan, is the drive that this file system is on the same drive as in
>  > > > this reported thread?
>  > > >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  > > >
>  > >
>  > > Yes it is the same drive.
>  > > Does that mean something is wrong with the drive ?
>  >
>  > It seems very likely -- either with the drive, or a cable, or the
>  > controller, or some other aspect of your system. You're getting serious
>  > errors regardless of what you're actually putting on the drive.
>  >
>
>  I beg to differ.
>
>  I never faced problems like this when I was on "thick/older" LVM with EXT4.
>
>  It is only when I switched to thin-pools and then to BTRFS that I
>  started getting these problems. 
This is an interesting data point. I will incorporate it. In the
meantime, we really need to stay focused on facts and not jump to
conclusions.

We can evaluate whether you stick with Btrfs, pros/cons. Bugs like
this are really slippery. The #1 reason they don't get caught and set
free, is tester exhaustion. This is a reasonable outcome. People have
better things to do than debug stuff. But it's sometimes a necessary
effort because we really are at significant confidence in Btrfs
stability equivalent to other file systems, even though we're aware of
its fragility in certain bad failure cases like this one. The only way
it gets better is to persevere.

Thanks,

-- 
Chris Murphy

[Tom Horsley]

On Mon, 11 Jan 2021 20:14:04 -0700
Chris Murphy wrote:

>  The #1 reason they don't get caught and set
>  free, is tester exhaustion. This is a reasonable outcome. People have
>  better things to do than debug stuff. But it's sometimes a necessary
>  effort because we really are at significant confidence in Btrfs
>  stability equivalent to other file systems, even though we're aware of
>  its fragility in certain bad failure cases like this one. 
Not to worry! I read that bcachefs is finally making it into
the linux kernel, so real soon now it will be the new shiny object
and btrfs will be obsolete :-).

[Sreyan Chakravarty]

On Tue, Jan 12, 2021 at 8:52 AM Tom Horsley <horsley1953(a)gmail.com&gt; wrote:
>  Not to worry! I read that bcachefs is finally making it into
>  the linux kernel, so real soon now it will be the new shiny object
>  and btrfs will be obsolete :-).
>  _______________________________________________ 
Let's hope it does a better job of preventing fragmentation on larger files.

Large files are a pain in BTRFS to manage.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 11, 2021 at 12:24 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Sun, Jan 10, 2021 at 3:10 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  > Sreyan, is the drive that this file system is on the same drive as in
>  > this reported thread?
>  >
> https://lists.fedoraproject.org/archives/list/[e-mail].o...
>  >
>  >
>
>  Yes it is the same drive.
>
>  Does that mean something is wrong with the drive ? 
Short version: Josef (Btrfs dev) and I agree there's probably
something wrong with the drive. The advice is to replace it, just
because trying to investigate further isn't worth your time, and also
necessarily puts your data at risk.

Longer version:

LVM thinp uses device-mapper as its backend to do all the work, and we
see checksum errors in the months old report. Where LVM thick has a
simpler on-disk format, so it's not as likely to discover such a
problem. And LUKS/dm-crypt is also using device-mapper as its backend
to do all work. So the two problems have two things in common: the
drive, and device-mapper. It's more probable there's an issue with the
drive, just because if there was a problem with device-mapper, we'd
have dozens of reports of it at the rate you're seeing this problem
once every couple of months (if that trend holds).

Is it possible LVM+ext4 on this same drive is more reliable? I think
that's specious. The problem can be masked due to much less stringent
integrity guarantees, i.e. there are no data checksums. Since the data
is overwhelmingly the largest target for corruption, just being a much
bigger volume compared to file system metadata. All things being
equal, there's a greater chance the problem affects data. On the other
hand, if it adversely impacts metadata, it could be true that e2fsck
has a better chance of fixing the damage than btrfsck right now. Of
course no fsck fixes your data.

So if you keep using the drive, you're in a catch-22. Btrfs is more
sensitive because everything is checksummed, so the good news is
you'll be informed about it fairly quickly, the bad news is that it's
harder to repair in this case. If you revert to LVM+ext4 the automatic
fsck at startup might fix up these problems as they occur, but it's
possible undetected data corruption shows up later or replicates into
your backups.

Regardless of what you decide to do, definitely keep frequent backups
of anything important.

-- 
Chris Murphy

[Sreyan Chakravarty]

On Tue, Jan 12, 2021 at 11:05 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Short version: Josef (Btrfs dev) and I agree there's probably
>  something wrong with the drive. The advice is to replace it, just
>  because trying to investigate further isn't worth your time, and also
>  necessarily puts your data at risk.
>
>  Longer version:
>
>  LVM thinp uses device-mapper as its backend to do all the work, and we
>  see checksum errors in the months old report. Where LVM thick has a
>  simpler on-disk format, so it's not as likely to discover such a
>  problem. And LUKS/dm-crypt is also using device-mapper as its backend
>  to do all work. So the two problems have two things in common: the
>  drive, and device-mapper. It's more probable there's an issue with the
>  drive, just because if there was a problem with device-mapper, we'd
>  have dozens of reports of it at the rate you're seeing this problem
>  once every couple of months (if that trend holds).
>
>  Is it possible LVM+ext4 on this same drive is more reliable? I think
>  that's specious. The problem can be masked due to much less stringent
>  integrity guarantees, i.e. there are no data checksums. Since the data
>  is overwhelmingly the largest target for corruption, just being a much
>  bigger volume compared to file system metadata. All things being
>  equal, there's a greater chance the problem affects data. On the other
>  hand, if it adversely impacts metadata, it could be true that e2fsck
>  has a better chance of fixing the damage than btrfsck right now. Of
>  course no fsck fixes your data.
>
>  So if you keep using the drive, you're in a catch-22. Btrfs is more
>  sensitive because everything is checksummed, so the good news is
>  you'll be informed about it fairly quickly, the bad news is that it's
>  harder to repair in this case. If you revert to LVM+ext4 the automatic
>  fsck at startup might fix up these problems as they occur, but it's
>  possible undetected data corruption shows up later or replicates into
>  your backups.
>
>  Regardless of what you decide to do, definitely keep frequent backups
>  of anything important.
>  
Ok first I don't mean to imply that I don't believe you or Josef when
you say there is something wrong with my HDD or that you are wrong.
But I have a lot of questions that I want to discuss :

1) Is it possible there is nothing wrong with my drive, but there is
something with my BIOS/HDD Firmware ? May be my firmware is not
capable of BTRFS's stringent write requirements ?

I say this because I have used Windows with NTFS on this machine, I
have used Ubuntu with EXT4, and Fedora with thick-LVM with EXT4. None
of these configurations gave me any such problems.

2) Since there is a high likelihood that my filesystem is not
completely fixed, then when I take a backup using partclone, dd or
clonezilla won't those errors be carried over ?

Even if I buy a new drive and restore the backup, I still might get crashes.

3) This is a weird question but can you recommend me a HDD that I can
buy which can handle BTRFS ? Or even which features I might look for
while buying (not a SSD but a HDD)

4) My manufacturer HP, does not make firmware updates for Linux, only
for Windows. So is there a way to update the firmware(if available)
without being on Windows ? Any ideas? Would a Windows PXE help ?

5) When you say "checksum errors in the month's old report" - which
report are you referring to ? The thin-LVM crash or the smartctl crash
?

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Wed, Jan 13, 2021 at 2:36 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  1) Is it possible there is nothing wrong with my drive, but there is
>  something with my BIOS/HDD Firmware ? May be my firmware is not
>  capable of BTRFS's stringent write requirements ? 
The sample size is too small to know for sure what kind of HDD defect
it is. If it were an actuator or read/write head, it would happen more
often. If it were a localized media surface defect, it's unlikely two
copies of metadata would be affected. Btrfs dup profile metadata
chunks aren't colocated. They aren't very far apart but far enough I'd
expect a lot more metadata and/or data corruption than just one
commit. If it were defective memory (used as cache) in the drive, I'd
expect it'd happen more often. I have discussed with folks who know
way more about myriad drive failures that they've seen cases where a
write failure results in all queued (cached) writes being dropped. Is
that what happened? *shrug* Speculation. And is it a firmware bug, or
is it some other transient problem with the drive? *shrug*

I don't think it has anything to do with BIOS, or logic board related
including memory. And I don't think it has anything to do with Btrfs
write patterns. Btrfs write pattern isn't that variable, so whatever
pattern triggers a problem is going to happen more often than once
every month or two. Probably hundreds of times per day or more. And
yet, this happened once with Btrfs in a bit over a month. Pretty
weird.

Another possibility is power supply. Either brown outs or noisy
incoming power. Or even made noisy by a power supply. I had a student
a while back with a high end imaging setup, all brand new equipment.
Constant crashes. Replaced memory first. Then other hardware. And got
to adding a UPS. Voila. All problems went away. Unfortunately if
you're in such a situation it is process of elimination. And if it
only reproduces every couple of months, that could take a while.

>  I say this because I have used Windows with NTFS on this machine, I
>  have used Ubuntu with EXT4, and Fedora with thick-LVM with EXT4. None
>  of these configurations gave me any such problems. 
Yeah it's a fair point. But you did have a problem with LVM thin
provisioning which is not Btrfs. But does use checksums for at least
some (maybe all, not sure) of its metadata.

NTFS doesn't checksum anything. ext4 checksums its own metadata but
not data. A lost metadata write on either will be immediately detected
on ext4 before it causes too much confusion where NTFS will need to
get confused before it realizes something is wrong.

A lost data write on NTFS or ext4 means the next time that data is
read, it's just not there. It's garbage. So the OS won't even care,
it'll just hand over what it finds to the application, and it'd be up
to the application to handle the fact it got back garbage. It could
manifest in all kinds of ways or not even at all.

So they are sufficiently different in this area that they're not that
comparable. The most comparable would be OpenZFS. It also checksums
all metadata and data, but it's not a supported file system in Fedora.
So you're kind on your own, but there are Fedora users using OpenZFS
for sysroot (maybe even /boot, GRUB supports it).

>  2) Since there is a high likelihood that my filesystem is not
>  completely fixed, then when I take a backup using partclone, dd or
>  clonezilla won't those errors be carried over ? 
Yes. I recommend a Pika Backup for a simple GUI solution to back
things up. It doesn't have any file system specific dependencies. I'm
sure if you look through the list archive for backups or start a new
thread with your requirements you'll get more suggestions.

>  Even if I buy a new drive and restore the backup, I still might get crashes. 
You definitely want a backup with its own independent file system. A
dd/ddrescue/clone is mainly for troubleshooting and disaster recovery.
It's not a great backup because a backup you want easy to keep up to
date. Daily or weekly, depending on your tolerance for loss.

>  3) This is a weird question but can you recommend me a HDD that I can
>  buy which can handle BTRFS ? Or even which features I might look for
>  while buying (not a SSD but a HDD) 
All the drive manufacturers have played enough musical chairs, I can't
keep track of who makes or made what. Every drive fails eventually.
HDD follow the bell curve, so they tend to either fail early or fail
late in their lifespan. You can't really game the system. The odds of
picking something that exhibits this same behavior is astronomical.
Except for the NVMe drive, which came in the laptop I'm using, I
pretty much use a mix of warranty and price. It's cheaper to mitigate
risk with a backup, which you need anyway even if you get an expensive
drive. So I just ignore all claims of reliability and I don't even
care about 5+ year warranties. No 90 day warranties. 1 year if it's
dirt cheap. Otherwise 3 years. And never buy an extended warranty.

But I gotta say for sysroot, a small inexpensive SSD is pretty
awesome. It's a major upgrade. And yeah we probably see more firmware
bugs with SSD than HDD, but at least Facebook is using the cheapest
consumer drives possible, with Btrfs. And it's fine. Until it's not.
So again, all things come back to the backup. Don't worry. Just
backup. And if you backup, you won't worry. Or at least, you'll worry
less.

>  4) My manufacturer HP, does not make firmware updates for Linux, only
>  for Windows. So is there a way to update the firmware(if available)
>  without being on Windows ? Any ideas? Would a Windows PXE help ? 
I don't think this is the problem. But also,
https://www.microsoft.com/en-us/software-download/windows10

Free download. If it only can update the logic board firmware with
Windows. It'll even work without a product key, just say you don't
have a product key at the part where it asks for one. It'll still
work, with some extra limitations that won't matter.

>  5) When you say "checksum errors in the month's old
> report" - which
>  report are you referring to ? The thin-LVM crash or the smartctl crash
>  ? 
LVM thin.


-- 
Chris Murphy

[Tim]

On Wed, 2021-01-13 at 23:53 -0700, Chris Murphy wrote:
>  Another possibility is power supply. Either brown outs or noisy
>  incoming power. Or even made noisy by a power supply. 
Don't discount cabling, either.

If connectors don't fit well, that can easily lead to failures from the
tiniest of vibrations.

And people mangle SATA leads.  If you kink them, that leads to data
corruption.  There should be no sharp bends, or folds.  Which could
mean that some cables are permanently ruined before you buy them, when
they tightly bundle them in the retail packaging.
 
-- 
 
uname -rsvp
Linux 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64
 
Boilerplate:  All unexpected mail to my mailbox is automatically deleted.
I will only get to see the messages that are posted to the mailing list.

[Sreyan Chakravarty]

On Fri, Jan 8, 2021 at 9:59 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  
> In the meantime, about your btrfs restore command:
>  
> # btrfs restore -sxmS -D /dev/mapper/dm_crypt
> /run/media/liveuser/Backup\ Plus/restore/
>  
> Try adding -i and remove -D. The -D means files aren't
restored. And
> -m and -x require files to be restored first or you get additional
> errors.
>  
I have a question.

Once I get my data out via btrfs restore, can I write that data back
to my root partition ?

I mean I really don't want to format and reinstall my OS again, but
what if I could exfiltrate the data format my root partition and then
again write the backed up data into it.

Would that work?

I am considering this as a last resort, if the fsck fix does not come through.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

1. If you don't have a backup of important data, you should first use
'btrfs restore' as I've described in previous emails. The fsck should
fix the problem, but there is a chance it makes things worse. No fsck
is guaranteed safe, because it's making changes to the file system.

2. Install the following updated btrfs-progs branch that has a tested
fix for this problem. And by the way your test image has helped
improve the Btrfs fsck. Pretty much all fsck improvements for all file
systems depend on getting good user reports and information. So thanks
for not giving up.

sudo dnf install
https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...

3. If it's a laptop, make sure it has power connected, and that you
can let it run until it finishes. Because it's going to take a long
time. I don't have a time estimate for what that means exactly, it
depends mostly on memory.

4. This is the command to run, good idea to save all the terminal
output because if it doesn't work we need to see what went wrong.

sudo btrfs check --repair /dev/sdXY

Near the end I'm told there will be complaints about missing csum
errors. Ignore that for now. And try to mount normally. You should be
able to use it normally at this point. And there should be no errors
in dmesg. If there are let me know before the next step.

5. There are some checksums that didn't make it to the drive when the
original problem happened. That has a separate repair:

sudo btrfs check --init-csum-tree /dev/sdXY


That's it, everything should be OK. If something goes wrong, best to
not try again, just ask here or on IRC.

--
Chris Murphy

[Sreyan Chakravarty]

On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  1. If you don't have a backup of important data, you should first use
>  'btrfs restore' as I've described in previous emails. The fsck should
>  fix the problem, but there is a chance it makes things worse. No fsck
>  is guaranteed safe, because it's making changes to the file system.
>  
Let me know if it makes more sense to use ddrescue instead of `btrfs restore`.

In case of ddrescue we can run an fsck again since the filesystem can
be restored back to a previous state.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Sat, Jan 9, 2021, 10:11 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:

>  On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy
> <lists(a)colorremedies.com&gt;
>  wrote:
>  >
>  > 1. If you don't have a backup of important data, you should first use
>  > 'btrfs restore' as I've described in previous emails. The fsck should
>  > fix the problem, but there is a chance it makes things worse. No fsck
>  > is guaranteed safe, because it's making changes to the file system.
>  >
>
>  Let me know if it makes more sense to use ddrescue instead of `btrfs
>  restore`.
>  
It's up to you. Both methods make a copy. Thats whats important.

ddrescue will take longer and needs more space than just pulling out files
with btrfs restore. But if you're already familiar with ddrescue, that's an
advantage.

--
Chris Murphy

>

[Sreyan Chakravarty]

On Sat, Jan 9, 2021 at 11:10 PM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  On Sat, Jan 9, 2021, 10:11 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
> >
> > On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
> > >
> > > 1. If you don't have a backup of important data, you should first use
> > > 'btrfs restore' as I've described in previous emails. The fsck
> should
> > > fix the problem, but there is a chance it makes things worse. No fsck
> > > is guaranteed safe, because it's making changes to the file system.
> > >
> >
> > Let me know if it makes more sense to use ddrescue instead of `btrfs restore`.
>
>
>  It's up to you. Both methods make a copy. Thats whats important.
>
>  ddrescue will take longer and needs more space than just pulling out files with btrfs
> restore. But if you're already familiar with ddrescue, that's an advantage.
>  
Going with ddrescue then.

I will try the fix after the backup has been created.

-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>   So thanks
>  for not giving up.
>  
Actually it is me who should be thanking you for not giving up.

For me giving up was not an option since that would involve me losing
all my data.

Thank you again for your time and patience, others would have told me
to reinstall my OS or restore my backups.


>  sudo dnf install
>
> https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...
>  
Ok, I finally have some good news and some minor bad news hopefully.

Good news first.

It worked, I can now mount my BTRFS root file system:

# mount -t btrfs /dev/mapper/dm_crypt /mnt/
# ll /mnt
# total 0
drwxr-xr-x. 1 root root   0 Jan 12 00:00 before_live_cd_exp
dr-xr-xr-x. 1 root root 178 Dec 14 14:58 root
dr-xr-xr-x. 1 root root 178 Dec 14 14:58 stable_15_Dec
drw-------. 1 root root  22 Dec 11 20:05 swap


Now to give you the output of the commands and the minor bad news:

1)   btrfs check --repair /dev/mapper/dm_crypt

    This command actually executed and exited normally, but I could
see some errors at the end of the command.

    Full output of repair command over here:
    https://drive.google.com/file/d/17EQ5TYJE5a4iwRGTXXj9Hh38btxFBLAJ/view?us...

2)  btrfs check --init-csum-tree /dev/mapper/dm_crypt

   This is where the minor bad news happens.

   This command exited with a "Segmentation Fault(Core Dumped)"

   Full output here:
   https://drive.google.com/file/d/1IYnLcw-vkZ2UeH0rOZje4kbhzB_tYaKy/view?us...

   I have my core dumps set to unlimited:

   # ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31030
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31030
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

But still I was unable to get coredump file as it was not generated:

# coredumpctl list btrfs
TIME                            PID   UID   GID SIG COREFILE  EXE
Mon 2021-01-11 22:58:18 IST    4337     0     0  11 none      /usr/sbin/btrfs

Which is weird.

All the outputs are compressed using GZIP, let me know if you are
unable to download them. The repair command was run before the
init-csum-tree command.

I have also used your enhanced version of btrfs-progs that you provided:

# dnf repoquery --installed btrfs-progs
btrfs-progs-0:5.9+git20210108.3783313-0.fc33.1.x86_64

Let me know the next steps ie. Do you need BTRFS logs to diagnose the
root cause of the problem ?

Thank you once again for your help and perseverance. Couldn't have
done it without you.
-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Tue, Jan 12, 2021 at 12:24 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  >
>  >  So thanks
>  > for not giving up.
>  >
>
>  Actually it is me who should be thanking you for not giving up.
>
>  For me giving up was not an option since that would involve me losing
>  all my data.
>
>  Thank you again for your time and patience, others would have told me
>  to reinstall my OS or restore my backups.
>
>
>  > sudo dnf install
>  >
> https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robu...
>  >
>
>  Ok, I finally have some good news and some minor bad news hopefully.
>
>  Good news first.
>
>  It worked, I can now mount my BTRFS root file system:
>
>  # mount -t btrfs /dev/mapper/dm_crypt /mnt/
>  # ll /mnt
>  # total 0
>  drwxr-xr-x. 1 root root   0 Jan 12 00:00 before_live_cd_exp
>  dr-xr-xr-x. 1 root root 178 Dec 14 14:58 root
>  dr-xr-xr-x. 1 root root 178 Dec 14 14:58 stable_15_Dec
>  drw-------. 1 root root  22 Dec 11 20:05 swap
>
>
>  Now to give you the output of the commands and the minor bad news:
>
>  1)   btrfs check --repair /dev/mapper/dm_crypt
>
>      This command actually executed and exited normally, but I could
>  see some errors at the end of the command.
>
>      Full output of repair command over here:
>      https://drive.google.com/file/d/17EQ5TYJE5a4iwRGTXXj9Hh38btxFBLAJ/view?us...
>
>  2)  btrfs check --init-csum-tree /dev/mapper/dm_crypt
>
>     This is where the minor bad news happens.
>
>     This command exited with a "Segmentation Fault(Core Dumped)"
>
>     Full output here:
>     https://drive.google.com/file/d/1IYnLcw-vkZ2UeH0rOZje4kbhzB_tYaKy/view?us...
>
>     I have my core dumps set to unlimited:
>
>     # ulimit -a
>  core file size          (blocks, -c) unlimited
>  data seg size           (kbytes, -d) unlimited
>  scheduling priority             (-e) 0
>  file size               (blocks, -f) unlimited
>  pending signals                 (-i) 31030
>  max locked memory       (kbytes, -l) 64
>  max memory size         (kbytes, -m) unlimited
>  open files                      (-n) 1024
>  pipe size            (512 bytes, -p) 8
>  POSIX message queues     (bytes, -q) 819200
>  real-time priority              (-r) 0
>  stack size              (kbytes, -s) 8192
>  cpu time               (seconds, -t) unlimited
>  max user processes              (-u) 31030
>  virtual memory          (kbytes, -v) unlimited
>  file locks                      (-x) unlimited
>
>  But still I was unable to get coredump file as it was not generated:
>
>  # coredumpctl list btrfs
>  TIME                            PID   UID   GID SIG COREFILE  EXE
>  Mon 2021-01-11 22:58:18 IST    4337     0     0  11 none      /usr/sbin/btrfs
>
>  Which is weird.
>
>  All the outputs are compressed using GZIP, let me know if you are
>  unable to download them. The repair command was run before the
>  init-csum-tree command.
>
>  I have also used your enhanced version of btrfs-progs that you provided:
>
>  # dnf repoquery --installed btrfs-progs
>  btrfs-progs-0:5.9+git20210108.3783313-0.fc33.1.x86_64
>
>  Let me know the next steps ie. Do you need BTRFS logs to diagnose the
>  root cause of the problem ?
>
>  Thank you once again for your help and perseverance. Couldn't have
>  done it without you. 
Sorry missed to include the dmesg error for the init-csum-tree command:

[34922.980911] btrfs[4337]: segfault at 60 ip 00005625672ebf81 sp
00007ffee2b38180 error 4 in btrfs[562567287000+96000]
[34922.980939] Code: a0 00 00 00 00 00 00 00 e8 5c a3 ff ff 8b 4c 24
50 4c 89 fe 85 c0 0f 44 44 24 38 45 31 c0 89 44 24 38 48 8b 84 24 a0
00 00 0
0 <8b> 40 60 49 29 87 d0 00 00 00 6a 00 55 4c 8b 9c 24 90 00 00 00 48
[35498.885297] i915 0000:00:02.0: [drm] *ERROR* CPU pipe A FIFO underrun

Guess it was caused by some sort of underrun.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Mon, Jan 11, 2021 at 11:55 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  Actually it is me who should be thanking you for not giving up.
>
>  For me giving up was not an option since that would involve me losing
>  all my data. 
No problem!

We should talk about a backup plan in another thread :D In my own
case, I never need them, but I have 1/2 dozen completely independent
backups, because I hate my data and I'm constantly trying to destroy
it. (This is turning into a really bad joke but hopefully it's weirdly
amusing. If you like your data you *definitely* should be backing up!
Doesn't matter what the file system is, or if it's on raid, or "safe"
in the cloud, etc.)

>      Full output of repair command over here:
>      https://drive.google.com/file/d/17EQ5TYJE5a4iwRGTXXj9Hh38btxFBLAJ/view?us...
>
>  2)  btrfs check --init-csum-tree /dev/mapper/dm_crypt
>
>     This is where the minor bad news happens.
>
>     This command exited with a "Segmentation Fault(Core Dumped)"
>
>     Full output here:
>     https://drive.google.com/file/d/1IYnLcw-vkZ2UeH0rOZje4kbhzB_tYaKy/view?us...
Forwarded this to Btrfs dev, so we'll just have to be patient.

I strongly suggest some kind of backup of the data, and confirm it's
good. Since you were stressed about maybe losing it? Even two copies!
Not anything because it's Btrfs but because it's important to you. Bad
things happen all the time that have nothing to do with the file
system.

Because of the --init-csum-tree failure i'm a bit concerned that it's
possible the --repair step didn't completely repair things. So... just
keep important things backed up. Top priority. If you told me you hate
your data then I wouldn't worry about it.


>  Let me know the next steps ie. Do you need BTRFS logs to diagnose
> the
>  root cause of the problem ? 
Not sure. I've asked. If there's a request to, we'll run it through
gdb and see what happens.

Can you tell me what you get for:

free -m
btrfs fi us /

It'll give an idea what the memory requirement is for the
init-csum-tree. It really should not be a lot, it's less complicated
than the repair by a lot. Glad that worked though, phew!


-- 
Chris Murphy

[Sreyan Chakravarty]

On Tue, Jan 12, 2021 at 9:04 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  We should talk about a backup plan in another thread :D In my own
>  case, I never need them, but I have 1/2 dozen completely independent
>  backups, because I hate my data and I'm constantly trying to destroy
>  it. (This is turning into a really bad joke but hopefully it's weirdly
>  amusing. If you like your data you *definitely* should be backing up!
>  Doesn't matter what the file system is, or if it's on raid, or "safe"
>  in the cloud, etc.)
>  
Now that the filesystem is working should I take a partclone backup or
a dd backup ?

Partclone is small but since as you say the filesystem is not
completely fixed, my backup may be messed up.

While a dd backup keeps things the way they are.

>  Because of the --init-csum-tree failure i'm a bit concerned that
> it's
>  possible the --repair step didn't completely repair things. So... just
>  keep important things backed up. Top priority. If you told me you hate
>  your data then I wouldn't worry about it.
>  
I too believe that the repair is not complete. I say this because when
I started my system my swap file could not be activated and System
Logging Service failed to start.

I have disabled my swap file and will recreate it and the System
Logging Service is now working after a reboot.

>  Can you tell me what you get for:
>
>  free -m
>  btrfs fi us /
>
>  It'll give an idea what the memory requirement is for the
>  init-csum-tree. It really should not be a lot, it's less complicated
>  than the repair by a lot. Glad that worked though, phew!
>  
# free -m
              total        used        free      shared  buff/cache   available
Mem:           7852        1045        5079         393        1727        6163
Swap:          3925           0        3925

# sudo btrfs fi us /
Overall:
    Device size:                 930.00GiB
    Device allocated:            115.02GiB
    Device unallocated:          814.97GiB
    Device missing:                  0.00B
    Used:                         96.11GiB
    Free (estimated):            831.30GiB      (min: 423.81GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              224.05MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:111.01GiB, Used:94.68GiB (85.29%)
   /dev/mapper/luks-1136a62b-955b-4391-b9a4-b48ab11a862d         111.01GiB

Metadata,DUP: Size:2.00GiB, Used:733.83MiB (35.83%)
   /dev/mapper/luks-1136a62b-955b-4391-b9a4-b48ab11a862d           4.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/luks-1136a62b-955b-4391-b9a4-b48ab11a862d          16.00MiB

Unallocated:
   /dev/mapper/luks-1136a62b-955b-4391-b9a4-b48ab11a862d         814.97GiB
-- 
Regards,
Sreyan Chakravarty

[Sreyan Chakravarty]

On Tue, Jan 12, 2021 at 9:04 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  Because of the --init-csum-tree failure i'm a bit concerned that it's
>  possible the --repair step didn't completely repair things. So... just
>  keep important things backed up. Top priority. If you told me you hate
>  your data then I wouldn't worry about it.
>  
Just letting you know, your suspicions were right. The fsck was not
fully successful.

I have lost all my previous snapshots, one of them is an empty
directory and one of them, while I can see files in it, can't be used
for restoration as the system won't boot.

My system is now a ticking time bomb for another crash.

-- 
Regards,
Sreyan Chakravarty

[Chris Murphy]

On Wed, Jan 13, 2021 at 1:09 PM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  On Tue, Jan 12, 2021 at 9:04 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  >
>  > Because of the --init-csum-tree failure i'm a bit concerned that it's
>  > possible the --repair step didn't completely repair things. So... just
>  > keep important things backed up. Top priority. If you told me you hate
>  > your data then I wouldn't worry about it.
>  >
>
>  Just letting you know, your suspicions were right. The fsck was not
>  fully successful.
>
>  I have lost all my previous snapshots, one of them is an empty
>  directory and one of them, while I can see files in it, can't be used
>  for restoration as the system won't boot.
>
>  My system is now a ticking time bomb for another crash. 
At this point I suggest not booting from it. Boot the live USB, mount
the file system, and copy the important files from the "home"
subvolume (or any of its snapshots) onto some other drive. Using the
file manager is sufficient.


-- 
Chris Murphy

[Chris Murphy]

On Sun, Jan 3, 2021 at 9:11 AM Sreyan Chakravarty <sreyan32(a)gmail.com&gt; wrote:
>  I don't know what happened, but I just logged into my system and
>  deleted some unused files in my home directory. Just some directories.
>
>  Suddenly everything on my system became read only. While rebooting I
>  think I saw messages like:
>
>  "BTRFS Error" 

What kernel version was running when this happened? (I want to know
the kernel version running at the time of the first instance of a
problem.)

Thanks,

-- 
Chris Murphy

[Chris Murphy]

I've filed a bug for tracking.
https://bugzilla.redhat.com/show_bug.cgi?id=1912598

[Sreyan Chakravarty]

On Tue, Jan 5, 2021 at 2:58 AM Chris Murphy <lists(a)colorremedies.com&gt; wrote:
>  What kernel version was running when this happened? (I want to know
>  the kernel version running at the time of the first instance of a
>  problem.)
>  
The kernel version was:

5.9.14-200.fc33.x86_64

-- 
Regards,
Sreyan Chakravarty