BTRFS/Rollback & Yum Snapshot Plugin

[Jorge Fábregas]

Hi everyone,

I'm playing around with btrfs (this time on a Fedora 20 VM).  I used the
default configuration for btrfs that Anaconda provides.  Right after
installation I did:

#btrfs subvolume list /
ID 257 gen 102 top level 5 path root
ID 258 gen 102 top level 5 path home

I then installed the yum-plugin-fs-snapshot package to try it out.  I
installed some updates and after it:

#btrfs subvolume list /
ID 257 gen 102 top level 5 path root
ID 258 gen 102 top level 5 path home
ID 278 gen 95 top level 257 path yum_20140130172422
ID 279 gen 97 top level 5 path home/yum_20140130172422

Fine.  yum created the snapshots.  I then wanted to rollback my system
(to the state prior to the updates) so I went ahead and changed the
rootflags=subvol... parameter in GRUB's kernel line in order to specify
the above snapshot for the root filesytem.  (I also changed /etc/fstab
to point /home to the snapshot subvolume).  After booting, everything
seemed right (I was running off the snapshot subvolumes).

Now, here's the thing.  I wanted to permanently run off these snapshots
so I wanted to delete the parent "subvolumes" for these snapshots (e.g.
"commit" the snapshot) by doing:

# mount -o subvolid=5 /dev/vda3 /mnt
# btrfs subvolume delete /mnt/root
Delete subvolume '/mnt/root'
ERROR: cannot delete '/mnt/root' - Device or resource busy

I believe this is happening because the snapshots are being created
WITHIN the subvolume they're snapshotting against.  That is, I can't
delete it because the currently running root snapshot,
yum_20140130172422, resides inside the "root" subvolume.   Is this
theory correct?  Can you modify the yum plugin so that it places its
snapshots within OTHER particular subvolume?

BTW,  is there anyone out there using this plugin with btrfs?

Thanks!
Jorge

p.d. I know snapshotting /home doesn't make sense at all for yum updates
but I followed it along...

[Chris Murphy]

On Jan 30, 2014, at 6:02 PM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:
>  
>  #btrfs subvolume list /
>  ID 257 gen 102 top level 5 path root
>  ID 258 gen 102 top level 5 path home
>  ID 278 gen 95 top level 257 path yum_20140130172422 
FYI note the top level is 257 which is root subvolume, therefore this entry is the same as
saying "top level 5 path root/yum_20140130172422"

>  
>  # mount -o subvolid=5 /dev/vda3 /mnt
>  # btrfs subvolume delete /mnt/root
>  Delete subvolume '/mnt/root'
>  ERROR: cannot delete '/mnt/root' - Device or resource busy 
>  I believe this is happening because the snapshots are being created
>  WITHIN the subvolume they're snapshotting against.   
Correct.

>  Can you modify the yum plugin so that it places its
>  snapshots within OTHER particular subvolume? 
I think a bug should be filed. It's a bug/RFE.

Another option is to look at snapper. It uses a subvolume at the top level called .snapper
that it puts snapshots into. It's in the fedora repo.


>  BTW,  is there anyone out there using this plugin with btrfs? 
I'm not, partly for the reason that I don't want snapshots available in the
normally mounted fs because I find it a bit confusing, and also because it does as you
say, it anchors root, boot, home subvolumes. Maybe I want to delete them if I get a
totally messed up update that's beyond repair or my interest.

Do you know if these are read only snapshots?

btrfs sub show /home/yum_20140130172422

>  p.d. I know snapshotting /home doesn't make sense at all for yum
> updates
>  but I followed it along… 
I'm not bothered by it but the purpose for snapshot-rollback of boot and root vs home
are different. Maybe for /home we'd want hourly snapshots. And we'd probably never
rollback /home for a bad system update. We'd keep existing home, and all of its
changes since updating. And then just rollback boot and root.

And on devel@ this was discussed that perhaps finer granularity is needed than what we
presently have. For example I have an additional subvolume called journald at the top
level which fstab mounts as:

UUID=xxx   /var/log/journal        btrfs   subvol=journald,compress=lzo,ssd 

This is because for now I've decided I don't want snapshots having their own
independent journal. I want a "master" journal kept up to date regardless of
what snapshot I boot. But it may turn out this is not a good idea since the journal
entries don't necessarily indicate what snapshot I've booted.



Chris Murphy

[Neal Becker]

Chris Murphy wrote:

>  
>  On Jan 30, 2014, at 6:02 PM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:
> > 
> > #btrfs subvolume list /
> > ID 257 gen 102 top level 5 path root
> > ID 258 gen 102 top level 5 path home
> > ID 278 gen 95 top level 257 path yum_20140130172422
>  
>  FYI note the top level is 257 which is root subvolume, therefore this entry is
>  the same as saying "top level 5 path root/yum_20140130172422"
>  
> > 
> > # mount -o subvolid=5 /dev/vda3 /mnt
> > # btrfs subvolume delete /mnt/root
> > Delete subvolume '/mnt/root'
> > ERROR: cannot delete '/mnt/root' - Device or resource busy
>  
> > I believe this is happening because the snapshots are being created
> > WITHIN the subvolume they're snapshotting against.
>  
>  Correct.
>  
> > Can you modify the yum plugin so that it places its
> > snapshots within OTHER particular subvolume?
>  
>  I think a bug should be filed. It's a bug/RFE.
>  
>  Another option is to look at snapper. It uses a subvolume at the top level
>  called .snapper that it puts snapshots into. It's in the fedora repo.
>  
>  
> > BTW,  is there anyone out there using this plugin with btrfs?
>  
>  I'm not, partly for the reason that I don't want snapshots available in the
>  normally mounted fs because I find it a bit confusing, and also because it
>  does as you say, it anchors root, boot, home subvolumes. Maybe I want to
>  delete them if I get a totally messed up update that's beyond repair or my
>  interest.
>  
>  Do you know if these are read only snapshots?
>  
>  btrfs sub show /home/yum_20140130172422
>  
> > p.d. I know snapshotting /home doesn't make sense at all for yum updates
> > but I followed it along…
>  
>  I'm not bothered by it but the purpose for snapshot-rollback of boot and root
>  vs home are different. Maybe for /home we'd want hourly snapshots. And we'd
>  probably never rollback /home for a bad system update. We'd keep existing
>  home, and all of its changes since updating. And then just rollback boot and
>  root.
>  
>  And on devel@ this was discussed that perhaps finer granularity is needed than
>  what we presently have. For example I have an additional subvolume called
>  journald at the top level which fstab mounts as:
>  
>  UUID=xxx   /var/log/journal        btrfs   subvol=journald,compress=lzo,ssd
>  
>  This is because for now I've decided I don't want snapshots having their own
>  independent journal. I want a "master" journal kept up to date regardless of
>  what snapshot I boot. But it may turn out this is not a good idea since the
>  journal entries don't necessarily indicate what snapshot I've booted.
>  
>  
>  
>  Chris Murphy
>  
I filed this BR: 
https://bugzilla.redhat.com/show_bug.cgi?id=1060241

[Chris Murphy]

On Jan 31, 2014, at 10:54 AM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:
>  
>  Ok, I might check that.  All these years I've been using Clonezilla to
>  do a "partition image" of my root filesystem before applying major
>  updates.  I have the ISO stored on my disk and GRUB2 is configured to
>  boot ("loopback") this image (this is great as I don't need to use a USB
>  stick or CD anymore).   Of course, using this btrfs snapshot
>  functionality with yum is by far way better & sophisticated than my
>  current method.  I wished it were more polished. That is, as soon as yum
>  snapshots your system, it should create the proper GRUB2 entry (pointing
>  to your snapshot volume so you can easily rollback) instead of you doing
>  it manually. 
Yes, there are challenges, much of which are the result of unanswered questions. For
example:

What should the layout be? (See opensuse's layout for an alternative to Fedora's.)
How discoverable should this be for mortal users vs experts? How far back should rootfs
snapshots be bootable? Should snapshots initially be read-only? Maybe. If a snapshot is
rw, and then booted, it's immediately changed, and if it's further changed by
being updated or modified by the user, is it really a snapshot of a file tree in a
particular state? No.

If snapshots are read-only, how do we boot them? What system changes are needed for rootfs
to always be ro in normal use, and only made rw when there's a system update? Or
alternatively, when doing the rollback, do we make a rw snapshot of the ro snapshot, and
boot the rw version? And then how do we clean up all of the ensuing snapshots?

Also, the fstab in all of the snapshots are wrong. An unmodified fstab in a snapshot
causes the parent subvolume of all snapshots to be mounted, not the snapshot. There are
multiple ways to solve this, it's not so much a technical problem as it is a
"determining best practices" by imagining many use cases, and figuring out the
liabilities of each potential solution.

Also, /boot quickly will contain updated kernels that can't boot old snapshots because
the snapshots only contain older kernel modules. So that implies /boot needs snapshotting.
Or we need limited snapshot/rollback to maybe just one older kernel. The main holdup for
/boot on Btrfs is an old grubby bug RHBZ# 864198. Also, the freedesktop bootloaderspec
calls for $BOOT being a non-snapshotable file system, while also being too small to
accumulate many kernel+initramfs files so that old snapshots can be booted.

For what it's worth, currently both the yum plugin and snapper presume the parent
subvols are the ones persistently used and modified; the snapshots are children and in
normal operation aren't ever used. If a rollback is needed, it's the
"child" snapshots that are used. This isn't the only way to do it. It's
equally valid to snapshot the parent, modify and use the child in normal operation, and
rollback to the parent. An advantage is that the parent subvol name and its fstab are
already properly in sync, the child snapshot subvolume(s) of course have new names and are
used in the modified fstab.

Note also that opensuse has a different layout for all of this than Fedora. They make the
default subvolume ID 5 (the top level of the Btrfs file system, the first subvolume, the
one that can't be deleted or named) the parent and mount it at /. And then create the
following subvolumes:

boot/grub2/x86_64-efi
home
opt
srv
tmp
usr/local
var/crash
var/log
var/opt
var/spool
var/tmp

So which is the more discoverable layout? Well it depends on one's point of view. The
expert who mounts a Fedora install on Btrfs doesn't see the linux FHS, and becomes
confused initially. They see what looks like two directories: root and home (on Fedora 19
they might also see boot if they opted to put /boot on Btrfs). Because the mount command
doesn't show the subvolume that's mounted, the assembly of the on-disk layout into
the mounted file system isn't obvious. It only becomes clear once understanding
subvolumes can be (almost completely) independently mounted, and looking at /etc/fstab
which shows the subvol= mount option.

Anyway, point is, even in the infant stage of Btrfs as a root file system, two distros
have two completely different layouts and snapshotting behaviors. I've argued that we
need some interdistro conversation on something like an FHS addenda that tackles some
standardization or best practices for how to organize such file systems and their
snapshots. It probably should also account for LVM thin provisioning, which enables
somewhat similar functionality. Or we'll just have to live with ensuing messiness.

And standardization is better for bootloader development too, so that we don't have
all of these different distro patches accounting for six ways to Sunday boot strategies.
More like 60 ways…

Not last, not least, is GRUB. It has several pieces, and relevant different behaviors on
BIOS and UEFI that I won't bring up here. However, there are some ideas how to make a
static grub.cfg dynamically produce entires for Btrfs snapshots on grub-devel@, and I
think that's more useful than always rewriting grub.cfg.

But yes, I wish it were more polished too.

>  
>  Of course, I'm just testing this on a VM.  My main system still runs
>  ext4 across the board.  I plan to start using btrfs gradually.  My
>  nearest plan is to switch my /backup partition (where my backup drive
>  is) to btrfs.  I plan to rsync my /home over there and then create the
>  snapshots there (and do some snapshot rotation of course).
>  
> > Do you know if these are read only snapshots?
> > 
> > btrfs sub show /home/yum_20140130172422
>  
>  They aren't.  I just checked now and could write to them. 
If they are read-only they have metadata at the Btrfs level that flags them such, which
can be seen with btrfs sub show <subvpath>. They aren't writable even by root.
Of course root can just snapshot the ro snapshot, which creates a rw snapshot that can be
modified. But then, if we wanted to audit all of this, we'd know that this happened
because each subvolume has a UUID. A snapshot is a subvolume, and each subvolume has a
UUID. A snapshot also references its parent's UUID.


Chris Murphy

[Chris Murphy]

On Feb 3, 2014, at 8:44 AM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:

>  On 01/30/2014 09:02 PM, Jorge Fábregas wrote:
> > Now, here's the thing.  I wanted to permanently run off these snapshots
> > so I wanted to delete the parent "subvolumes" for these snapshots (e.g.
> > "commit" the snapshot) by doing:
> > 
> > # mount -o subvolid=5 /dev/vda3 /mnt
> > # btrfs subvolume delete /mnt/root
> > Delete subvolume '/mnt/root'
> > ERROR: cannot delete '/mnt/root' - Device or resource busy
>  
>  Well, I found out you can rename the subvolumes and not only that;  you
>  can also rename them ("move" them) while they're being used.  This is
>  what I'm doing when I want to rollback my system to the previous state:
>  
>  1) Modify GRUB2:  Change "rootflags=subvol=root/yum_*" accordingly & reboot
It seems to me that there will be some error message in the log as the (stale) fstab tries
to do a rw mount of a different subvol on top of the one specified by rootflags=. Or maybe
it succeeds silently, I haven't recently tested this scenario.

>  
>  Since I used the default setup for btrfs, Anaconda created a /boot of
>  its own (ext4). (I presume a btrfs /boot is not ready yet). 
Yes, I mentioned this in the monolithic post I just sent. It's ready except for a
grubby bug.

>   This
>  creates some time paradox issues when I want to run "yum upgrade" again:
>  
>  - yum wants to install a kernel that I already have (the one I'm
>  currently running with)
>  - yum wants  to remove the oldest kernel (which it already removed on
>  previous existence) 
Yes I mention this in the long email also. I think /boot simply needs to be snapshot and
rolled back in parity with a rootfs snapshot: either by making /boot a directory within
the rootfs subvolume; or if boot and root are separate then they need to always be
snapshot and rolled back together. OpenSUSE does the former (/boot is a directory not a
subvolume), and since they make /boot/grub2/<arch> a subvolume, it's excluded
from the rootfs snapshot.

That brings up a point of possible confusion. A snapshot of a subvolume only snapshots
that FS tree, it does not recursively snapshot any subvolumes contained within
subvolumes.


>  All of this due to the fact that I restored the previous RPM
> database
>  when I rollbacked my system. 
Yes.


>  There is more to this yum-plugin/btrfs/rollback than meets the eye :(
The main issue you've run into is /boot is newer than rootfs, because it too
wasn't rolled back when rootfs was.

There probably should be some sane limitation (by default, of course users can modify
this) on system snapshot and rollbacks. I think it makes sense to have a max of five
system states: current, previous, previous 2, previous 3, original. The original would be
"as-installed" so that you could go back to a totally clean system, or, very
cool, you can create a derivative installation without installing ;-) Just fork off the
originally installed system, and create a whole new system.



Chris Murphy

[Chris Murphy]

On Feb 5, 2014, at 4:24 AM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:

>  On 02/03/2014 03:16 PM, Chris Murphy wrote:
> > It seems to me that there will be some error message in the log 
> > as the (stale) fstab tries to do a rw mount of a different subvol on
> > top of the one specified by rootflags=. Or maybe it succeeds silently,
> > I haven't recently tested this scenario.
>  
>  Yes, I wondered about this. I even questioned the purpose of specifying
>  the root filesystem in fstab since it is already present as a kernel
>  parameter in GRUB.  Is this the flow of events?
>  
>  - kernel mounts read-only the root filesystem specified as kernel
>  parameter in GRUB
>  - init/systemd reads fstab and remounts root-filesystem read-write
>  (along with specified mount options) 
Yes. It's a throwback from when file systems needed an fsck to even determine if they
had a problem, and then to fix them, because they didn't have journals.

man fsck.xfs

It's not really doing an fsck at all. The unattended repair from crash or unclean
mount is done at mount time by the XFS kernel code replaying the journal. If that
doesn't work, then user involvement is needed because chances are it's a hardware
problem: bad sector read errors, flaky controller cable, bad RAM, a LUN has died. In which
case there may be real file system damage, but that's what xfs_repair is for. First,
every physical device that makes up the file system needs to be present and working, and
then probably best to do a dry run through with xfs_repair -n. But even if you don't,
if you haven't first tried to mount XFS, xfs_repair will say sorry, do not collect
$200, go back and mount the file system so the journal can be replayed, unmount, and then
run me.

Btrfs is similar in concept but there are a lot more things to try before running btrfs
repair (which is what btrfsck points to these days). There isn't even an fsck.btrfs
placeholder. Before running btrfs repair, one should try mount option recovery. If
it's multiple devices, then try mount option degraded. And btrfs repair should be
preceded with btrfs check and posting those results to linux-btrfs@. The repair tool is
practically a last resort and sometimes still can make things worse.

I'm not sure where ext4 repair code in the kernel is at, but like XFS it has a journal
and is designed to mostly repair itself from crashes and unclean shutdowns simply by
mounting the file system. It may be there's still useful unattended repair code in
user space fsck.ext4.

So yeah, we probably could safely mount a Btrfs rootfs rw from the get go, and apply mount
options with rootflags= as a boot parameter, and forego an fstab entry. I've done this
quite a bit as a test so I know it works.

Chris Murphy

[Jorge Fábregas]

On 02/03/2014 03:16 PM, Chris Murphy wrote:
>  It seems to me that there will be some error message in the log 
>  as the (stale) fstab tries to do a rw mount of a different subvol on
>  top of the one specified by rootflags=. Or maybe it succeeds silently,
>  I haven't recently tested this scenario. 
Yes, I wondered about this. I even questioned the purpose of specifying
the root filesystem in fstab since it is already present as a kernel
parameter in GRUB.  Is this the flow of events?

- kernel mounts read-only the root filesystem specified as kernel
parameter in GRUB
- init/systemd reads fstab and remounts root-filesystem read-write
(along with specified mount options)

Thanks,
Jorge

[Jorge Fábregas]

On 02/03/2014 03:16 PM, Chris Murphy wrote:
>  Yes, there are challenges, much of which are the result of
> unanswered
>  questions. For example:
>  
>  What should the layout be? (See opensuse's layout for an alternative to
>  Fedora's.) How discoverable should this be for mortal users vs experts?
>  How far back should rootfs snapshots be bootable? Should snapshots
>  initially be read-only? Maybe. If a snapshot is rw, and then booted, it's
>  immediately changed, and if it's further changed by being updated or
>  modified by the user, is it really a snapshot of a file tree in a
>  particular state? No.
>  
>  If snapshots are read-only, how do we boot them? What system changes are
>  needed for rootfs to always be ro in normal use, and only made rw when
>  there's a system update? Or alternatively, when doing the rollback, do we
>  make a rw snapshot of the ro snapshot, and boot the rw version? And then
>  how do we clean up all of the ensuing snapshots? 
These are all excellent questions.  Some which I never imagined!

>  Also, the fstab in all of the snapshots are wrong. An unmodified
> fstab in
>  a snapshot causes the parent subvolume of all snapshots to be mounted,
>  not the snapshot. There are multiple ways to solve this, it's not so much
>  a technical problem as it is a "determining best practices" by imagining
>  many use cases, and figuring out the liabilities of each potential
>  solution. 
I haven't follow the openSUSE community for years.  Do you think they've
nailed most of these issues? or all distros on the same boat ?

My impression is that , until it gets to be the default filesystem on
most distros, these issues (of rolling back system via snapshots) won't
get much attention.  Do you agree?


>  Also, /boot quickly will contain updated kernels that can't boot
> old
>  snapshots because the snapshots only contain older kernel modules. So
>  that implies /boot needs snapshotting. Or we need limited
>  snapshot/rollback to maybe just one older kernel. The main holdup for
>  /boot on Btrfs is an old grubby bug RHBZ# 864198. Also, the freedesktop
>  bootloaderspec calls for $BOOT being a non-snapshotable file system,
>  while also being too small to accumulate many kernel+initramfs files so
>  that old snapshots can be booted. 
Interesting stuff about /boot.

>  For what it's worth, currently both the yum plugin and snapper
> presume
>  the parent subvols are the ones persistently used and modified; the
>  snapshots are children and in normal operation aren't ever used. If a
>  rollback is needed, it's the "child" snapshots that are used. This
> isn't
>  the only way to do it. It's equally valid to snapshot the parent, modify
>  and use the child in normal operation, and rollback to the parent. An
>  advantage is that the parent subvol name and its fstab are already
>  properly in sync, the child snapshot subvolume(s) of course have new
>  names and are used in the modified fstab. 
At this point I don't see any value for the yum plugin.  It simply does
the autoamtic snapshotting before applying updates.  It flies the
airplaine but doesn't land it.  That is, the advanced user who knows how
to: modify GRUB, fstab, specify subvolume, in order to rollback its
system, already knows how to create the snapshot in the first place.

I haven't played with snapper.  I don't know if it just a tool to
create/delete/manage/rotate snapshots or if it automates all of these
things in order to rollback your system.  I'm still learning the basics
of btrs so I wanted to avoid any front-end tools.


>  Note also that opensuse has a different layout for all of this than
>  Fedora. They make the default subvolume ID 5 (the top level of the Btrfs
>  file system, the first subvolume, the one that can't be deleted or named)
>  the parent and mount it at /. And then create the following subvolumes:
>  
>  boot/grub2/x86_64-efi
>  home
>  opt
>  srv
>  tmp
>  usr/local
>  var/crash
>  var/log
>  var/opt
>  var/spool
>  var/tmp 
Interesting.  I think Fedora's way looks more simple.  I guess there are
pros & cons for each.

>  So which is the more discoverable layout? Well it depends on
> one's point
>  of view. The expert who mounts a Fedora install on Btrfs doesn't see the
>  linux FHS, and becomes confused initially. They see what looks like two
>  directories: root and home (on Fedora 19 they might also see boot if they
>  opted to put /boot on Btrfs). Because the mount command doesn't show the
>  subvolume that's mounted, the assembly of the on-disk layout into the
>  mounted file system isn't obvious. It only becomes clear once
>  understanding subvolumes can be (almost completely) independently
>  mounted, and looking at /etc/fstab which shows the subvol= mount option.
>  
>  Anyway, point is, even in the infant stage of Btrfs as a root file
>  system, two distros have two completely different layouts and
>  snapshotting behaviors. I've argued that we need some interdistro
>  conversation on something like an FHS addenda that tackles some
>  standardization or best practices for how to organize such file systems
>  and their snapshots. It probably should also account for LVM thin
>  provisioning, which enables somewhat similar functionality. Or we'll just
>  have to live with ensuing messiness.
>  
>  And standardization is better for bootloader development too, so that we
>  don't have all of these different distro patches accounting for six ways
>  to Sunday boot strategies. More like 60 ways… 
I totally agree with all of the above.

>  Not last, not least, is GRUB. It has several pieces, and relevant
>  different behaviors on BIOS and UEFI that I won't bring up here. However,
>  there are some ideas how to make a static grub.cfg dynamically produce
>  entires for Btrfs snapshots on grub-devel@, and I think that's more
>  useful than always rewriting grub.cfg.
>  
>  But yes, I wish it were more polished too. 
Thanks a lot for your thoughtful insight about the current situation.
It's very much appreciated.

All the best,
Jorge

[Chris Murphy]

On Feb 4, 2014, at 12:23 AM, James Hogarth <james.hogarth(a)gmail.com&gt; wrote:
>  The other fun thing I encountered the other day is that cp
> --reflink=always still doesn't work across mount points even if they are part of the
> same btrfs pool and therefore capable of using COW between subvolumes.
>   
There are patches upstream for this. I'm running 3.14rc1 which has a large pile of
Btrfs patches, but reflink between mounted subvols is still not yet possible. The
workaround is to mount a subvolume they have in common and then it works.

Chris Murphy

[Jorge Fábregas]

On 01/30/2014 09:02 PM, Jorge Fábregas wrote:
>  Now, here's the thing.  I wanted to permanently run off these
> snapshots
>  so I wanted to delete the parent "subvolumes" for these snapshots (e.g.
>  "commit" the snapshot) by doing:
>  
>  # mount -o subvolid=5 /dev/vda3 /mnt
>  # btrfs subvolume delete /mnt/root
>  Delete subvolume '/mnt/root'
>  ERROR: cannot delete '/mnt/root' - Device or resource busy 
Well, I found out you can rename the subvolumes and not only that;  you
can also rename them ("move" them) while they're being used.  This is
what I'm doing when I want to rollback my system to the previous state:

1) Modify GRUB2:  Change "rootflags=subvol=root/yum_*" accordingly & reboot

2) Make sure you're running off the snapshot subvolume:
# cat /proc/selfmountinfo | grep btrfs

3) Move/Rename snapshot subvol & delete old root fs
# mount -o subvolid=5 /dev/vda3 /mnt
# mv /mnt/root/yum_20140203080009 /mnt
# btrfs subvolume delete /mnt/root
# mv /mnt/yum_20140203080009 /mnt/root

4) Modify GRUB2:  revert rootflags=subvol=root/yum_* to subvol=root

Since I used the default setup for btrfs, Anaconda created a /boot of
its own (ext4). (I presume a btrfs /boot is not ready yet).  This
creates some time paradox issues when I want to run "yum upgrade" again:

- yum wants to install a kernel that I already have (the one I'm
currently running with)
- yum wants  to remove the oldest kernel (which it already removed on
previous existence)

All of this due to the fact that I restored the previous RPM database
when I rollbacked my system.

There is more to this yum-plugin/btrfs/rollback than meets the eye :(

-- 
Jorge

[Jorge Fábregas]

On 01/31/2014 10:51 AM, Neal Becker wrote:
>  I filed this BR: 
>  https://bugzilla.redhat.com/show_bug.cgi?id=1060241 
Thanks!

-- 
Jorge

[Jorge Fábregas]

On 01/31/2014 12:29 AM, Chris Murphy wrote:

>  FYI note the top level is 257 which is root subvolume, therefore this
>
>  entry is the same as saying "top level 5 path root/yum_20140130172422" 
Indeed.  What a coincidence. The other day I was confused about what
"top level" meant and after searching for:  btrfs "top level is" ...I
found a linux-btrfs' thread where your explanation nailed it.  Not only
that, I've been searching for other btrfs issues and keep bumping into
your replies in that mailing list where you help others.  So, my hat off
to you.  Thanks for your time helping others in the community.


>  Another option is to look at snapper. It uses a subvolume at the top
>
>  level called .snapper that it puts snapshots into. It's in the fedora repo. 
Ok, I might check that.  All these years I've been using Clonezilla to
do a "partition image" of my root filesystem before applying major
updates.  I have the ISO stored on my disk and GRUB2 is configured to
boot ("loopback") this image (this is great as I don't need to use a USB
stick or CD anymore).   Of course, using this btrfs snapshot
functionality with yum is by far way better & sophisticated than my
current method.  I wished it were more polished. That is, as soon as yum
snapshots your system, it should create the proper GRUB2 entry (pointing
to your snapshot volume so you can easily rollback) instead of you doing
it manually.

Of course, I'm just testing this on a VM.  My main system still runs
ext4 across the board.  I plan to start using btrfs gradually.  My
nearest plan is to switch my /backup partition (where my backup drive
is) to btrfs.  I plan to rsync my /home over there and then create the
snapshots there (and do some snapshot rotation of course).

>  Do you know if these are read only snapshots?
>  
>  btrfs sub show /home/yum_20140130172422 
They aren't.  I just checked now and could write to them.

Thanks,
Jorge

[Chris Murphy]

On Feb 5, 2014, at 4:23 AM, Jorge Fábregas <jorge.fabregas(a)gmail.com&gt; wrote:
>  
>  I haven't follow the openSUSE community for years.  Do you think they've
>  nailed most of these issues? or all distros on the same boat ? 
I'd say it's all pretty much up in the air. openSUSE uses snapper by default, so
they take snapshots right before and right after an update. But it also has some
interactivity, so you can list things and say you want to rollback to a certain state/time
and it will do that for you.

>  
>  My impression is that , until it gets to be the default filesystem on
>  most distros, these issues (of rolling back system via snapshots) won't
>  get much attention.  Do you agree? 
Yes. It's going to be like a first time cattle rustle. It's not like you get to do
a trial run for practice. When you do one for real, some people simply are going to get
broken legs. The lack of urgency on many issues is due to it not being default. 

But everyone keeps at least two pristine backups, right? And they've done a full blown
restore procedure to confirm both backup strategies are really working (i.e. restorable),
right?


>  At this point I don't see any value for the yum plugin.  It
> simply does
>  the autoamtic snapshotting before applying updates.  It flies the
>  airplaine but doesn't land it.  That is, the advanced user who knows how
>  to: modify GRUB, fstab, specify subvolume, in order to rollback its
>  system, already knows how to create the snapshot in the first place. 
It's a good proof of concept. But it is a bit like a game of cards with just a dealer.
But that just means we need some decisions on other details, and find someone to implement
them!


>  I haven't played with snapper.  I don't know if it just a
> tool to
>  create/delete/manage/rotate snapshots or if it automates all of these
>  things in order to rollback your system.  I'm still learning the basics
>  of btrs so I wanted to avoid any front-end tools. 
It automates some of this, I'm not sure of the extent by which it "ages"
snapshots and cleans them up. It doesn't automatically do rollbacks, but if you tell
it to go back to a state then it uses those snapshots to do it. I don't know for sure,
but I think, by default it will rollback just the system, separate from home. That's
the default behavior we'd probably want.

And yes it's understandable to use the btrfs tools directly to gain familiarity.


>  
>  
> > Note also that opensuse has a different layout for all of this than
> > Fedora. They make the default subvolume ID 5 (the top level of the Btrfs
> > file system, the first subvolume, the one that can't be deleted or named)
> > the parent and mount it at /. And then create the following subvolumes:
> > 
> > boot/grub2/x86_64-efi
> > home
> > opt
> > srv
> > tmp
> > usr/local
> > var/crash
> > var/log
> > var/opt
> > var/spool
> > var/tmp
>  
>  Interesting.  I think Fedora's way looks more simple.  I guess there are
>  pros & cons for each. 
It's all about granularity of what should be rolled back and when. The Fedora method
is simple. If we rollback all of root that's pretty safe from a strictly system
rollback perspective, but it does mean you're also rolling back logs (which users may
expect, I'm not sure), and if you're running a server it means rolling back
/var/spool which actually may be a problem if mail is stored there.

While I think the FHS needs updating (it's 10 years old), the reality is there's
no way to organize the file system for a one size fits all snapshot-rollback policy. The
data loss [1] with a single policy would almost always be unacceptable. A bad update might
take an hour, day, or even a week to realize that a rollback is a good way to solve the
problem, in the meantime /home has accumulated valuable user data. The simplest practical
policy, is a dual policy: rollback /home, or rollback everything except /home. It only
gets more involved there, i.e. there isn't such a thing is one kind of rollback.
It'll depend on the circumstances.

[1] Not really lost, it's just not available. It's in a snapshot somewhere.



Chris Murphy

[James Hogarth]

On 3 Feb 2014 18:26, "Chris Murphy" <lists(a)colorremedies.com&gt; wrote:
>  Note also that opensuse has a different layout for all of this than
Fedora. They make the default subvolume ID 5 (the top level of the Btrfs
file system, the first subvolume, the one that can't be deleted or named)
the parent and mount it at /. And then create the following subvolumes:
>  boot/grub2/x86_64-efi
>  home
>  opt
>  srv
>  tmp
>  usr/local
>  var/crash
>  var/log
>  var/opt
>  var/spool
>  var/tmp 
The other fun thing I encountered the other day is that cp --reflink=always
still doesn't work across mount points even if they are part of the same
btrfs pool and therefore capable of using COW between subvolumes.