Snapshotting for rollback after updates was[ Re: Drawing lessons from fatal SELinux bug #1054350]

[Simo Sorce]

On Sat, 2014-01-25 at 17:46 +0100, Tomasz Torcz wrote:
> On Fri, Jan 24, 2014 at 03:10:04PM -0700, Chris Murphy wrote:
> > 
> > On Jan 24, 2014, at 11:19 AM, Kevin Fenzi <kevin at scrye.com> wrote:
> > 
> > > On Fri, 24 Jan 2014 09:41:13 -0800
> > > Adam Williamson <awilliam at redhat.com> wrote:
> > > 
> > >> AIUI there is/was a long-term plan to integrate this as core
> > >> functionality using btrfs snapshots - in fact that was one of the
> > >> major attractions of the idea of switching to btrfs-by-default in the
> > >> first place. I believe those involved didn't think the LVM-based
> > >> implementation was clean/robust enough to use by default, but a
> > >> btrfs-based implementation would be. Do correct me if I'm wrong.
> > > 
> > > I don't think snapshots are a partcularly good solution, unless there's
> > > some way to only roll back the rpm/yum transaction without also rolling
> > > back unrelated changes. 
> > 
> > 
> > If there is a directory that contains update and non-update related file
> > changes, that's a problem. If there's segmentation, then this can be done.  
> > 
> > Clearly /home needs to be separate (it's OK to take a snapshot but just don't
> > use it by default in a rollback) or we lose changes in /home in a rollback from
> > the time of the snapshot to the time of the decision to rollback.  
> > 
> > Another possible case it's /etc/ where the either a package or the user could
> > make changes during the update. Btrfs allows per file snapshots with cp
> > --reflink so there might be a way to carve the snapshot with a scalpel but I
> > prefer doing it with subvolume granularity. Plus that granularity translates to
> > LVM.  
> 
>   Note that this situation is perfectly handled by Offline Updates.
> After reboot, there aren't collateral changes to filesystem, only upgrade-related
> ones. So if there's a need for revert, the previous state is clearly defined.

Sorry, but this is simply not true.

I would really like to DISABUSE people of the notion that automated (or
not) rollbacks can be easily done in bulk, by the magic wand of file
system snapshots.

The ONLY way to do that is if you do not care at all about user's data
and simply accept that a rollback will also remove user data.

The reason is simple: lot's of software *changes* data as part of its
normal functioning, including and often in rollback-incompatible ways.

You cannot assume that upgrading a program that uses a database X from
version A to B can still work if you keep database X unchanged and then
rollback from B to A. Lot of applications apply changes to database at
upgrade time, either in the rpm scriplets or automatically as soon as a
new version binary is run.

It is basically impossible to find applications that handle the case
where you downgrade, in any more graceful way than punting and failing
to start in the *good* case. In the bad case they start and trash the
database.

And by database, do not think SQL/NOSQL engines only, it can be any
simple dataset in a file, including configuration files in user's homes.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York