Technical Spec, better upgrade/rollback control

[Chris Murphy]

On Feb 21, 2014, at 5:21 PM, Colin Walters <walters at verbum.org> wrote:

> On Fri, Feb 21, 2014 at 4:51 PM, Chris Murphy <lists at colorremedies.com> wrote:
>> The snapper, yum plugin (and the manually performed user) convention is to take a snapshot that is then set aside for safe keeping and rollback; and it is the active parent tree that's upgraded. 
> 
> Yes.  The downside of this is that it doesn't solve the issue where updates actively break running processes, like say the default web browser of the "default offering".
> 
> I think it's *utterly insane* to ship an update system that does that.  I won't continue to attach my name to one that does - instead, I will get around at some point to researching and implementing *completely race-free* live upgrades only *where possible*.
> 
> I'm pretty sure I'll have such a subset of live upgrades on top of a known safe and tested basis working much faster than anyone can try to retrofit complete safety on top of yum/rpm + snapshot tool.

I just tried creating rw btrfs snapshots, mounting them and bind mounting /dev /proc et al in a chroot and doing a dnf update from a clean F20 to current, then fixing up the grub.cfg and fstab, rebooting, and it sorta mostly appears to work. [1][2] Maybe this is better/safer done in a systemd container.

> 
>> But I see no reason why an implementation couldn't update the snapshot instead of the active parent. If it fails, then clean up the snapshot. If it succeeds, reboot when convenient. Isn't this the OSTree convention? Create a new tree and update it, not the active tree?
> 
> Correct.  There is an important aspect here, which is that doing all of this safely needs a system which can do *coordinated* changes to both /boot and the root filesystem.

Yes. Snapper on openSUSE is doing this already on Btrfs. I'm not sure how it's dealt with on LVM thinp since /boot has to be outside LVM thinp because while GRUB groks conventional LVM, it doesn't get thinp yet. GRUB does understand /boot on Btrfs, but Fedora's grubby has a problem with it [1]. I've also been making /var/log a separate subvolume making it immune to rootfs snapshots and rollbacks.

>  It needs something aware of bootloaders, kernels, as well as higher level concepts such as the semantics of /etc and /var.  OSTree is such a system.  

Yes. I started some conversations about this on the grub-devel list and there seems a tentative way to get it to dynamically show snapshots rather than always having to modify grub.cfg. Also they really don't want /boot/grub to be snapshot/rolled back because it's not good for core.img to be out of sync with grub modules (not dissimilar to the kernel on /boot being disconnected from /lib/modules). 

Is there good chance of optimizing OSTree to use LVMthin and Btrfs snapshots instead of hardlinks, while still being in charge of the proper semantic enforcement?


> Projects like snapper and the yum fs-snapshot plugin are basically glued to the side of rpm (or dpkg) - they aren't in a position to enforce reliable semantics. 

Yes I also don't consider one kind of "rollback" since there can be different contexts. A user rolling back their /home doesn't mean rolling back any other user's, or the system. Conversely rolling back the system doesn't mean rolling back user /home or logs or some other things. 

> For example, even though the yum fs-snapshot plugin is linked into yum, it isn't ultimately in control of kernel installation - that's just a shell script forked off in a %posttrans.  OSTree, when installing a kernel (or making any other change at all), will cleanly handle ENOSPC and unwind, leaving your current system completely untouched.  You could then provision new space, and retry the upgrade.

Sounds pretty neat.


Chris Murphy



[1] grubby doesn't update grub.cfg when /boot is on a (nested) Btrfs subvolume.
    https://bugzilla.redhat.com/show_bug.cgi?id=864198

[2] A bunch of /usr/sbin binaries including sshd and bash have the wrong selinux label, unconfined_u:object_r:bin_t:s0, and so they fail to work unless I relabel or boot enforcing=0.