Questions: - Does rpm-ostree keep an upgrade log of what it did during the upgrade? If so where is it? - Is there a config file to permit it to keep more than two trees at a time? - Is there a way to correlate ostree tree hashes and their release versions?
The problem:
I have a Fedora-Cloud_Atomic-x86_64-23.iso with a btrfs root and ext4 boot, in an otherwise conventional (flat, no subvolumes) layout.
After today's 'rpm-ostree upgrade' the new tree isn't in the /boot/efi/EFI/fedora/grub.cfg. The old tree is still there, but the path to that old tree has changed so even trying to boot that old tree fails.
I'm still in the middle of the autopsy but what appears to be the case:
1. The /boot/loader.0/grub.cfg contains the 23.29 and 23.34 trees. 2. This is a UEFI system which doesn't use that grub.cfg, it uses the one at /boot/efi/EFI/fedora/grub.cfg 3. That ESP grub.cfg wasn't updated, it's stale. I don't know why it wasn't updated. It contains the 23.29 and 23 tree entries, both of which look for ostree=/ostree/boot.1 4. The next problem is on rootfs there is no /ostree/boot.1, it's /ostree/boot.0 which is a symlink to /ostree/boot.0.1.
And hence the double fail.
So there are in effect two fails: A. The EFI System partition grub.cfg was not updated at all. B. The /ostree path changed in such a way that made it impossible for a stale grub.cfg to boot the old working tree that's still present.
I think there's a bug here somewhere, even if it might be user induced (exposed) due to a non-standard installation. But off hand I can't figure out how the non-standard install may have affected this upgrade. The 23 to 23.29 upgrade worked, except for root=UUID= was wrong, and that's due to... skip that part right now.
I think there's a bug if it's intended for the priot /ostree/boot* path to change in an upgrade. That seems fragile because if either that change or the grub.cfg change doesn't happen, then the rollback is going to fail too. I think that old (current prior to upgrade reboot) tree needs to stay the same no matter.
Non-standard install gory details: "ostree+grub2 will not auto-regenerate root=UUID=" https://bugzilla.redhat.com/show_bug.cgi?id=1290296
Gist is, that bug has an easy work around, and doesn't involve the ostree= parameter or a changed /ostree/boot* path at all which is why I don't think it's related. But I'm still suspicious because, well otherwise other people would be affected by now and I wouldn't be the first one posting this problem.
At the grub menu, I hit, c to get to a grub shell, and then use 'configfile' command pointed to /boot/loader.0/grub.cfg - it reads that configuration file instead of the one on /boot/efi/EFI/fedora/grub.cfg, and both 23.29 and 23.34 tree menu entries appear.
Both contain an error, however. They both use 'linux16' and 'initrd16' instead of 'linuxefi' and 'initrdefi'. Something is very confused about whether this is a BIOS or UEFI system.
If I change those commands to linuxefi and initrdefi, I can boot either tree.
There is something... maybe. My fstab looks like this:
UUID=908cb4df-410b-47e4-afb1-872255bd1244 /boot ext4 defaults 1 2 UUID=5956-63D8 /boot/efi vfat umask=0077,shortname=winnt,x-systemd.automount,noauto 0 2 UUID=8b0c4840-4fc7-4782-a4c0-25fec8a40dd4 / btrfs defaults 0 0
Normally grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg run manually causes /boot/efi to be mounted automatically in a split second. So, is rpm-ostree looking to see first if /boot/efi exists for any reason? What determines whether grub2-mkconfig -o is directed to /boot/efi/EFI/fedora, vs /boot/grub2? Thing is, there is no /boot/grub2/grub.cfg at all... neither of the correct locations got a grub.cfg. The correct grub.cfg (minus the wrong linux command) is in /boot/loader.0.
Wonky.
Chris Murphy
Successfully booted using 'configfile' and editing the grub.cfg to use linuxefi/initrdefi instead of linux16 and initrd16...
# bash -x grub2-mkconfig
http://fpaste.org/301941/30720114/
That's a bug. I just don't know whose bug it is. This is definitely a UEFI system, the CSM is not used (efibootmgr works, and Secure Boot is enabled). So something's got grub2-mkconfig awfully confused about what kind of firmware this system has. And then it also tells grub the -o destination path incorrectly.
Chris Murphy
rpm-ostree entry .conf
http://fpaste.org/301944/45030746/
What translates this file's linux/initrd into either linux16/initrd16 vs linuxefi/initrdefi?
OK really weird.
Tree 29.34 is deployed, and I just ran:
# rpm-ostree rollback
This writes out a correct grub.cfg (uses linuxefi/initrdefi) and it wrote the grub.cfg in the correct location (/boot/efi/EFI/fedora/grub.cfg). And this grub.cfg works regardless of which menu entry I pick in GRUB.
So the bug affected rpm-ostree upgrade, and possibly only the version of that command in the 23.29 tree.
OK part of my confusion is that 'grub2-mkconfig' does not work when called directly. Doing that results in a malformed grub.cfg.
What is the correct way, on atomic builds, to recreate the grub.cfg? How does rpm-ostree do this?
The closest I get to a command is 'ostree admin instutil grub2-generate' but this fails with ** ERROR:src/libostree/ostree-bootloader-grub2.c:154:_ostree_bootloader_grub2_generate_config: assertion failed: (grub2_boot_device_id != NULL) Aborted (core dumped)
I'm not sure what it wants.
Chris Murphy
----- Original Message -----
OK part of my confusion is that 'grub2-mkconfig' does not work when called directly. Doing that results in a malformed grub.cfg.
What is the correct way, on atomic builds, to recreate the grub.cfg? How does rpm-ostree do this?
Yeah, I've been confused by that as well. I haven't bothered investigating more, but it seems like running grub2-mkconfig on a fresh boot works, whereas calling it after some rpm-ostree operation such as upgrade/rebase will cause no output from 15_ostree.
The closest I get to a command is 'ostree admin instutil grub2-generate' but this fails with ** ERROR:src/libostree/ostree-bootloader-grub2.c:154:_ostree_bootloader_grub2_generate_config: assertion failed: (grub2_boot_device_id != NULL) Aborted (core dumped)
I'm not sure what it wants.
It's only meant to be called by the /etc/grub.d/15_ostree script, which sets up some env vars for it. That said, it should probably error out more gracefully.
Jonathan
On Wed, Dec 23, 2015 at 2:37 PM, Jonathan Lebon jlebon@redhat.com wrote:
----- Original Message -----
OK part of my confusion is that 'grub2-mkconfig' does not work when called directly. Doing that results in a malformed grub.cfg.
What is the correct way, on atomic builds, to recreate the grub.cfg? How does rpm-ostree do this?
Yeah, I've been confused by that as well. I haven't bothered investigating more, but it seems like running grub2-mkconfig on a fresh boot works, whereas calling it after some rpm-ostree operation such as upgrade/rebase will cause no output from 15_ostree.
For me, now on 23.39, even after fresh boot, and no matter where I direct -o to write the file, 15_ostree is empty. It's seems like it's not meant to be directly called. I've never had that command from user space produce a correct grub.cfg. But with one exception, within rpm-ostree updates or rollbacks, it produces correct grub.cfgs. So I think something else is calling that script, and also telling it where to put the grub.cfg (which goes in different locations depending on the firmware, because rabbits).
I've variably gotten no menu entry grub.cfgs, and ones with linux16/initrd16 instead of linuxefi/initrdefi and I can't tell why this flips around other than there's some kind of state change that doesn't happen when it's correctly called.
The closest I get to a command is 'ostree admin instutil grub2-generate' but this fails with ** ERROR:src/libostree/ostree-bootloader-grub2.c:154:_ostree_bootloader_grub2_generate_config: assertion failed: (grub2_boot_device_id != NULL) Aborted (core dumped)
I'm not sure what it wants.
It's only meant to be called by the /etc/grub.d/15_ostree script, which sets up some env vars for it. That said, it should probably error out more gracefully.
Well that sorta answers this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1293986
But on a normal system /etc/default/grub is consumed by grub2-mkconfig, but that's not true on atomic. I've made changes to that file and yet those changes aren't rolled into the grub.cfg. So OK, there's 'ostree admin instutil set-karg' but I run into this problem https://bugzilla.redhat.com/show_bug.cgi?id=1293987
So now I have no idea whether 'ostree admin instutil' is user domain or just meant as helpers for some other scripts. So I think we need to know what the deprecated and new knobs are.
This is calling grub2-mkconfig at line 340 https://git.gnome.org/browse/ostree/tree/src/libostree/ostree-bootloader-gru...
And line 153 says this must have been called from a wrapper script. I'm pretty much thinking grub2-mkconfig is not meant to be directly called by the user either, and is envisioned to only get called by e.g. ostree admin deploy/switch, or rpm-ostree rollback/upgrade, etc. That's fine, it's just not obvious what user space tools belong to the user.
Chris Murphy
-
Chris Murphy -
Jonathan Lebon