Should a working fedup in Fedora N's stable repository be a release criterion for N+1?

[Adam Williamson]

On Wed, 2013-12-18 at 11:22 -0800, Andrew Lutomirski wrote:
> OK, so I'll re-ask my original question.  Fedora 20 was released with
> a broken update path from F19.  Should the release criteria be
> amended?  This particular issue would have been avoided if F19's fedup
> were frozen along with F20 and if all of the destined-for-stable
> versions were tested together as a release criterion.

It really isn't that simple. There are more working parts to this than
you realize.

fedup uses a custom initramfs. A kernel and initramfs that are used for
all fedup runs are built as part of repo compose. See:

https://dl.fedoraproject.org/pub/fedora/linux/releases/20/Fedora/x86_64/os/images/pxeboot/

When you run fedup, its first stage downloads all packages, and
downloads the vmlinuz and upgrade.img from there (well...not ALWAYS from
there, see below) and puts them in /boot. The 'System Upgrade'
bootloader menu entry uses that vmlinuz and upgrade.img (renamed); the
upgrade.img is an initramfs with a special upgrade environment in it,
implemented by the 'fedup' dracut module which is shipped in the
fedup-dracut package.

So the question of 'what version' you're using to do a fedup upgrade
isn't as simple as 'rpm -q fedup'. It is significant - sometimes very
significant - what version of the upgrade kernel and initramfs you're
using.

fedup-dracut has bugs and needs fixes just like any other bit of
software. As well as the relatively late bump of fedup to 0.8, we got a
relatively late bump of fedup-dracut to 0.8.

When we're testing fedup, there's three significant pairs of
kernel/initramfs to keep in mind: the one from the last milestone (so,
when we're testing Final, Beta), the one in the current day's Branched
compose, and the one in the current TC/RC (which will also be the one
that goes to release). Each of these could potentially be different.

fedup asks mirrormanager which location to get the kernel and initramfs
from, if you don't tell it with --instrepo . Usually, during a
pre-release period, it'll go and get the one from the last milestone if
you don't pass it --instrepo . The test case currently instructs you to
override it with the one from the current development branch.

So why am I bringing up this morass? Because it's significant. We *did*
test fedup 0.7 prior to release, and it *did* work. Multiple times, for
multiple testers. You seem to be making the assumption that we never
tested 0.7 because the test case says to use updates-testing and u-t
currently has 0.8, but in practice we did (I know I did, personally) -
0.8 wasn't sent to updates-testing until quite late, so tests before
that were with 0.7, and some of us do intentionally test without u-t,
whatever the test case says. But it matters which kernel and initramfs
you test with, too.

My current working theory for what happened here is that fedup 0.7 can
upgrade to F20 if you use an older kernel/initramfs built with
fedup-dracut 0.7, but not a newer one. (This isn't something you can
always assume - despite the paired versioning, it's not always expected
that the versions must match for things to work, I don't believe). And
by bad luck we never happened to test the 0.7/0.8 combination, or if we
did, not enough times to catch the pattern that it was entirely broken.

There is definitely stuff we can improve here, but it's a more complex
area than you realize, and not as simple to resolve as you suggest.
Particularly when we need to allow for late updates to both fedup and
fedup-dracut, as we have had to do for F18, F19 *and* F20.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net