another upgrade, another disaster

[Adam Williamson]

On Tue, 2012-05-29 at 16:50 -0400, Tom Callaway wrote:
> On 05/29/2012 04:46 PM, Corey Richardson wrote:
> > I've heard nothing but bad things about preupgrade from lots of people,
> > and I've heard the developers either never hear about it, ignore it, or
> > don't care. I tried a preupgrade and it half-succeeded, I had to fix
> > the initramfs and a few other things afterwards though. IMO, it isn't
> > worth the pain.
> 
> We do test preupgrade as part of the pre-release testing. The problem is
> that people do all sorts of wacky things to their systems that cause it
> to not work well. I'm sure that QA would love to have more people
> testing preupgrade during the beta->RC window.
> 
> So, your statement that devs don't care isn't valid. It may still not be
> worth the pain, depending on how much you have installed outside of the
> Fedora Package universe.

Not entirely valid, but if we're being completely honest, there is only
one preupgrade maintainer - hughsie - and he has a couple of other
full-time jobs. So I think it's fair to say it doesn't get as much
active maintenance as it might. The preupgrade bug list is quite long -
127 bugs, http://bugz.fedoraproject.org/preupgrade - and I know for a
fact it contains quite a few valid reports which could be fixed
by...development.

Your broader point is of course also true. upgrading is a fundamentally
unsupportable operation: the amount of variables in play when it comes
to upgrading a system which a person has been using, normally, for some
time is so huge as to be probably incalculable. What we test in a
programmed way is that you can do a clean installation of F16, update it
to current packages, then run 'preupgrade' and wind up with an F17
system that boots. That is the extent of the preupgrade validation
testing. Any further testing relies on people trying it and filing bugs,
as Tom says.

The particular failure scenario involved here - 'user let
SOME_THIRD_PARTY_THING screw up RPM-managed files' - is obviously quite
a long way down the list of things we're going to care about. There has
to be a limit to what we can achieve with the resources available to a
project like Fedora. We don't have a 2,000 system test lab stashed away
somewhere with a few hundred people in it running upgrade tests in
various arbitrary scenarios, 24 hours a day. We just don't. All we can
do is ask people to test and file bugs prior to release, and I think we
certainly do enough of that...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net