F20 -> F21 test matrix and scope

[Adam Williamson]

On Wed, 2014-09-17 at 20:55 -0600, Chris Murphy wrote:
> On Sep 17, 2014, at 1:14 AM, Adam Williamson <adamwill at fedoraproject.org> wrote:
> 
> > On Mon, 2014-09-15 at 15:50 -0600, Chris Murphy wrote:
> >> For reference:
> >> https://lists.fedoraproject.org/pipermail/desktop/2014-September/010657.html
> >> 
> >> I'm uncertain what this means for F20->F21 testing scope. For sure,
> >> fedup upgrade from Fedora 20 to Fedora 21, just like any other
> >> release. But do we need to test the 'yum swap' and 'yum install
> >> <product>-environment' conversion after doing the upgrade?
> >> 
> >> Presumably it'd be for final release, but there's some float between
> >> final go/no-go and actual release day.
> > 
> > Man, why did I not know 'yum swap'? I've been using 'yum shell' to do
> > that since forever.
> > 
> > It's an interesting question, I'd say we should certainly test it, and
> > if it's going to be promoted/mentioned in the Fedora 21 publicity it
> > should probably block final. Do you want to write up a test case? Thanks
> > for spotting this!
> 
> Time frame for the write up?

By Final, I guess?

> What's the metric for fail vs success? Ideally there'd be some way to
> compare clean F21 <product> *rpm, to
> F20->F21"standard"->F21"productized" *rpm, and they'd be identical.

I'm not sure that'd quite be the goal. I think the Products are
basically considered to be an 'at least' thing: to have the Product
installed you must have 'at least' the Product's defined package base,
but you *can* have anything else installed. So I'm not sure
upgraded/converted installs can be expected to be identical to cleanly
installed ones.

>  Maybe some fc20 rpms are expected to remain in the latter case, I'm
> not sure whether those are cleaned up.
> 
> What do you suppose the delta download is between standard and each of
> the three products?

I have absolutely no idea.

> I definitely think my best usage here is ways to cheat and be lazy by
> using snapshotting to avoid having to install F20 three times to do
> these four tests. My bet is that fedup would be the part most
> sensitive to problems related to esoteric partitioning/layouts. So if
> that step works, I don't expect 'yumswap' + 'yuminstall'
> "productizing" steps to give a hoot about layout, or even baremetal vs
> VM. Therefore it should be fine to predicate the test being done on a
> VM, and using qcow2 (or vbox) snapshots.

That certainly sounds like the most efficient way to do it, yeah, and it
seems perfectly valid as it's about packaging stuff, not hardware or
really the installer.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net