Draft: compendium image validation testing matrix

[Adam Williamson]

On Thu, 2012-08-09 at 05:49 -0400, Kamil Paral wrote:

> > QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will
> > simply check that the compendium images themselves boot
> 
> I don't understand what this means.
> 
> To clarify, multidvd structure is this:
> -> default DVD boot (auto-selects architecture)
> -> select arch manually -> i386
>                         -> x86_64
> -> basic video -> both 2 options
> -> memory test
> -> local drive boot
> 
> So you probably mean the "default DVD boot" here, right?

No. I simply mean 'do you get to the sub-image selection menu'.

> > when written
> > to
> > a physical dual-layer DVD and booted as normal.
> 
> How important is here to really burn it? Is ISO VM booting regarded as
> a reliable test or not? (Note: We don't burn classic Fedora images
> when testing either.)

I would want to, yes. I do burn classic images when testing; the tests
https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Dvd and
https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot specifically
require writing the images to media. (They currently refer to USB as
well as optical media, but I'm going to propose changing that, because
we have specific tests for USB now; those tests are intended to cover
precisely the case of actually booting the images from the optical media
to which they were originally intended to be written).

_In theory_ booting a VM which is treating the image file as an optical
disc always ought to be good enough for testing, but we all know how
well theory works when it comes to QA =)

> >  We _may_ also want to
> > test booting them when written to USB using one specific method.
> 
> I don't think so. This is just to be pressed to DVDs. If Red Hat ever
> starts to give away USB sticks, we can reconsider. Until then, it's
> redundant work.

Right, that was the scenario I was considering; if there aren't any
immediate plans to start distributing USB sticks, we can skip it.

> "auto_boot" is when you select the default option and correct
> architecture is chosen for you. To test this properly, you need both
> i386 and x86_64 machines (or VMs, if accepted).

For the sub-image boot testing I think using a VM is okay. I hadn't
considered the automatic arch selection wrinkle, I'll think up a way to
adjust for that - thanks.

> I am wondering whether we need installation tests at all.
> Theoretically the ISOs are unchanged so there should be no difference
> at all. If they start to boot, it should be fine. But according to
> Murphy's Law...

Exactly. _Theoretically_, you're correct. ;)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net