Draft: compendium image validation testing matrix

[Adam Williamson]

On Fri, 2012-08-10 at 03:23 -0400, Kamil Paral wrote:
> > > How important is here to really burn it? Is ISO VM booting regarded
> > > as
> > > a reliable test or not? (Note: We don't burn classic Fedora images
> > > when testing either.)
> > 
> > I would want to, yes. I do burn classic images when testing; the
> > tests
> > https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Dvd and
> > https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot
> > specifically
> > require writing the images to media.
> 
> Oops, and I was doing it wrong all the time.

Welp, robatino says he always read it the same way you did, so it's
definitely not clear. I think my reading actually reflects the initial
intent of the tests, but we could definitely clarify things.

So, this is how I think it ought to work:

* the boot_iso, boot_methods_dvd, and live_image_boot tests should
specifically test booting of an actual physical optical disc (we should
explain this more clearly, and remove the USB references now we have
dedicated USB tests)

* the 'USB stick' table tests should specifically test booting of
physical USB media (this is already clear, so that's fine)

* the current_KVM and previous_KVM tests (and Xen_Para_Virt, for that
matter) cover booting of the images attached to VMs (not really in as
much depth, but in practice I think it's fine, as we do lots of testing
in KVMs anyhow)

* all the other 'general tests' are tests of installer functionality,
and can be done from any boot method

Is this general scheme OK with everyone? If so, I'll propose a few test
revisions to make it much more clear. Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net