Draft: compendium image validation testing matrix

[Adam Williamson]

On Thu, 2012-08-09 at 05:49 -0400, Kamil Paral wrote:
> > Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307
> > , I
> > think we need to add a process for validating the multi-live and
> > multi-install images that we've been building for the last few
> > releases.
> 
> It is ironic that before F17 release I have talked about this to cwickert (IIRC) and he assured me no QA help is needed and that nothing can go wrong. When they build the image, they just ensure everything boots and that's it. There's just a small syslinux glue, nothing else is changed. Well, it didn't go according to the plan, apparently.
> 
> > I've put together a draft matrix template for this validation here:
> > 
> > https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_validation_results_template
> > 
> > It's similar to the other validation matrices, and the changes are
> > mostly obvious. The key bit is the layout of the matrix and the test
> > cases themselves. I haven't actually written any of the specific test
> > cases yet, but I tried to think of the best way to organize things.
> > We
> > obviously need to do the basic sanity tests on each image, then we
> > need
> > to test that each image boots and the menu that lets you select which
> > sub-image to boot works properly, and also that each sub-image boots
> > and
> > installs properly. I think that set of tests ought to pretty much
> > cover
> > the intended functionality of the compendium images. When actually
> > writing the test cases, we might be able to come up with ways to take
> > advantage of some of the existing install/base/desktop validation
> > test
> > cases, but I haven't quite thought that far ahead yet.
> > 
> > I tried to organize the planned test cases as efficiently as
> > possible;
> > this is the best layout I could come up with, but someone might be
> > able
> > to suggest improvements. In case it's not obvious, here's roughly
> > what
> > each test case does:
> > 
> > QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums
> > -
> > these test cases already exist and can apply without modification to
> > the
> > compendium images, they simply check that they're within required
> > size
> > limits (dual-layer DVD size for these images) and the published
> > checksums match the images.
> > 
> > QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will
> > simply check that the compendium images themselves boot
> 
> I don't understand what this means.
> 
> To clarify, multidvd structure is this:
> -> default DVD boot (auto-selects architecture)
> -> select arch manually -> i386
>                         -> x86_64
> -> basic video -> both 2 options
> -> memory test
> -> local drive boot
> 
> So you probably mean the "default DVD boot" here, right? But which architecture? The result might differ on i386 and on x86_64 machine.
> 
> But multilive structure is this:
> -> default Live GNOME (auto-selects architecture)
> -> default Live KDE (auto-selects architecture)
> -> default Live LXDE (auto-selects architecture)
> -> default Live SoaS (auto-selects architecture)
> -> default Live XFCE (auto-selects architecture)
> -> select arch manually -> i386 -> Live GNOME
>                                 -> Live KDE
>                                 -> Live LXDE
>                                 -> Live SoaS
>                                 -> Live XFCE
>                         -> x86_64 -> Live GNOME
>                                   -> Live KDE
>                                   -> Live LXDE
>                                   -> Live SoaS
>                                   -> Live XFCE
> -> basic video -> all 10 options here
> -> memory test
> -> local drive boot
> 
> What does the test refer to for multilive?

> I have to say I'm somewhat confused from the current matrices structure. But maybe when the test cases are written it'll clearer.
> 
> My idea personally was:
> 
> MultiLive matrix                | GNOME | KDE | LXDE | SoaS | XFCE
> ------------------------------------------------------------------
> multilive_auto_boot (i386)      |
> multilive_auto_boot (x86_64)    |
> multilive_manual_boot (i386)    |
> multilive_manual_boot (x86_64)  |
> multilive_install (anyarch)     |
> 
> "auto_boot" is when you select the default option and correct architecture is chosen for you. To test this properly, you need both i386 and x86_64 machines (or VMs, if accepted).
> "manual_boot" is when you select the architecture manually from the submenu. A single x86_64 machine is sufficient to check both cases.
> 
> MultiDVD matrix                | result
> ---------------------------------------
> multidvd_auto_boot (i386)      |
> multidvd_auto_boot (x86_64)    |
> multidvd_manual_boot (i386)    |
> multidvd_manual_boot (x86_64)  |
> multidvd_install (anyarch)     |
> 
> I am wondering whether we need installation tests at all. Theoretically the ISOs are unchanged so there should be no difference at all. If they start to boot, it should be fine. But according to Murphy's Law...

OK, I revised the middle table somewhat, I tried to figure it out myself
but it turns out fairly similar to yours:

https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_validation_results_template#Boot_and_sub-image_selection

so 'multi_auto' would just be 'leave it, and check it does the right
thing', 'multi_manual' would be a test case that's something like 'try
every entry besides basic video / memtest / local boot and make sure
they respond appropriately', and the other three are obvious - we
probably don't really need to do those three on both arches, we could
split out a small table for those, I guess.

The multi_sub_boot test now becomes redundant, and I suppose we could
move sub_install up into this new big table; we can probably get away
without testing install of every DE, since they all use anaconda. The
case I can see where install might break on the compendium images is the
case where anaconda, for whatever reason, examines the actual medium
it's booting from. It _does_ do this, in some cases - like it tries to
except the USB stick it's booting from (if it is) from the target device
list. (This test may well not work on the compendium images, actually).
I can possibly imagine a case where anaconda goes to look what image
it's booting from, and gets confused if it's the compendium disc. But
that shouldn't vary between desktops.

This is a pretty tricky problem space and I'm not at all confident I'm
wrapping my head around it perfectly, more comments and improvements of
course welcome =) Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net