installation matrix environment adjustments

[Adam Williamson]

On Fri, 2014-11-07 at 13:02 -0500, Kamil Paral wrote:
> I have a proposal regarding installation matrix [1] related to
> environments. Adam did a truly stellar job in his latest results table
> restructuring, he eliminated a lot of duplicated testing and
> simplified the tables a lot (for example, there's no point in testing
> additional HTTP repos in anaconda on all three major architectures,
> because that functionality it architecture independent). We don't have
> limitless resources and must keep focused on areas where it's likely
> for a bug to occur, instead of testing everything. Here's a proposal
> of some last touches to installation tables:
> 
> 1. Testcase_Anaconda_User_Interface_Basic_Video_Driver was changed to
> a generic x86 test. In this case, it is problematic, because BIOS and
> UEFI fallback video driver are completely different code paths. I have
> a personal experience from that past when one of those worked and the
> other didn't. We should distinguish BIOS and UEFI in this case. To fix
> this, I see two viable options:

> a) move the test case to Miscellaneous section, so that the table in
> User interface section doesn't get more complicated

> b) keep the test case in User interface section, but create a separate
> table for it (right under the first table). This new table would have
> x86 BIOS and x86 UEFI columns. Again, the purpose is to keep the
> original table simple (avoid adding columns which are not required for
> anything else).

The tests in that table are really covering somewhat different things,
now I look at them. Graphical, and Text actually exercise two of
anaconda's UIs (we seem to be missing a test for cmdline, unless one of
the other test cases hits it in passing). The others are different ways
of accessing those same UIs: the VNC tests are different ways to get at
graphical, and serial_console is another way to get at text.

It makes sense for Graphical and Text to not be split between i686,
x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
other tests in deciding the right thing to do - what's the appropriate
split for VNC and serial console?

> 2. Testcase_Install_to_Previous_KVM and
> Testcase_Install_to_Current_KVM differentiate BIOS and UEFI. I always
> forget whether this is about the host or the guest (or both, or which
> combination). But I think host differentiation doesn't make any
> difference - libvirt should work the same, I assume. It makes sense to
> differentiate the guests. Unfortunately I believe the UEFI mode for
> libvirt/kvm [2] is not available under an open source license,
> therefore it's not packaged and not trivial to use. Which means not
> that many people probably use it. I wonder whether we really need to
> keep this separated. I would gladly merge this together to a single
> x86 column and split up once OVMF is available under a better license.
> If we keep it separate, we should provide better instructions for the
> UEFI part.

Well, the licensing issue is a bit subtle. OVMF itself is freely
licensed (BSD or MIT IIRC). The bit that has a license issue is the FAT
driver used by EDK II, on which OVMF is built. It has this extra term in
its license:

Additional terms: In addition to the forgoing, redistribution and use
of the code is conditioned upon the FAT 32 File System Driver and all
derivative works thereof being used for and designed only to read
and/or write to a file system that is directly managed by Intel's
Extensible Firmware Initiative (EFI) Specification v. 1.0 and later
and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI
Specifications v.2.0 and later (together the "UEFI Specifications");
only as necessary to emulate an implementation of the UEFI
Specifications;
and to create firmware, applications, utilities and/or drivers.

which obviously makes it non-free. I believe the *reason* is something
to do with Microsoft patent(s) believed to be related to FAT, and I
don't think it's prudent to say anything more specific than that.

There therefore isn't a legal issue with redistributing OVMF builds, and
there is a commonly-used repository which contains them, updated daily:

https://www.kraxel.org/repos/

they just can't go into Fedora, because of Fedora's policies about
non-freely-licensed software. It's pretty easy to set up a UEFI KVM
using the packages from that repo, I have one here and just spent all
morning using it to test UEFI multi-boot.

We certainly *can* block Fedora releases on things that have to do with
software that's not part of Fedora's repositories, if we want to - viz
the Windows or OS X boot criteria, for e.g. It'd be interesting to see,
if a blocker was proposed, if we actually want to block on UEFI KVM
guest operation, but I think at least it's worth having it in the
matrix.

> PS: I know nothing about Xen, which is also in the table, so I have no
> idea whether it's a similar story or not.

We have a commitment from Oracle to provide the testing for that
criterion - basically we agreed to have the criterion so long as they
make sure the testing gets done. Konrad Wilk has been doing that testing
ever since we added the criterion, and he usually does it promptly and
thoroughly, so we haven't really had any trouble with it.

> 3. Testcase_Anaconda_save_traceback_to_bugzilla,
> Testcase_Anaconda_updates.img_via_URL,
> Testcase_Anaconda_user_creation,
> Testcase_Anaconda_updates.img_via_installation_source and
> Testcase_Anaconda_traceback_debug_mode all differentiate BIOS and
> UEFI. I think that's a waste of our energy. None of these testcases
> should IMO be affected by firmware. We should make them generic x86
> tests. We could go even further and make them completely
> arch-independent (at least some of them), but I'm a bit careful around
> ARM, so not completely sure about that.

ACK. Honestly I think this is just because they used to be in the
miscellaneous tests table (some of them still are, I think) and that
thing was freaking huge, and I kinda ran out of steam when I was doing
the revision and just left some of 'em alone).

It's probably worth noting the meaning of a box being available or grey
isn't *entirely* solidly nailed down. It's never been strictly the case
that we require every white box for the appropriate release level to be
filled - back when all we had was i386 and x86_64, we just had a box for
each for every test, and it was kind of a case of 'fill in as many as we
can'. Since we started getting ARM and UEFI it's got a bit more complex
and I think to an extent we're starting to think 'if a box is there it
really needs to be filled'.

In practice when creating and revising the tables (as the person who
does it most of the time) I tend to be slightly fuzzy between 'the boxes
should express the range of all environments for which we really need
this test to happen' or 'the boxes should express the range of
environments in which it's *possible* to do this test, even if we don't
really need to do it in every one of them'. I think we've been trending
towards the former interpretation, though, and that's probably not a bad
idea, for the sake of clarity and reducing the sheer weight of tests.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net