installation matrix environment adjustments

[Kamil Paral]

> > 1. Testcase_Anaconda_User_Interface_Basic_Video_Driver was changed to
> > a generic x86 test. In this case, it is problematic, because BIOS and
> > UEFI fallback video driver are completely different code paths. I have
> > a personal experience from that past when one of those worked and the
> > other didn't. We should distinguish BIOS and UEFI in this case. To fix
> > this, I see two viable options:
> 
> > a) move the test case to Miscellaneous section, so that the table in
> > User interface section doesn't get more complicated
> 
> > b) keep the test case in User interface section, but create a separate
> > table for it (right under the first table). This new table would have
> > x86 BIOS and x86 UEFI columns. Again, the purpose is to keep the
> > original table simple (avoid adding columns which are not required for
> > anything else).
> 
> The tests in that table are really covering somewhat different things,
> now I look at them. Graphical, and Text actually exercise two of
> anaconda's UIs (we seem to be missing a test for cmdline, unless one of
> the other test cases hits it in passing). The others are different ways
> of accessing those same UIs: the VNC tests are different ways to get at
> graphical, and serial_console is another way to get at text.
> 
> It makes sense for Graphical and Text to not be split between i686,
> x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
> have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
> other tests in deciding the right thing to do - what's the appropriate
> split for VNC and serial console?

For VNC, we require the vnc binary (and networking) to work correctly. Of course there can be arch-specific bugs. But that's true for almost anything, we're working with likelihood-to-break-in-just-a-single-arch estimates here. From that POV, I think the test case is very similar to NFS/NFSISO repo testcase - again, you need the NFS binaries to work correctly. And we have just a single Result column for that.
As I said, I don't feel well acquainted with ARM to guess which areas are more or less likely to work differently than on x86, and thus likely to contain arch-specific bugs. But I'd either keep current x86 & ARM separation, or unify that into a single test result, as with NFS.

Serial console is likely to be more different between x86 and ARM than VNC (I seem to remember some ARM specific serial console bugs from this cycle), so the current separation seems reasonably there.

Another question is whether we should keep the same approach for Text interface test case, or whether that can be united into a single result. Not sure here.

> 
> > 2. Testcase_Install_to_Previous_KVM and
> > Testcase_Install_to_Current_KVM differentiate BIOS and UEFI. I always
> > forget whether this is about the host or the guest (or both, or which
> > combination). But I think host differentiation doesn't make any
> > difference - libvirt should work the same, I assume. It makes sense to
> > differentiate the guests. Unfortunately I believe the UEFI mode for
> > libvirt/kvm [2] is not available under an open source license,
> > therefore it's not packaged and not trivial to use. Which means not
> > that many people probably use it. I wonder whether we really need to
> > keep this separated. I would gladly merge this together to a single
> > x86 column and split up once OVMF is available under a better license.
> > If we keep it separate, we should provide better instructions for the
> > UEFI part.
> 
> Well, the licensing issue is a bit subtle. OVMF itself is freely
> licensed (BSD or MIT IIRC). The bit that has a license issue is the FAT
> driver used by EDK II, on which OVMF is built. It has this extra term in
> its license:
> 
> Additional terms: In addition to the forgoing, redistribution and use
> of the code is conditioned upon the FAT 32 File System Driver and all
> derivative works thereof being used for and designed only to read
> and/or write to a file system that is directly managed by Intel's
> Extensible Firmware Initiative (EFI) Specification v. 1.0 and later
> and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI
> Specifications v.2.0 and later (together the "UEFI Specifications");
> only as necessary to emulate an implementation of the UEFI
> Specifications;
> and to create firmware, applications, utilities and/or drivers.
> 
> which obviously makes it non-free. I believe the *reason* is something
> to do with Microsoft patent(s) believed to be related to FAT, and I
> don't think it's prudent to say anything more specific than that.
> 
> There therefore isn't a legal issue with redistributing OVMF builds, and
> there is a commonly-used repository which contains them, updated daily:
> 
> https://www.kraxel.org/repos/
> 
> they just can't go into Fedora, because of Fedora's policies about
> non-freely-licensed software. It's pretty easy to set up a UEFI KVM
> using the packages from that repo, I have one here and just spent all
> morning using it to test UEFI multi-boot.
> 
> We certainly *can* block Fedora releases on things that have to do with
> software that's not part of Fedora's repositories, if we want to - viz
> the Windows or OS X boot criteria, for e.g. It'd be interesting to see,
> if a blocker was proposed, if we actually want to block on UEFI KVM
> guest operation, but I think at least it's worth having it in the
> matrix.

Thanks for explanation, I adjusted the test cases a bit:
https://fedoraproject.org/w/index.php?title=QA%3ATestcase_Install_to_Previous_KVM&diff=394349&oldid=374818
https://fedoraproject.org/w/index.php?title=QA%3ATestcase_Install_to_Current_KVM&diff=394350&oldid=374816


> > 3. Testcase_Anaconda_save_traceback_to_bugzilla,
> > Testcase_Anaconda_updates.img_via_URL,
> > Testcase_Anaconda_user_creation,
> > Testcase_Anaconda_updates.img_via_installation_source and
> > Testcase_Anaconda_traceback_debug_mode all differentiate BIOS and
> > UEFI. I think that's a waste of our energy. None of these testcases
> > should IMO be affected by firmware. We should make them generic x86
> > tests. We could go even further and make them completely
> > arch-independent (at least some of them), but I'm a bit careful around
> > ARM, so not completely sure about that.
> 
> ACK. Honestly I think this is just because they used to be in the
> miscellaneous tests table (some of them still are, I think) and that
> thing was freaking huge, and I kinda ran out of steam when I was doing
> the revision and just left some of 'em alone).

All of them are from  Miscellaneous section. Do you have a good idea how to make then x86 generic without making the tables too complex?

I can add a fourth 'x86' column, and gray out BIOS and UEFI columns for these discussed test cases. Which is usable, but not completely nice (the table source code grows a lot).
Or I can create a separate table in this section. I think this is nicer and we'll end up doing this in some sections in the future anyway. Do you see some disadvantage in it?

> 
> It's probably worth noting the meaning of a box being available or grey
> isn't *entirely* solidly nailed down. It's never been strictly the case
> that we require every white box for the appropriate release level to be
> filled - back when all we had was i386 and x86_64, we just had a box for
> each for every test, and it was kind of a case of 'fill in as many as we
> can'. Since we started getting ARM and UEFI it's got a bit more complex
> and I think to an extent we're starting to think 'if a box is there it
> really needs to be filled'.
> 
> In practice when creating and revising the tables (as the person who
> does it most of the time) I tend to be slightly fuzzy between 'the boxes
> should express the range of all environments for which we really need
> this test to happen' or 'the boxes should express the range of
> environments in which it's *possible* to do this test, even if we don't
> really need to do it in every one of them'. I think we've been trending
> towards the former interpretation, though, and that's probably not a bad
> idea, for the sake of clarity and reducing the sheer weight of tests.

Yes, this is exactly the reason why I try to reduce the number of boxes. Because even though in theory it would be nice to have fields for every possible combination and people would fill them according to the real priority (e.g. it's better to test 2 different test cases on a single arch, than a single test case on 2 arches), in practice often the opposite happens. We don't have this priority specified anywhere ("is it really important to test run test case on all arches or not?"), and even if we did, it would probably make the matrix horribly complicated. So people tend to do the opposite approach - I have read this test case, I remember it now, so let's run it for all architectures available, and only then continue to the next one. I do that often as well, even though it's not efficient from test coverage POV, because it's much easier especially when you're juggling with several tasks at the same time. And it's also easier than to try to decide which white boxes are mandatory and which ones are optional.

The second thing is that we don't usually have too much spare time for running extra test cases anyway, so it's better to optimize the required path as much as possible, and if people end up with spare time, usually they find more bugs when they're just playing with software and experimenting, than testing the same stuff on a bit different hardware.