installation matrix environment adjustments

[Adam Williamson]

On Mon, 2014-11-10 at 05:13 -0500, Kamil Paral wrote:

> > It makes sense for Graphical and Text to not be split between i686,
> > x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
> > have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
> > other tests in deciding the right thing to do - what's the appropriate
> > split for VNC and serial console?
> 
> For VNC, we require the vnc binary (and networking) to work correctly.
> Of course there can be arch-specific bugs. But that's true for almost
> anything, we're working with likelihood-to-break-in-just-a-single-arch
> estimates here. From that POV, I think the test case is very similar
> to NFS/NFSISO repo testcase - again, you need the NFS binaries to work
> correctly. And we have just a single Result column for that.
> As I said, I don't feel well acquainted with ARM to guess which areas
> are more or less likely to work differently than on x86, and thus
> likely to contain arch-specific bugs. But I'd either keep current x86
> & ARM separation, or unify that into a single test result, as with
> NFS.
> 
> Serial console is likely to be more different between x86 and ARM than
> VNC (I seem to remember some ARM specific serial console bugs from
> this cycle), so the current separation seems reasonably there.
> 
> Another question is whether we should keep the same approach for Text
> interface test case, or whether that can be united into a single
> result. Not sure here.

If you want my guess, I guess I'd suggest we make VNC non-arch-specific,
keep serial console split ARM/x86, and make text non-interface-specific,
it seems like as long as you get a console at all, that ought to work.

> > ACK. Honestly I think this is just because they used to be in the
> > miscellaneous tests table (some of them still are, I think) and that
> > thing was freaking huge, and I kinda ran out of steam when I was doing
> > the revision and just left some of 'em alone).
> 
> All of them are from  Miscellaneous section. Do you have a good idea
> how to make then x86 generic without making the tables too complex?
> 
> I can add a fourth 'x86' column, and gray out BIOS and UEFI columns
> for these discussed test cases. Which is usable, but not completely
> nice (the table source code grows a lot).
> Or I can create a separate table in this section. I think this is
> nicer and we'll end up doing this in some sections in the future
> anyway. Do you see some disadvantage in it?

No, I agree, I was kinda working to split the misc section up. It fits
the design most nicely if you can come up with some sort of 'topic'
connection between the cases that live together in a table, but the
*functional* reason to have split tables is indeed to group tests with
similar result environments, and I don't like having too much greying
out of the rows either.

We should really limit the greying-out hack and use it only when a test
case is *clearly* part of a set with a lot of others but the
environments that otherwise apply to them all don't apply to it - i.e.
it should be a 'rare exception' case, we shouldn't be building complex
sets of result columns where each one might have only one or two white
boxes.

> Yes, this is exactly the reason why I try to reduce the number of
> boxes. Because even though in theory it would be nice to have fields
> for every possible combination and people would fill them according to
> the real priority (e.g. it's better to test 2 different test cases on
> a single arch, than a single test case on 2 arches), in practice often
> the opposite happens. We don't have this priority specified anywhere
> ("is it really important to test run test case on all arches or
> not?"), and even if we did, it would probably make the matrix horribly
> complicated. So people tend to do the opposite approach - I have read
> this test case, I remember it now, so let's run it for all
> architectures available, and only then continue to the next one. I do
> that often as well, even though it's not efficient from test coverage
> POV, because it's much easier especially when you're juggling with
> several tasks at the same time. And it's also easier than to try to
> decide which white boxes are mandatory and which ones are optional.
> 
> The second thing is that we don't usually have too much spare time for
> running extra test cases anyway, so it's better to optimize the
> required path as much as possible, and if people end up with spare
> time, usually they find more bugs when they're just playing with
> software and experimenting, than testing the same stuff on a bit
> different hardware.

Yeah, I more or less agree entirely with all of this. You can count me
generally in favour of any change which rationalizes the tables on the
basis of moving more towards 'white boxes express required testing'.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net