Installation validation matrix revision

[Adam Williamson]

Hey, folks!

I spent some time today fiddling around with the installation validation
matrix. I haven't applied the changes to Beta TC1 to give us some time
to review/tweak them, but they're in the template:

https://fedoraproject.org/wiki/Template:Installation_test_matrix

so, I did a few things:

* Split several test groups out from the big ugly "Miscellaneous" table
into their own tables
* Moved a few test cases from "Miscellaneous" into the storage tables
* Dropped three duplicated tests
* Cleaned up the table formatting:
** We don't need the ugly nested tables to have collapsible tables with
nice header lines, you can do it within a single table like this
** Dropped all the icky hard-coded column widths, let's just let
mediawiki sort it out
** Made the tables span the full page width (more space!)
** Renamed "Release Level" to "Milestone" (it's shorter, and it's the
term we've been standardizing on across the docs)
** Dropped all remaining "test area" columns (not needed with enough
sub-tables)
* Moved the instructions and notes we have for a few of the tables
inside the tables themselves
* Dropped the sub-sections from the matrix, instead you can wrap table
titles in <h4></h4> and they show up in the ToC (trick I found in the
Mediawiki docs)
* Tried to give each of the zillion tables we now have a color, it's not
the prettiest - anyone with a better eye than me can find the HTML color
list at http://www.w3schools.com/tags/ref_color_tryit.asp?color=White
and go nuts.

Where I created new tables I tried to be strategic about the
'environments', because we clearly have just too many tests now to run
them all in every possible environment. So I tried to reduce the
environments where possible without hurting test coverage too much. If
anyone thinks it simplifies things too much in terms of arch coverage,
yell.

Thoughts: wow, the page is getting long. The separate tables make it a
bit longer, but the table formatting improvements and loss of section
heads make it a bit shorter, so the changes are kind of a wash, but
that's *really* a lot of tests on one page.

I think it would be feasible to split it into three: something like
sanity tests ("Image sanity tests", "Default boot and install", "ARM
disk images", "Cloud images", "PXE boot tests", "USB stick tests",
"Virtualization tests"), installation storage tests (all the storage
stuff) and installer functionality tests (the rest). I actually used
this split, more or less, to come up with the color scheme (each of
those 'areas' uses variations on one color). Does that seem like a good
idea? We'd have more results pages per compose, but each one would be
shorter. With relval the creation of the results pages is just as easy
either way (doing it by hand, it gets tedious if there are too many).

Thoughts #2: there's a little bit of instruction text at the top of the
page:

"Please click [show] in each table to view the tests of each media
installation, and click [edit] to post your test results using the
syntax in Key Section. "

which suggests we initially meant the collapsible tables to be
*collapsed* by default. Does anyone remember if we ever did that? Does
anyone think it might be a good idea? (I'm also thinking of collapsing
the page ToC by default, because it sure takes up a lot of vertical
space).

Feedback and improvements welcome! Thanks :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net