Announcing: template-driven validation test result page generation

[Adam Williamson]

Well, I set out this morning to move the rescue mode test case from
Alpha to Beta and somehow wound up doing this instead.

Opening a release validation test page 'template', copy 'n' pasting the
contents into a new page, and hand-editing bits until your eyes glaze
over? That's the old thing. The new thing is this:

{{subst:Validation_results|testtype=Base|release=21|milestone=Alpha|
compose=RC1}}

Create a new wiki page with just that as the contents, and you get a 21
Alpha RC1 Base validation results page. Neato, eh?

It works for any 'testtype' which has a matrix template. I ripped the
tables out of the old templates and made them into bare templates:

https://fedoraproject.org/wiki/Category:QA_test_matrix_templates

So you can pass 'Base', 'Cloud', 'Desktop', 'Installation' etc as the
testtype.

This system uses two other templates, along with the matrix templates:

https://fedoraproject.org/wiki/Template:Validation_results
https://fedoraproject.org/wiki/Template:Release_validation_instructions

When you substitute Validation_results into a page, it makes the page
*transclude* the instructions template and *substitute* the appropriate
matrix template. The upshot is that if you change
https://fedoraproject.org/wiki/Template:Release_validation_instructions
the changes will be reflected in any test results page (past, present
and future), but when you edit a matrix template, the changes will only
affect pages created later. If you need to add, remove or modify a test
case and the change should apply to the build currently under test,
you'll have to edit that page as well as the template page.

The Release_validation_instructions template is a meld of the
instructions from the old Install template and the newer Desktop
template (which was cloned for Base, Server, Cloud etc etc), with some
new improvements as well. Roshi, you might want to check the results are
good enough for Cloud, and carefully tweak the instructions template if
not (but bear in mind it has to remain generic, applicable to all test
types).

There's one unfortunate inconsistency here: the 'installation' results
pages have always been called Test_Results:Fedora_(whatever)_Install,
but the redirect page is Test_Results:Current_Installation_Test and the
category is Category:Installation_validation_testing . I went with the
majority vote and stuck with 'Installation' in this approach - you have
to set testtype=Installation , not testtype=Install . We can still call
the page _Install if we want, but it'd probably be more consistent to
call it _Installation going forward.

There's one known bug, which is that the created pages are put in the
category Fedora_(release)_(milestone)_(compose)_Test_Results . That
sounds right, but actually, it's meant to be _RC_Test_Results or
_TC_Test_Results.

Aside from that, the links and categories should all be 'magically'
correct, they should not need hand-editing after page creation.

There's a few things we can do there. Either we can just go with the
more-specific categories, and remember to add them all to the higher
level categories. Or we can drop the compose type entirely and just have
21_Alpha_Test_Results and 21_Beta_Test_Results and so on; I'm not sure
we really need to split TC and RC. Or we can wait for the infra folks to
add an extension to mediawiki which does string parsing, then I can
parse the compose parameter to get the correct result, but they say
that's probably going to happen for F22, not F21. Or we can simply
hand-edit the pages after creating them, it's not so arduous. :)

I have vague plans to write a little script using one or other of the
zillion python mediawiki interfaces out there that would make creating
the pages and handling the categories for a new TC/RC build a one-line
operation. If I don't get distracted by something else shiny in the
meantime!

I think I lined up all the categories and updated the relevant docs -
https://fedoraproject.org/wiki/QA:Release_validation_test_plan and
https://fedoraproject.org/wiki/QA/SOP_Release_Validation_Test_Event are
updated with the new procedure, and I did some related category cleanup
along the way. The new templates all have documentation.

Please let me know if you see any issues or possible improvements!
Thanks folks :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net