proposal: 'base' validation matrix

[James Laska]

On Fri, 2011-06-24 at 12:53 -0700, Adam Williamson wrote:
> Hey, folks. So, right now, as most of you know, we have two release
> validation 'matrices' - sets of tests we run and document on each
> candidate build of Fedora to check compliance with the release criteria.
> 
> A few of us have been working in ticket #151 -
> https://fedorahosted.org/fedora-qa/ticket/151 - to ensure the tests are
> in line with the criteria, particularly that we have tests to cover each
> criterion, and we've introduced a few new tests to cover criteria that
> weren't previously covered.
> 
> Some of these don't really fit easily into the category of
> 'installation' tests or 'desktop' tests, so I'm proposing we introduce a
> new, third matrix, which I'm currently thinking of as 'base'. There have
> already been three tests written in this proposed new category:
> 
> https://fedoraproject.org/wiki/QA:Testcase_base_firstboot
> https://fedoraproject.org/wiki/QA:Testcase_base_startup
> https://fedoraproject.org/wiki/QA:Testcase_base_system_logging
> 
> there will likely be more as we go through the Beta and Final criteria.
> I've thrown up a quick draft of how the matrix might look here:
> 
> https://fedoraproject.org/wiki/User:Adamwill/Draft_base_validation_matrix
> 
> Just wanted to make sure the group knows about this week, and ask for
> feedback on the test cases, the matrix and the overall plan here - do
> you agree a third matrix is the best way to organize things? 

Good thinking, an additional matrix seems appropriate.  I'm sure as we
start adding more tests, a theme or organizational scheme might surface.
Until then, just tossing them into this additional 'base' matrix seems
like a great idea!

Are any SOP updates needed to ensure this matrix is included when
setting up test runs in Fedora 16?

> Any ideas
> for naming, etc? Any comments on the test cases written so far? Thanks!

One thought, the 'base' matrix page looks a bit sparse.  I was about to
suggest that we add a standard header to the new 'base' matrix with some
basic instructions, but I wonder if something else might be more
appropriate.  When Andre and Hurry announce test runs, instead of
linking to each individual test matrix, we'd send a link to the
appropriate Category:Test_Results page (e.g.
https://fedoraproject.org/wiki/Category:Fedora_15_Final_RC_Test_Results).  

What if we move the stock "instructions" (see
https://fedoraproject.org/wiki/Test_Results:Fedora_15_Final_RC1_Install#Instructions) out of each individual page, and into the category page?  Or at the very least, make it an trancludable [1] wiki template {{Fedora_15_Test_Results_Instructions}}.

> (also thanks to Athmane Madjouj and Rui He for helping out with this
> work.)

Yes, thank you! :D

Thanks,
James

[1] http://www.mediawiki.org/wiki/Help:Transclusion
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20110630/b1edd754/attachment.bin