Rawhide validation testing matrix proposal

[Adam Williamson]

On Fri, 2014-04-04 at 07:00 +0000, "Jóhann B. Guðmundsson" wrote:
> On 04/04/2014 04:05 AM, Adam Williamson wrote:
> > Does this look OK to everyone? Thanks!
> 
> Is not better to have this separated/sorted by milestones ( 
> alpha/beta/final)

Well, the tests within each table are usually ordered that way (though
the 'general' table has gotten rather messy over time). Do you mean a
heavier separation than that? And do you mean specifically for this
Rawhide testing, or is it just a general suggestion that would apply to
the Alpha/Beta/Final matrices as well?

>  as well as aligned with Mike's test maps?

I'm not sure exactly how we could 'align' them - the point of the 'test
maps' effort is that there are really quite a lot of possible 'routes'
through the test matrix, and we can't really represent them all in a
flat '2D' view. It'd also not be compatible with the previous idea
(ordering by milestone), as often the tests you can do together / in
sequence are for different milestones.

Really I think this comes back to the matrices being kind of
unsatisfactory for the amount of tests we have at this point, and
needing a better system for finding tests and reporting results. The
'test maps' thing is working towards that, in the long run. It'd be nice
if we could combine that idea plus testcase_stats plus some kind of
result reporting/tracking system into a new thing that entirely
supersedes the matrix system.

> And this probably should also be tagged with what's applicable to each 
> workgroup so they know which test cases belong to them and they need to 
> be testing

Yeah, in principle we should do that, but I think it doesn't actually
apply to a lot of the tests on the installation matrix - I think most of
them apply to all the products. I'll have another look through and
check, though. I guess a lot of them don't apply to Cloud, so we could
note that.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net