The heroes of Fedora 18 Final testing - wiki matrices

[Adam Williamson]

On Mon, 2013-01-14 at 12:24 +0000, "Jóhann B. Guðmundsson" wrote:
> On 01/14/2013 12:16 PM, Michael Schwendt wrote:
> > On Mon, 14 Jan 2013 09:56:11 +0000, Jóhann B. Guðmundsson wrote:
> >
> >> On 01/14/2013 09:54 AM, Kamil Paral wrote:
> >>> I'd like to thank all who participated in Fedora 18 Final testing. I have published some statistics here:
> >>>
> >>> http://kparal.wordpress.com/2013/01/14/the-heroes-of-fedora-18-final-testing-wiki-matrices/
> >>>
> >>> Thanks again.
> >> Can you please remove any paid Red Hatter from that statistic since no
> >> community member can compete with that thanks.
> > That doesn't matter here at all. It's more important that actual tests are
> > performed, and painstakingly. It's great to see people working on this.
> >
> > A bit of background for these statistics might be enlightening, however.
> > I only remember the first Wiki "matrices", which are not everyone's cup of tea,
> > and certainly there are lots of community people reporting installer bugs
> > into bugzilla separately from this Wiki based activity.
> >
> 
> Which is another point this only collects what got written in the wiki 
> not what's reported in Bugzilla which excludes a largest group of our 
> reporters and some of our reporters might also be writing QA related 
> wiki pages and doing another QA related work. Dont those indviduals 
> deserve credit as well?
> 
> The wiki was only supposed to be a a short term solution because we 
> never found a testing system to that quite suited our needs.
> 
> I've ping Adam and Tim to see if it's on the plan to look ( again ) at 
> test systems and see if we find something that suits our need.
> 
> If anyone knows of any feel free to share it.

This comes up periodically, and of course in the RH context, Beaker is
always on the table. Tim and I keep half an eye on it. But so far as I
know, nothing major has changed here since Rui left RH: there still
isn't an off-the-shelf TCMS, whether Beaker or anything else, which
really matches Fedora's needs. Beaker is written to RHEL's needs, which
are pretty different from Fedora's. We evaluated several other TCMSes
the last time we took a concerted run at this, but none of them really
came out significantly ahead of the wiki-as-TCMS setup.

The last time Tim and I chatted about it, the conclusion we came to was
that a good dedicated TCMS could theoretically be significantly better
than the wiki-as-TCMS system, but given the idiosyncratic needs of a
project like Fedora QA, it's not very likely that _any_ 'off the shelf'
TCMS is likely to be what we need, now or in the future. Conventional
TCMSes tend to be written around a rather different set of needs: what a
closed group of professional QA testers needs to test a piece of
software. The requirements for a Fedora TCMS would be 'what a very open,
distributed and diverse group of mostly non-professional testers needs
to test ten thousand glued-together pieces of software'. It's a pretty
big gap.

So, we rather came to the conclusion we'd have to write something
ground-up or modify an existing project very heavily. Right now the RH
team doesn't have the resources to dedicate someone to doing this on a
paid basis, so if it's going to happen, it'll have to be a
volunteer/community effort, or it'll have to wait till RH has the
resources for it. Or, you know, maybe another distro will come up with
something awesome we can steal.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net