F17 PM Test day late :) recap

[Adam Williamson]

On Tue, 2012-07-31 at 14:14 -0500, Michael Cronenworth wrote:
> Adam Williamson wrote:
> > Still, I agree it's not an optimal system, and if anyone has any
> > suggestions for alternatives we'd be glad to have them.
> 
> Sounds like you need a client app that will talk to a Fedora infra
> server and dump the data into a database. A wiki extension could be made
> to connect to that database and display the data.
> 
> I use something[1] like I described to display Bugzilla bugs into wiki
> pages.
> 
> [1] http://www.mediawiki.org/wiki/Extension:Bugzilla_Reports

I didn't really cover the problem space very well, sorry for that.

It's one of those things that gets bigger each time you look at it. In
traditional 'big grown-up QA' terms, what we're doing is using the Wiki
as a TCMS - test case management system. Real big grown-up QA tends to
base a lot of work around these, which are pretty complex projects that
aim to handle both the creation and management of test cases, and
tracking test results.

Test Days are really just one application, is the thing - at least
looking at things from the traditional perspective, it wouldn't make a
lot of sense to write a small, Test Day-specific client/server system,
because really Test Days are just events where we get lots of people
together to iterate intensively over a specific set of test cases. In
theory the test cases aren't 'special' - they can be run in other
contexts besides Test Days, which might want different things from the
results. TCMSes tend to wind up as big sophisticated beasts which can
track results in lots of different ways.

TCMSes unfortunately tend to be built with an assumption that they'll be
used by a small group of trusted and relatively savvy users: they'll be
used by a dedicated QA team in a 'closed' environment, essentially. So
they don't go out of their way to be easy to use and often aren't
architected in a way which would make them particularly easy to deploy
in an environment like Fedora, where we want to be open to engagement by
very casual testers and people who aren't necessarily trusted and
experienced QA members.

There are several open source TCMSes and similar systems that we could,
in theory, use for Fedora QA; we've evaluated several of them in the
past. The 'obvious' candidate for Fedora is Nitrate, which is Red Hat's
TCMS - https://fedoraproject.org/wiki/Nitrate . Red Hat QE, which is a
much more grown-up and 'proper' effort than Fedora QA (but also
basically a closed shop within RH), bases all its work around Nitrate.
Even Nitrate, though, has quite a lot of disadvantages compared to the
Wiki when it comes to the unique context of Fedora testing -
https://fedoraproject.org/wiki/Tcms_Comparison is an evaluation of
nitrate against the various features of 'wiki as a TCMS' which we've
found useful or which have become integrated into the way in which we do
testing for Fedora. Other open source TCMSes all have similar issues, or
more, in the context of Fedora QA. None of them would be a
straightforward drop-in replacement for the way we currently order
things, with clear advantages and only a few drawbacks - they'd all
represent a significant trade-off in capability and would take a lot of
engineering work and re-design of our processes. This doesn't mean we
shouldn't do it, of course, but it's not a straightforward call.

There are, of course, always other ways of looking at things. You could
take the point of view that a lot of successful open source code
resulted from someone starting out by doing something small in a
simplistic, unsophisticated way, and building from there. Perhaps the
fact that we've never found a drop-in TCMS that entirely makes sense for
Fedora is an indication that we really ought to grow our own, and making
a simple limited system specifically for Test Days, or some other
specific use Fedora makes of test cases, wouldn't be such a bad way to
start out: I'm certainly sympathetic to the view that sometimes it works
out better to do a good job on a small task in a relatively short time
with concrete results than to look at everything Fedora QA would
ultimately want from a TCMS and try to knock it all off on the first
try.

So that's a very longwinded way of saying - 'yes, with reservations' :).
I could certainly see value in an effort to write a relatively simple
system for handling test cases and results for Fedora test days,
tailored to the somewhat unusual requirements of collaborative testing
for an open project like Fedora. But I think if anyone's going to do it,
they should do it with all of the above in mind - bear in mind that
they're not exactly breaking new ground, but approaching a relatively
mature field from a slightly unique angle. Be aware that, ultimately,
what we really want is a more complex and flexible system around which
we could base all of Fedora's QA efforts; something that can _only_ ever
work for the Test Day format would be a bit of a dead end.

Hope that made sense and made things a bit clearer :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net