The heroes of Fedora 18 Final testing - wiki matrices

[Adam Williamson]

On Mon, 2013-01-14 at 14:25 +0100, Michael Schwendt wrote:

> Has anyone ever before retrieved and refined statistics from bugzilla?

Well, sure. Technically it's not difficult to do. What can be quite
difficult is designing a query that actually makes sense and gives you
useful data. What would we want to take out of Bugzilla in this case?
All bugs filed against Fedora 18? All bugs filed against anaconda? Bugs
that got fixed? Bugs that were blockers or NTH? Which of those is a
'useful' data set? It's a bit of a rabbit hole.

We had an effort for a while to try and draw some long-term statistics
from Bugzilla on an ongoing basis, related to bug triage work. That was
a pretty complex project and never quite entirely got off the ground.

This thread in general is a pretty good example of why it's very tricky
to try and 'statistically' list out and thank contributors to F/OSS
projects because there's just so many forms of contribution and it's
practically impossible to nail them all down into numbers. I'm always
hesitant to do this kind of thing for that reason. But I still applaud
Kamil's effort and I think before anyone worries that it's undervaluing
some contributions and overvaluing others, please just remember that
context: any such effort is necessarily going to be incomplete, so relax
and take it with a pinch of salt! No-one is being graded on their
performance, it was just an incomplete attempt to say thanks to at least
*some* of the people who've worked so hard on F18 validation.

> > The wiki was only supposed to be a a short term solution because we 
> > never found a testing system to that quite suited our needs.
> 
> It demands special efforts of the people who act in that area. In similar
> ways, Fedora Packaging requires special efforts of people who do packages.
> Hurdles everywhere. ;)

Well, I mean, just about any kind of 'planned testing' is going to
require 'special effort' of some kind. Fundamentally, we want to do
specific testing of specific composes. That means we need in some way to
denote 'this test is part of the validation testing for this compose'.
The production and communication of that information cannot possibly be
free. It is going to 'cost' someone, somewhere, some labor, at some
point. No whizzy improved TCMS will change that.

> I once had a look at what it would take to contribute "an official" test
> of Fedora Test Release media (e.g. my pet peeve scenario "DVD image on HDD"),
> but the Wiki pages were like a maze, very brief or for insiders only, new
> images were thrown out quickly (with no rsync access), with hardly any
> activity on test-list but probably heavy activity only on IRC. As I wrote,
> not everyone's cup of tea.

There's only a limited amount we can do about any of that. Again, we're
working with fundamental constraints: we have a very short time to do
image validation and a lot of validation to do (and, usually, a lot of
bugs to fix). The fast revision cycle of TC/RCs, the limited download
options, and the amount of test cases are all basically unavoidable
given the release cycle we're operating within. I find it odd that you'd
find the Wiki pages to be 'like a maze', though - there are simply three
primary pages for each TC/RC and all three are directly linked to in the
release announcement for each TC/RC. It's really not any more
complicated than that. Each TC/RC has Install, Base and Desktop pages,
with a bunch of test cases on each one. It doesn't seem like a hugely
complex setup and I don't think it can really be made any simpler within
the Wiki context.

The current setup works better for the workflow 'I want to help test
this build, let's go and see what tests need doing' than 'I want to do
this specific test on the current build, I'll do it and then figure out
where to put my result', admittedly. If anyone has ideas for improving
that it'd be great, but again I'm not sure it can be made too much
better within the Wiki and possibly it couldn't be improved even if we
replaced the wiki - even in a 'better system' we'd have several dozen
test cases and I'm not sure how much it's possible to 'automate' this
particular flow, where a tester has a specific test they like to do, but
doesn't know whether that test is one of the 'official' validation tests
and, if it is, which one it is. That seems like a rather difficult thing
to do intrinsically, to me.

Anyhow, for the record, I can solve this particular one for you with the
magic of the human brain: the closest 'official' test case to what
you're talking about is
https://fedoraproject.org/wiki/QA:Testcase_install_repository_Hard_drive_variation , and it's part of the 'General Tests' matrix on https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test . We probably need to resurrect the 'graphical' version of that test case, since newUI does let you graphically select an ISO as an install source now.

> If community people take the time to become familiar with all that,
> that's great.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net