The QA Problem

[Will Woods]

On Wed, 2007-01-03 at 13:02 -0500, Greg Dekoenigsberg wrote:
> On Wed, 3 Jan 2007, Bill Nottingham wrote:
> 
> >>>> I'm wondering if we could provide solutions for both users: One update
> >>>> channel that only gets security updates and important bugfixes while the
> >>>> other is a bit more bold -- we for example could have firefox2 in the
> >>>> bold channel for FC6 while shipping the latest firefox 1.5.x in the more
> >>>> conservative channel.
> >>>
> >>> So, an idea like this:
> >>>
> >>> - starts to exponentially expand the QA problem
> >>
> >> Bill, I love you, but I'm going to have to call bullshit on this one.
> >> The answer is not "avoid making QA harder", the answer is to SOLVE THE QA
> >> PROBLEM.
> >
> > Sure, but which of these plans make more sense:
> >
> > 1)
> >
> > - Solve the QA problem for our repo configurations as they exist
> > - Expand the QA solution to new, multiple, disparate and conflicting
> >  repositories
> >
> > 2)
> >
> > - Expand into new, multiple, disparate and conflicting repositories
> > - Then try to solve the QA problem
> >
> > Honestly, before we can do multiple experimental repositories of this,
> > that, and the other, we need to get our *OWN* house in order.
> 
> I agree with this completely -- if, and only if, we actually make a 
> concerted effort at fixing the QA problem.
> 
> Which means articulating the QA problem, actually.

Okay. "The QA Problem" is a big, fat, multi-headed monster. I mentally
group QA into four tasks, ordered by priority:

Task #1: Testing updates to stable releases
Task #2: Testing new releases before they go out (rawhide, TestX, etc.)
Task #3: Bug triage (this is the stuff we miss in #1 and #2)
Task #4: Writing tools and docs to make the previous tasks easier

Examining the first task - we don't currently have the *dedicated*
manpower to *guarantee* that every package will be tested and approved
by the QA group before it leaves updates-testing and goes into updates.

I don't think we actually lack for *available* manpower. AFAIK there's
plenty of people willing and able to install packages from
updates-testing. The problems here are, I believe, a good summary of
"The QA Problem".

Problem #1: Testing currently requires a lot of skill, which reduces
usable manpower.

We lack how-to-test documents, so each tester must know how to set up
and test any given package/feature on his own. New features don't
necessarily come with much documentation (e.g. iSCSI).

- Possible solution(s): More docs would lower the barrier to entry.
Setting up an official Fedora QA group will help keep track of team
strength and help everyone work together.


Problem #2: Testing requires a lot of time.

The simple tasks - e.g. "start httpd and make sure it can actually serve
files" - are not automated. Nobody knows if anyone else has already
tested a package, so there's duplicated work.

- Possible solutions: Automation (beaker) and infrastructure work
(fedora-updates-system) will help cut down on wasted time.


Problem #3: Motivation to report bugs / testresults is low.

Even though it's easy to install packages from updates-testing,
reporting problems with them is far harder than it needs to be. The same
old bugs get reported over and over while some new problems don't get
reported because the tester didn't want to spend 15 minutes looking for
duplicate bugs, figuring out the appropriate component/version/etc.

- Possible solutions: The updates system should have an easy way to
report common problems with packages in updates-testing. A modified
bug-buddy for Fedora would be very helpful here. These tools should also
show the user commonly-reported bugs, and allow them to easily add a "me
too" comment. A "Bugzilla RPG" or other ranking system (like GNOME's
point system or Launchpad's Karma ranking) makes bug reporting and
triage more interesting.


So. Does that define The QA Problem? Or are there other issues I'm
forgetting?

-w
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/advisory-board/attachments/20070103/8927811a/attachment.bin