The QA Problem

Stephen John Smoogen smooge at gmail.com
Wed Jan 3 23:22:10 UTC 2007


On 1/3/07, Will Woods <wwoods at redhat.com> wrote:
> On Wed, 2007-01-03 at 13:02 -0500, Greg Dekoenigsberg wrote:
> > On Wed, 3 Jan 2007, Bill Nottingham wrote:
> >
> > >>>> I'm wondering if we could provide solutions for both users: One update
> > >>>> channel that only gets security updates and important bugfixes while the
> > >>>> other is a bit more bold -- we for example could have firefox2 in the
> > >>>> bold channel for FC6 while shipping the latest firefox 1.5.x in the more
> > >>>> conservative channel.
> > >>>
> > >>> So, an idea like this:
> > >>>
> > >>> - starts to exponentially expand the QA problem
> > >>
> > >> Bill, I love you, but I'm going to have to call bullshit on this one.
> > >> The answer is not "avoid making QA harder", the answer is to SOLVE THE QA
> > >> PROBLEM.
> > >
> > > Sure, but which of these plans make more sense:
> > >
> > > 1)
> > >
> > > - Solve the QA problem for our repo configurations as they exist
> > > - Expand the QA solution to new, multiple, disparate and conflicting
> > >  repositories
> > >
> > > 2)
> > >
> > > - Expand into new, multiple, disparate and conflicting repositories
> > > - Then try to solve the QA problem
> > >
> > > Honestly, before we can do multiple experimental repositories of this,
> > > that, and the other, we need to get our *OWN* house in order.
> >
> > I agree with this completely -- if, and only if, we actually make a
> > concerted effort at fixing the QA problem.
> >
> > Which means articulating the QA problem, actually.
>
> Okay. "The QA Problem" is a big, fat, multi-headed monster. I mentally
> group QA into four tasks, ordered by priority:
>
> Task #1: Testing updates to stable releases
> Task #2: Testing new releases before they go out (rawhide, TestX, etc.)
> Task #3: Bug triage (this is the stuff we miss in #1 and #2)
> Task #4: Writing tools and docs to make the previous tasks easier
>
> Examining the first task - we don't currently have the *dedicated*
> manpower to *guarantee* that every package will be tested and approved
> by the QA group before it leaves updates-testing and goes into updates.
>
> I don't think we actually lack for *available* manpower. AFAIK there's
> plenty of people willing and able to install packages from
> updates-testing. The problems here are, I believe, a good summary of
> "The QA Problem".
>
> Problem #1: Testing currently requires a lot of skill, which reduces
> usable manpower.
>
> We lack how-to-test documents, so each tester must know how to set up
> and test any given package/feature on his own. New features don't
> necessarily come with much documentation (e.g. iSCSI).
>
> - Possible solution(s): More docs would lower the barrier to entry.
> Setting up an official Fedora QA group will help keep track of team
> strength and help everyone w uork together.
>
>

This would be great. Knowing when you have tested enough, and covered
the 'standard' way a package is meant to be used would help cut down
on the trying to hit every corner case.




>
> Problem #3: Motivation to report bugs / testresults is low.
>
> Even though it's easy to install packages from updates-testing,
> reporting problems with them is far harder than it needs to be. The same
> old bugs get reported over and over while some new problems don't get
> reported because the tester didn't want to spend 15 minutes looking for
> duplicate bugs, figuring out the appropriate component/version/etc.
>

Or when you do take the 15 minutes you don't find the duplicate
because it was described differently than you would have done so
(little solution to that). I sent an intern last month to report a bug
he found.. frankly bugzilla scared him when trying to pin something
down.. there are 14 states that an existing bug could be in (but only
really 3 maybe are used.) Putting stuff for RHEL and Fedora probably
causes confusion on the RH side also.

The biggest grief for me is when I find a bug and then go to search
and find that there are 30+ open NEW bugs on the package all the way
back from 1999 or so. This is the immediate "It is obvious that the
Developer refuses to use bugzilla or does not care about this package
so why post another bug that wont be looked at."

I would second the point system.. and I would like to add a wall of
shame. If a developer doesnt move a bug from NEW to ASSIGNED (LOOKED
AT?) within a week his mug is posted to Mugshot. At the end of the
quarter, their fate is voted on by members of the community (will it
be the dish rack? the cushions? the COMFY CHAIR? 2 hours of a board
meeting?)


> - Possible solutions: The updates system should have an easy way to
> report common problems with packages in updates-testing. A modified
> bug-buddy for Fedora would be very helpful here. These tools should also
> show the user commonly-reported bugs, and allow them to easily add a "me
> too" comment. A "Bugzilla RPG" or other ranking system (like GNOME's
> point system or Launchpad's Karma ranking) makes bug reporting and
> triage more interesting.
>
>
> So. Does that define The QA Problem? Or are there other issues I'm
> forgetting?
>
> -w
>
>
> _______________________________________________
> fedora-advisory-board mailing list
> fedora-advisory-board at redhat.com
> http://www.redhat.com/mailman/listinfo/fedora-advisory-board
>
>
>
>


-- 
Stephen J Smoogen. -- CSIRT/Linux System Administrator
How far that little candle throws his beams! So shines a good deed
in a naughty world. = Shakespeare. "The Merchant of Venice"




More information about the advisory-board mailing list