Spins proposals

[Will Woods]

On Wed, 2008-04-02 at 09:06 -0800, Jeff Spaleta wrote:

>         * Fedora QA is tasked with testing all the release spins.
>         Nominally being part of that team, I don't think QA has the
>         resources (ie) enough people involved considering that we have
>         a six month release cycle and lots of bugs to deal with
> 
> How QA gets the testing done is QA's perview. But the tasking is most
> certaintly correctly scoped.  
> 
> I will be blunt.  I think we've had a significant problem with the QA
> process for multiple releases now.   We must find a way to organize
> volunteer labor better with regard to QA..generally.

And here we have a beautifully ironic illustration of the real problem
with QA:

It took me a day to respond to this email because I was too busy:
- running the QA meeting,
- triaging the F9Blocker/F9Tracker lists,
- retesting bugs that should be fixed,
- testing upgrades from F8,
- testing fixed builds for problems in rawhide, and
- filing bug reports for new problems found.

Get it? I'm too busy *doing* QA to *improve* QA. I'm too understaffed to
work on staffing up. Chicken, meet egg.

Now. If you'd care to expand on *your* thoughts on the problems with the
QA Process, I'm all ears.

> If QA wants to officially get on the record as saying they aren't
> going to take the responsibility of testing the composed images that
> RelEng has selected and working on as part of a release cycle... then
> by all means get on the record....as a group put your foot down in
> protest...so I can come in with my riot gear and tear gas and bust
> some skulls.

Seriously? That's how we're going to start the discussion? You're gonna
bust my head in unless I agree to take on *more* work when the entire
problem with QA is that I'm *already* too overworked to do anything?

Something is not quite right here.

> Or QA as a group try to get involved with the formation of the Spin
> SIG and make sure they are well prepared to do a significant amount of
> the QA as part of Kickstart Pool management.

Getting involved in the Spin SIG: yes. 
Kickstart Pool management: Also yes. 

Doing QA for every spin produced: Very no.

Here's how it's gonna happen: QA will provide the same plans,
guidelines, and tools that we use for Fedora now, and we'll provide
advice on using and improving them. But *the spins are responsible for
performing their own testing*.

QA already provides these things for the standard releases:
Release guidelines
http://fedoraproject.org/wiki/QA/ReleaseCriteria
A detailed installation test plan..
http://fedoraproject.org/wiki/QA/TestPlans/Fedora9Install
..composed of dozens of individual test cases..
http://fedoraproject.org/wiki/QA/TestCases
..along with templates for creating new test cases and test plans.
http://fedoraproject.org/wiki/QA/TestCases/TestCaseTemplate
http://fedoraproject.org/wiki/QA/TestPlans/TestPlanTemplate
And a test summary page where we track our testing results.
http://fedoraproject.org/wiki/QA/TestResults/TestSummaryTemplate

For each release, we triage bugs according to the ReleaseCriteria,
update the installation test cases to reflect new features, execute the
installation test cases, and track and report results.

The owners of each spin should be responsible for:
- Maintaining spin-specific ReleaseCriteria or TestCases, if needed
- Tracking spin-specific bugs,
- Executing test plans for their spin, and
- Tracking the results thereof.

Now - I'm not saying they have to do all the same work. The spin owners
can definitely *choose* to skip some testing, if they like. And I want
to avoid duplicate work - it's perfectly fine to assume that if
raid1-root works in the normal Fedora spin, it probably works in the
Electronics Lab spin as well. 

And, really, all I'm asking is that each spin spend some time doing QA
on their stuff. It helps the overall test effort by making more official
"QA People", spreading the test load over all the spins, and will create
new test cases. And hopefully this will drive an effort to improve the
tools for managing and tracking test plans and execution.

But this is the important thing: THE SPIN OWNERS ARE ULTIMATELY
RESPONSIBLE FOR GETTING THEIR SPIN TESTED. 

You can find your own QA guys, you can do the testing yourself,
whatever. But the current QA team is already far, far too busy to
actually sit down and test all the bits we produce *now*. We cannot -
and will not - test *new* spins that are dropped in our laps, so long as
that would take time away from testing Fedora proper.

This will probably remain the case until such time as we've managed to
build the robust automated testing infrastructure that Fedora deserves.

> The Spin SIG may very well find that its in their mutual best interest
> in getting involved more directly in the QA process..even before a
> spin has been selected for release.  And it very well maybe in QA's
> best interest to find a way to guide the Spin SIGs involvement in
> post-release testing as well.  But that's a conversation for the Spin
> SIG and QA to have at some point between now and the start of the F10
> release cycle.

Oh, definitely. I have a feeling the Spin SIG and QA and the Kickstart
Pool will all be very chummy in the near future. 

Until then, though, we've got a release to work on.

-w

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/advisory-board/attachments/20080403/f17c5981/attachment.bin