RawhideBlocker [was Re: QA group activities + goals discussion]

[James Laska]

On Fri, 2009-02-20 at 18:18 +0000, Mark McLoughlin wrote:
> On Mon, 2009-02-16 at 14:38 -0800, Adam Williamson wrote:
> 
> > 1. Increase participation in Rawhide: it provides a huge benefit in
> > terms of identifying issues and having them fixed quickly and early in
> > the cycle.
> 
> Agreed - rawhide has a bad rep, and lots of people tend to avoid it. It
> has been improving lately, but we should keep trying out new things to
> get it to the stage that anyone involved in Fedora development should be
> able to run rawhide.
> 
> Here's a snippet from an old email on time based release processes that
> still really resonates with me when thinking about rawhide:
> 
>   http://mail.gnome.org/archives/gnome-hackers/2002-June/msg00041.html
> 
>   The goal of the unstable branch is to be an exciting and 
>   forward-moving but USABLE BY TESTERS piece of software. This is just 
>   the "release early, release often" principle.
>   ...
>   The unstable branch must always be dogfood-quality. If testers can't
>   test it by using it daily, they can't make the jump. If the unstable
>   branch becomes too unstable, we can't release it on a reliable
>   schedule, so we have to start breaking the stable branch as a stopgap.
> 
> Here's a suggestion:
> 
>   1) Come up with a rough definition of what it means for rawhide to be 
>      considered "dogfoodable" - e.g. installs, boots, networking works, 
>      desktop and core apps usable, etc. etc.

This is a great idea!  Several folks helped flesh out what rawhide is
and what the exit criteria are for delivering that content (full details
at
https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10#Defining_GOOD).

We broke rawhide out into several groups:
     1. Rawhide has a package repo
     2. Rawhide as an installation repo
     3. Rawhide as a snapshot candidate
     4. Rawhide as a milestone (alpha, beta, preview) candidate

The work that Jesse and Will are doing with the autoqa
(http://git.fedorahosted.org/git/?p=autoqa.git;a=summary) project should
help address #1.

I plan to piggy-back on their efforts with the lab-in-a-box (a pool of
virt install test systems managed by cobbler/koan) to address #2.

>   2) Create a RawhideBlocker tracker bug - add anything to it that 
>      breaks something on the dogfoodable list

This is bug escalation by public shaming?  I'm mixed on this.  Not that
I think it's bad, but that it's solving a different problem.

We currently have a problem of 1) not having an automated rawhide health
check and 2) not having a well defined method for defect escalation of
rawhide blockers.  The current mileage for fedora defect escalation
seems to be in the form of blocker bugs (instead of priority+severity).
Are there other solutions?  Does this mechanism work for development?
What are the needs of the maintainers/developers for providing a work
queue?

I'd like to focus first on gathering and presenting data on the health
of rawhide.  Once we master that domain ... we can address prioritizing
the plethora [1] of issues we discover outside of the current F#Blocker
bug method.

>   3) Post details to fedora-devel-list of any new bug added to
>      RawhideBlocker, cc-ing the package owners
> 
>   4) Harass package owners, and encourage others to help, until the 
>      issue is resolved

While if we have time, and we enjoy harassing package owners ... we can
do this.  I subscribe to the report and present the data model of
testing.  QA should definitely work with development to help
diagnose/debug any issues.  

>   5) Keep the list small, so that "OMG, it's blocking rawhide" is a
>      real incentive for people to sort problems out. If the list grows 
>      too long, consider lowering the dogfoodable bar.

Thanks,
James

[1] One of my favorites ... http://www.imdb.com/title/tt0092086/quotes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20090224/c678b6b2/attachment.bin