Increasing user testing and feedback (was Re: Bad-mouthing and hostility)

[Tom "spot" Callaway]

On 03/11/2010 12:29 AM, Ryan Rix wrote:
> Can I put forth a suggestion to ease this process for contributors (and 
> users!) to easily contribute to this testing, and make it as easily as 
> possible for applications and libraries to have themselves "heard."
> 
> The creation of a Fedora-updates-testing mailing list.

First of all, thank you for proposing a solution. Now, for my
constructive criticism:

* I'm not sure yet-another-mailing-list will go far enough to solve the
problem. Even if it is placed prominently on the wiki or on the main
fedoraproject website.

A mailing list is a flood of email that a user has to really really look
at and keep up with, then proceed to a manual process. I'd argue that
the majority of community members willing to watch a mailing list for
updates that need testing are already doing so on the existing testing
mailing list.

I posit an alternative suggestion:

* At firstboot, the installing user is asked if they would be willing to
participate in user-driven updates testing. It is explained to them that
in Fedora, updates to packages need to be tested by users, and that if
they opt-in, they will be prompted from PackageKit about updates which
need user testing. They can choose an update which needs testing from a
list. Once an update is selected from the list, PackageKit will apply
the update from updates-testing, then open a new window which contains:

* General update testing advice
* Package specific update testing advice (this can live on the wiki)
* A graphical selector for giving +1 (works great!), 0 (cannot determine
state) or -1 (something didn't work)
* A text box for inputting comments

The user then submits the results, which go into Bodhi. Once results are
submitted, that update no longer appears in the PackageKit "updates
which need testing" list.

If they report a 0 or -1, they are then prompted to back out the update
by PackageKit (at their choice).

* On the backend, should a user choose to opt-in, they would be prompted
to create a FAS account (or authenticate to an existing FAS account)
(e.g. RHN handling in the past). They would _NOT_ be required to sign
the Fedora CLA in order to participate in user-driven testing, as
reported results from QA testing has already been determined to be
non-copyrightable and thus, not considered a contribution.

Each user who opts-in to perform user-driven testing will have it
flagged in their account. Each successful update testing submission will
be minimally logged (package, target, timedate stamp) and a count
incremented for unique update feedback performed.

In thanks for their testing, users will be informed (at firstboot) that
they will receive Fedora swag, both in random drawings and at certain
threshold points (give good feedback on N updates and get a Fedora
Tester T-shirt).

Users can choose to opt out at any time.

*****

Yes, it's more coding change than a mailing list, but I think it will
gather the attention of a lot more potential testers. What it will also
need is the efforts of existing QA contributors in the Fedora community
to help draft the general update testing advice and work with
maintainers (and upstreams) to create package specific update testing
advice.

Note that this goes beyond the "things we can test in an automated
fashion", I'm not looking for this as a mechanism for our users to
mindlessly run "rpmlint" for us, but instead as a mechanism to perform
human testing which cannot be easily automated. It also gives our users
a low-impact way to be involved with Fedora.

~spot