Increasing user testing and feedback (was Re: Bad-mouthing and hostility)

[Paul W. Frields]

On Thu, Mar 11, 2010 at 08:39:45AM -0500, Tom spot Callaway wrote:
> On 03/11/2010 12:29 AM, Ryan Rix wrote:
> > Can I put forth a suggestion to ease this process for contributors (and 
> > users!) to easily contribute to this testing, and make it as easily as 
> > possible for applications and libraries to have themselves "heard."
> > 
> > The creation of a Fedora-updates-testing mailing list.
> 
> First of all, thank you for proposing a solution. Now, for my
> constructive criticism:
> 
> * I'm not sure yet-another-mailing-list will go far enough to solve the
> problem. Even if it is placed prominently on the wiki or on the main
> fedoraproject website.
> 
> A mailing list is a flood of email that a user has to really really look
> at and keep up with, then proceed to a manual process. I'd argue that
> the majority of community members willing to watch a mailing list for
> updates that need testing are already doing so on the existing testing
> mailing list.

I agree -- mailing lists are a higher barrier tool than some sort of
GUI helper that's well integrated with the installation of Fedora.

> I posit an alternative suggestion:
> 
> * At firstboot, the installing user is asked if they would be willing to
> participate in user-driven updates testing. It is explained to them that
> in Fedora, updates to packages need to be tested by users, and that if
> they opt-in, they will be prompted from PackageKit about updates which
> need user testing. They can choose an update which needs testing from a
> list. Once an update is selected from the list, PackageKit will apply
> the update from updates-testing, then open a new window which contains:
> 
> * General update testing advice
> * Package specific update testing advice (this can live on the wiki)
> * A graphical selector for giving +1 (works great!), 0 (cannot determine
> state) or -1 (something didn't work)
> * A text box for inputting comments
> 
> The user then submits the results, which go into Bodhi. Once results are
> submitted, that update no longer appears in the PackageKit "updates
> which need testing" list.
> 
> If they report a 0 or -1, they are then prompted to back out the update
> by PackageKit (at their choice).
> 
> * On the backend, should a user choose to opt-in, they would be prompted
> to create a FAS account (or authenticate to an existing FAS account)
> (e.g. RHN handling in the past). They would _NOT_ be required to sign
> the Fedora CLA in order to participate in user-driven testing, as
> reported results from QA testing has already been determined to be
> non-copyrightable and thus, not considered a contribution.
> 
> Each user who opts-in to perform user-driven testing will have it
> flagged in their account. Each successful update testing submission will
> be minimally logged (package, target, timedate stamp) and a count
> incremented for unique update feedback performed.
> 
> In thanks for their testing, users will be informed (at firstboot) that
> they will receive Fedora swag, both in random drawings and at certain
> threshold points (give good feedback on N updates and get a Fedora
> Tester T-shirt).
> 
> Users can choose to opt out at any time.

It's a brilliant idea and also gets us toward one of the concepts
you've proposed in the past, which is a sort of "bonus point" system
for contributors where we can give additional rewards to people.  Some
people don't require those types of rewards to participate, others
will find it delightful.  Providing small incentives for people to
contribute is a good way to cement a growing relationship with the
Fedora Project.

> *****
> 
> Yes, it's more coding change than a mailing list, but I think it will
> gather the attention of a lot more potential testers. What it will also
> need is the efforts of existing QA contributors in the Fedora community
> to help draft the general update testing advice and work with
> maintainers (and upstreams) to create package specific update testing
> advice.
> 
> Note that this goes beyond the "things we can test in an automated
> fashion", I'm not looking for this as a mechanism for our users to
> mindlessly run "rpmlint" for us, but instead as a mechanism to perform
> human testing which cannot be easily automated. It also gives our users
> a low-impact way to be involved with Fedora.

Is this something that would make sense for a summer coding project?

-- 
Paul W. Frields                                http://paul.frields.org/
  gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233  5906 ACDB C937 BD11 3717
  http://redhat.com/   -  -  -  -   http://pfrields.fedorapeople.org/
          Where open source multiplies: http://opensource.com