old_testing_critpath notifications

Toshio Kuratomi a.badger at gmail.com
Wed Dec 1 23:53:04 UTC 2010


On Wed, Dec 01, 2010 at 02:17:32PM -0800, Adam Williamson wrote:
> 
> The concept of having a policy requiring updates to be tested before
> they're issued is really no different. I think one point where we've
> fallen over is that it wasn't sufficiently well discussed / communicated
> in advance that this testing wasn't just going to 'get done' by some
> independent group and no-one else would have to worry about it, but
> would require a lot of people to chip in. In the same way that there
> isn't some separate independent group that does package reviews, it's
> just all maintainers chipping in when they can. I think perhaps those
> who supported and voted for the policy kind of assumed this would
> happen, and many others weren't actually aware of it.
> 
I think this is the heart of the matter.  Communication and buy-in.

The difference between package reviews/guidelines and testing is a matter of
history -- package reviews are an expectation of maintainer responsibility
from fedora.us days.  Recruitment of testers is an increase in expectations
of maintainers that's happened in the last year.  Without buy-in from
maintainers that they want to do this, you don't get maintainers actually
working on testing packages.  Actually, thinking back to fedora.us, testing
of each update was actually done in fedora.us and abandoned for lack of
manpower.  However the way we tested was a lot different -- each update went
through a new package review, not just a build being installed and rated as
to whether or not it worked.

The comparison to package reviews is also interesting in several ways.

There is an ad hoc group of a few package maintainers that do most of the
reviews.  So this is similar to what you're currently seeing with the
testers.  An ad hoc group of a (relatively) few package maintainers is
testing updates.while the majority of packagers do not participate.

The queue of packages often seem larger than the available manpower to
review packages.

Recently the queue of packages has been going down.  I attribute a large
part of this to tibbs's efforts where he's done a couple things:
* Closing out old reviews where the package submitter no longer responds to
  the review request.
* Actively seeking to put together domain-specific reviewers with packages
  that fit those domains (hooking up active python-sig members with python
  package reviews for instance).

One encouraged and somewhat popular method of getting packages reviewed is
to trade reviews with other packagers.  we don't have that recommendation
for testing at the moment.

> I do think that for update testing to work well going forward we need to
> engage more groups with it and make it clear it's not something that
> some separate QA group is just going to do for everyone and no-one has
> to worry about it. We can get, and already have got, some enthusiastic
> people to sign up to run updates-testing and provide testing feedback
> for the packages they use anyway, but the concept of there being a
> hardcore group of dedicated testers who will go out of their way to
> install, configure and test software they wouldn't usually use is not
> one that's likely to fly, I don't think.
> 
> When software is packaged it's reasonable to expect that someone,
> somewhere, uses it; if they don't, it probably shouldn't be packaged. We
> need to find those people and engage them in the testing process, and it
> seems to me that the maintainers of packages are as well placed as
> anyone to help find and engage their users in this process.
> 
Allowing anonymous karma to count is something that I think targets this.

> In many cases it's easier than that; a lot of packages are maintained by
> more than one person. It's not only perfectly okay but more or less
> *what we want to happen* for co-maintainers to sign up as proven testers
> and test each others' updates. There's a bunch of people in the anaconda
> group, for instance; it's perfectly fine for you all to sign up as
> proven testers and test each other's code. The testing doesn't have to
> come from some impartial outside body, all we need is a sanity check.
> 
> I don't really see any reason why *everyone* who's a packager shouldn't
> also have signed up to be a proven tester by now. I'd like to ask if
> anyone has a perception that it's a hard process to get involved in, or
> if they got the impression that they *shouldn't* get engaged in it, or
> something like that. Maybe we can improve the presentation to make it
> clear that this really ought to be a very wide-based process.
>
With that in mind,  perhaps we should have being added to the packager group
automatically put you in the proventester group.  If you turn out to be
a problem we can then remove you from the proventester group until you've
learned how you should be testing.  (On the implementation side, we should
have this ability in fas since we do something similar to put people who
sign the cla into the cla_done group automatically).

And in answer to your question -- my perception is that it's a separate
thing that I could join just as I've joined infrastructure as well as
packaging.  So in the sense that it's not something that's automatically
there for me to do by virtue of being in packager, it is hard to get
involved in.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20101201/6e921168/attachment.bin 


More information about the devel mailing list