old_testing_critpath notifications

[Doug Ledford]

On 12/01/2010 05:17 PM, Adam Williamson wrote:
> On Wed, 2010-12-01 at 16:55 -0500, Doug Ledford wrote:
> 
>> The comparison is 100% fair because it points out the fundamental
>> problem with the current policy: if you don't have a paid staff of
>> testers to make sure testing is done in a timely fashion, then you have
>> absolutely no business gating updates on a testing staff that doesn't
>> exist.  It's nice in theory to think we can force testing of updates
>> prior to their release, but if the testing staff simply isn't there,
>> then you aren't improving the product, you're just stopping progress.
> 
> The gating is not on 'a testing staff'. The gating is on *testing*.
> 
> I want to say again that I'm not particularly wedded to the current
> policy and I don't mind at all if it changes, but I think we need to be
> careful of the mindset that says 'we can't enforce any standards in
> Fedora because it's a volunteer project so we must just accept what
> people are willing to give us'.

My package in question (mdadm) is only used in certain circumstances,
but if it isn't right, systems fail to boot.  I can certainly see why
something that can render a machine unbootable should be critpath.
However, because only a few people use it, testing is sparse at best.
I'll get one or two testers actually testing the package, but I won't
know until a release is made whether or not it truly works for the
masses because it isn't until then that I hit enough critical mass to know.

That being the case, I test the package fairly rigorously myself.  But
this process doesn't take that into account.  I test far more things
than you get with a few people just doing smoke tests, but the smoke
tests are actually the gating factor.  If you changed the process so a
maintainer can indicate they've done their own fairly extensive testing,
that would satisfy me.  But that also opens the door for abuse, so you
would have to watch maintainers once you enabled this ability.

> Even though packaging in Fedora is a volunteer process, we still have
> fairly rigorous packaging guidelines and a review process. We don't just
> accept any package someone turns up and submits. i.e., we're enforcing
> standards of quality, despite this being an entirely volunteer effort
> and no-one being compelled to show up and provide packages of a
> particular quality.

And packages *can* and *do* languish in the review process seemingly
interminably at times.

> The concept of having a policy requiring updates to be tested before
> they're issued is really no different.

I'm fine with this at a conceptual level, but the current policy takes
the update out of my hands no matter how much testing I do.  That I'm
not fine with.

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20101202/dd7a2dbd/attachment.bin