old_testing_critpath notifications

[Clyde E. Kunkel]

On 12/02/2010 02:25 PM, Adam Williamson wrote:
> On Thu, 2010-12-02 at 14:10 -0500, Doug Ledford wrote:
>
>> My package in question (mdadm) is only used in certain circumstances,
>> but if it isn't right, systems fail to boot.  I can certainly see why
>> something that can render a machine unbootable should be critpath.
>> However, because only a few people use it, testing is sparse at best.
>
> I think probably more people use it than current testing indicates, and
> we should do a better job of getting more people involved in testing.
>
>> I'll get one or two testers actually testing the package, but I won't
>> know until a release is made whether or not it truly works for the
>> masses because it isn't until then that I hit enough critical mass to know.
>>
>> That being the case, I test the package fairly rigorously myself.  But
>> this process doesn't take that into account.  I test far more things
>> than you get with a few people just doing smoke tests, but the smoke
>> tests are actually the gating factor.  If you changed the process so a
>> maintainer can indicate they've done their own fairly extensive testing,
>> that would satisfy me.  But that also opens the door for abuse, so you
>> would have to watch maintainers once you enabled this ability.
>
> I've posted in the thread earlier that I'd actually like to do this,
> others seem opposed however.

What about putting the maintainer's tests somewhere a tester can get 
them and use on their own system(s)? Maybe even require critical path 
packages to actually provide test cases.  This is not to say we don't 
believe the maintainer, it is just a validation that the package passes 
reasonable testing of the actual changes.

I continue to be frustrated by not being able to always test the actual 
change.  In most cases all I can do is make sure there are no 
regressions and then that means all I have done is use the system with 
the changed package installed and see if anything at all fails.

Regards,
OldFart