The new Update Acceptance Criteria are broken

[Tom Lane]

"Clyde E. Kunkel" <clydekunkel7734 at cox.net> writes:
> On 11/12/2010 02:32 PM, Tom Lane wrote:
>> It's absolutely crystal clear to me that we don't have enough tester
>> manpower to make the current policy workable; it's past time to stop
>> denying that.  I'd suggest narrowing the policy to a small number of
>> critical packages, for which there might be some hope of it actually
>> working as designed.

> Test cases would help alleviate manpower issues.  Many of the security 
> updates and regular updates are outside my area and I feel some 
> frustration that I have to bypass providing karma; however, I am used to 
> doing QA work with test cases.  Are they so hard to provide?  Maybe 
> certain updates should have test cases, i.e., security updates and 
> critical path updates.

The major packages that I work with have regression test suites,
which in fact get run as part of the RPM build sequence.  It's not
apparent to me that I should need to invent some more tests.

The likely failure cases that I can see are of two types:

1. Upstream screwed up and introduced a regression into what was
supposed to be a minor bug-fix or security update.  This does happen,
for sure, but there's pretty much 0 chance that I as packager am going
to catch it if it gets past the built-in regression tests.
Unfortunately, there is also pretty much 0 chance that Fedora testers
are going to notice such a problem in the limited time window for sanity
testing.  It hasn't ever happened for any of my packages that Fedora
testers caught such things in time.

2. I screwed up and introduced a packaging bug, for instance bad
dependencies or inability to "yum update".  That's been known to happen
too.  But I have a lot more faith in autoqa being able to catch that
kind of problem in a timely fashion than I do in manual testing catching
it.

I guess what this boils down to is that I'd be happier with the testing
process if it were actually successful at finding problems.  In my
experience, it's a week's delay for exactly zero return.

			regards, tom lane