The new Update Acceptance Criteria are broken
tgl at redhat.com
Sat Nov 13 04:14:02 UTC 2010
"Clyde E. Kunkel" <clydekunkel7734 at cox.net> writes:
> On 11/12/2010 02:32 PM, Tom Lane wrote:
>> It's absolutely crystal clear to me that we don't have enough tester
>> manpower to make the current policy workable; it's past time to stop
>> denying that. I'd suggest narrowing the policy to a small number of
>> critical packages, for which there might be some hope of it actually
>> working as designed.
> Test cases would help alleviate manpower issues. Many of the security
> updates and regular updates are outside my area and I feel some
> frustration that I have to bypass providing karma; however, I am used to
> doing QA work with test cases. Are they so hard to provide? Maybe
> certain updates should have test cases, i.e., security updates and
> critical path updates.
The major packages that I work with have regression test suites,
which in fact get run as part of the RPM build sequence. It's not
apparent to me that I should need to invent some more tests.
The likely failure cases that I can see are of two types:
1. Upstream screwed up and introduced a regression into what was
supposed to be a minor bug-fix or security update. This does happen,
for sure, but there's pretty much 0 chance that I as packager am going
to catch it if it gets past the built-in regression tests.
Unfortunately, there is also pretty much 0 chance that Fedora testers
are going to notice such a problem in the limited time window for sanity
testing. It hasn't ever happened for any of my packages that Fedora
testers caught such things in time.
2. I screwed up and introduced a packaging bug, for instance bad
dependencies or inability to "yum update". That's been known to happen
too. But I have a lot more faith in autoqa being able to catch that
kind of problem in a timely fashion than I do in manual testing catching
I guess what this boils down to is that I'd be happier with the testing
process if it were actually successful at finding problems. In my
experience, it's a week's delay for exactly zero return.
regards, tom lane
More information about the devel