FESCo wants to ban direct stable pushes in Bodhi (urgent call for feedback)

[Will Woods]

On Mon, 2010-03-01 at 11:57 -0500, Tom "spot" Callaway wrote:
> On 03/01/2010 11:52 AM, Peter Jones wrote:
> > If you think this isn't the right way
> > to provide a safety net for package maintainers - what is?
> 
> With the understanding that you're not specifically asking me that
> question, I'd say that I'd prefer to first try to automate checks for
> the most frequent update issues:
> 
> * Causes broken deps

This is in progress, and we have a working (albeit very alpha) test. See
the depcheck ticket/milestone:

https://fedorahosted.org/autoqa/ticket/111
https://fedorahosted.org/autoqa/milestone/autoqa%20depcheck

> * Has ABI/API change (and is a Critical Path package)

This should be handled by the current rpmguard test:
https://fedorahosted.org/autoqa/wiki/rpmguard

since changing the ABI/API should generally change the soname/version,
thus changing the package Provides. Furthermore there's plans to
actually diff the symbol tables themselves, for extra safety:
https://fedorahosted.org/autoqa/ticket/75

> * Breaks clean upgrade path between releases

That's an interesting test case, actually. I'm not sure we currently
check packages against the corresponding versions *other* releases. 

I've added ticket 123 to track this:
https://fedorahosted.org/autoqa/ticket/123

> * Fails to pass any package specific sanity tests (as written by either
> the maintainer, QA, rel-eng, or qualified contributors)

Handling package-specific tests is definitely a goal of AutoQA, and
allowing maintainers to add tests to their packages is something I'm
trying to keep in mind for the dist-git planning. But this is a big goal
and it'll come after the previous three.

> AutoQA has the potential to do this. I'd rather see energy and effort
> spent on taking out these low hanging fruit. If, after that, we're still
> having broken updates pushed directly to stable, then I'd be willing to
> consider a policy with an enforced delay in "testing".

I'm all for extra energy and effort toward getting AutoQA up and
running, but automated testing does not obviate the need for manual
testing. It just changes which tests need to be performed manually.

Let's be clear about the current state of the test process, and its
future:

1) We have some unspoken minimum standards of quality.
2) We are not meeting those standards.

3) Meeting those standards currently requires manual testing.
4) We can automate most tests required to meet those standards.

5) We would like to raise our standards of quality.

Meeting those new standards would require new tests, which have not yet
been automated. 6 GOTO 3.

It should be obvious that the process by which we improve the quality of
the distribution will always require some amount of manual testing.

So I think it would be shortsighted for FESCo to refuse to even discuss
a policy about what manual testing is currently required, since any plan
for improving the quality of the distribution will always require some
amount of manual testing.

-w