updates improvements/changes ideas

Michael Schwendt mschwendt at gmail.com
Tue Nov 30 12:00:54 UTC 2010


On Tue, 30 Nov 2010 12:24:32 +0100, Matthias wrote:

> The bug reporter will probably verify the fixture for his bug. But,
> given the fact, this new version breaks other things, this could be
> possibly covered by a test case (in best case).

A _new version_ will need to stay in updates-testing to allow for _more_
testers to get an opportunity to test the software with daily usage. It
will be beneficial to wait for more testers instead of rushing out an
update within 24 hours based on possibly superficial testing done by
update-freaks and koji-leechers.

 => bodhi karma automatism should be _off_ by default
 => packagers, who turn it on, must be aware of the consequences

> IMHO you should write test cases only once (a general sheet, what to test).

What is needed is brave Fedora Users, who actually use the software
and know where/how individual packages fit into the system.

> I see, we need to get more testers. But those people need to know what
> to test, sometimes even: how to test.... If we don't provide those
> information, we will get a lot of feedback: +1, "works for me". What do
> we really know then? What was tested? Does it prevent us from shipping
> broken updates? Definitely not.

Oh so true, ... but that isn't any excuse for lazy/poor test results IMO.
Unfortunately, the system so far encourages people to post quick +1 because
that will speed up the release of the update and add to the metrics.

Packagers need to become aware of their accountability with regard to broken
updates and think twice before relying on bodhi karma automatism. And the user
community needs to understand that some types of updates won't be marked
stable unless more users give positive feedback. It's the packager's
responsibility to tell what specific test results are still needed.


More information about the test mailing list