Update testing policy: how to use Bodhi

[Mathieu Bridon]

Hi,

On Fri, Mar 26, 2010 at 23:49, Adam Williamson <awilliam at redhat.com> wrote:
[... snip ...]
> 1. I have tried this update in my regular day-to-day use and seen no
> regressions.
>
> 2. I have tried this update in my regular day-to-day use and seen a
> regression: bug #XXXXXX.
>
> 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does fix bug #XXXXXX.
>
> 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does not fix bug #XXXXXX.
>
> 5. I have performed the following planned testing on the update: (link
> to test case / test plan) and it passes.
>
> 6. I have performed the following planned testing on the update: (link
> to test case / test plan) and it fails: bug #XXXXXX.

This is basically what Doug had proposed, except that you added 5. and 6.

I know Luke also likes the idea, and I've been toying with
implementing Doug's idea in the Bodhi2 branch. Adding those 2 more
karma types shouldn't be too hard.

>Testers should be able to file multiple types of feedback in one
> operation - for instance, 4+1 (the update didn't fix the bug it claimed
> to, but doesn't seem to cause any regressions either). Ideally, the
> input of feedback should be 'guided' with a freeform element, so there's
> a space to enter bug numbers, for instance.

That's what I had in mind, but the Bodhi2 branch currently doesn't
have any controllers/template, so it will be some time before I have
something to show.

> I think Bill's proposed policy can be modified quite easily to fit this.
> All it would need to say is that for 'important' updates to be accepted,
> they would need to have one 'type 1' feedback from a proven tester, and
> no 'type 2' feedback from anyone (or something along those lines; this
> isn't the main thrust of my post, please don't sidetrack it too
> much :>).

That's actually one of the points I was missing : rules for
(auto-)push / (auto-)unpush. But yeah, it can be agreed on later.

> The system could do a count of how many of each type of feedback any
> given update has received, but I don't think there's any way we can
> sensibly do some kind of mathematical operation on those numbers and
> have a 'rating' for the update. Such a system would always give odd /
> undesirable results in some cases, I think (just as the current one
> does). I believe the above system would be sufficiently clear that there
> would be no need for such a number, and we would be able to evaluate
> updates properly based just on the information listed.
>
> What are everyone's thoughts on this? Thanks!

I totally agree that this would be far more desirable than the current
+1/-1 system.


----------
Mathieu Bridon