A Glorious Vision of Our Shared Update Feedback Future (bodhi, karma, and proventesters, oh my)

[Adam Williamson]

Hey, folks.

So in the recent proven tester discussion, and in various other threads,
I've oft stated that the limits of the current Bodhi karma system are a
significant problem, and the planned Bodhi 2.0 karma system has to
potentially to significantly improve our update testing process. But it
occurred to me that I haven't really laid out why in much detail, and
those who aren't as involved with the process as we in QA are might not
have a really clear vision of why this is so important.

So, I thought I'd lay it out in the form of a glorious vision of the
future! Note: I have zero UI design skills. This UI as described would
suck. But the idea is that there would be a decent UI which *represents
all the described choices*.

In the Great Bodhi Of The Future, for any given update, a tester will
not simply have a comment box and a drop-down for -1, 0 or +1. They will
see:

* The list of test cases associated with the package, with a PASS / FAIL
choice for each

* Checkboxes for 'This update does / does not break critical path
functionality' (with a link out to the critpath definition)

* A checkbox for 'I installed this update and continued to use my system
as normal and did not note any regressions'

* Any custom choices the package maintainer opts to provide, via some
kind of interface to Bodhi

This is the kind of flexibility that would make karma massively more
useful. The tighter definition of what feedback actually *means*
provides far more information to the maintainer, and enables us to
automate certain outcomes much more aggressively.

For me, one the principal benefits of such a system would be that we
could make the 'This update breaks critical path functionality' checkbox
an absolutely red flashing light, wailing siren emergency button. I
mean, you hit that thing and trucks roll from Fedora HQ, metaphorically
speaking. It would have a confirmation page which clearly described the
impact of asserting that an update broke critpath, so we could be
confident that it didn't get falsely triggered very often. I'd suggest
that:

1. Any update that is marked as 'critpath breaking' by a FAS-registered
tester would be blocked from going any further in the update process
without manual intervention (no autopushes at all)

2. Any update marked as 'critpath breaking' by a proven tester would be
blocked from being pushed stable at all - automatically or manually -
until the PT modified the feedback or it was overridden by someone with
appropriately godlike powers (TBD, but probably not just the maintainer)

3. Any update marked as 'critpath breaking' should probably get
announced on at least test-announce and/or devel-announce

4. Any update marked as 'critpath breaking' *after it has already been
pushed* would similarly trigger a major response: notify maintainer very
hard, notify lists, generally do stuff to make sure it gets immediate
attention

We would also obviously be able to offer maintainers more nuanced
options for autopushing updates - require a certain number of passes on
a certain set of test cases, for instance.

To go a bit into the theory, the really nasty limitation of the current
system is we simply have very little easily consumable indication of
what the karma provided on an update *means*. A +1 can mean anything
from 'I booted and nothing exploded' to 'I tested every line of this
code, then did it backwards standing on my head'. A -1 can mean 'I
booted this update and then my cat got sick, I think there's a
connection!', 'this update breaks $REALLY OBSCURE FEATURE Z', or 'this
update exploded my monitor'. We just _don't know_. You can get the
information from comments, usually, but that's obviously a complete
non-starter for any kind of programmatized or automated action based on
the feedback. In particular, we currently can't do anything dramatic
based on negative feedback because of the uncertainty around exactly
what it means. We do have a few policies about when to file negative
feedback, but even this only very slightly mitigates the problem - we
can't say 'only file negative feedback if the update breaks critpath',
that's just not feasible. So we can't identify the 'really really
negative' feedback and respond appropriate drastically to it without
causing lots of pain through false positives (or rather, false
negatives...)

With a more advanced feedback system we can identify the really big
problems and be much more aggressive about handling them, without
over-reacting to smaller problems. We could, correspondingly, be a bit
less strict about how much *positive* feedback you need to push an
update, if we can be a bit more confident that we'll definitely identify
the really bad problems through negative feedback.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Sandro "red" Mathys]

On Tue, Nov 22, 2011 at 22:03, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  2. Any update marked as 'critpath breaking' by a proven
> tester would be
>  blocked from being pushed stable at all - automatically or manually -
>  until the PT modified the feedback or it was overridden by someone with
>  appropriately godlike powers (TBD, but probably not just the maintainer) 
Here I'd add something like "three proventesters can turn over one
proventester" and only if that doesn't work activate someone with
godlike powers. Just because godlike powers should never have to be
used in an ideal community/world and I think this overturn rule can
help to get us a little closer to ideal ;)

-- red

[Adam Williamson]

On Tue, 2011-11-22 at 22:17 +0100, Sandro "red" Mathys wrote:
>  On Tue, Nov 22, 2011 at 22:03, Adam Williamson
> <awilliam(a)redhat.com&gt; wrote:
>  > 2. Any update marked as 'critpath breaking' by a proven tester would be
>  > blocked from being pushed stable at all - automatically or manually -
>  > until the PT modified the feedback or it was overridden by someone with
>  > appropriately godlike powers (TBD, but probably not just the maintainer)
>  
>  Here I'd add something like "three proventesters can turn over one
>  proventester" and only if that doesn't work activate someone with
>  godlike powers. Just because godlike powers should never have to be
>  used in an ideal community/world and I think this overturn rule can
>  help to get us a little closer to ideal ;) 
Possibly, but in that case, I'd want the mechanism to require the
'overturning' PTs to explicitly consider the 'omg raptors' PT. i.e., if
3 PTs check 'doesn't break critpath' and then one PT checks 'breaks
critpath!', those three earlier tests should not be considered to
override the one later one. Other PTs would have to explicitly say that
they tested whatever the 'omg raptors!' PT said was broken and could not
reproduce, in order to overturn their feedback.

(And then we'd go after the incorrect PT with the 2x4 with a rusty nail
in it...)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Sandro "red" Mathys]

On Tue, Nov 22, 2011 at 22:59, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  (And then we'd go after the incorrect PT with the 2x4 with a
> rusty nail
>  in it...) 
Sometimes as a PT I feel like I would like to +1 an update, but I
don't feel like waving this critpath update through. Mostly because I
don't know the software good enough or because another tester had a
negative experience. In that situations I'd like to upkarma the update
without giving it the critpath-OK.

Take this example:
https://admin.fedoraproject.org/updates/FEDORA-2011-16237/kernel-3.1.2-1....

Right now, a PT has voiced that he's seen a regression. He also states
that the issue he's seeing might have other reasons. So I'd like to
have him eventually +1/-1 this update, but still I have tested this
and would like to upkarma it, to contribute to the total number of
necessary karma.

Maybe there should simply be another checkbox "treat me as a non-PT
this single time".

-- red

[Tom Callaway]

On 11/22/2011 04:03 PM, Adam Williamson wrote:
>  Hey, folks.
>  
>  So in the recent proven tester discussion, and in various other threads,
>  I've oft stated that the limits of the current Bodhi karma system are a
>  significant problem, and the planned Bodhi 2.0 karma system has to
>  potentially to significantly improve our update testing process. But it
>  occurred to me that I haven't really laid out why in much detail, and
>  those who aren't as involved with the process as we in QA are might not
>  have a really clear vision of why this is so important.
>  
>  So, I thought I'd lay it out in the form of a glorious vision of the
>  future! Note: I have zero UI design skills. This UI as described would
>  suck. But the idea is that there would be a decent UI which *represents
>  all the described choices*. 
IMHO, this seems like the sort of thing that would benefit from a
discussion session at the upcoming FUDCon where there will be members of
the Fedora Design team and the main coders of Bodhi. Not that I want to
do it in secret, but coming up with a solid battle plan that also has
the decent UI (in a mockup) as a first step seems like a better plan to
success.

~tom

==
Fedora Project

[Adam Williamson]

On Fri, 2011-12-02 at 17:24 -0500, Tom Callaway wrote:

>  IMHO, this seems like the sort of thing that would benefit from a
>  discussion session at the upcoming FUDCon where there will be members of
>  the Fedora Design team and the main coders of Bodhi. Not that I want to
>  do it in secret, but coming up with a solid battle plan that also has
>  the decent UI (in a mockup) as a first step seems like a better plan to
>  success. 
That certainly seems like a good idea.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net