This "karma" stuff is a pain!

[Adam Williamson]

On Fri, 2012-03-16 at 12:15 -0400, John Ellson wrote:
> On 03/16/2012 05:13 AM, Michael Scherer wrote:
> > Le jeudi 15 mars 2012 à 12:49 -0400, John Ellson a écrit :
> >> Can we just generate "karma" from a comment in bugzilla please?  Having
> >> to find some other weird place to indicate that a fix "works for me" is
> >> a real pain.
> > Have you tried fedora-easy-karma ?
> > https://fedoraproject.org/wiki/Fedora_Easy_Karma
> >
> 
> This is supposed to be easy?
>         "Run fedora-cert, included with fedora-easy-karma as a 
> dependency, to set up a certificate with your FAS credentials. "
> 
> I'm sure "karma" is useful to Release-Engineering.   I just think the 
> scope is wrong for a bug reporter.
> 
> Take, for example:
>      
> https://admin.fedoraproject.org/updates/NetworkManager-0.9.3.995-0.6.git20120314.fc17
> 
> The update contains fixes for three problems:  800690, 798102, 802540
> 
> I contributed to the first bug, 800690, and duly tested and reported 
> "works for me", but I had no involvement in the other two, so I'm not in 
> a position to judge their "karma".
> 
> I think these release updates should automatically gain partial, per-bug 
> karma from "works for me" in the bug reports.
> 
> "karma" for the update in total needs to come from people in a 
> "release-engineering" role, rather than people in a "bug 
> reporting/fixing/testing role".
> 
> I agree that people using an "update testing" repository are reasonable 
> candidates for the "release-engineering" role, but "bug 
> reporting/fixing/testing" role doesn't require "update testing".   The 
> bugs fixes might be tested directly from koji, or from some private 
> builds, or even from local patching.
> 
> I am trying to be constructive here.  We're all busy people.  I just 
> think that "karma" is outside of a reasonable workflow for a bug reporter.

The 'karma' relates to the update as a whole, not any specific bug. What
we're principally concerned with in the 'updates testing' process is not
'does this update fix the bugs it claims to fix' but 'does this update
cause any major functionality regressions'.

It's useful to read https://fedoraproject.org/wiki/Proven_tester in this
context. It is/was intended as instructions for proven testers but it's
useful reading for anyone in filing karma.

The current system is clearly limited in quite a lot of ways. The
single, numeric karma system really isn't sophisticated enough. I've
mentioned this several times, and wrote a fairly long post explaining
the advantages of a more complex system (and hence, by implication, the
drawbacks of the current system) at
https://lists.fedoraproject.org/pipermail/test/2011-November/104579.html .

Luke has had Bodhi 2.0 in the works for a while, now. A large part of
what Bodhi 2.0 will do is what's described in that post - allow for
multiple, possibly-dynamically-definable types of feedback on updates,
rather than a single 'karma number' for each update.

That might be an appropriate time to try and work some kind of
connection between Bugzilla and Bodhi. But I still think it might be
very difficult to do; it's very difficult to parse a freeform Bugzilla
comment and be sure whether it means 'the update's good' or 'the
update's bad', and implementing some kind of 'tick here if the update
works' box in Bugzilla requires downstream patching of Bugzilla, which
we're currently quite heavily opposed to.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net