exclude people from giving karma?

Thu Feb 27 21:43:50 UTC 2014

On Thu, 2014-02-27 at 09:07 -0800, Adam Williamson wrote:
> On Thu, 2014-02-27 at 02:18 +0100, Kevin Kofler wrote:
> > Miloslav Trmač wrote:
> > > I fully agree with you testers giving +1 is not even close to proper
> > > validation, but what alternative to get proper validation do you propose
> > > as an improvement?  Dropping autokarma would replace broken validation
> > > with *no* validation; that's not an improvement.
> > 
> > The proposal here is only to abolish AUTOkarma, i.e. to require the 
> > maintainer to manually push the update even if it has enough karma. In other 
> > words, remove the "[ ] Enable karma automatism" checkbox and the 2 
> > thresholds that go with it from Bodhi. (Just ensure that non-critpath 
> > updates can always be manually pushed once they get a karma of +1, because 
> > last I checked, this was STILL broken in Bodhi and required changing the 
> > stable threshold to 1 to be able to push the update.)
> > 
> > This is not about the policies to not allow the maintainer to push without 
> > karma, though I happen to think those should also be abolished. (If you ask 
> > why: Because I think the maintainer is better placed to judge the quality of 
> > his/her update than an arbitrary number of people with nothing more than a 
> > FAS account. Feedback from the testers should only be informative.)
> > 
> > But again, I think that even with no other policy change, just removing the 
> > "karma automatism" misfeature from Bodhi would be an improvement.
> 
> So here's an idea I do want to float on devel@ before taking it to FESCo
> (and run by the AutoQA folks), but how about this?
> 
> We don't currently 'enforce' the AutoQA depcheck and upgradepath tests
> as we don't consider them reliable enough. OK. But are they at least
> reliable enough that we could disable karma automatism if one fails,
> with a note to the update submitter that they can push the update
> manually or re-enable automatism if they find the failure is a false
> negative?
> 
> We've had two unfortunate broken-depcheck updates in the past week, now
> - libreoffice and dnf. In the libreoffice case, oddly, depcheck was
> reported as "PASSED" even though the log shows all sorts of dep issues -
> http://autoqa.fedoraproject.org/results/758667-autotest/virt06.qa/depcheck/results/libreoffice-4.2.1.1-.html - but depcheck did catch the dnf one:
> 
>  autoqa - 2014-02-24 23:19:06
> AutoQA: depcheck test FAILED on i386. Result log:
> http://autoqa.fedoraproject.org/report/1ccpl (results are informative
> only) 

Just as a follow up: I've spent the morning going through depcheck
failures from the last few weeks and manually inspecting them, and found
several other incorrect updates:

https://admin.fedoraproject.org/updates/FEDORA-2014-1676/drupal7-webform-4.0-0.1.beta1.fc19
https://admin.fedoraproject.org/updates/nodejs-cssom-0.3.0-1.fc19
https://admin.fedoraproject.org/updates/FEDORA-2014-3048/NetworkManager-0.9.9.0-31.git20131003.fc20 (upgradepath problem, not depcheck)
https://admin.fedoraproject.org/updates/FEDORA-2014-2747/rubygem-openshift-origin-node-1.18.0.1-1.fc19
https://admin.fedoraproject.org/updates/testdisk-6.14-3.fc20,ntfs-3g-2014.2.15-1.fc20 (rapidly fixed, thanks spot)
https://admin.fedoraproject.org/updates/FEDORA-2014-2300/cutter-1.2.3-1.fc19
https://admin.fedoraproject.org/updates/FEDORA-2014-2231/python-wstool-0.1.1-1.fc19,python-vcstools-0.1.32-2.fc19,python-rosinstall-0.7.3-1.fc19
https://admin.fedoraproject.org/updates/FEDORA-2014-2991/rubygem-tins-1.0.0-2.fc20
https://admin.fedoraproject.org/updates/FEDORA-2014-3143/shogun-3.2.0-1.fc19
https://admin.fedoraproject.org/updates/FEDORA-2014-2750/bmake-20140214-1.fc19 (upgradepath problem, now fixed, thanks)
https://admin.fedoraproject.org/updates/FEDORA-2014-1372/dnssec-tools-2.0-9.fc19
https://admin.fedoraproject.org/updates/FEDORA-2014-1666/freeipa-3.3.4-3.fc20 (caused by a Bodhi bug, not the packager's fault - releng is cleaning it up)
https://admin.fedoraproject.org/updates/guacamole-client-0.8.3-5.fc19

I've provided feedback on all of these via appropriate channels.

On the other hand, there certainly are false negatives from depcheck.
The most common thing is that it considers explicit Conflicts: to be
failures. The tools folks tell me it's not entirely trivial even just to
filter out runs that 'fail' only on conflicts, so that may be a bit of a
problem. Still, I think it's worth considering.

At the very least, let me plead with packagers again: *please* read your
AutoQA test results when they fail. Yes, sometimes they are false
negatives, but often they are identifying problems you should fix.

(Before KK jumps in claiming that this proves Bodhi is useless and
AutoQA is useless and everything is useless - I also noticed several
cases where packagers clearly *had* read the AutoQA results, because I
saw that shortly after AutoQA posted the failure notification comment,
the update was edited in some way, and a subsequent test run passed.
Those packagers rock! Keep it up!)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net