Proven tester wiki love

[Adam Williamson]

On Wed, 2010-07-07 at 19:22 +0100, mike cloaked wrote:
> On Wed, Jul 7, 2010 at 1:44 AM, Adam Williamson <awilliam at redhat.com> wrote:
> > On Mon, 2010-07-05 at 19:33 -0500, Aaron Faanes wrote:
> >> I went to work a bit on jdulaney's fork of the proven tester page to
> >> make the mentorship-merging stuff fit a little more smoothly, and I
> >> sort of got carried away. I'd normally just add this as a new revision
> >> for Proven_testers, but it's pretty substantially revised. It probably
> >> needs to get re-reviewed:
> >>
> >> https://fedoraproject.org/wiki/User:Dafrito/Proven_tester
> >>
> >> I can write up a summary of the major differences, but I figure I
> >> should only toss out one wall-of-text at a time. ;) Basically, I took
> >> the current one, and tried to expand on places that were shallow (like
> >> testing criteria) and made other sections flow a lot better (like the
> >> old Feedback section).
> >>
> >> I feel the need to mention I refer to critpath packages as just
> >> critical packages and critical updates, instead of critical path
> >> packages and critical path updates. This is purely a stylistic change:
> >> when I was reading it out loud, I trip over the alliteration of "path
> >> packages" a lot. Let me know if this (or anything else) is a mortal
> >> sin.
> >
> > I understand - I've had the same feeling - but I think we may want to
> > stick with the 'critical path' naming, as it's what's used elsewhere in
> > Fedora-land.
> 
> In general I am happy with the draft - however there is one area that
> I am a little uncertain about, and maybe others could clarify. This is
> the situation where for example there is currently a kernel in
> updates-testing for both f12 and f13, and each of these packages has a
> series of bug fixes.  I am perfectly happy to install and test but I
> may not be able to test all the bugs reported as fixed. In this
> instance if the kernel installs without issue and the machine boots,
> with normal logins, normal networking for my particular wireless card
> on a laptop, and no issues with any tests that I conduct, should I
> report +ve karma if all the tests I can do pass without incident, or
> should I as proventester not add karma (i.e. neutral) since I can't
> check that all the bugs reported are not problematic.
> 
> In the current case I see that for
> https://admin.fedoraproject.org/updates/kernel-2.6.32.16-141.fc12?_csrf_token=c14a79deed966770749b2d7990491e42a15cb89c
> I can check 559153 but not 558002 for example. So should I add a
> report and have neutral karma or +ve? The instructions would indicate
> neutral karma is appropriate here. (Prior to becoming a proventester I
> would have given +ve karma in this situation.
> 
> Similarly for https://admin.fedoraproject.org/updates/kernel-2.6.33.6-147.fc13?_csrf_token=c14a79deed966770749b2d7990491e42a15cb89c
> where there is an even longer list of bugs fixed.  Clearly I can only
> test on the hardware that I am installed on for either f12 or f13 ....
> 
> Would be nice to have this clarified - I guess that reading the
> proventester instructions again this case should have neutral
> feedback. It seems that the only situation for giving positive karma
> as a proventester is if previously reported negative karma is
> demonstrated to have been wrong - please correct me if I am wrong.

This certainly isn't the intention. In general if you use it and find it
works for you - in that it doesn't break the critical path actions - you
should file +ve feedback.

Actually, re-reading them, both versions tend to emphasize the negative
feedback scenarios more than the positive. We should fix that. Positive
feedback is in a way more important than negative, because we *need*
proven testers to leave at least +1 on each critpath update (that
doesn't have a problem, obviously) or it will not get published. We
should probably explicitly call out that, in general, if you don't hit
any of the negative or neutral feedback scenarios discussed, you should
leave positive feedback.

Thinking about it, though, we could consider a slightly different
process for the kernel, as it's a component that's *extremely* subject
to different experiences for different users. I'm not sure the workflow
we've designed will work terribly well for kernels. I suspect it'll be
all too easy for a kernel which actually contains a major regression to
be approved; all it needs is for a proventester who doesn't happen to
own the hardware concerned to find it works fine on their system, and
file a +1, and anyone else to file a +1 too, and it'd be approved, even
though someone who does own the hardware might come by and test an hour
later and find the problem...

we might want to design a system for the kernel where all proventesters
hold off posting positive feedback for a day or two, until several
proventesters and regular testers have had the chance to check for
regressions.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net