Rethinking proventester and critpath

[Matthew Garrett]

It's a common complaint that it's too difficult to get updates to 
critpath packages through the update system at the moment. We've been 
looking into trying to make that easier without just dropping the 
critpath requirements, and one thing we looked at was whether the 
requirement for positive karma from proventesters was a net benefit.

Thankfully this is the kind of thing that we can actually generate 
numbers for. Luke pulled some statistics which are available at 
http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html . 
The relevant section here is the set of packages that have (a) 
sufficient positive karma to be pushed, but (b) negative proventester 
karma - that is, the packages where negative proventester karma 
prevented a push.

Straight off, we can see that these amount to 1-2% of all critpath 
updates. It's simply not common for proventester to make a difference to 
the outcome. If we look at the individual packages, things get even more 
interesting. Many of the updates receive a mixture of proventester 
karma, so even with the negative the push would still go ahead. As far 
as I can tell:

https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14

are the only two updates where the proventester karma requirement would 
have made a difference, out of 1942 critpath updates that made it to 
stable. That doesn't seem like a great hit rate.

So, assuming I'm not grossly misanalysing the data, it seems that we 
could drop the proventester requirement from critical path updates with 
a negligable change in the quality of the updates. Thoughts?

-- 
Matthew Garrett | mjg59(a)srcf.ucam.org

[drago01]

On Tue, Nov 1, 2011 at 2:59 PM, Matthew Garrett <mjg59(a)srcf.ucam.org&gt; wrote:
>  It's a common complaint that it's too difficult to get
> updates to
>  critpath packages through the update system at the moment. We've been
>  looking into trying to make that easier without just dropping the
>  critpath requirements, and one thing we looked at was whether the
>  requirement for positive karma from proventesters was a net benefit.
>
>  Thankfully this is the kind of thing that we can actually generate
>  numbers for. Luke pulled some statistics which are available at
>  http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html .
>  The relevant section here is the set of packages that have (a)
>  sufficient positive karma to be pushed, but (b) negative proventester
>  karma - that is, the packages where negative proventester karma
>  prevented a push.
>
>  Straight off, we can see that these amount to 1-2% of all critpath
>  updates. It's simply not common for proventester to make a difference to
>  the outcome. If we look at the individual packages, things get even more
>  interesting. Many of the updates receive a mixture of proventester
>  karma, so even with the negative the push would still go ahead. As far
>  as I can tell:
>
>  https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
>  https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14
>
>  are the only two updates where the proventester karma requirement would
>  have made a difference, out of 1942 critpath updates that made it to
>  stable. That doesn't seem like a great hit rate. 
Well this looks like just a coincidence to me that the people finding
issues where proventesters in this cases.

>  So, assuming I'm not grossly misanalysing the data, it seems that
> we
>  could drop the proventester requirement from critical path updates with
>  a negligable change in the quality of the updates. Thoughts? 
I'd say go for it and I doubt there would be any quality change.

[Jóhann B. Guðmundsson]

On 11/01/2011 01:59 PM, Matthew Garrett wrote:
>  So, assuming I'm not grossly misanalysing the data, it seems that
> we
>  could drop the proventester requirement from critical path updates with
>  a negligable change in the quality of the updates. Thoughts? 
Agreed flag it as tried and tested stats dont lie...

JBG

[Adam Williamson]

On Tue, 2011-11-01 at 13:59 +0000, Matthew Garrett wrote:
>  It's a common complaint that it's too difficult to get
> updates to 
>  critpath packages through the update system at the moment. We've been 
>  looking into trying to make that easier without just dropping the 
>  critpath requirements, and one thing we looked at was whether the 
>  requirement for positive karma from proventesters was a net benefit.
>  
>  Thankfully this is the kind of thing that we can actually generate 
>  numbers for. Luke pulled some statistics which are available at 
>  http://lists.fedoraproject.org/pipermail/test/2011-October/104084.html . 
>  The relevant section here is the set of packages that have (a) 
>  sufficient positive karma to be pushed, but (b) negative proventester 
>  karma - that is, the packages where negative proventester karma 
>  prevented a push.
>  
>  Straight off, we can see that these amount to 1-2% of all critpath 
>  updates. It's simply not common for proventester to make a difference to 
>  the outcome. If we look at the individual packages, things get even more 
>  interesting. Many of the updates receive a mixture of proventester 
>  karma, so even with the negative the push would still go ahead. As far 
>  as I can tell:
>  
>  https://admin.fedoraproject.org/updates/xorg-x11-drv-geode-2.11.9-1.fc14
>  https://admin.fedoraproject.org/updates/system-setup-keyboard-0.8.6-2.fc14
>  
>  are the only two updates where the proventester karma requirement would 
>  have made a difference, out of 1942 critpath updates that made it to 
>  stable. That doesn't seem like a great hit rate.
>  
>  So, assuming I'm not grossly misanalysing the data, it seems that we 
>  could drop the proventester requirement from critical path updates with 
>  a negligable change in the quality of the updates. Thoughts? 
So, we discussed this at the QA meeting a couple of weeks back.
Initially the plan was for anyone who had concerns to reply to this
mail, but that doesn't seem to have happened, so I will try to summarize
the various responses from that meeting. You can check the full summary
and log of the meeting:

https://fedoraproject.org/wiki/QA/Meetings/20111107

if you want to make sure I'm not misrepresenting anything.

adamw: 1-2% of all critpath updates (where proventester process is seen
to have 'had an impact') is a small number, but it is not 0. It only
takes *one* really bad update to cause big problems.

adamw: it's possible we could use proven tester feedback more
effectively: there have been cases where bad updates went out despite
negative feedback, because we currently don't give negative feedback -
even from proventesters - much significance at all. So there may have
been more cases where proventester input *may have been significant*
with a system where negative karma is more strongly considered. e.g.
updates could be blocked from being auto-pushed if there is any negative
feedback from a proven tester

red_alert (sandro mathys): critpath packages should have detailed test
plans, as currently proventesters often do not know exactly what they
need to verify with some critpath packages: "the process as its done
right now doesn't work (i.e. is no improvement to not having pts) but
having pts with a better process would surely add to the overall
quality"

tflink: we have to keep in mind the reason for the proposal -
maintainers frustrated that packages get stuck in updates-testing for
weeks - is a valid reason and a significant problem. "if we object to
getting rid of the proventester process, do we have any solutions to the
root of their complaints?"

brunowolff: how about keeping the PT karma requirement but adopting the
plan to allow critpath updates to go through without 'required' karma
after a two week wait

adamw: "i assume the stats made the assumption that the proventester
feedback on any update would still have been present but treated it no
different from regular feedback. so, that's not necessarily a safe
assumption: proventesters may feel a stronger 'responsibility to test',
and if you cancel the process, they might stop doing so. but that's hard
to gauge."

red_alert: proposes a meeting during FUDCon NA to try and come up with a
better pt process

That's about all the concrete thoughts / suggestions I can filter out of
the log.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Bruno Wolff III]

On Mon, Nov 21, 2011 at 10:32:06 -0800,
  Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  That's about all the concrete thoughts / suggestions I can filter
> out of
>  the log. 
I don't remember which meeting I noted it at, but another idea is to
weight proventester feedback higher (maybe count as +2 or -2) in the
short term (until we have something more than a running count).

[Nils Philippsen]

On Mon, 2011-11-21 at 10:32 -0800, Adam Williamson wrote:

>  red_alert (sandro mathys): critpath packages should have detailed
> test plans 
Hm. The list of (implicitly labeled) critpath packages seems to have
proliferated recently: a few days ago I submitted an update for
sane-backends and then found out that it's on critpath, probably by way
of critical-path-gnome -> control-center -> colord ->
sane-backends-libs. On the one hand it would probably be a good idea to
notify maintainers of such packages being implicitly pulled in (in this
case when BR: colord-devel was added to control-center). On the other
hand, sane-backends is a particularly nasty case and a detailed test
plan would probably start with this:

0. Have a lot of scanners handy, or at least models affected by the
changes.

Not even I get past that point in most cases: I have one USB scanner in
the office, and two rather ancient SCSI models at home. The only thing I
can meaningfully test is these models/their drivers and the
HW-independent parts of the package, and I don't expect more from the
testers. With the package being critpath now, I fear that updates for
sane-backends might never see the light of day, depending on the HW
affected, unless Bruno's proposal (two weeks of wait for critpath
without needed and no negative karma) or enough (proven)testers are
content with only cursory testing :-/.

Don't get me wrong: I'd really like sane-backends to get as much testing
as possible before turning updates loose on the unsuspecting public. I
just don't see how it can be done right at the moment.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Ken Dreyer]

On Tue, Nov 22, 2011 at 6:06 AM, Nils Philippsen <nils(a)redhat.com&gt; wrote:
>  Hm. The list of (implicitly labeled) critpath packages seems to have
>  proliferated recently 
Why not impose a self-restriction on the number of critpath packages?
Make a rule like "The ratio of proventesters to critpath packages must
be x or higher". When the critpath numbers start creeping up, drop a
few other packages from critpath. This will help focus testing
efforts, and allow it to expand along with the growth of the
community.

[Bill Nottingham]

Ken Dreyer (ktdreyer(a)ktdreyer.com) said: 
>  On Tue, Nov 22, 2011 at 6:06 AM, Nils Philippsen
> <nils(a)redhat.com&gt; wrote:
>  > Hm. The list of (implicitly labeled) critpath packages seems to have
>  > proliferated recently
>  
>  Why not impose a self-restriction on the number of critpath packages?
>  Make a rule like "The ratio of proventesters to critpath packages must
>  be x or higher". When the critpath numbers start creeping up, drop a
>  few other packages from critpath. This will help focus testing
>  efforts, and allow it to expand along with the growth of the
>  community. 
This then implies creation of a process that involves continual review of
the critpath package list, where it was designed to be able to be simply
computed without a lot of manual work.

Bill

[Nils Philippsen]

On Tue, 2011-11-22 at 07:42 -0700, Ken Dreyer wrote:
>  On Tue, Nov 22, 2011 at 6:06 AM, Nils Philippsen
> <nils(a)redhat.com&gt; wrote:
>  > Hm. The list of (implicitly labeled) critpath packages seems to have
>  > proliferated recently
>  
>  Why not impose a self-restriction on the number of critpath packages?
>  Make a rule like "The ratio of proventesters to critpath packages must
>  be x or higher". When the critpath numbers start creeping up, drop a
>  few other packages from critpath. This will help focus testing
>  efforts, and allow it to expand along with the growth of the
>  community. 
But that wouldn't really help, would it? I assume packages are
explicitly labeled critpath for a reason: if they break, the system
might not boot up, or users might not be able to log in (or several
other reasons).

I think the current system is a bit too simplistic when it comes to the
packages upon which critpath components depend, however. For instance
(picking this example because it affects me), I don't think that logging
in should fail because of problems the scanner library may have. Or
because (drawing examples from a hat) the desktop background can't be
set, or there's an issue with the camera in your laptop. Yet, by way of
simply following package dependencies, we do as if that were the case.

Nils
-- 
Nils Philippsen      "Those who would give up Essential Liberty to purchase 
Red Hat               a little Temporary Safety, deserve neither Liberty
nils(a)redhat.com       nor Safety."  --  Benjamin Franklin, 1759
PGP fingerprint:      C4A8 9474 5C4C ADE3 2B8F  656D 47D8 9B65 6951 3011

[Adam Williamson]

On Tue, 2011-11-22 at 17:04 +0100, Nils Philippsen wrote:

>  (picking this example because it affects me), I don't think that
> logging
>  in should fail because of problems the scanner library may have. Or 
Well, you'd think so, but reality frequently offends. ;)

There is, for instance, a bug in Fedora 16 which results in GNOME
failing to start if colord can't read a color profile in the user's home
directory due to it being incorrectly SELinux-labelled.

You might think this comes down to idiotic coding on the part of the
GNOME team, but actually it turns out to be rather more complex than
that. I'm too dumb to recap it accurately, but essentially, it's not
something that can actually be very easily fixed, though you'd think
it'd be easy to make sure the entirety of GNOME doesn't fall over
because of such a trivial sub-function, it actually isn't.

The general statement of this is 'it is sometimes the case that things
you'd never imagine could possibly break really critical functions, do
break really critical functions'.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Adam Williamson]

On Tue, 2011-11-22 at 14:06 +0100, Nils Philippsen wrote:
>  On Mon, 2011-11-21 at 10:32 -0800, Adam Williamson wrote:
>  
>  > red_alert (sandro mathys): critpath packages should have detailed test plans
>  
>  Hm. The list of (implicitly labeled) critpath packages seems to have
>  proliferated recently: a few days ago I submitted an update for
>  sane-backends and then found out that it's on critpath, probably by way
>  of critical-path-gnome -> control-center -> colord ->
>  sane-backends-libs. On the one hand it would probably be a good idea to
>  notify maintainers of such packages being implicitly pulled in (in this
>  case when BR: colord-devel was added to control-center). On the other
>  hand, sane-backends is a particularly nasty case and a detailed test
>  plan would probably start with this:
>  
>  0. Have a lot of scanners handy, or at least models affected by the
>  changes.
>  
>  Not even I get past that point in most cases: I have one USB scanner in
>  the office, and two rather ancient SCSI models at home. The only thing I
>  can meaningfully test is these models/their drivers and the
>  HW-independent parts of the package, and I don't expect more from the
>  testers. With the package being critpath now, I fear that updates for
>  sane-backends might never see the light of day, depending on the HW
>  affected, unless Bruno's proposal (two weeks of wait for critpath
>  without needed and no negative karma) or enough (proven)testers are
>  content with only cursory testing :-/.
>  
>  Don't get me wrong: I'd really like sane-backends to get as much testing
>  as possible before turning updates loose on the unsuspecting public. I
>  just don't see how it can be done right at the moment. 
This touches on a couple of issues. I know part of FESCo's response to
the proventesters question was to look at ways to slim down critpath.

It also hits the question of things which are critpath indirectly (via
dependencies). Part of the idea of having a test plan for all components
identified as critpath is to help identify those which are indirect
critpath packages, and make it clear what needs to be the case for those
packages to pass critpath validation - usually only that they don't
cause the actual critical path function to fail. In the sane_backends
case, for instance, in order to validate the critical path, you only
have to check the update doesn't somehow stop GNOME (and possibly
control-center) running. You don't have to make sure it doesn't break
scanning.

Providing a test plan is actually a good way to do this, if you take a
look at
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_pl... and
https://fedoraproject.org/wiki/QA:SOP_test_case_creation : with the scheme explained there
for categorizing test cases, you can both associate test cases with packages, and mark
them as critical path or not.

So the 'intended' way to do things in this case would be to write a
bunch of test cases that actually test sane-backends, and put them in
the 'Category:Package_sane-backends_test_cases' category - but *not* to
put them in the 'Category:Critical_path_test_cases' category. You would
also add an existing test case for basic GNOME functionality - say,
https://fedoraproject.org/wiki/QA:Testcase_base_startup - to the
'Category:Package_sane-backends_test_cases' category, and make sure it
was in the 'Category:Critical_path_test_cases' category too.

The result of that would be that, when a sane-backends update appeared,
the Bodhi page for it would list all the sane-backends test cases and
also the 'base_startup' test case, and identify the 'base_startup' test
case as being a critical path test case - thus indicating that the way
to really validate whether sane-backends *works* is to do the detailed,
specific tests, but the way to validate whether it breaks the critical
path is just to see if the system boots to a working desktop.

Then you'd need Bodhi 2.0 so you'd be able to provide feedback of "This
update doesn't break critical path" or "This update actually works for
scanning", rather than simply numeric feedback, as is the case now.

This may be a bit complex, I admit, and we haven't really completed the
'intended design' for many packages to show how it's supposed to work.
(That 'base_startup' test case should be associated with just about
every critpath package in existence, for instance, and it should be in
the critical path category too, which it currently isn't). But all the
bits are more or less there already, and just need hooking up. And we
need Bodhi 2.0 (yes, I know that's my catchphrase).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net