Update testing policy: how to use Bodhi

[Adam Williamson]

Hi, folks. At the last QA meeting, I volunteered (dumb of me!) to draft
a policy for testing updates - basically, a policy for what kind of
feedback should be posted in Bodhi for candidate updates.

This turns out to be pretty hard. =) Thinking about it from an
high-level perspective like this, I think it becomes pretty clear that
the current system is just broken.

The major problem is it attempts to balance things that don't really
balance. It lets you say 'works for me' or 'doesn't work' and then
sums
the two and subtracts the second from the first to give you a 'rating'
for the update.

This doesn't really mean anything. As has been rehashed many times,
there are situations where an update with a positive rating shouldn't go
out, and situations where an update with a negative rating should. So
the current system isn't really that great.

I can't think of a way to draft a policy to guide the use of the current
system in such a way that it will be really reliable. I think it'd be
much more productive to revise the Bodhi feedback system alongside
producing a policy.

So, here's a summary of what the new system should aim for. 

At the high level, what is this system for? It's there for three
purposes:

1) to provide maintainers with information they can use in deciding
whether to push updates.

2) to provide a mechanism for mandating a certain minimum level of
manual testing for 'important' packages, under Bill Nottingham's current
update acceptance criteria proposal.

3) to provide an 'audit trail' we can use to look back on how the
release of a particular update was handled, in the case where there are
problems.

Given the above, we need to capture the following types of feedback, as
far as I can tell. I don't think there is any sensible way to assign
numeric values to any of this feedback. I think we have to trust people
to make sensible decisions as long as it's provided, in accordance with
any policy we decide to implement on what character updates should have.

1. I have tried this update in my regular day-to-day use and seen no
regressions.

2. I have tried this update in my regular day-to-day use and seen a
regression: bug #XXXXXX.

3. (Where the update claims to fix bug #XXXXXX) I have tried this update
and found that it does fix bug #XXXXXX.

4. (Where the update claims to fix bug #XXXXXX) I have tried this update
and found that it does not fix bug #XXXXXX.

5. I have performed the following planned testing on the update: (link
to test case / test plan) and it passes.

6. I have performed the following planned testing on the update: (link
to test case / test plan) and it fails: bug #XXXXXX.

Testers should be able to file multiple types of feedback in one
operation - for instance, 4+1 (the update didn't fix the bug it claimed
to, but doesn't seem to cause any regressions either). Ideally, the
input of feedback should be 'guided' with a freeform element, so there's
a space to enter bug numbers, for instance.

There is one type of feedback we don't really want or need to capture:
"I have tried this update and it doesn't fix bug #XXXXXX", where the
update doesn't claim to fix that bug. This is a quite common '-1' in the
current system, and one we should eliminate.

I think Bill's proposed policy can be modified quite easily to fit this.
All it would need to say is that for 'important' updates to be accepted,
they would need to have one 'type 1' feedback from a proven tester, and
no 'type 2' feedback from anyone (or something along those lines; this
isn't the main thrust of my post, please don't sidetrack it too
much :>).

The system could do a count of how many of each type of feedback any
given update has received, but I don't think there's any way we can
sensibly do some kind of mathematical operation on those numbers and
have a 'rating' for the update. Such a system would always give odd /
undesirable results in some cases, I think (just as the current one
does). I believe the above system would be sufficiently clear that there
would be no need for such a number, and we would be able to evaluate
updates properly based just on the information listed.

What are everyone's thoughts on this? Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[Mathieu Bridon]

Hi,

On Fri, Mar 26, 2010 at 23:49, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
[... snip ...]
>  1. I have tried this update in my regular day-to-day use and seen no
>  regressions.
>
>  2. I have tried this update in my regular day-to-day use and seen a
>  regression: bug #XXXXXX.
>
>  3. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  and found that it does fix bug #XXXXXX.
>
>  4. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  and found that it does not fix bug #XXXXXX.
>
>  5. I have performed the following planned testing on the update: (link
>  to test case / test plan) and it passes.
>
>  6. I have performed the following planned testing on the update: (link
>  to test case / test plan) and it fails: bug #XXXXXX. 
This is basically what Doug had proposed, except that you added 5. and 6.

I know Luke also likes the idea, and I've been toying with
implementing Doug's idea in the Bodhi2 branch. Adding those 2 more
karma types shouldn't be too hard.

> Testers should be able to file multiple types of feedback in one
>  operation - for instance, 4+1 (the update didn't fix the bug it claimed
>  to, but doesn't seem to cause any regressions either). Ideally, the
>  input of feedback should be 'guided' with a freeform element, so there's
>  a space to enter bug numbers, for instance. 
That's what I had in mind, but the Bodhi2 branch currently doesn't
have any controllers/template, so it will be some time before I have
something to show.

>  I think Bill's proposed policy can be modified quite easily to
> fit this.
>  All it would need to say is that for 'important' updates to be accepted,
>  they would need to have one 'type 1' feedback from a proven tester, and
>  no 'type 2' feedback from anyone (or something along those lines; this
>  isn't the main thrust of my post, please don't sidetrack it too
>  much :>). 
That's actually one of the points I was missing : rules for
(auto-)push / (auto-)unpush. But yeah, it can be agreed on later.

>  The system could do a count of how many of each type of feedback any
>  given update has received, but I don't think there's any way we can
>  sensibly do some kind of mathematical operation on those numbers and
>  have a 'rating' for the update. Such a system would always give odd /
>  undesirable results in some cases, I think (just as the current one
>  does). I believe the above system would be sufficiently clear that there
>  would be no need for such a number, and we would be able to evaluate
>  updates properly based just on the information listed.
>
>  What are everyone's thoughts on this? Thanks! 
I totally agree that this would be far more desirable than the current
+1/-1 system.


----------
Mathieu Bridon

[Adam Williamson]

On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
>  Hi,
>  
>  On Fri, Mar 26, 2010 at 23:49, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  [... snip ...]
>  > 1. I have tried this update in my regular day-to-day use and seen no
>  > regressions.
>  >
>  > 2. I have tried this update in my regular day-to-day use and seen a
>  > regression: bug #XXXXXX.
>  >
>  > 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  > and found that it does fix bug #XXXXXX.
>  >
>  > 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  > and found that it does not fix bug #XXXXXX.
>  >
>  > 5. I have performed the following planned testing on the update: (link
>  > to test case / test plan) and it passes.
>  >
>  > 6. I have performed the following planned testing on the update: (link
>  > to test case / test plan) and it fails: bug #XXXXXX.
>  
>  This is basically what Doug had proposed, except that you added 5. and 6. 
Great, glad to hear we're thinking in the same direction from different
angles :) Do you have a link to his proposal? I don't recall reading it.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[Mathieu Bridon]

On Sat, Mar 27, 2010 at 00:31, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
> > This is basically what Doug had proposed, except that you added 5. and 6.
>
>  Great, glad to hear we're thinking in the same direction from different
>  angles :) Do you have a link to his proposal? I don't recall reading it. 
Here it is:
   http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html


----------
Mathieu Bridon

[Adam Williamson]

On Sat, 2010-03-27 at 00:35 +0100, Mathieu Bridon wrote:
>  On Sat, Mar 27, 2010 at 00:31, Adam Williamson
> <awilliam(a)redhat.com&gt; wrote:
>  > On Sat, 2010-03-27 at 00:11 +0100, Mathieu Bridon wrote:
>  >> This is basically what Doug had proposed, except that you added 5. and 6.
>  >
>  > Great, glad to hear we're thinking in the same direction from different
>  > angles :) Do you have a link to his proposal? I don't recall reading it.
>  
>  Here it is:
>     http://lists.fedoraproject.org/pipermail/devel/2010-March/131799.html 
Oh, yeah, that one. I even replied to it, but it obviously fell prey to
my mind-erase of the whole mess of a thread =)

in that case, yep, my proposal is more or less just a slight elaboration
on Doug's indeed. I may even have subconsciously brought in some
elements from Doug's when writing it, so thanks Doug!

if you're working in this direction in improving Bodhi, then, from the
'update testing policy' angle, I think I'll just leave it at 'wait and
see' for the new Bodhi version. Do you have a vague ETA on when we can
expect the improved Bodhi?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[Till Maas]

On Fri, Mar 26, 2010 at 03:49:28PM -0700, Adam Williamson wrote:

>  1. I have tried this update in my regular day-to-day use and seen no
>  regressions.
>  
>  2. I have tried this update in my regular day-to-day use and seen a
>  regression: bug #XXXXXX.
>  
>  3. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  and found that it does fix bug #XXXXXX.
>  
>  4. (Where the update claims to fix bug #XXXXXX) I have tried this update
>  and found that it does not fix bug #XXXXXX.
>  
>  5. I have performed the following planned testing on the update: (link
>  to test case / test plan) and it passes.
>  
>  6. I have performed the following planned testing on the update: (link
>  to test case / test plan) and it fails: bug #XXXXXX. 
I have some additions:

7. fixes bug X, but does not claim to fix it

This can often happen with hardware related bugs, e.g. with the kernel
where something starts to work again

8. The package updated sucessfully, but was not used intentionally. No
breakage noticed.

This shows, that at least on the test machine, there are no broken deps,
conflicts or broken scriptlets.

Also it would be nice to provide hardware testing feedback, e.g. for Xorg
updates to say "Works with nouveau, Geforce XY, using VGA out and XV",
which then shows that e.g. 3D support, DVI out or multi screen support
was not tested. This is kind of related to testing with a test plan, but
having this data available in a format that can be easily parsed, would
be nice, too. Maybe this could be done with adding smolt information in
the feedback and the tested features (XV, VGA, DVI, 3D, ...) and the
update needs to have some meta data, which kind of devices are supported
(e.g. only Geforce devices for the nouveau driver package).

Regards
Till

[Matthew Woehlke]

Till Maas wrote:
>  Also it would be nice to provide hardware testing feedback, e.g. for
> Xorg
>  updates to say "Works with nouveau, Geforce XY, using VGA out and XV",
>  which then shows that e.g. 3D support, DVI out or multi screen support
>  was not tested. 
That sounds like a social problem, or at least just the reason to not 
kill free-form comments. Maybe it would help to have a free-form comment 
field specifically asking the submitter to describe the testing they did 
and/or any details of their setup that seem relevant.

-- 
Matthew
Please do not quote my e-mail address unobfuscated in message bodies.
-- 
Microsoft has become the next IBM; a dinosaur struggling to survive in 
the age of more able-to-adapt mammals (FLOSS). It remains to be seen if 
they'll be able to adapt before they go extinct.

[Adam Williamson]

On Sat, 2010-03-27 at 16:33 +0100, Till Maas wrote:

>  I have some additions:
>  
>  7. fixes bug X, but does not claim to fix it
>  
>  This can often happen with hardware related bugs, e.g. with the kernel
>  where something starts to work again 
Oh, yes, I like that. Then the update text could be updated to claim a
fix for that bug, is what you're thinking, right?

>  8. The package updated sucessfully, but was not used intentionally.
> No
>  breakage noticed.
>  
>  This shows, that at least on the test machine, there are no broken deps,
>  conflicts or broken scriptlets. 
In my head I sort of had this wrapped up with 'no regressions', but you
might be right that it's best to split it out as a choice.

>  Also it would be nice to provide hardware testing feedback, e.g. for
> Xorg
>  updates to say "Works with nouveau, Geforce XY, using VGA out and XV",
>  which then shows that e.g. 3D support, DVI out or multi screen support
>  was not tested. This is kind of related to testing with a test plan, but
>  having this data available in a format that can be easily parsed, would
>  be nice, too. Maybe this could be done with adding smolt information in
>  the feedback and the tested features (XV, VGA, DVI, 3D, ...) and the
>  update needs to have some meta data, which kind of devices are supported
>  (e.g. only Geforce devices for the nouveau driver package). 
I sort of agree, but it's quite a complex topic and probably would
require some serious design and thinking about of its own, so I was
kinda leaving it out at this stage :)

The infrastructure team is already working along these lines following
Doug's earlier proposal, BTW, so I hope they'll pick up your
suggestions.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[Mathieu Bridon]

On Wed, Mar 31, 2010 at 01:21, Adam Williamson <awilliam(a)redhat.com&gt; wrote:
>  On Sat, 2010-03-27 at 16:33 +0100, Till Maas wrote:
> > 8. The package updated sucessfully, but was not used intentionally. No
> > breakage noticed.
> >
> > This shows, that at least on the test machine, there are no broken deps,
> > conflicts or broken scriptlets.
>
>  In my head I sort of had this wrapped up with 'no regressions', but you
>  might be right that it's best to split it out as a choice. 
I'm not convinced by the value of this one. IMHO, testing is supposed
to be a conscious action. You don't test "unintentionally", and if you
installed an update unintentionally, it means (to me) that you didn't
test it. I guess it's related to the issue of "do we want to only
provide updates that add value or do we want to push any updates that
don't break anything"?

In any case, like I said I've started playing with the idea in the
Bodhi tg2 branch (no ETA yet, I do that on my spare time and I have
very few, and lots of other stuff have to be written in the tg2 branch
anyway). The way I did it, several karma types can be defined (and I'm
taking suggestions on those types from these emails), the only thing
that matters is: are they global to the update or are they tied to a
specific bug? That's what will be most important when displaying the
comment form.

Until now, the only proposed thing that wouldn't really fit is number
7: fixes bug X, but does not claim to fix it. It could probably be a
special case of the "karma type tied to a specific bug", except the
bug is not known in advance but rather specified by the tester. Gotta
think about it. :)


----------
Mathieu Bridon

[Till Maas]

On Wed, Mar 31, 2010 at 11:37:33AM +0200, Mathieu Bridon wrote:
>  On Wed, Mar 31, 2010 at 01:21, Adam Williamson
> <awilliam(a)redhat.com&gt; wrote:
>  > On Sat, 2010-03-27 at 16:33 +0100, Till Maas wrote:
>  >> 8. The package updated sucessfully, but was not used intentionally. No
>  >> breakage noticed.
>  >>
>  >> This shows, that at least on the test machine, there are no broken deps,
>  >> conflicts or broken scriptlets.
>  >
>  > In my head I sort of had this wrapped up with 'no regressions', but you
>  > might be right that it's best to split it out as a choice.
>  
>  I'm not convinced by the value of this one. IMHO, testing is supposed
>  to be a conscious action. You don't test "unintentionally", and if you
>  installed an update unintentionally, it means (to me) that you didn't
>  test it. I guess it's related to the issue of "do we want to only
>  provide updates that add value or do we want to push any updates that
>  don't break anything"? 
IMHO the question what updates to provide is not related to this. Even
if an update is there to fix something, it does not mean that one can or
will test it completely (special hardware might be required). In this
case it is still interesting to know, whether it installs cleanly or
not. And testing whether it updates cleanly can still be done
intentionally, even when the package is not used. Just taking a look at
the output of "yum --enablerepo=*-testing update" is enough for this.

Also there are packages like libraries, that will probably never be
intentionally manually tested, because it is not that obvious when a
library is used by another program and which code paths are used.

Regards
Till

[Björn Persson]

Till Maas wrote:
>  Even
>  if an update is there to fix something, it does not mean that one can or
>  will test it completely (special hardware might be required). In this
>  case it is still interesting to know, whether it installs cleanly or
>  not. And testing whether it updates cleanly can still be done
>  intentionally, even when the package is not used. Just taking a look at
>  the output of "yum --enablerepo=*-testing update" is enough for this. 
I thought the plan was to have AutoQA verify that packages install cleanly. 
Isn't it better to automate such simple checks and save human resources for 
the things that can't be automated? Or do you think even just installing 
updates is too complex to be tested automatically?

Björn Persson

[Till Maas]

On Wed, Mar 31, 2010 at 01:13:18PM +0200, Björn Persson wrote:
>  Till Maas wrote:
>  > Even
>  > if an update is there to fix something, it does not mean that one can or
>  > will test it completely (special hardware might be required). In this
>  > case it is still interesting to know, whether it installs cleanly or
>  > not. And testing whether it updates cleanly can still be done
>  > intentionally, even when the package is not used. Just taking a look at
>  > the output of "yum --enablerepo=*-testing update" is enough for this.
>  
>  I thought the plan was to have AutoQA verify that packages install cleanly. 
>  Isn't it better to automate such simple checks and save human resources for 
>  the things that can't be automated? Or do you think even just installing 
>  updates is too complex to be tested automatically? 
I agree, that automation is the way to do this, but still it might not
catch everything and there is not much effort required to do this
testing (just look at the yum output) and report it. But there are also
updates that might interfer with RPMFusion, like it happened with
gstreamer. Here AutoQA might not be able to test this, because of legal
issues.

Regards
Till