Update testing policy: how to use Bodhi

[Till Maas]

On Fri, Mar 26, 2010 at 03:49:28PM -0700, Adam Williamson wrote:

> 1. I have tried this update in my regular day-to-day use and seen no
> regressions.
> 
> 2. I have tried this update in my regular day-to-day use and seen a
> regression: bug #XXXXXX.
> 
> 3. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does fix bug #XXXXXX.
> 
> 4. (Where the update claims to fix bug #XXXXXX) I have tried this update
> and found that it does not fix bug #XXXXXX.
> 
> 5. I have performed the following planned testing on the update: (link
> to test case / test plan) and it passes.
> 
> 6. I have performed the following planned testing on the update: (link
> to test case / test plan) and it fails: bug #XXXXXX.

I have some additions:

7. fixes bug X, but does not claim to fix it

This can often happen with hardware related bugs, e.g. with the kernel
where something starts to work again

8. The package updated sucessfully, but was not used intentionally. No
breakage noticed.

This shows, that at least on the test machine, there are no broken deps,
conflicts or broken scriptlets.

Also it would be nice to provide hardware testing feedback, e.g. for Xorg
updates to say "Works with nouveau, Geforce XY, using VGA out and XV",
which then shows that e.g. 3D support, DVI out or multi screen support
was not tested. This is kind of related to testing with a test plan, but
having this data available in a format that can be easily parsed, would
be nice, too. Maybe this could be done with adding smolt information in
the feedback and the tested features (XV, VGA, DVI, 3D, ...) and the
update needs to have some meta data, which kind of devices are supported
(e.g. only Geforce devices for the nouveau driver package).

Regards
Till
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20100327/f95d6f7c/attachment.bin