On Tue, 21 Dec 2010 17:11:15 +0000
Adam Williamson <awilliam(a)redhat.com> wrote:
Hi, everyone. So, in the recent debate about the update process it
again became clear that we were lacking a good process for providing
package-specific test instructions, and particularly specific
instructions for testing critical path functions.
...snip...
Comments, suggestions and rotten fruit welcome :) I'm
particularly
interested in feedback from package maintainers and QA contributors in
whether you feel, just after reading these pages, that you'd be
confident in going ahead and creating some test cases, or if there's
stuff that's scary or badly explained or that you feel like something
is missing and you wouldn't know where to start, etc.
...snip...
Great work Adam and QA folks. ;)
From a quick glance over this looks good to me, and I would be happy to
start trying to make test cases. A few random things:
* Would it be worth noting that anyone can make a test case, it doesn't
need to be the maintainer, right?
* Also, might be worth noting that if you run into a specific bug as a
maintainer and fix it, thats a great time to go add a test case to
specifically test that (since it's fresh in your mind).
* finally, it's ok to just start in on some tests and add them over
time, right? We don't want to care if a package has incomplete
coverage right off the bat right?
it also mentions one big current omission: dependencies. For instance,
it would be very useful to be able to express 'when yum is updated, we
should also run the PackageKit test plan' (because it's possible that
a change in yum could be fine 'within itself', and all the yum test
cases pass, but could break PackageKit). That's rather complex,
though, especially with a Wiki-based system. If anyone has any bright
ideas on how to achieve this, do chip in! Thanks.
Yeah, tough one. Not sure how best to handle that. Perhaps just a
'Dependencies:' type header asking you to make sure you test dependent
packages and see their test cases?
Thanks again for working on this!
kevin