Package-specific test case and critical path test case project: drafts for review

[Kevin Fenzi]

On Tue, 21 Dec 2010 17:11:15 +0000
Adam Williamson <awilliam at redhat.com> wrote:

> Hi, everyone. So, in the recent debate about the update process it
> again became clear that we were lacking a good process for providing
> package-specific test instructions, and particularly specific
> instructions for testing critical path functions.

...snip...

> Comments, suggestions and rotten fruit welcome :) I'm particularly
> interested in feedback from package maintainers and QA contributors in
> whether you feel, just after reading these pages, that you'd be
> confident in going ahead and creating some test cases, or if there's
> stuff that's scary or badly explained or that you feel like something
> is missing and you wouldn't know where to start, etc.

...snip...

Great work Adam and QA folks. ;) 

From a quick glance over this looks good to me, and I would be happy to
start trying to make test cases. A few random things: 

* Would it be worth noting that anyone can make a test case, it doesn't
  need to be the maintainer, right? 

* Also, might be worth noting that if you run into a specific bug as a
  maintainer and fix it, thats a great time to go add a test case to
  specifically test that (since it's fresh in your mind). 

* finally, it's ok to just start in on some tests and add them over
  time, right? We don't want to care if a package has incomplete
  coverage right off the bat right? 

> 
> it also mentions one big current omission: dependencies. For instance,
> it would be very useful to be able to express 'when yum is updated, we
> should also run the PackageKit test plan' (because it's possible that
> a change in yum could be fine 'within itself', and all the yum test
> cases pass, but could break PackageKit). That's rather complex,
> though, especially with a Wiki-based system. If anyone has any bright
> ideas on how to achieve this, do chip in! Thanks.

Yeah, tough one. Not sure how best to handle that. Perhaps just a
'Dependencies:' type header asking you to make sure you test dependent
packages and see their test cases? 

Thanks again for working on this!

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/devel/attachments/20101221/040a5ea3/attachment.bin