Package-specific test case and critical path test case project: drafts for review

[James Laska]

On Tue, 2010-12-21 at 15:53 -0700, Kevin Fenzi wrote:
+AD4 On Tue, 21 Dec 2010 17:11:15 +-0000
+AD4 Adam Williamson +ADw-awilliam+AEA-redhat.com+AD4 wrote:
+AD4 
+AD4 +AD4 Hi, everyone. So, in the recent debate about the update process it
+AD4 +AD4 again became clear that we were lacking a good process for providing
+AD4 +AD4 package-specific test instructions, and particularly specific
+AD4 +AD4 instructions for testing critical path functions.
+AD4 
+AD4 ...snip...
+AD4 
+AD4 +AD4 Comments, suggestions and rotten fruit welcome :) I'm particularly
+AD4 +AD4 interested in feedback from package maintainers and QA contributors in
+AD4 +AD4 whether you feel, just after reading these pages, that you'd be
+AD4 +AD4 confident in going ahead and creating some test cases, or if there's
+AD4 +AD4 stuff that's scary or badly explained or that you feel like something
+AD4 +AD4 is missing and you wouldn't know where to start, etc.
+AD4 
+AD4 ...snip...
+AD4 
+AD4 Great work Adam and QA folks. +ADs) 
+AD4 
+AD4 From a quick glance over this looks good to me, and I would be happy to
+AD4 start trying to make test cases. A few random things: 
+AD4 
+AD4 +ACo Would it be worth noting that anyone can make a test case, it doesn't
+AD4   need to be the maintainer, right? 

Oh that raises another good point.  Unlike the Test+AF8-Results: wiki name
space used for Test Days and similar test events, the QA: namespace does
not allow anonymous edits.

+AD4 +ACo Also, might be worth noting that if you run into a specific bug as a
+AD4   maintainer and fix it, thats a great time to go add a test case to
+AD4   specifically test that (since it's fresh in your mind). 

Great point+ACE  I've also seen folks use placeholders in bugs when
triaging or fixing them.  For example, there are bugzilla keywords that
can be used to tag bugs for future test case review, such as
'NeedsTestCase' (see https://bugzilla.redhat.com/describekeywords.cgi).

+AD4 +ACo finally, it's ok to just start in on some tests and add them over
+AD4   time, right? We don't want to care if a package has incomplete
+AD4   coverage right off the bat right? 
+AD4 
+AD4 +AD4 
+AD4 +AD4 it also mentions one big current omission: dependencies. For instance,
+AD4 +AD4 it would be very useful to be able to express 'when yum is updated, we
+AD4 +AD4 should also run the PackageKit test plan' (because it's possible that
+AD4 +AD4 a change in yum could be fine 'within itself', and all the yum test
+AD4 +AD4 cases pass, but could break PackageKit). That's rather complex,
+AD4 +AD4 though, especially with a Wiki-based system. If anyone has any bright
+AD4 +AD4 ideas on how to achieve this, do chip in+ACE Thanks.
+AD4 
+AD4 Yeah, tough one. Not sure how best to handle that. Perhaps just a
+AD4 'Dependencies:' type header asking you to make sure you test dependent
+AD4 packages and see their test cases? 

It occurred to me that transclusion might work in a pinch
(http://www.mediawiki.org/wiki/Help:Transclusion).  It can be
non-obvious, but it might allow us to include PackageKit test cases in
the yum test case category by doing
+AHsAew-Category:Package+AF8-PackageKit+AF8-test+AF8-cases+AH0AfQ.  I've never tried this with
Categories, so not sure if this works.  I'll have to test this out.

+AD4 Thanks again for working on this+ACE

Agreed+ACE  Great to see this coming together.

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application+AC8-pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http:+AC8ALw-lists.fedoraproject.org+AC8-pipermail+AC8-test+AC8-attachments+AC8-20101221+AC8-5fa7751b+AC8-attachment.bin