Package-specific test case and critical path test case project: drafts for review

James Laska jlaska at redhat.com
Tue Dec 21 23:12:47 UTC 2010


Thanks Adam for getting the ball rolling on this topic.

On Tue, 2010-12-21 at 17:11 +-0000, Adam Williamson wrote: 
+AD4 Hi, everyone. So, in the recent debate about the update process it again
+AD4 became clear that we were lacking a good process for providing
+AD4 package-specific test instructions, and particularly specific
+AD4 instructions for testing critical path functions.
+AD4 
+AD4 I've been working on a process for this, and now have two draft Wiki
+AD4 pages up for review which together describe it:
+AD4 
+AD4 https://fedoraproject.org/wiki/User:Adamwill/Draft+AF8-QA+AF8-SOP+AF8-test+AF8-case+AF8-creation
+AD4 https://fedoraproject.org/wiki/User:Adamwill/Draft+AF8-QA+AF8-SOP+AF8-package+AF8-test+AF8-plan+AF8-creation
+AD4 
+AD4 the first isn't particularly specific to this, but it was a prerequisite
+AD4 that I discovered was missing: it's a guide to test case creation in
+AD4 general, explaining the actual practical process of how you create a
+AD4 test case, and the best principles to consider in doing it.

Nice job here, this is something that's difficult to explain if you've
done it a lot, but I think you've captured the key points.  If possible,
it might be helpful to highlight a few existing examples that stand out
for the different characteristics you mention (comprehensive, but able
to stand the test of time).

Another thought, any reason that we wouldn't want to keep all wiki tests
in the QA: namespace (and with the prefix QA:Testcase+AF8)?  The door is
left open for other names, I wonder if we want to cut that off ahead of
time to keep our sanity by having all tests in the same namespace?

The page also talks about using +AFsAWw-Category:Test+AF8-Cases+AF0AXQ.  I worry if we
are too lax in categorizing new tests we'll end up with a large amount
of random tests in the main +AFsAWw-Category:Test+AF8-Cases+AF0AXQ making it a
maintenance nightmare to cleanup that category.  Should we instead
direct users to your other page
(https://fedoraproject.org/wiki/User:Adamwill/Draft+AF8-QA+AF8-SOP+AF8-package+AF8-test+AF8-plan+AF8-creation) for guidance on categorizing test cases?

+AD4 The second is what's really specific to this subject. It describes how
+AD4 to create a set of test cases for a particular package, and a proposed
+AD4 standardized categorization scheme which will allow us to denote test
+AD4 cases as being associated with specific packages, and also denote them
+AD4 as concerning critical path functionality.

I think I mentioned this previously, in the section 'Preparation', I
appreciate the distinction of 'core' and 'extended'.  But I it resonates
with me better under the context of test +ACI-priority+ACI.  I don't see why we
can't keep using the terms 'core' and 'extended', but just want to
clarify their purpose.  They're intended to add some sense of execution
priority to a list of test cases, right?  Where critpath comes first,
then core, then extended, then other?  Also, you describe
categorizing/grouping test cases in more detail below, maybe just link
to that instead?

In the section, 'Simple (required)', would it help to add a link to
http://fedoraproject.org/wiki/BugZappers/CorrectComponent+ACM-Which+AF8-component+AF8-is+AF8-it.3F (or similar page).  Something to help testers find the right src.rpm name of the component under test?  Side note, this might also be a maintenance task we can define where I, or anyone interested, could manually scrub (or script) finding Categories:Test+AF8-Cases searching incorrectly named category pages.

Also in 'Simple (required)', we don't tell the author to add their
'Category:Package+AF8AJAB7-sourcename+AH0AXw-test+AF8-cases' to 'Category:Test+AF8-Cases'.  I
think we want all newly created package categories anchored under
'Category:Test+AF8-Cases'.

General comment.  I know we've got an eye towards integrating this work
with bodhi and/or f-e-k.  Until that work is complete, I wonder if those
notes will introduce confusion/speculation.  Should we leave out the
bits about possible future tool integration until such support is
active?

+AD4 Given that mediawiki has a handy API which also allows you to deal with
+AD4 categories, this should make it easy to both manually and
+AD4 programmatically derive a list of test cases for a given package, and a
+AD4 list of +ACo-critical path+ACo test cases for a given package. You can do this
+AD4 manually, but I also envision Bodhi and fedora-easy-karma utilizing the
+AD4 API so that when an update is pushed for a package for which test cases
+AD4 have been created under this system, they will link to those test cases+ADs
+AD4 and when an update is pushed for a critical path package, they will be
+AD4 able to display separately (and more prominently, perhaps) the list of
+AD4 test cases relevant to the critical path functionality of the package.

I should add too that we've explored the mediawiki remote API used to
extract data from the wiki by other scripts/tools, and are confident
that the desired queries, and data, are available. 

+AD4 Comments, suggestions and rotten fruit welcome :) I'm particularly
+AD4 interested in feedback from package maintainers and QA contributors in
+AD4 whether you feel, just after reading these pages, that you'd be
+AD4 confident in going ahead and creating some test cases, or if there's
+AD4 stuff that's scary or badly explained or that you feel like something is
+AD4 missing and you wouldn't know where to start, etc.

Agreed, would love to hear from others.

+AD4 The trac ticket on this is probably valuable for background, explaining
+AD4 why some things in the proposal are the way they are:
+AD4 
+AD4 https://fedorahosted.org/fedora-qa/ticket/154
+AD4 
+AD4 it also mentions one big current omission: dependencies. For instance,
+AD4 it would be very useful to be able to express 'when yum is updated, we
+AD4 should also run the PackageKit test plan' (because it's possible that a
+AD4 change in yum could be fine 'within itself', and all the yum test cases
+AD4 pass, but could break PackageKit). That's rather complex, though,
+AD4 especially with a Wiki-based system. If anyone has any bright ideas on
+AD4 how to achieve this, do chip in+ACE Thanks.

Certainly seems like a feature worth noting on the TCMS requirements
page Hurry has been building
(https://fedoraproject.org/wiki/Rhe/tcms+AF8-requirements+AF8-proposal).  I've
added that to the Talk page.

Thanks,
James 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application+AC8-pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http:+AC8ALw-lists.fedoraproject.org+AC8-pipermail+AC8-test+AC8-attachments+AC8-20101221+AC8-886e1605+AC8-attachment.bin 


More information about the test mailing list