Package-specific test case and critical path test case project: drafts for review

[James Laska]

On Wed, 2010-12-22 at 17:29 +-0000, Adam Williamson wrote:
+AD4 On Tue, 2010-12-21 at 18:12 -0500, James Laska wrote:
+AD4 
+AD4 +AD4 +AD4 the first isn't particularly specific to this, but it was a prerequisite
+AD4 +AD4 +AD4 that I discovered was missing: it's a guide to test case creation in
+AD4 +AD4 +AD4 general, explaining the actual practical process of how you create a
+AD4 +AD4 +AD4 test case, and the best principles to consider in doing it.
+AD4 +AD4 
+AD4 +AD4 Nice job here, this is something that's difficult to explain if you've
+AD4 +AD4 done it a lot, but I think you've captured the key points.  If possible,
+AD4 +AD4 it might be helpful to highlight a few existing examples that stand out
+AD4 +AD4 for the different characteristics you mention (comprehensive, but able
+AD4 +AD4 to stand the test of time).
+AD4 
+AD4 Thanks. I'll see if I can find some and add them.
+AD4 
+AD4 +AD4 Another thought, any reason that we wouldn't want to keep all wiki tests
+AD4 +AD4 in the QA: namespace (and with the prefix QA:Testcase+AF8)?  The door is
+AD4 +AD4 left open for other names, I wonder if we want to cut that off ahead of
+AD4 +AD4 time to keep our sanity by having all tests in the same namespace?
+AD4 
+AD4 I was a bit unsure on that one. I think I thought of some possible
+AD4 scenario where you might want to write a test case in a different name
+AD4 space, but I'm not entirely sure I remember what it was. I can just
+AD4 change it to say test cases should always go in the qa namespace, I
+AD4 guess.
+AD4 
+AD4 +AD4 The page also talks about using +AFsAWw-Category:Test+AF8-Cases+AF0AXQ.  I worry if we
+AD4 +AD4 are too lax in categorizing new tests we'll end up with a large amount
+AD4 +AD4 of random tests in the main +AFsAWw-Category:Test+AF8-Cases+AF0AXQ making it a
+AD4 +AD4 maintenance nightmare to cleanup that category.  Should we instead
+AD4 +AD4 direct users to your other page
+AD4 +AD4 (https://fedoraproject.org/wiki/User:Adamwill/Draft+AF8-QA+AF8-SOP+AF8-package+AF8-test+AF8-plan+AF8-creation) for guidance on categorizing test cases?
+AD4 
+AD4 This was something I wanted to call out for discussion and forgot - so
+AD4 far we've put all test cases directly into the Test+AF8-Cases category, but
+AD4 like you, I'm worried that really won't scale. I did wonder whether
+AD4 others would agree we should stop doing that and instead have them
+AD4 usually go into a more specific category which in turn would be a
+AD4 sub-category of Test+AF8-Cases, and only have test cases be members of
+AD4 Test+AF8-Cases directly if it really made no sense to have them in a more
+AD4 specific category.

Agreed ... I think it makes sense to keep Category:Test+AF8-Cases as just a
container for sub-categories if possible.  Mainly for the reasons you
note around +ACo-trying+ACo to keep content organized.

+AD4 +AD4 +AD4 The second is what's really specific to this subject. It describes how
+AD4 +AD4 +AD4 to create a set of test cases for a particular package, and a proposed
+AD4 +AD4 +AD4 standardized categorization scheme which will allow us to denote test
+AD4 +AD4 +AD4 cases as being associated with specific packages, and also denote them
+AD4 +AD4 +AD4 as concerning critical path functionality.
+AD4 +AD4 
+AD4 +AD4 I think I mentioned this previously, in the section 'Preparation', I
+AD4 +AD4 appreciate the distinction of 'core' and 'extended'.  But I it resonates
+AD4 +AD4 with me better under the context of test +ACI-priority+ACI.  I don't see why we
+AD4 +AD4 can't keep using the terms 'core' and 'extended', but just want to
+AD4 +AD4 clarify their purpose.  They're intended to add some sense of execution
+AD4 +AD4 priority to a list of test cases, right?  Where critpath comes first,
+AD4 +AD4 then core, then extended, then other?  Also, you describe
+AD4 +AD4 categorizing/grouping test cases in more detail below, maybe just link
+AD4 +AD4 to that instead?

Was I accurate in my understanding above of your proposed groupings
(critpath, core and extended)?  Are they intended to convey an execution
priority of the tests?

+AD4 well, the idea is that the two are complementary: if you're going to
+AD4 separate the test cases into 'core' and 'extended' groups, then why not
+AD4 identify which functionality is 'core' and which is 'extended' at the
+AD4 time you're identifying functionality to write test cases for? I'm not
+AD4 quite sure what your proposal is here - could you draft it up in terms
+AD4 of an actual change to the page so I can see it more clearly? thanks+ACE

I articulated several layouts in previous comments in the ticket.  See
https://fedorahosted.org/fedora-qa/ticket/154+ACM-comment:12 and
https://fedorahosted.org/fedora-qa/ticket/154+ACM-comment:18.

I guess I'm hesitant about introducing new terminology (+ACI-core+ACI and
+ACI-extended+ACI) when I'm more familiar with prioritizing test cases using
the term  +ACI-priority+ACI.  I'm not saying we shouldn't use them, I'm just
trying to understand the context.  I'm also trying to ensure your
project ties in nicely with the work Hurry is doing with regards to
scoping out a TCMS (http://fedorahosted.org/fedora-qa/ticket/152).  My
question (I guess I already re-stated above) was whether you consider
the terms +ACI-core+ACI and +ACI-extended+ACI as a designation of test case priority?

Outside of the terminology, I have some concerns whether this is within
the scope of the initial project, or something we want to leave as a
phase+ACM-2 effort.  We definitely need to think about it as non-critpath
tests will come in, I just hope we don't spend all our collective energy
on defining non-critpath tests and then we are still exposed to a lack
of test documentation for the critpath.

+AD4 +AD4 In the section, 'Simple (required)', would it help to add a link to
+AD4 +AD4 http://fedoraproject.org/wiki/BugZappers/CorrectComponent+ACM-Which+AF8-component+AF8-is+AF8-it.3F (or similar page).  Something to help testers find the right src.rpm name of the component under test?  Side note, this might also be a maintenance task we can define where I, or anyone interested, could manually scrub (or script) finding Categories:Test+AF8-Cases searching incorrectly named category pages.
+AD4 +AD4 
+AD4 +AD4 Also in 'Simple (required)', we don't tell the author to add their
+AD4 +AD4 'Category:Package+AF8AJAB7-sourcename+AH0AXw-test+AF8-cases' to 'Category:Test+AF8-Cases'.  I
+AD4 +AD4 think we want all newly created package categories anchored under
+AD4 +AD4 'Category:Test+AF8-Cases'.
+AD4 
+AD4 yup, indeed, oversight - will add it. thanks+ACE
+AD4 
+AD4 +AD4 General comment.  I know we've got an eye towards integrating this work
+AD4 +AD4 with bodhi and/or f-e-k.  Until that work is complete, I wonder if those
+AD4 +AD4 notes will introduce confusion/speculation.  Should we leave out the
+AD4 +AD4 bits about possible future tool integration until such support is
+AD4 +AD4 active?
+AD4 
+AD4 possibly. I was meaning those bits to be read simply as a potential
+AD4 illustration of programmatic use of the categories to illustrate why
+AD4 consistent categorization is important, but if you think it's confusing,
+AD4 we could take it out.

No strong opinions here.  I thought I learned somewhere that one should
avoid future leading statements when documenting process.  I could have
sworn that was in the Fedora doc guide ... but I could be making it up.

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application+AC8-pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http:+AC8ALw-lists.fedoraproject.org+AC8-pipermail+AC8-test+AC8-attachments+AC8-20110103+AC8-bd1a36fb+AC8-attachment.bin