Package-specific test case and critical path test case project: drafts for review

[Adam Williamson]

On Tue, 2010-12-21 at 18:12 -0500, James Laska wrote:

> > the first isn't particularly specific to this, but it was a prerequisite
> > that I discovered was missing: it's a guide to test case creation in
> > general, explaining the actual practical process of how you create a
> > test case, and the best principles to consider in doing it.
> 
> Nice job here, this is something that's difficult to explain if you've
> done it a lot, but I think you've captured the key points.  If possible,
> it might be helpful to highlight a few existing examples that stand out
> for the different characteristics you mention (comprehensive, but able
> to stand the test of time).

Thanks. I'll see if I can find some and add them.

> Another thought, any reason that we wouldn't want to keep all wiki tests
> in the QA: namespace (and with the prefix QA:Testcase_)?  The door is
> left open for other names, I wonder if we want to cut that off ahead of
> time to keep our sanity by having all tests in the same namespace?

I was a bit unsure on that one. I think I thought of some possible
scenario where you might want to write a test case in a different name
space, but I'm not entirely sure I remember what it was. I can just
change it to say test cases should always go in the qa namespace, I
guess.

> The page also talks about using [[Category:Test_Cases]].  I worry if we
> are too lax in categorizing new tests we'll end up with a large amount
> of random tests in the main [[Category:Test_Cases]] making it a
> maintenance nightmare to cleanup that category.  Should we instead
> direct users to your other page
> (https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_SOP_package_test_plan_creation) for guidance on categorizing test cases?

This was something I wanted to call out for discussion and forgot - so
far we've put all test cases directly into the Test_Cases category, but
like you, I'm worried that really won't scale. I did wonder whether
others would agree we should stop doing that and instead have them
usually go into a more specific category which in turn would be a
sub-category of Test_Cases, and only have test cases be members of
Test_Cases directly if it really made no sense to have them in a more
specific category.

> > The second is what's really specific to this subject. It describes how
> > to create a set of test cases for a particular package, and a proposed
> > standardized categorization scheme which will allow us to denote test
> > cases as being associated with specific packages, and also denote them
> > as concerning critical path functionality.
> 
> I think I mentioned this previously, in the section 'Preparation', I
> appreciate the distinction of 'core' and 'extended'.  But I it resonates
> with me better under the context of test "priority".  I don't see why we
> can't keep using the terms 'core' and 'extended', but just want to
> clarify their purpose.  They're intended to add some sense of execution
> priority to a list of test cases, right?  Where critpath comes first,
> then core, then extended, then other?  Also, you describe
> categorizing/grouping test cases in more detail below, maybe just link
> to that instead?

well, the idea is that the two are complementary: if you're going to
separate the test cases into 'core' and 'extended' groups, then why not
identify which functionality is 'core' and which is 'extended' at the
time you're identifying functionality to write test cases for? I'm not
quite sure what your proposal is here - could you draft it up in terms
of an actual change to the page so I can see it more clearly? thanks!

> In the section, 'Simple (required)', would it help to add a link to
> http://fedoraproject.org/wiki/BugZappers/CorrectComponent#Which_component_is_it.3F (or similar page).  Something to help testers find the right src.rpm name of the component under test?  Side note, this might also be a maintenance task we can define where I, or anyone interested, could manually scrub (or script) finding Categories:Test_Cases searching incorrectly named category pages.
> 
> Also in 'Simple (required)', we don't tell the author to add their
> 'Category:Package_${sourcename}_test_cases' to 'Category:Test_Cases'.  I
> think we want all newly created package categories anchored under
> 'Category:Test_Cases'.

yup, indeed, oversight - will add it. thanks!

> General comment.  I know we've got an eye towards integrating this work
> with bodhi and/or f-e-k.  Until that work is complete, I wonder if those
> notes will introduce confusion/speculation.  Should we leave out the
> bits about possible future tool integration until such support is
> active?

possibly. I was meaning those bits to be read simply as a potential
illustration of programmatic use of the categories to illustrate why
consistent categorization is important, but if you think it's confusing,
we could take it out.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net