[Fedora QA] #154: Tracker: critical path test case creation

[Fedora QA]

#154: Tracker: critical path test case creation
--------------------------+-------------------------------------------------
  Reporter:  adamwill     |       Owner:  adamwill 
      Type:  enhancement  |      Status:  new      
  Priority:  major        |   Milestone:  Fedora 15
 Component:  Wiki         |     Version:           
Resolution:               |    Keywords:           
--------------------------+-------------------------------------------------
Comment (by adamwill):

 sigh, I keep getting ready to send this out to lists and then hitting one
 more snag. =)

 so, in discussion with jlaska, we both think it makes most sense to only
 create two new types of category: one for associating test cases with
 specific packages, one for marking them as critical path test cases.

 Critical path test cases should be in the category
 Critical_path_Test_Cases . Note that this means *any test case which is
 primarily concerned with verifying critical path test cases* should be in
 this category. It doesn't mean any test case associated with a critical
 path package should be in the category, because critical path packages can
 have non-critical path uses and hence non-critical path test cases: a test
 case to test, say, tape drive functionality in the kernel is associated
 with a critical path package (the kernel) but is not a critical path test.

 The category name for specific source packages should be:

 Package_(srcrpmname)_Test_Cases

 same reasoning as my initial proposal for the test case name: we need to
 specify that (srcrpmname), in this context, is specifically a source RPM
 name. I can imagine, for instance, we might want to create a generic
 category for Python test packages, called Category:Python_Test_Cases, and
 put test cases into that category which are not actually test cases for
 the Python .src.rpm. So we'd also have a
 Category:Package_python_Test_Cases . That could probably be a member of
 Category:Python_Test_Cases , but not vice versa.

 I think the page name of each test case can be a bit more flexible and we
 don't really need to worry about defining it, the categories are the
 crucial thing.

 Do we also want to suggest a category for each .src.rpm along the lines
 of:

 Category:Package_python_core_Test_Cases

 ? Rationale: I'm thinking about how we want the bodhi or fedora-easy-karma
 listing for each package to look. I'm thinking of something like:

 -------------------------------------------------------

 Update: python-2.7.9-blahblah

 This update does blah blah foo foo whee whee poop.

 The following test cases can be used to check the functionality of this
 package.

 '''Critical path tests'''

 These tests verify critical path functionality (link to critical path
 process page) and should take highest priority:

 * links to all tests in Category:Package_python_Test_Cases *and*
 Category:Critical_path_Test_Cases

 '''Core functionality tests'''

 These tests verify core functionality of the package and should take
 highest priority after critical path tests:

 * links to all tests in Category:Package_python_core_Test_Cases (but not
 Category:Critical_path_Test_Cases)

 '''Remaining tests'''

 These tests verify non-critical path, non-core functionality:

 * links to all other tests (in Category:Package_python_Test_Cases but not
 Category:Package_python_core_Test_Cases or
 Category:Critical_path_Test_Cases)

 ------------------------------------------------------
 ------------------------------------------------------

 The thinking being that some packages will grow fairly large sets of tests
 (abrt already has, for instance) and it would help to isolate core
 functionality tests for rapid sanity checking.

 random side observation: our naming of test case pages and categories
 currently stinks.

 next big concern: where do we effectively document this? All we currently
 have on test cases is:

 https://fedoraproject.org/wiki/Category:Test_Cases

 I can add some overview and stuff to that page but it's not a terribly
 visible page, really. I guess the best approach is to edit that page and
 add some links to it from strategic other places, especially developer
 instruction pages. Thoughts?

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/154#comment:16>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance