[Fedora QA] #154: Tracker: critical path test case creation

Fedora QA trac at fedorahosted.org
Fri Dec 17 15:13:31 UTC 2010 

#154: Tracker: critical path test case creation
--------------------------+-------------------------------------------------
  Reporter:  adamwill     |       Owner:  adamwill 
      Type:  enhancement  |      Status:  new      
  Priority:  major        |   Milestone:  Fedora 15
 Component:  Wiki         |     Version:           
Resolution:               |    Keywords:           
--------------------------+-------------------------------------------------
Comment (by jlaska):

 Replying to [comment:16 adamwill]:
 > The category name for specific source packages should be:
 >
 > Package_(srcrpmname)_Test_Cases

 That works fine.  I think the "Package_" prefix is not needed, but it
 certainly doesn't hurt anything to have it (just longer wiki page names
 "Category:Package_mobile-broadband-provider-info_Test_Cases" vs "Category
 :Mobile-broadband-provider-info_Test_Cases", but that's certainly not
 horrible).

 > same reasoning as my initial proposal for the test case name: we need to
 specify that (srcrpmname), in this context, is specifically a source RPM
 name. I can imagine, for instance, we might want to create a generic
 category for Python test packages, called Category:Python_Test_Cases, and
 put test cases into that category which are not actually test cases for
 the Python .src.rpm. So we'd also have a
 Category:Package_python_Test_Cases . That could probably be a member of
 Category:Python_Test_Cases , but not vice versa.
 >
 > I think the page name of each test case can be a bit more flexible and
 we don't really need to worry about defining it, the categories are the
 crucial thing.

 Definitely!

 > Do we also want to suggest a category for each .src.rpm along the lines
 of:
 >
 > Category:Package_python_core_Test_Cases
 >
 > ? Rationale: I'm thinking about how we want the bodhi or fedora-easy-
 karma listing for each package to look. I'm thinking of something like:
 >
 > -------------------------------------------------------
 >
 > Update: python-2.7.9-blahblah
 >
 > This update does blah blah foo foo whee whee poop.
 >
 > The following test cases can be used to check the functionality of this
 package.
 >
 > '''Critical path tests'''
 >
 > These tests verify critical path functionality (link to critical path
 process page) and should take highest priority:
 >
 > * links to all tests in Category:Package_python_Test_Cases *and*
 Category:Critical_path_Test_Cases
 >
 > '''Core functionality tests'''
 >
 > These tests verify core functionality of the package and should take
 highest priority after critical path tests:
 >
 > * links to all tests in Category:Package_python_core_Test_Cases (but not
 Category:Critical_path_Test_Cases)
 >
 > '''Remaining tests'''
 >
 > These tests verify non-critical path, non-core functionality:
 >
 > * links to all other tests (in Category:Package_python_Test_Cases but
 not Category:Package_python_core_Test_Cases or
 Category:Critical_path_Test_Cases)
 >
 > ------------------------------------------------------
 > ------------------------------------------------------
 >
 > The thinking being that some packages will grow fairly large sets of
 tests (abrt already has, for instance) and it would help to isolate core
 functionality tests for rapid sanity checking.

 The way I read your organization above is that the front-ends will present
 the tests by priority (critical path, core, *).  I really like this
 visualization, but wonder if it is outside the scope for the first pass.
 Meaning, I think I'd want to see more focus on just getting critpath tests
 defined for as many critpath packages, rather than spending time on non-
 critpath or non-core stuff.  With the setup you've designed, it doesn't
 exclude people from creating non-critpath tests.

 An aside, I think the previously discussed Category: organization would
 lend better to allowing maintainers to prioritize tests themselves.  For
 example ...

 bodhi and f-e-k will look for wiki pages under
 Category:Package_%{sourcerpm}_Test_cases.  Any tests in that category
 would be listed, any tests in sub-categories of that page will be
 presented as you demonstrate above.  So to achieve your desired output,
 you might organize tests like ...

 {{{
 Category:Package_python_Test_Cases
 |
 +--> Category:Critpath_python_Test_Cases
 |    |
 |    +-> QA:Testcase_python_do_something_1 [1]
 |    +-> QA:Testcase_python_do_something_2 [1]
 |    +-> QA:Testcase_python_do_something_3 [1]
 |
 +--> Category:Core_python_Test_Cases
 |    |
 |    +-> QA:Testcase_python_do_something_4
 |    +-> QA:Testcase_python_do_something_5
 |
 +--> Category:Low_python_Test_Cases
 |    |
 |    +-> QA:Testcase_python_do_something_6
 |    +-> QA:Testcase_python_do_something_7
 |
 |--> QA:Testcase_python_do_something_8
 |--> QA:Testcase_python_do_something_9

 [1] Could also exist in Category:Critpath_test_cases, but not required
 }}}

 This would allow maintainers to group the tests as they see fit, but holds
 that everything must be anchored out of
 Category:Package_python_Test_Cases.  We would still focus on prioritizing
 development of critpath tests, but the support exists for
 maintainers/packagers to organize as they feel is appropriate.  Also, the
 mediawiki API provides the needed methods to find and present the
 information listed above.

 Another something to keep in mind, any heavy/complex wiki organization we
 do may require extra work when migrating data to another system (which is
 a possible mid-term future scenario).

 > random side observation: our naming of test case pages and categories
 currently stinks.

 Sure, page naming wasn't as important when we weren't attempting to
 integrate it with other tools so I gather some wiki renaming may be
 required.  Or did it stink in another regard?

 > next big concern: where do we effectively document this? All we
 currently have on test cases is:
 >
 > https://fedoraproject.org/wiki/Category:Test_Cases

 > I can add some overview and stuff to that page but it's not a terribly
 visible page, really. I guess the best approach is to edit that page and
 add some links to it from strategic other places, especially developer
 instruction pages. Thoughts?

 Agreed.  Do we need an SOP/HOWTO for each of the use cases where
 proventesters, or maintainers, will interact with this setup?  Such as ...
  1. How to create a test case
  2. How to organize test cases
  3. How to post results for test cases
  4. ... ?

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/154#comment:18>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance

More information about the test mailing list