[test-case] kickstarts and release notes TC

[Kamil Paral]

> Hi,
> 
> as part of my work on final criteria/test cases interconnection I
> wrote
> this test case [1]. It should check that criteria:
> 
> * A spin-kickstarts package which contains the exact kickstart files
> used to build the release must be present in the release repository.
> The
> included kickstarts must define the correct set of release
> repositories
> *  The final branded release notes from the Documentation team must
> be
> present on ISO media and the appropriately versioned generic release
> notes must be available in the online release repository
> *  A fedora-release package containing the correct names, information
> and repository configuration for a final Fedora release (as opposed
> to a
> pre-release) must be present on ISO media while the appropriately
> versioned Package-x-generic-16.pnggeneric-release package must be
> available in the online release repository

Hmm, I wonder whether we should have this criteria at all. The first and last one looks like RelEng criteria, not QA. The second one could be ours.

> 
> are properly tested.
> 
> Please if send me any suggestions or objections you have.
> 
> Thank you
> Petr Schindler
> 
> [1]
> https://fedoraproject.org/wiki/User:Pschindl/Draft_QA_Testcase_Repository_completeness

I don't want to sound like I don't appreciate your work. But I don't like this test case much in our current workflow. We don't have the manpower to check non-trivial stuff. And our current approach says "the matrices must be complete" and we try to check all the test cases for every TC/RC. If we don't want to change that, we should keep the list of test cases to the absolutely essential stuff.

By the way, this test case doesn't even make sense to check for TCs, just RCs. Our wiki matrices are not prepared for that.

Some of that stuff is untestable by QA - "spin-kickstarts package which contains the exact kickstart files used to build the release" - only by RelEng (we don't know which kickstarts they used, they know).

If we want to have test cases like these, I think we need to:
1. leave the idea of testing every test case for every compose
2. create a set of test cases which are essential and are tested very often
3. mark the rest of test cases as "complementary" - we can test some of them if we have time, and we can try to have them completed for RCs, but it's not mandatory

I think the latter approach would be better. It's better to have these low-impact test cases than not to have them at all (even though the drawback is that it adds to the maintenance burden). But they don't fit to our current workflow, and it should be changed.

Of course you might argue "this one test case doesn't add too much extra work". Sure, it doesn't. I was referring to the general direction we're heading, not to this particular test case.

We could add this test case (and any other similar future test cases) to our matrix and mark it as Optional, until we decide how do we want to handle the test case explosion.