F18 Criterions/Testcases interconnection

[Kamil Paral]

> Hi,
> 
> I started to connect criterions to testcases for the F18 testing. So
> far, the Alpha is done (the messy page with criterions & links to
> testcases can be found here
> <https://fedoraproject.org/wiki/User:Jskladan/Sandbox:F18_Criteria_Testcases_Alpha>),
> and it looks all right. Some problems I've come around:

I think we don't have to have a test case for every criterion (sometimes it might not even be possible to write a generic test case), but of course it's good to have most of them reasonably covered.

> 
> --------------------------
> 
> 10. The installer must be able to install each of the release
> blocking desktops, as well as the minimal package set, with each
> supported installation method
>     - We have testcase for the minimal install
>     https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install
>     but IMHO no testcases for the desktops


We have this:
https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_Install

That covers GNOME. We could modify it and ask the tester to pick any one of the release blocking desktops to be installed. Do we want to make it fuzzy? (You won't know exactly what was tested). Or we can add a new KDE test case. Or we can leave it as it is. In this case, all variants seem OK to me. I would probably make it fuzzy or do nothing.

> 
> 13. The installer must allow the user to select which of the disks
> connected to the system will be affected by the installation
> process. Disks not selected as installation targets must not be
> affected by the installation process in any way
>     - IMHO the same testcases as #12 (partitioning related), but none
>     of them requires, that disks not selected must not be affected.
> 
> 19. When booting a system installed without a graphical environment,
> or when using a correct configuration setting to cause an installed
> system to boot in non-graphical mode, the system should boot to a
> state where it is possible to log in through at least one of the
> default virtual consoles
>     - Maybe we'd like to add some specific testcase like
>     https://fedoraproject.org/wiki/QA:Testcase_base_startup seems to
>     be for graphical startup

That "expected result" can be added here:
https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install

> 
> 22. The default desktop background must be different from that of the
> two previous stable releases
>     - IMHO no testcase coveres this (we have testcases that require
>     'proper' artwork for beta/final though)

This can be added as an "expected result" to some of existing test cases. I, personally, miss a generic test case "system boots and user logs in", where this could be added. I wouldn't create a separate test case just for this criterion.

> 
> 23. Any component which prominently identifies a Fedora release
> version number, code name or phase (Alpha, Beta, Final) must do so
> correctly
>     - Same as #22 IMHO

Same as above. I also think it is fine to have this without a test case.

> 
> 25. It must be possible to trigger a system shutdown using standard
> console commands, and the system must shut down in such a way that
> storage volumes (e.g. simple partitions, LVs and PVs, RAID arrays)
> are taken offline safely and the system's BIOS or EFI is correctly
> requested to power down the system
>     - IMHO there is no testcase covering this criterion

It actually is here:
https://fedoraproject.org/wiki/QA:Testcase_desktop_login

But it is merged with testing keymaps, and I don't like it a bit. Those should be two different test cases.