F18 Criterions/Testcases interconnection

[Adam Williamson]

On Thu, 2012-10-25 at 06:51 -0400, Kamil Paral wrote:

> > 10. The installer must be able to install each of the release
> > blocking desktops, as well as the minimal package set, with each
> > supported installation method
> >     - We have testcase for the minimal install
> >     https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install
> >     but IMHO no testcases for the desktops
> 
> 
> We have this:
> https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_Install
> 
> That covers GNOME. We could modify it and ask the tester to pick any
> one of the release blocking desktops to be installed. Do we want to
> make it fuzzy? (You won't know exactly what was tested). Or we can add
> a new KDE test case. Or we can leave it as it is. In this case, all
> variants seem OK to me. I would probably make it fuzzy or do nothing.

I think Josef's probably right here, the most straightforward thing to
do would seem to be to just test it explicitly. It's in the criteria,
writing the test case(s) should be simple, and they're not onerous tests
to do, so I don't really see a problem.

I think 'the default package set install works' is a Thing in itself, so
I'd probably be more in favour of adding a separate test case which
covers installing each of the release-blocking desktops and can be
marked as 'pass' only if they all work. In practice, whoever's doing the
test obviously can 'count' the default_package_install result towards
the release_blocking_package_sets_install (or whatever we call it) test
case if the default package set at the time happens to correspond to one
of the release-blocking desktops, as it does now.

So I'd imagine this would happen:

1. Tester checks that a default install works, marks
default_package_install test as pass
2. Tester checks that a KDE install works, and then marks
release_blocking_package_sets_install as pass

in the future if our default package set is somehow not a release
blocking desktop any more, or we get more release blocking desktops,
everything continues to work out, testers just have a bit more testing
to do before marking release_blocking_package_sets_install as pass.

> > 19. When booting a system installed without a graphical environment,
> > or when using a correct configuration setting to cause an installed
> > system to boot in non-graphical mode, the system should boot to a
> > state where it is possible to log in through at least one of the
> > default virtual consoles
> >     - Maybe we'd like to add some specific testcase like
> >     https://fedoraproject.org/wiki/QA:Testcase_base_startup seems to
> >     be for graphical startup
> 
> That "expected result" can be added here:
> https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_Install

I believe this is in fact covered in
https://fedoraproject.org/wiki/QA:Testcase_base_startup .

> > 22. The default desktop background must be different from that of the
> > two previous stable releases
> >     - IMHO no testcase coveres this (we have testcases that require
> >     'proper' artwork for beta/final though)
> 
> This can be added as an "expected result" to some of existing test
> cases. I, personally, miss a generic test case "system boots and user
> logs in", where this could be added. I wouldn't create a separate test
> case just for this criterion.

This is again covered by
https://fedoraproject.org/wiki/QA:Testcase_base_startup (it's easy to
forget about the Base tests, but don't!) I think the 'expected results'
need a bit of tweaking for the revisions we made to the artwork
criterion, though.

> > 25. It must be possible to trigger a system shutdown using standard
> > console commands, and the system must shut down in such a way that
> > storage volumes (e.g. simple partitions, LVs and PVs, RAID arrays)
> > are taken offline safely and the system's BIOS or EFI is correctly
> > requested to power down the system
> >     - IMHO there is no testcase covering this criterion
> 
> It actually is here:
> https://fedoraproject.org/wiki/QA:Testcase_desktop_login
> 
> But it is merged with testing keymaps, and I don't like it a bit.
> Those should be two different test cases.

For the record this happened the other way around - we had the 'login'
test case and sort of stretched it to cover keymap issues, instead of
creating a new test case. For me this works in practical terms, it kind
of works out efficiently to test those things together. But it's not the
most elegant organization ever, no.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net