F18 Criterions/Testcases interconnection

[Kamil Paral]

> > We have this:
> > https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_Install
> > 
> > That covers GNOME. We could modify it and ask the tester to pick
> > any
> > one of the release blocking desktops to be installed. Do we want to
> > make it fuzzy? (You won't know exactly what was tested). Or we can
> > add
> > a new KDE test case. Or we can leave it as it is. In this case, all
> > variants seem OK to me. I would probably make it fuzzy or do
> > nothing.
> 
> I think Josef's probably right here, the most straightforward thing
> to
> do would seem to be to just test it explicitly. It's in the criteria,
> writing the test case(s) should be simple, and they're not onerous
> tests
> to do, so I don't really see a problem.
> 
> I think 'the default package set install works' is a Thing in itself,
> so
> I'd probably be more in favour of adding a separate test case which
> covers installing each of the release-blocking desktops and can be
> marked as 'pass' only if they all work. In practice, whoever's doing
> the
> test obviously can 'count' the default_package_install result towards
> the release_blocking_package_sets_install (or whatever we call it)
> test
> case if the default package set at the time happens to correspond to
> one
> of the release-blocking desktops, as it does now.
> 
> So I'd imagine this would happen:
> 
> 1. Tester checks that a default install works, marks
> default_package_install test as pass
> 2. Tester checks that a KDE install works, and then marks
> release_blocking_package_sets_install as pass
> 
> in the future if our default package set is somehow not a release
> blocking desktop any more, or we get more release blocking desktops,
> everything continues to work out, testers just have a bit more
> testing
> to do before marking release_blocking_package_sets_install as pass.

That is algorithmically perfect, but over-engineered [1]. I think it's much easier to have a default_package_install and a KDE_package_install and adjust it if the defaults changes, than to think about the correct workflow every time we execute the tests (I understand I'll be fired for saying this, but I tend to forget these things. I also want to make release validation more accessible for newcomers).

[1] $ python -c 'import this'