Proposal to change a test case to meet more of the criteria

[Lukas Ruzicka]

Hello,

I was checking the release criteria and I spotted this one:
https://fedoraproject.org/wiki/Fedora_29_Final_Release_Criteria#Release_i...
which does not have a test case connected to it. However, there is one test
case that does pretty much the same, although uses a different attitude. It
is the
https://fedoraproject.org/wiki/QA:Testcase_base_edition_self_identification

I think that the check for installed packages could be added to the above
mentioned test case and kill two birds with one stone:


   1. If using a specific release of Fedora, check that the *fedora-release*
   package is installed:
      - rpm -qi fedora-release
   2. Check that the *generic-release* package is in the repository:
      - dnf search generic-release

The added criteria to pass the test would be:

   1. The fedora-release package is installed.
   2. The generic-release package is in the repository.

What do you think about it?

-- 

Lukáš Růžička

FEDORA QE, RHCE

Red Hat

<https://www.redhat.com>

Purkyňova 115

612 45 Brno - Královo Pole

lruzicka(a)redhat.com
TRIED AND PERSONALLY TESTED, ERGO TRUSTED. <https://redhat.com/trusted>

[Adam Williamson]

On Wed, 2018-05-30 at 14:23 +0200, Lukas Ruzicka wrote:
>  Hello,
>  
>  I was checking the release criteria and I spotted this one:
>  https://fedoraproject.org/wiki/Fedora_29_Final_Release_Criteria#Release_i...
>  which does not have a test case connected to it. However, there is one test
>  case that does pretty much the same, although uses a different attitude. It
>  is the
>  https://fedoraproject.org/wiki/QA:Testcase_base_edition_self_identification
>  
>  I think that the check for installed packages could be added to the above
>  mentioned test case and kill two birds with one stone:
>  
>  
>     1. If using a specific release of Fedora, check that the *fedora-release*
>     package is installed:
>        - rpm -qi fedora-release
>     2. Check that the *generic-release* package is in the repository:
>        - dnf search generic-release
>  
>  The added criteria to pass the test would be:
>  
>     1. The fedora-release package is installed.
>     2. The generic-release package is in the repository.
>  
>  What do you think about it? 
Unfortunately it's not quite sufficient. Just checking that a fedora-
release package is installed does not enforce that criterion; there is
pretty much always going to be a fedora-release package unless we
really screwed up, but it may not contain the correct 'names,
information and repository configuration'. There's two particular
requirements we always have to remember to check: the release must be
*at least* 1 (it cannot be 0.x, as this causes the 'Timbuktu warning'
to appear in some installer modes), and the updates-testing repo *must*
be disabled by default. Both those things are usually false until quite
late in the release cycle; somewhere around Final freeze time releng
will do a fedora-release package update that bumps the release to -1
and disables updates-testing.

As for generic-release - doing a 'dnf search' from an installed system
again doesn't quite cut the mustard, as that's looking in the 'fedora'
repo, and until updates-testing is disabled, it's looking in the
'updates-testing' repo. At least conceivably, the package could be
there but not in the actual release repository. It also may not be
'appropriately versioned' (again, the release should be -1 or higher).

These have always been sort of magic things that we and releng
magically know, and it's certainly true that we should write them into
a test case to formalize the knowledge. I would not want to extend this
test case to cover that, though. Lately I've actually been trying to
make the test cases vaguely related to identification *more* specific
and *less* general, in the interests of making them easier to automate.
If we were going to extend an existing test case,
https://fedoraproject.org/wiki/QA:Testcase_base_artwork_release_identific...
might be a better choice, but really, I think it would be best to have
a new separate test case. One tricky thing is that we shouldn't mark it
as a fail early in the cycle when fedora-release has u-t enabled and a
release of < 1; this is correct at that time. The test case should
cover this somehow - it should have two - or, in fact, three, thinking
about Rawhide composes - different expected results for what the
packages should be versioned and what repositories should be enabled at
different points in the cycle.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net