F18 Criterions/Testcases interconnection

[Josef Skladanka]

Hi,

I started to connect criterions to testcases for the F18 testing. So far, the Alpha is
done (the messy page with criterions & links to testcases can be found here
<https://fedoraproject.org/wiki/User:Jskladan/Sandbox:F18_Criteria_Testcas...>),
and it looks all right. Some problems I've come around:

--------------------------

10. The installer must be able to install each of the release blocking desktops, as well
as the minimal package set, with each supported installation method
    - We have testcase for the minimal install
https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_I... but IMHO
no testcases for the desktops

13. The installer must allow the user to select which of the disks connected to the system
will be affected by the installation process. Disks not selected as installation targets
must not be affected by the installation process in any way
    - IMHO the same testcases as #12 (partitioning related), but none of them requires,
that disks not selected must not be affected.

19. When booting a system installed without a graphical environment, or when using a
correct configuration setting to cause an installed system to boot in non-graphical mode,
the system should boot to a state where it is possible to log in through at least one of
the default virtual consoles
    - Maybe we'd like to add some specific testcase like
https://fedoraproject.org/wiki/QA:Testcase_base_startup seems to be for graphical startup

22. The default desktop background must be different from that of the two previous stable
releases
    - IMHO no testcase coveres this (we have testcases that require 'proper'
artwork for beta/final though)

23. Any component which prominently identifies a Fedora release version number, code name
or phase (Alpha, Beta, Final) must do so correctly
    - Same as #22 IMHO

25. It must be possible to trigger a system shutdown using standard console commands, and
the system must shut down in such a way that storage volumes (e.g. simple partitions, LVs
and PVs, RAID arrays) are taken offline safely and the system's BIOS or EFI is
correctly requested to power down the system
    - IMHO there is no testcase covering this criterion

--------------------------

Of couse, I could have missed something, so feel free to point me to the respective
testcases. In the mean time, I'm going to get my hands dirty with Beta and Final
criterion pairing.

J.

[Kamil Paral]

>  Hi,
>  
>  I started to connect criterions to testcases for the F18 testing. So
>  far, the Alpha is done (the messy page with criterions & links to
>  testcases can be found here
>
> <https://fedoraproject.org/wiki/User:Jskladan/Sandbox:F18_Criteria_Testcas...>),
>  and it looks all right. Some problems I've come around: 
I think we don't have to have a test case for every criterion (sometimes it might not
even be possible to write a generic test case), but of course it's good to have most
of them reasonably covered.

>  
>  --------------------------
>  
>  10. The installer must be able to install each of the release
>  blocking desktops, as well as the minimal package set, with each
>  supported installation method
>      - We have testcase for the minimal install
>      https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_I...
>      but IMHO no testcases for the desktops 

We have this:
https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_I...

That covers GNOME. We could modify it and ask the tester to pick any one of the release
blocking desktops to be installed. Do we want to make it fuzzy? (You won't know
exactly what was tested). Or we can add a new KDE test case. Or we can leave it as it is.
In this case, all variants seem OK to me. I would probably make it fuzzy or do nothing.

>  
>  13. The installer must allow the user to select which of the disks
>  connected to the system will be affected by the installation
>  process. Disks not selected as installation targets must not be
>  affected by the installation process in any way
>      - IMHO the same testcases as #12 (partitioning related), but none
>      of them requires, that disks not selected must not be affected.
>  
>  19. When booting a system installed without a graphical environment,
>  or when using a correct configuration setting to cause an installed
>  system to boot in non-graphical mode, the system should boot to a
>  state where it is possible to log in through at least one of the
>  default virtual consoles
>      - Maybe we'd like to add some specific testcase like
>      https://fedoraproject.org/wiki/QA:Testcase_base_startup seems to
>      be for graphical startup 
That "expected result" can be added here:
https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_I...

>  
>  22. The default desktop background must be different from that of the
>  two previous stable releases
>      - IMHO no testcase coveres this (we have testcases that require
>      'proper' artwork for beta/final though) 
This can be added as an "expected result" to some of existing test cases. I,
personally, miss a generic test case "system boots and user logs in", where this
could be added. I wouldn't create a separate test case just for this criterion.

>  
>  23. Any component which prominently identifies a Fedora release
>  version number, code name or phase (Alpha, Beta, Final) must do so
>  correctly
>      - Same as #22 IMHO 
Same as above. I also think it is fine to have this without a test case.

>  
>  25. It must be possible to trigger a system shutdown using standard
>  console commands, and the system must shut down in such a way that
>  storage volumes (e.g. simple partitions, LVs and PVs, RAID arrays)
>  are taken offline safely and the system's BIOS or EFI is correctly
>  requested to power down the system
>      - IMHO there is no testcase covering this criterion 
It actually is here:
https://fedoraproject.org/wiki/QA:Testcase_desktop_login

But it is merged with testing keymaps, and I don't like it a bit. Those should be two
different test cases.

[Adam Williamson]

On Thu, 2012-10-25 at 06:51 -0400, Kamil Paral wrote:

>  > 10. The installer must be able to install each of the release
>  > blocking desktops, as well as the minimal package set, with each
>  > supported installation method
>  >     - We have testcase for the minimal install
>  >     https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_I...
>  >     but IMHO no testcases for the desktops
>  
>  
>  We have this:
>  https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_I...
>  
>  That covers GNOME. We could modify it and ask the tester to pick any
>  one of the release blocking desktops to be installed. Do we want to
>  make it fuzzy? (You won't know exactly what was tested). Or we can add
>  a new KDE test case. Or we can leave it as it is. In this case, all
>  variants seem OK to me. I would probably make it fuzzy or do nothing. 
I think Josef's probably right here, the most straightforward thing to
do would seem to be to just test it explicitly. It's in the criteria,
writing the test case(s) should be simple, and they're not onerous tests
to do, so I don't really see a problem.

I think 'the default package set install works' is a Thing in itself, so
I'd probably be more in favour of adding a separate test case which
covers installing each of the release-blocking desktops and can be
marked as 'pass' only if they all work. In practice, whoever's doing the
test obviously can 'count' the default_package_install result towards
the release_blocking_package_sets_install (or whatever we call it) test
case if the default package set at the time happens to correspond to one
of the release-blocking desktops, as it does now.

So I'd imagine this would happen:

1. Tester checks that a default install works, marks
default_package_install test as pass
2. Tester checks that a KDE install works, and then marks
release_blocking_package_sets_install as pass

in the future if our default package set is somehow not a release
blocking desktop any more, or we get more release blocking desktops,
everything continues to work out, testers just have a bit more testing
to do before marking release_blocking_package_sets_install as pass.

>  > 19. When booting a system installed without a graphical
> environment,
>  > or when using a correct configuration setting to cause an installed
>  > system to boot in non-graphical mode, the system should boot to a
>  > state where it is possible to log in through at least one of the
>  > default virtual consoles
>  >     - Maybe we'd like to add some specific testcase like
>  >     https://fedoraproject.org/wiki/QA:Testcase_base_startup seems to
>  >     be for graphical startup
>  
>  That "expected result" can be added here:
>  https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Minimal_Package_I... 
I believe this is in fact covered in
https://fedoraproject.org/wiki/QA:Testcase_base_startup .

>  > 22. The default desktop background must be different from that
> of the
>  > two previous stable releases
>  >     - IMHO no testcase coveres this (we have testcases that require
>  >     'proper' artwork for beta/final though)
>  
>  This can be added as an "expected result" to some of existing test
>  cases. I, personally, miss a generic test case "system boots and user
>  logs in", where this could be added. I wouldn't create a separate test
>  case just for this criterion. 
This is again covered by
https://fedoraproject.org/wiki/QA:Testcase_base_startup (it's easy to
forget about the Base tests, but don't!) I think the 'expected results'
need a bit of tweaking for the revisions we made to the artwork
criterion, though.

>  > 25. It must be possible to trigger a system shutdown using
> standard
>  > console commands, and the system must shut down in such a way that
>  > storage volumes (e.g. simple partitions, LVs and PVs, RAID arrays)
>  > are taken offline safely and the system's BIOS or EFI is correctly
>  > requested to power down the system
>  >     - IMHO there is no testcase covering this criterion
>  
>  It actually is here:
>  https://fedoraproject.org/wiki/QA:Testcase_desktop_login
>  
>  But it is merged with testing keymaps, and I don't like it a bit.
>  Those should be two different test cases. 
For the record this happened the other way around - we had the 'login'
test case and sort of stretched it to cover keymap issues, instead of
creating a new test case. For me this works in practical terms, it kind
of works out efficiently to test those things together. But it's not the
most elegant organization ever, no.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Kamil Paral]

>  > We have this:
>  > https://fedoraproject.org/wiki/QA:Testcase_Package_Sets_Default_Package_I...
>  > 
>  > That covers GNOME. We could modify it and ask the tester to pick
>  > any
>  > one of the release blocking desktops to be installed. Do we want to
>  > make it fuzzy? (You won't know exactly what was tested). Or we can
>  > add
>  > a new KDE test case. Or we can leave it as it is. In this case, all
>  > variants seem OK to me. I would probably make it fuzzy or do
>  > nothing.
>  
>  I think Josef's probably right here, the most straightforward thing
>  to
>  do would seem to be to just test it explicitly. It's in the criteria,
>  writing the test case(s) should be simple, and they're not onerous
>  tests
>  to do, so I don't really see a problem.
>  
>  I think 'the default package set install works' is a Thing in itself,
>  so
>  I'd probably be more in favour of adding a separate test case which
>  covers installing each of the release-blocking desktops and can be
>  marked as 'pass' only if they all work. In practice, whoever's doing
>  the
>  test obviously can 'count' the default_package_install result towards
>  the release_blocking_package_sets_install (or whatever we call it)
>  test
>  case if the default package set at the time happens to correspond to
>  one
>  of the release-blocking desktops, as it does now.
>  
>  So I'd imagine this would happen:
>  
>  1. Tester checks that a default install works, marks
>  default_package_install test as pass
>  2. Tester checks that a KDE install works, and then marks
>  release_blocking_package_sets_install as pass
>  
>  in the future if our default package set is somehow not a release
>  blocking desktop any more, or we get more release blocking desktops,
>  everything continues to work out, testers just have a bit more
>  testing
>  to do before marking release_blocking_package_sets_install as pass. 
That is algorithmically perfect, but over-engineered [1]. I think it's much easier to
have a default_package_install and a KDE_package_install and adjust it if the defaults
changes, than to think about the correct workflow every time we execute the tests (I
understand I'll be fired for saying this, but I tend to forget these things. I also
want to make release validation more accessible for newcomers).

[1] $ python -c 'import this'

[Adam Williamson]

On Thu, 2012-10-25 at 18:47 -0400, Kamil Paral wrote:

>  > I think 'the default package set install works' is a
> Thing in itself,
>  > so
>  > I'd probably be more in favour of adding a separate test case which
>  > covers installing each of the release-blocking desktops and can be
>  > marked as 'pass' only if they all work. In practice, whoever's doing
>  > the
>  > test obviously can 'count' the default_package_install result towards
>  > the release_blocking_package_sets_install (or whatever we call it)
>  > test
>  > case if the default package set at the time happens to correspond to
>  > one
>  > of the release-blocking desktops, as it does now.
>  > 
>  > So I'd imagine this would happen:
>  > 
>  > 1. Tester checks that a default install works, marks
>  > default_package_install test as pass
>  > 2. Tester checks that a KDE install works, and then marks
>  > release_blocking_package_sets_install as pass
>  > 
>  > in the future if our default package set is somehow not a release
>  > blocking desktop any more, or we get more release blocking desktops,
>  > everything continues to work out, testers just have a bit more
>  > testing
>  > to do before marking release_blocking_package_sets_install as pass.
>  
>  That is algorithmically perfect, but over-engineered [1]. I think it's
>  much easier to have a default_package_install and a
>  KDE_package_install and adjust it if the defaults changes, than to
>  think about the correct workflow every time we execute the tests (I
>  understand I'll be fired for saying this, but I tend to forget these
>  things. I also want to make release validation more accessible for
>  newcomers).
>  
>  [1] $ python -c 'import this' 
Well I see your point, but it kind of cuts both ways - Josef clearly got
confused by a few cases where we overload test cases to test several
different things, so you can argue that it's actually more 'accessible'
when we try to stick to 'one test case tests one thing'. But your
approach would achieve what we need to achieve indeed.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Josef Skladanka]

>  Well I see your point, but it kind of cuts both ways - Josef clearly
>  got
>  confused by a few cases where we overload test cases to test several
>  different things, so you can argue that it's actually more
>  'accessible'
>  when we try to stick to 'one test case tests one thing'. But your
>  approach would achieve what we need to achieve indeed. 
/me feels a bit ashamed :D

I'm all for 'one test case tests one thing', even though I understand, that
having several 'similar' testcases which differ in just few words is nonsense (as
I got equally confused with the _almost the same_ criterions in alpha/beta).

J.

[Josef Skladanka]

Beta is done
<https://fedoraproject.org/wiki/User:Jskladan/Sandbox:F18_Criteria_Testcas...;,
some issues I've come around:

--------------------------
5. The installer must be able to use the HTTP, FTP and either NFS or NFSISO remote package
source options 
    - The testcase
<https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFS_variation
https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFSISO_vari... is
marked as Final

6. It must be possible to install by booting the installation kernel directly, including
via PXE, and correctly specifying a remote source for the installer itself, using
whichever protocols are required to work for package retrieval at the current phase
(Alpha, Beta, Final). This must work if the remote source is not a complete repository but
contains only the files necessary for the installer itself to run.
    - <https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot> Marked as
Alpha

9.  The installer must be able to complete an installation using automatic partitioning to
a validly-formatted disk with sufficient empty space, using the empty space and installing
a bootloader but leaving the pre-existing partitions and data untouched
    - Testcases as #12 Alpha criterion
    - Testcases do not mention that pre-existing partitions must be untouched

16. When booting a system installed without a graphical environment, or when using a
correct configuration setting to cause an installed system to boot in non-graphical mode,
the system should provide a working login prompt without any unintended user intervention
when boot is complete, and all virtual consoles intended to provide a working login prompt
should do so
    - This is exactly the same as #19 Aplha criterion, maybe we'd like to leave it
just Alpha or Beta 

23.  All release-blocking desktops' offered mechanisms (if any) for shutting down,
logging out and rebooting must work 
    - IMHO no testcase covers this 
--------------------------

J.

[Kamil Paral]

>  Beta is done
>
> <https://fedoraproject.org/wiki/User:Jskladan/Sandbox:F18_Criteria_Testcas...;,
>  some issues I've come around:
>  
>  --------------------------
>  5. The installer must be able to use the HTTP, FTP and either NFS or
>  NFSISO remote package source options
>      - The testcase
>      <https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFS_variation
>      https://fedoraproject.org/wiki/QA:Testcase_install_repository_NFSISO_vari...
>      is marked as Final 
Since only one of them has to work for Beta, we could mark both of them as Beta/Final.

>  
>  6. It must be possible to install by booting the installation kernel
>  directly, including via PXE, and correctly specifying a remote
>  source for the installer itself, using whichever protocols are
>  required to work for package retrieval at the current phase (Alpha,
>  Beta, Final). This must work if the remote source is not a complete
>  repository but contains only the files necessary for the installer
>  itself to run.
>      -
>      <https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Pxeboot>
>      Marked as Alpha 
mark as Beta

>  
>  9.  The installer must be able to complete an installation using
>  automatic partitioning to a validly-formatted disk with sufficient
>  empty space, using the empty space and installing a bootloader but
>  leaving the pre-existing partitions and data untouched
>      - Testcases as #12 Alpha criterion
>      - Testcases do not mention that pre-existing partitions must be
>      untouched
>  
>  16. When booting a system installed without a graphical environment,
>  or when using a correct configuration setting to cause an installed
>  system to boot in non-graphical mode, the system should provide a
>  working login prompt without any unintended user intervention when
>  boot is complete, and all virtual consoles intended to provide a
>  working login prompt should do so
>      - This is exactly the same as #19 Aplha criterion, maybe we'd
>      like to leave it just Alpha or Beta 
No, this is a game of "spot five differences" :-) At least one versus all
virtual consoles. This can be part of the same test case, marked as Alpha/Beta

>  
>  23.  All release-blocking desktops' offered mechanisms (if any) for
>  shutting down, logging out and rebooting must work
>      - IMHO no testcase covers this 
Same as in Alpha feedback:
https://fedoraproject.org/wiki/QA:Testcase_desktop_login

[Josef Skladanka]

----- Original Message -----
>  From: "Kamil Paral" <kparal(a)redhat.com&gt; 
>  No, this is a game of "spot five differences" :-) At least
> one versus
>  all virtual consoles. This can be part of the same test case, marked
>  as Alpha/Beta 
OK, I'll update the summary page/matrix accordingly. Also I'll try to highlight
this "at least one/all" difference on the criteria page.

J.