testcase proposal: joining bios+uefi in system upgrade section

[Kamil Paral]

Now that we have system-upgrade instead of fedup, I don't think we need to split every
single upgrade test case between bios and uefi, as we have it right now:
https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade

The difference is that fedup created its own upgrade environment, a new boot menu, a
special initrd, contained a lot of systemd-related hacks. System-upgrade uses the standard
"offline upgrade" environment, so if offline upgrades ever worked for you, there
should be no difference. It doesn't create its own boot menu, it doesn't have a
separate initrd. It uses the standard way of booting your system - if your system boots,
the upgrade process should boot as well :) Likewise for adding a new boot entries during
the upgrade - if kernel updates work well for you, there should be no problem adding a new
kernel entry during upgrade, it's the same operation.

So, I'd like to save some unnecessary work and merge "x86_64 BIOS" and
"x86_64 UEFI" columns into "x86_64" column in that wiki matrix.

I'd also like to add two new test cases:
QA:Testcase_upgrade_dnf_previous_bios
QA:Testcase_upgrade_dnf_previous_uefi

This will only serve as a way to track that we have tested both environments, but it no
longer matters which product was used to perform the testing, and also we no longer need
to test all combinations. So, if you tested workstation on bios, you can fill out two test
cases immediately (_workstation and _bios). If you tested minimal on uefi, the same
applies.

I think it's valuable to keep bios vs uefi differentiation in the table at least in
the form of these two additional test cases, because it's still a system upgrade, and
there are some critical components which can break (like a new major release of efibootmgr
or grub). We should make sure we've tested both variants. But this way, it adds
virtually no extra work (compared to testing all combinations of it) and still gives us
almost the same test value.

Thoughts?

Kamil

[Kamil Paral]

>  Now that we have system-upgrade instead of fedup, I don't think
> we need to
>  split every single upgrade test case between bios and uefi, as we have it
>  right now:
>  https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
>  
>  The difference is that fedup created its own upgrade environment, a new boot
>  menu, a special initrd, contained a lot of systemd-related hacks.
>  System-upgrade uses the standard "offline upgrade" environment, so if
>  offline upgrades ever worked for you, there should be no difference. It
>  doesn't create its own boot menu, it doesn't have a separate initrd. It uses
>  the standard way of booting your system - if your system boots, the upgrade
>  process should boot as well :) Likewise for adding a new boot entries during
>  the upgrade - if kernel updates work well for you, there should be no
>  problem adding a new kernel entry during upgrade, it's the same operation.
>  
>  So, I'd like to save some unnecessary work and merge "x86_64 BIOS" and
>  "x86_64 UEFI" columns into "x86_64" column in that wiki matrix.
>  
>  I'd also like to add two new test cases:
>  QA:Testcase_upgrade_dnf_previous_bios
>  QA:Testcase_upgrade_dnf_previous_uefi
>  
>  This will only serve as a way to track that we have tested both environments,
>  but it no longer matters which product was used to perform the testing, and
>  also we no longer need to test all combinations. So, if you tested
>  workstation on bios, you can fill out two test cases immediately
>  (_workstation and _bios). If you tested minimal on uefi, the same applies.
>  
>  I think it's valuable to keep bios vs uefi differentiation in the table at
>  least in the form of these two additional test cases, because it's still a
>  system upgrade, and there are some critical components which can break (like
>  a new major release of efibootmgr or grub). We should make sure we've tested
>  both variants. But this way, it adds virtually no extra work (compared to
>  testing all combinations of it) and still gives us almost the same test
>  value. 
Oh, one more thing - similarly to _bios and _uefi test cases, I'd like to change
existing _workstation_encrypted test case to just _encrypted test case - it shouldn't
matter which package set is installed to verify that there's no issue with unlocking
your drives. This way you can easily fill 3 test cases with just a single test (e.g.
minimal + encrypted + bios).

[Adam Williamson]

On Mon, 2015-11-23 at 10:36 -0500, Kamil Paral wrote:

>  Oh, one more thing - similarly to _bios and _uefi test cases,
> I'd
>  like to change existing _workstation_encrypted test case to just
>  _encrypted test case - it shouldn't matter which package set is
>  installed to verify that there's no issue with unlocking your drives.
>  This way you can easily fill 3 test cases with just a single test
>  (e.g. minimal + encrypted + bios). 
Just be aware that the upgrade test cases are based on templates,
please don't break that setup as it saves a lot of work in duplicating
content between all the various upgrade tests. I understand why you
want to make the changes in the way you do, but I do think it'd be nice
if possible to always treat UEFI vs. BIOS as an 'environment'
difference, i.e. in the result columns, not in the test case name.

As an alternative perhaps we could have workstation_encrypted (or 'any
package set encrypted', if I can tweak the template a bit) with BIOS
and UEFI columns, and other tests with just x86 and ARM columns?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kamil Paral]

>  On Mon, 2015-11-23 at 10:36 -0500, Kamil Paral wrote:
>  
>  > Oh, one more thing - similarly to _bios and _uefi test cases, I'd
>  > like to change existing _workstation_encrypted test case to just
>  > _encrypted test case - it shouldn't matter which package set is
>  > installed to verify that there's no issue with unlocking your drives.
>  > This way you can easily fill 3 test cases with just a single test
>  > (e.g. minimal + encrypted + bios).
>  
>  Just be aware that the upgrade test cases are based on templates,
>  please don't break that setup as it saves a lot of work in duplicating
>  content between all the various upgrade tests. I understand why you
>  want to make the changes in the way you do, but I do think it'd be nice
>  if possible to always treat UEFI vs. BIOS as an 'environment'
>  difference, i.e. in the result columns, not in the test case name.
>  
>  As an alternative perhaps we could have workstation_encrypted (or 'any
>  package set encrypted', if I can tweak the template a bit) with BIOS
>  and UEFI columns, and other tests with just x86 and ARM columns? 
Hmm, the environments as columns work well if we want to test all combinations, but not
that well if we want to test it just once, for any test case. I'd really like to say
"if you have tested upgrade with uefi, whatever package set or configuration, you can
tick this box", and not "tick this box only if you have tested _encrypted_
installation with uefi". That puts additional constraints on testing, and it seems to
me that it's better to live with two additional lines (_uefi and _bios) rather than
changing our testing to accommodate a nicer 2D structure of the matrix.

As a middle ground, what about having:
   .---------------------------------------------------.
   |                                      BIOS     UEFI|
   |Testcase_upgrade_dnf_previous_any                  |
   '---------------------------------------------------'

I think that's reasonable? Maybe we don't even need to modify and use the template
content, just reference other test cases and say that any of them is fine.

[Adam Williamson]

On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:

>  As a middle ground, what about having:
>     .---------------------------------------------------.
>     |                                      BIOS     UEFI|
>     |Testcase_upgrade_dnf_previous_any                  |
>     '---------------------------------------------------'
>  
>  I think that's reasonable? Maybe we don't even need to modify and use
>  the template content, just reference other test cases and say that
>  any of them is fine. 
Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a
better way of saying 'out of these tests, at least one needs to be done
for each environment' and calculating coverage and such, but this is
Wikitcms ;)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kamil Paral]

>  On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:
>  
>  > As a middle ground, what about having:
>  >    .---------------------------------------------------.
>  >    |                                      BIOS     UEFI|
>  >    |Testcase_upgrade_dnf_previous_any                  |
>  >    '---------------------------------------------------'
>  > 
>  > I think that's reasonable? Maybe we don't even need to modify and use
>  > the template content, just reference other test cases and say that
>  > any of them is fine.
>  
>  Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a
>  better way of saying 'out of these tests, at least one needs to be done
>  for each environment' and calculating coverage and such, but this is
>  Wikitcms ;) 
I have made the changes:
https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade

This is the new test case:
https://fedoraproject.org/wiki/QA:Testcase_upgrade_dnf_previous_any

Originally I wanted to use something like this in the test case:
{{Testcase upgrade|(any flavor)}}

However, that didn't work and I could wrap my mind around all the templates, so I
ended up with that bare-bones description. I hope it's fine, patches welcome :)

[Adam Williamson]

On Thu, 2015-11-26 at 10:23 -0500, Kamil Paral wrote:
>  > On Tue, 2015-11-24 at 06:33 -0500, Kamil Paral wrote:
>  > 
>  > > As a middle ground, what about having:
>  > >    .---------------------------------------------------.
>  > >    |                                      BIOS     UEFI|
>  > >    |Testcase_upgrade_dnf_previous_any                  |
>  > >    '---------------------------------------------------'
>  > > 
>  > > I think that's reasonable? Maybe we don't even need to modify and use
>  > > the template content, just reference other test cases and say that
>  > > any of them is fine.
>  > 
>  > Yeah, that would be OK, I think. Of course in a sane TCMS we'd have a
>  > better way of saying 'out of these tests, at least one needs to be done
>  > for each environment' and calculating coverage and such, but this is
>  > Wikitcms ;)
>  
>  I have made the changes:
>  https://fedoraproject.org/wiki/Template:Installation_test_matrix#Upgrade
>  
>  This is the new test case:
>  https://fedoraproject.org/wiki/QA:Testcase_upgrade_dnf_previous_any
>  
>  Originally I wanted to use something like this in the test case:
>  {{Testcase upgrade|(any flavor)}}
>  
>  However, that didn't work and I could wrap my mind around all the templates, so I
> ended up with that bare-bones description. I hope it's fine, patches welcome :)
I like it the way it is anyway, it's clearer that it's just a
'synthetic' entry in the table. Thanks.

We will need to adjust openqa_fedora_tools for this, I think - I'll
check that.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net