Installation validation test change proposal: merge USB tests into 'default boot and install', add more environment columns

[Adam Williamson]

Hi folks! Sending this to devel@ as well as test@ as there's been some
relevant discussion there recently. We've been kicking around a couple
of issues lately:

1. Exactly what do we need to test and block on, in terms of writing
images to USB sticks?

2. 'Default boot and install' table was supposed to require bare metal
optical media testing at release time, but this is not clear, and
openQA fills in results despite not testing real media on bare metal.

So here's my proposal to address both topics. It involves changes to
the Boot_default_install test:

https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_...

and the Installation matrix template page:

https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installat...

to summarize the changes - we improve the test case's wording to give
explicit instructions for all three media types we care about (virtual
disc attached to a VM, real optical disc in a real machine, real USB
stick in a real machine), we ditch the USB test matrices entirely, and
we give the 'Default boot and install' tables extra result columns so
they can now reflect results for VM, CD/DVD and USB testing (for BIOS
and UEFI in all three cases, for x86_64).

I didn't get too finicky about the wording for the virt case, as that's
basically there to get filled in by openQA anyway. For the CD/DVD case,
the test case instructs you to write the disc according to the
'official instructions', with a URL that is supposed to be always
provided by the docs project for that purpose. For the USB case, the
test case instructs you to write the stick according to
https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling
you to use the 'most prominently-recommended method'; the intent being
that, as things stand, we would only block on Fedora Media Writer
issues. I didn't explicitly cover dd (e.g. with separate FMW and dd
columns) for two reasons:

1. It should pretty reliably be the case that if FMW works, dd works
2. Some people are going to test plain dd anyway, it's just such a
convenient 'expert' method that I can't imagine we won't have people
using it throughout the test period

I didn't cover livecd-iso-to-disk because it seems like, overall, the
feedback to cmurf's "F26 proposal: Make Fedora Media Writer the
officially supported USB install media creator" was fairly positive and
I think justifies not guaranteeing testing for it.

To be clear, this is a trade-off between 'in an ideal world we'd test
everything' and 'but we really don't have time'. Each additional method
adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the
test). We can discount the virt tests, as openQA will do those, so in
the proposal, there are 24 mandatory tests for humans; if we added dd
as its own thing there'd be 36, if we added litd as well there'd be 48.

One note is that this doesn't do much to ensure that FMW is working at
release time in all expected environments (probably at least the two
previous stable Fedora releases, macOS, and Windows). On the other
hand, neither does the current matrix; we explicitly list FMW, but we
don't specify what environments it should be run in. We could consider
adding a USB table with a slightly different focus, the idea being to
test the *writing tool* rather than test that the *image* can be
written and works properly. We could make that a follow-up proposal. If
we had such a table, it could maybe cover testing livecd-iso-to-disk's
advanced features as optional tests, or something.

At present I'm not proposing any changes to the *release criteria* (we
can maybe consider that later). This is just a proposal for what we
will actually guarantee testing of via the validation process.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Chris Murphy]

On Fri, Oct 7, 2016 at 2:08 PM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  Hi folks! Sending this to devel@ as well as test@ as there's been
> some
>  relevant discussion there recently. We've been kicking around a couple
>  of issues lately:
>
>  1. Exactly what do we need to test and block on, in terms of writing
>  images to USB sticks?
>
>  2. 'Default boot and install' table was supposed to require bare metal
>  optical media testing at release time, but this is not clear, and
>  openQA fills in results despite not testing real media on bare metal.
>
>  So here's my proposal to address both topics. It involves changes to
>  the Boot_default_install test:
>
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
>
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_...
>
>  and the Installation matrix template page:
>
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
>
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installat...
>
>  to summarize the changes - we improve the test case's wording to give
>  explicit instructions for all three media types we care about (virtual
>  disc attached to a VM, real optical disc in a real machine, real USB
>  stick in a real machine), we ditch the USB test matrices entirely, and
>  we give the 'Default boot and install' tables extra result columns so
>  they can now reflect results for VM, CD/DVD and USB testing (for BIOS
>  and UEFI in all three cases, for x86_64).
>
>  I didn't get too finicky about the wording for the virt case, as that's
>  basically there to get filled in by openQA anyway. For the CD/DVD case,
>  the test case instructs you to write the disc according to the
>  'official instructions', with a URL that is supposed to be always
>  provided by the docs project for that purpose. For the USB case, the
>  test case instructs you to write the stick according to
>  https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling
>  you to use the 'most prominently-recommended method'; the intent being
>  that, as things stand, we would only block on Fedora Media Writer
>  issues. I didn't explicitly cover dd (e.g. with separate FMW and dd
>  columns) for two reasons:
>
>  1. It should pretty reliably be the case that if FMW works, dd works
>  2. Some people are going to test plain dd anyway, it's just such a
>  convenient 'expert' method that I can't imagine we won't have people
>  using it throughout the test period
>
>  I didn't cover livecd-iso-to-disk because it seems like, overall, the
>  feedback to cmurf's "F26 proposal: Make Fedora Media Writer the
>  officially supported USB install media creator" was fairly positive and
>  I think justifies not guaranteeing testing for it.
>
>  To be clear, this is a trade-off between 'in an ideal world we'd test
>  everything' and 'but we really don't have time'. Each additional method
>  adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the
>  test). We can discount the virt tests, as openQA will do those, so in
>  the proposal, there are 24 mandatory tests for humans; if we added dd
>  as its own thing there'd be 36, if we added litd as well there'd be 48.
>
>  One note is that this doesn't do much to ensure that FMW is working at
>  release time in all expected environments (probably at least the two
>  previous stable Fedora releases, macOS, and Windows). On the other
>  hand, neither does the current matrix; we explicitly list FMW, but we
>  don't specify what environments it should be run in. We could consider
>  adding a USB table with a slightly different focus, the idea being to
>  test the *writing tool* rather than test that the *image* can be
>  written and works properly. We could make that a follow-up proposal. If
>  we had such a table, it could maybe cover testing livecd-iso-to-disk's
>  advanced features as optional tests, or something.
>
>  At present I'm not proposing any changes to the *release criteria* (we
>  can maybe consider that later). This is just a proposal for what we
>  will actually guarantee testing of via the validation process. 

Looks and sounds good to me.

-- 
Chris Murphy

[Kamil Paral]

>  Hi folks! Sending this to devel@ as well as test@ as there's been
> some
>  relevant discussion there recently. We've been kicking around a couple
>  of issues lately:
>
>  1. Exactly what do we need to test and block on, in terms of writing
>  images to USB sticks?
>
>  2. 'Default boot and install' table was supposed to require bare metal
>  optical media testing at release time, but this is not clear, and
>  openQA fills in results despite not testing real media on bare metal.
>
>  So here's my proposal to address both topics. It involves changes to
>  the Boot_default_install test:
>
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
>
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_...
>
>  and the Installation matrix template page:
>
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
>
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installat...
>
>  to summarize the changes - we improve the test case's wording to give
>  explicit instructions for all three media types we care about (virtual
>  disc attached to a VM, real optical disc in a real machine, real USB
>  stick in a real machine), we ditch the USB test matrices entirely, and
>  we give the 'Default boot and install' tables extra result columns so
>  they can now reflect results for VM, CD/DVD and USB testing (for BIOS
>  and UEFI in all three cases, for x86_64). 
It's a lot of empty fields, which can't be automated by definition. The new matrix
version seems to be more time consuming that the previous version. I wonder what we can do
to cut down the time spent on this. Here are a few ideas:

a) could we decouple boot from install?

I know we've had a few problems in the past that occurred only on bare metal and not
in VMs. But do I remember correctly that all of those were related to booting the
installer/live system, and not the installation? What we do here is that we repeat the
default installation over and over again, e.g. Workstation Live using BIOS and UEFI (which
results in a slightly different partition layout), from different sources (DVD in VM, DVD
on bare metal, USB). But do those environments (VM vs bare metal, DVD vs USB) actually
matter, is there a good reason to repeat them that many times? What if we reduce all of
this to:

Default install
===============
                        Result
Workstation Live         [ ]
Workstation netinst      [ ]
Everything netinst       [ ]
Server netinst           [ ]
Server DVD               [ ]
KDE Live                 [ ]

which would be satisfiable using ANY environment and it would just check that anaconda can
use that particular installation source correctly? The BIOS vs UEFI partitioning is
already covered thoroughly in the "Guided storage configuration" table.

The currently proposed "Default boot and install" section would stay the same,
but be renamed to "Default boot" and would just check that the image *boots*
into the installer in that particular environment.


b) do we need to insist on performing the *default* install?

If the previous idea doesn't fly, this is another way to improve testing time. For
example, we're installing the Server package set 12 times in the currently proposed
table ((Server DVD + Server netinst) * environments). Is that really necessary? What if we
did it once (in any environment), and then could do just a minimal install the remaining
11 times? Because installation takes most of the time here, esp. from optical media. And I
can't imagine how the environments could differ in such a way that would break default
installation set just in a particular environment but not a different installation set in
the same environment (esp. when heavily pushing dd-like based approach).

So again we would have a table like this:

Default package set
===================
                        Result
Workstation netinst      [ ]
Server netinst           [ ]
Server DVD               [ ]

which would be satisfiable in any environment (we already have "Package sets"
section in which this would fit nicely), and in the "Default boot and install"
table you could choose a different package set for installation. We could go even further
and allow arbitrary partitioning to be used in "Default boot and install", so
that it can be combined with our partitioning test cases. Again, that would allow us to
test multiple features together and save time, while not sacrificing the core thing to
test here (running the particular medium in a particular environment).


>  I didn't get too finicky about the wording for the virt case, as that's
>  basically there to get filled in by openQA anyway. For the CD/DVD case,
>  the test case instructs you to write the disc according to the
>  'official instructions', with a URL that is supposed to be always
>  provided by the docs project for that purpose. For the USB case, the
>  test case instructs you to write the stick according to
>  https://fedoraproject.org/wiki/How_to_create_and_use_Live_USB , telling
>  you to use the 'most prominently-recommended method'; the intent being
>  that, as things stand, we would only block on Fedora Media Writer
>  issues. I didn't explicitly cover dd (e.g. with separate FMW and dd
>  columns) for two reasons:
>
>  1. It should pretty reliably be the case that if FMW works, dd works
>  2. Some people are going to test plain dd anyway, it's just such a
>  convenient 'expert' method that I can't imagine we won't have people
>  using it throughout the test period
>
>  I didn't cover livecd-iso-to-disk because it seems like, overall, the
>  feedback to cmurf's "F26 proposal: Make Fedora Media Writer the
>  officially supported USB install media creator" was fairly positive and
>  I think justifies not guaranteeing testing for it. 
All of that sounds fine to me.

>  To be clear, this is a trade-off between 'in an ideal world we'd test
>  everything' and 'but we really don't have time'. Each additional method
>  adds 12 mandatory tests (6 images, both UEFI and BIOS variants of the
>  test). We can discount the virt tests, as openQA will do those, so in
>  the proposal, there are 24 mandatory tests for humans; if we added dd
>  as its own thing there'd be 36, if we added litd as well there'd be 48.
>
>  One note is that this doesn't do much to ensure that FMW is working at
>  release time in all expected environments (probably at least the two
>  previous stable Fedora releases, macOS, and Windows). On the other
>  hand, neither does the current matrix; we explicitly list FMW, but we
>  don't specify what environments it should be run in. We could consider
>  adding a USB table with a slightly different focus, the idea being to
>  test the *writing tool* rather than test that the *image* can be
>  written and works properly. We could make that a follow-up proposal. If
>  we had such a table, it could maybe cover testing livecd-iso-to-disk's
>  advanced features as optional tests, or something. 
Makes sense.

>  At present I'm not proposing any changes to the *release criteria* (we
>  can maybe consider that later). This is just a proposal for what we
>  will actually guarantee testing of via the validation process. 
If we go full steam ahead with optical disk testing as proposed (in the past, the
"default boot and install" test case said you can choose either USB or DVD, now
we have separate fields for both implying we need to test both) I'd suggest optical
media being only in the Final criteria.

[Adam Williamson]

On Mon, 2016-10-10 at 10:25 -0400, Kamil Paral wrote:

>  > Hi folks! Sending this to devel@ as well as test@ as there's
> been some
>  > relevant discussion there recently. We've been kicking around a couple
>  > of issues lately:
>  > 
>  > 1. Exactly what do we need to test and block on, in terms of writing
>  > images to USB sticks?
>  > 
>  > 2. 'Default boot and install' table was supposed to require bare metal
>  > optical media testing at release time, but this is not clear, and
>  > openQA fills in results despite not testing real media on bare metal.
>  > 
>  > So here's my proposal to address both topics. It involves changes to
>  > the Boot_default_install test:
>  > 
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
>  >
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Testcase_...
>  > 
>  > and the Installation matrix template page:
>  > 
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
>  >
> https://fedoraproject.org/w/index.php?title=User:Adamwill/Draft_Installat...
>  > 
>  > to summarize the changes - we improve the test case's wording to give
>  > explicit instructions for all three media types we care about (virtual
>  > disc attached to a VM, real optical disc in a real machine, real USB
>  > stick in a real machine), we ditch the USB test matrices entirely, and
>  > we give the 'Default boot and install' tables extra result columns so
>  > they can now reflect results for VM, CD/DVD and USB testing (for BIOS
>  > and UEFI in all three cases, for x86_64). 
It's a lot of empty fields, which can't be automated by definition. 

Yep, yep it is.

>  The new matrix version seems to be more time consuming that the
>  previous version. 
In fact, the count of 'empty spaces' is identical in both versions. In
the old version there were 24 empty spaces that humans were *supposed*
to be filling in (though this wasn't always really happening): 12 in
the "Default boot and install" table, and 12 in the USB table (it's a
4x3 table). In the new version, there are...24 empty spaces for humans
to fill in in the "Default boot and install (x86_64)" table.

The coverage is slightly *different*, but the amount of tests humans
are intended to run has not in fact changed. We lose the fudge factor
of openQA filling in the spaces I really intended humans to test on
real machines with real shiny discs, but that's kind of the *point* of
the change, so it's not a reason to not do it...

>   I wonder what we can do to cut down the time spent on this. Here
> are
>  a few ideas:
>  
>  a) could we decouple boot from install?
>  
>  I know we've had a few problems in the past that occurred only on
>  bare metal and not in VMs. But do I remember correctly that all of
>  those were related to booting the installer/live system, and not the
>  installation? What we do here is that we repeat the default
>  installation over and over again, e.g. Workstation Live using BIOS
>  and UEFI (which results in a slightly different partition layout),
>  from different sources (DVD in VM, DVD on bare metal, USB). But do
>  those environments (VM vs bare metal, DVD vs USB) actually matter, is
>  there a good reason to repeat them that many times? What if we reduce
>  all of this to:
>  
>  Default install
>  ===============
>                          Result
>  Workstation Live         [ ]
>  Workstation netinst      [ ]
>  Everything netinst       [ ]
>  Server netinst           [ ]
>  Server DVD               [ ]
>  KDE Live                 [ ]
>  
>  which would be satisfiable using ANY environment and it would just
>  check that anaconda can use that particular installation source
>  correctly? The BIOS vs UEFI partitioning is already covered
>  thoroughly in the "Guided storage configuration" table. 
>  The currently proposed "Default boot and install" section
> would stay
>  the same, but be renamed to "Default boot" and would just check that
>  the image *boots* into the installer in that particular environment. 
I think this is going a bit too far. There are, IIRC, at least two
known places beyond just initial boot where things can go wrong in ways
related to the install medium.

One is install target selection. anaconda has code for filtering out
'protected' devices which is intended (in part) to prevent you trying
to install to the USB stick you're running the install from. That code
can go wrong in various ways, including over-aggressive filtering, and
the bugs can be media-specific (e.g. there's no problem in a VM or from
a real optical disc, but it goes wrong when booting from a USB stick).

The other is use of the on-medium repository. This is mainly for the
Server DVD image. When you install from that image the install should
use the packages from the medium itself as the base repository (and not
go out and get them from the network); I believe we've seen a case
before where the Server DVD written to USB would boot OK, and even
install OK, but it didn't actually use the USB stick as a repository,
it acted like a network install image. When written to a DVD it worked
OK. That was a long time ago, though, and I don't have the bug
reference handy.

This is why the test case has this text:

"If the image under test is intended to act as a package source, it
must be the default repository and installation must in fact use the
media as the package source."

We *could* reduce the test case to 'boot, select a target disk and
start the installation process if using a DVD image', I guess. Would
this sufficiently address your concerns?

Of course it's *possible* that some weird bug could show up which made
bootloader configuration not work properly in some media-dependent way,
or something. But I don't recall us running into such a bug before.

>  > 
>  > At present I'm not proposing any changes to the *release criteria* (we
>  > can maybe consider that later). This is just a proposal for what we
>  > will actually guarantee testing of via the validation process.
>  
>  If we go full steam ahead with optical disk testing as proposed (in
>  the past, the "default boot and install" test case said you can
>  choose either USB or DVD, now we have separate fields for both
>  implying we need to test both) I'd suggest optical media being only
>  in the Final criteria. 
That was my intent, but it's a bit hard to express with the way we lay
out the wiki tables, unfortunately (hard to specify different
milestones for different environments). I can add an admon note above
the table, I guess, explaining precisely what's actually required for
each milestone. How does that sound?

Thanks for the feedback!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Kamil Paral]

>  > The new matrix version seems to be more time consuming that the
>  > previous version.
>  
>  In fact, the count of 'empty spaces' is identical in both versions. In
>  the old version there were 24 empty spaces that humans were *supposed*
>  to be filling in (though this wasn't always really happening): 12 in
>  the "Default boot and install" table, and 12 in the USB table (it's a
>  4x3 table). In the new version, there are...24 empty spaces for humans
>  to fill in in the "Default boot and install (x86_64)" table. 
Yes, their number is the same, but I meant the amount of time required. The USB test cases
allowed us to combine them with other test cases (e.g. partitioning) or perform a minimal
install, while the new test case doesn't (a default install is mandatory).

>  > a) could we decouple boot from install?
>  > 
>  > I know we've had a few problems in the past that occurred only on
>  > bare metal and not in VMs. But do I remember correctly that all of
>  > those were related to booting the installer/live system, and not the
>  > installation? What we do here is that we repeat the default
>  > installation over and over again, e.g. Workstation Live using BIOS
>  > and UEFI (which results in a slightly different partition layout),
>  > from different sources (DVD in VM, DVD on bare metal, USB). But do
>  > those environments (VM vs bare metal, DVD vs USB) actually matter, is
>  > there a good reason to repeat them that many times? What if we reduce
>  > all of this to:
>  > 
>  > Default install
>  > ===============
>  >                         Result
>  > Workstation Live         [ ]
>  > Workstation netinst      [ ]
>  > Everything netinst       [ ]
>  > Server netinst           [ ]
>  > Server DVD               [ ]
>  > KDE Live                 [ ]
>  > 
>  > which would be satisfiable using ANY environment and it would just
>  > check that anaconda can use that particular installation source
>  > correctly? The BIOS vs UEFI partitioning is already covered
>  > thoroughly in the "Guided storage configuration" table.
>  
>  > The currently proposed "Default boot and install" section would stay
>  > the same, but be renamed to "Default boot" and would just check that
>  > the image *boots* into the installer in that particular environment.
>  
>  I think this is going a bit too far. There are, IIRC, at least two
>  known places beyond just initial boot where things can go wrong in ways
>  related to the install medium.
>  
>  One is install target selection. anaconda has code for filtering out
>  'protected' devices which is intended (in part) to prevent you trying
>  to install to the USB stick you're running the install from. That code
>  can go wrong in various ways, including over-aggressive filtering, and
>  the bugs can be media-specific (e.g. there's no problem in a VM or from
>  a real optical disc, but it goes wrong when booting from a USB stick).
>  
>  The other is use of the on-medium repository. This is mainly for the
>  Server DVD image. When you install from that image the install should
>  use the packages from the medium itself as the base repository (and not
>  go out and get them from the network); I believe we've seen a case
>  before where the Server DVD written to USB would boot OK, and even
>  install OK, but it didn't actually use the USB stick as a repository,
>  it acted like a network install image. When written to a DVD it worked
>  OK. That was a long time ago, though, and I don't have the bug
>  reference handy.
>  
>  This is why the test case has this text:
>  
>  "If the image under test is intended to act as a package source, it
>  must be the default repository and installation must in fact use the
>  media as the package source." 
OK, you're right that there are some cases which could bypass our testing. That's
why I assumed this idea might not fly, even though I'd personally have no problem to
focus on "reasonable coverage" instead of full coverage. But there might be
better approaches to handle this than my "a)" proposal.

From all that you described above, it seems that none of the important features to be
tested should be negatively affected if we allow arbitrary partitioning or a different
package set (provided the default installation source is used). Or even other, like VNC or
basic video mode. What needs to stay fixed is just the install media + environment +
install source. Which essentially my "b)" proposal in the original email. That
would allow us to be more effective when going through "Default boot and
install" table while (hopefully) not sacrificing any test coverage. What do you think
about that?

>  
>  We *could* reduce the test case to 'boot, select a target disk and
>  start the installation process if using a DVD image', I guess. Would
>  this sufficiently address your concerns? 
That would definitely speed things up. But it feels to me a bit ugly to kill installations
during execution as an official test approach. Would the idea written above serve better?
It's not that fast and easy (the installations should still finish and you should
verify the system boots), but since it can be combined with other test cases, it should
not effectively slow us down (too much).

>  
>  Of course it's *possible* that some weird bug could show up which made
>  bootloader configuration not work properly in some media-dependent way,
>  or something. But I don't recall us running into such a bug before.
>  
>  > > 
>  > > At present I'm not proposing any changes to the *release criteria* (we
>  > > can maybe consider that later). This is just a proposal for what we
>  > > will actually guarantee testing of via the validation process.
>  > 
>  > If we go full steam ahead with optical disk testing as proposed (in
>  > the past, the "default boot and install" test case said you can
>  > choose either USB or DVD, now we have separate fields for both
>  > implying we need to test both) I'd suggest optical media being only
>  > in the Final criteria.
>  
>  That was my intent, but it's a bit hard to express with the way we lay
>  out the wiki tables, unfortunately (hard to specify different
>  milestones for different environments). I can add an admon note above
>  the table, I guess, explaining precisely what's actually required for
>  each milestone. How does that sound? 
I'd put "Alpha/Final" into the Milestone in the "Default boot and
install" table. And the testcase would contain an admon bar linking to a Final
criterion saying "installation must work from optical media". That could be
obvious enough (at least for us, initiated testers) and follow the same pattern we use for
other such mixed-milestone test cases. Wdyt?

[Adam Williamson]

On Tue, 2016-10-18 at 07:54 -0400, Kamil Paral wrote:

<snip>

>  OK, you're right that there are some cases which could bypass
> our
>  testing. That's why I assumed this idea might not fly, even though
>  I'd personally have no problem to focus on "reasonable coverage"
>  instead of full coverage. But there might be better approaches to
>  handle this than my "a)" proposal.
>  
>  From all that you described above, it seems that none of the
>  important features to be tested should be negatively affected if we
>  allow arbitrary partitioning or a different package set (provided the
>  default installation source is used). Or even other, like VNC or
>  basic video mode. What needs to stay fixed is just the install media
>  + environment + install source. Which essentially my "b)" proposal in
>  the original email. That would allow us to be more effective when
>  going through "Default boot and install" table while (hopefully) not
>  sacrificing any test coverage. What do you think about that? 
<snip>

>  That would definitely speed things up. But it feels to me a bit ugly
>  to kill installations during execution as an official test approach.
>  Would the idea written above serve better? It's not that fast and
>  easy (the installations should still finish and you should verify the
>  system boots), but since it can be combined with other test cases, it
>  should not effectively slow us down (too much). 
<snip>

So after some further consideration I think I'm on board with this, and
it turned out not to be too hard to implement. I have updated the draft
test case:

https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...

so it now basically says 'you can run any kind of install that boots
from the medium under test and uses the default repo, and if it
satisfies the Expected Results, we're good'.

Does this look better to you now?

>  > > If we go full steam ahead with optical disk testing as
> proposed (in
>  > > the past, the "default boot and install" test case said you can
>  > > choose either USB or DVD, now we have separate fields for both
>  > > implying we need to test both) I'd suggest optical media being only
>  > > in the Final criteria.
>  > 
>  > That was my intent, but it's a bit hard to express with the way we lay
>  > out the wiki tables, unfortunately (hard to specify different
>  > milestones for different environments). I can add an admon note above
>  > the table, I guess, explaining precisely what's actually required for
>  > each milestone. How does that sound?
>  
>  I'd put "Alpha/Final" into the Milestone in the "Default boot and
>  install" table. And the testcase would contain an admon bar linking
>  to a Final criterion saying "installation must work from optical
>  media". That could be obvious enough (at least for us, initiated
>  testers) and follow the same pattern we use for other such mixed-
>  milestone test cases. Wdyt? 
I think this is going a bit too far. I'm OK with not requiring complete
coverage at Alpha or Beta, but I think explicitly moving optical boot
to the Final criteria is too much. I think if we do find out that
optical boot is broken for a release-blocking medium at Alpha or Beta
stage that should still be a blocker. At least for now, until the
project as a whole decides we don't care about optical media any more.

So I went with a halfway house. I've updated the matrix draft too:

https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix

so it now lists the milestone as 'Alpha / Final' and has an admon that
explicitly explains the 'Expected coverage' for the section, saying we
expect a reasonable sample of tests for Alpha and Beta, and full
coverage for Final. Do you think that's sufficient?

Note that with the current wikitcms code, testcase_stats considers the
milestone for 'Alpha / Final' rows to be 'Final'...because it works
like this:

        for mile in ('Alpha', 'Beta', 'Final', 'Optional',
'Tier1', 'Tier2', 'Tier3'):
            if mile in cells[0]:
                milestone = mile

obviously that would be very trivial to tweak, I just thought I'd
mention it. I might tweak it to prefer the first milestone it finds
rather than the last, I'll just have to look and see if that would
cause any unexpected effects in other cases (I don't think we have
many, though).
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Chris Murphy]

On Wed, Oct 26, 2016 at 11:04 AM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:

>  So after some further consideration I think I'm on board with
> this, and
>  it turned out not to be too hard to implement. I have updated the draft
>  test case:
>
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_... 
LGTM.

VM BIOS/UEFI assumes attaching the ISO file to a virtual optical
device; not a virtual drive, correct? Does it make sense to make this
explicit?

I take it there's been actual cases of ISO files booting in the VM as
a virtual optical device; that when burned did not boot from physical
optical media?


-- 
Chris Murphy

[Adam Williamson]

On Wed, 2016-10-26 at 11:59 -0600, Chris Murphy wrote:
>  On Wed, Oct 26, 2016 at 11:04 AM, Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  
>  > So after some further consideration I think I'm on board with this, and
>  > it turned out not to be too hard to implement. I have updated the draft
>  > test case:
>  > 
>  > https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
>  
>  LGTM.
>  
>  VM BIOS/UEFI assumes attaching the ISO file to a virtual optical
>  device; not a virtual drive, correct? Does it make sense to make this
>  explicit? 
As I said in the original proposal, I'm not worrying too hard about the
VM case, because it's more or less entirely there to get filled in by
openQA. Humans can add results if they like, but it's not really that
important.

The test case does actually state this, though:

"If testing in a virtual machine, attach the image to the virtual
machine as a CD or DVD."

>  I take it there's been actual cases of ISO files booting in the
> VM as
>  a virtual optical device; that when burned did not boot from physical
>  optical media? 
Yes, that was the case with the F21 syslinux bug -
https://bugzilla.redhat.com/show_bug.cgi?id=1148087 - which is why it
took us a while to catch it.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Chris Murphy]

On Wed, Oct 26, 2016 at 2:46 PM, Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:

>  Yes, that was the case with the F21 syslinux bug -
>  https://bugzilla.redhat.com/show_bug.cgi?id=1148087 - which is why it
>  took us a while to catch it. 
Reading that and the syslinux change log, I'm guessing it was less a
syslinux bug than it was a work around in isolinux for buggy BIOS :-)
Small distinction I suppose.


-- 
Chris Murphy

[Kamil Paral]

>  So after some further consideration I think I'm on board with
> this, and
>  it turned out not to be too hard to implement. I have updated the draft
>  test case:
>  
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
>  
>  so it now basically says 'you can run any kind of install that boots
>  from the medium under test and uses the default repo, and if it
>  satisfies the Expected Results, we're good'.
>  
>  Does this look better to you now? 
Yes, thanks.

>  I think this is going a bit too far. I'm OK with not requiring
> complete
>  coverage at Alpha or Beta, but I think explicitly moving optical boot
>  to the Final criteria is too much. I think if we do find out that
>  optical boot is broken for a release-blocking medium at Alpha or Beta
>  stage that should still be a blocker. At least for now, until the
>  project as a whole decides we don't care about optical media any more. 
OK. However, unless you're strongly opposed, I'd like to start discussing exactly
this (requiring optical media to work only for Final, not before) and have it agreed on
for F26. From my POV, there's little gain in making sure optical works for Alpha/Beta.
Optical is important for conservative audience with very old machines, and is probably in
a similar position to i386. Also for giveaways. All of that is important for Final, but
isn't really for Alpha and Beta. I don't think we have too many users which
regularly install the latest and greatest betaversions from optical drives. (Also, the
risk of breaking just optical boot on bare metal seems pretty low, even though it happened
in the past).

>  
>  So I went with a halfway house. I've updated the matrix draft too:
>  
>  https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
>  
>  so it now lists the milestone as 'Alpha / Final' and has an admon that
>  explicitly explains the 'Expected coverage' for the section, saying we
>  expect a reasonable sample of tests for Alpha and Beta, and full
>  coverage for Final. Do you think that's sufficient? 
Just to have an idea, say that we have both VM* columns fully covered by OpenQA, but for
DVD columns with have just a single BIOS+UEFI result for Workstation Live. Would that be
sufficient for Beta? What do we consider "a reasonable coverage"?

>  
>  Note that with the current wikitcms code, testcase_stats considers the
>  milestone for 'Alpha / Final' rows to be 'Final'...because it works
>  like this:
>  
>          for mile in ('Alpha', 'Beta', 'Final',
> 'Optional', 'Tier1', 'Tier2',
>          'Tier3'):
>              if mile in cells[0]:
>                  milestone = mile
>  
>  obviously that would be very trivial to tweak, I just thought I'd
>  mention it. I might tweak it to prefer the first milestone it finds
>  rather than the last, I'll just have to look and see if that would
>  cause any unexpected effects in other cases (I don't think we have
>  many, though). 
Yeah, tweaking that would probably make sense.

[Adam Williamson]

On Thu, 2016-10-27 at 06:56 -0400, Kamil Paral wrote:
>  Just to have an idea, say that we have both VM* columns fully
> covered
>  by OpenQA, but for DVD columns with have just a single BIOS+UEFI
>  result for Workstation Live. Would that be sufficient for Beta? What
>  do we consider "a reasonable coverage"? 
Mmm, I kinda wanted to leave it a bit vague to give us wiggle room, but
I did put some guidelines in the text:

"at least some testing for all three media types, both firmware types,
and each major class of deliverable (netinst, live and DVD)"

But yeah, I think we can kinda work it out as we go along.

As it seems people are generally OK with the current version of the
proposal, I'm gonna go ahead and implement it now. I'm totally open to
discussing downgrading the importance of optical media in future, but I
think we can make that a separate thing.

Thanks for the feedback!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Matthew Miller]

On Fri, Oct 07, 2016 at 01:08:13PM -0700, Adam Williamson wrote:
>  explicit instructions for all three media types we care about
> (virtual
>  disc attached to a VM, real optical disc in a real machine, real USB
>  stick in a real machine), we ditch the USB test matrices entirely, and 
I think we should plan to drop out real optical disks sometime in the
next couple releases. (In line with "'in an ideal world we'd test
everything' and 'but we really don't have time'".) I think the
hardware
on which Fedora will run well which cannot boot from USB media is
vanishingly small.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Adam Williamson]

On Mon, 2016-10-10 at 11:42 -0400, Matthew Miller wrote:
>  On Fri, Oct 07, 2016 at 01:08:13PM -0700, Adam Williamson wrote:
>  > explicit instructions for all three media types we care about (virtual
>  > disc attached to a VM, real optical disc in a real machine, real USB
>  > stick in a real machine), we ditch the USB test matrices entirely, and
>  
>  
>  I think we should plan to drop out real optical disks sometime in the
>  next couple releases. (In line with "'in an ideal world we'd test
>  everything' and 'but we really don't have time'".) I think the
> hardware
>  on which Fedora will run well which cannot boot from USB media is
>  vanishingly small. 
Well, that's not the only consideration though. There's two other
factors I can think of:

1) Are there still cases where an admin would want to use optical media
on a machine that *could* boot via USB, for legitimate reasons? (I
genuinely don't know the answer to this).

2) What do we do about handouts (giving away media at conferences)?
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Matthew Miller]

On Mon, Oct 10, 2016 at 08:58:43AM -0700, Adam Williamson wrote:
>  Well, that's not the only consideration though. There's two
> other
>  factors I can think of:
>  1) Are there still cases where an admin would want to use optical media
>  on a machine that *could* boot via USB, for legitimate reasons? (I
>  genuinely don't know the answer to this). 
I can't think of in the Fedora case. I can image some situations where
install media is mandated to be read-only, but I doubt it's Fedora
being installed in those situations. The only other one I can think of
is "we've always done it that way", which I guess shouldn't be taken
completely for granted.


>  2) What do we do about handouts (giving away media at conferences)?
This is already a problem. We're giving away DVDs because it's
something to give away, not because it's what people really still want.
We need to find out some other take-homes that people find
useful and desirable *anyway*.

Based on feedback from Ambassadors, DVD images may still be useful
giveaways in regions with less access to bandwidth. I'm not sure what
to do about that.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Chris Murphy]

On Mon, Oct 10, 2016 at 2:59 PM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:

>  Based on feedback from Ambassadors, DVD images may still be useful
>  giveaways in regions with less access to bandwidth. I'm not sure what
>  to do about that. 
U.S. rural areas? :-D

Are we talking about the DVDs being useful in areas with no internet
at all, or just slow internet? And would an optical specific Live
Workstation spin for Ambassadors be a useful work around?


-- 
Chris Murphy

[Matthew Miller]

On Mon, Oct 10, 2016 at 10:00:05PM -0600, Chris Murphy wrote:
>  > Based on feedback from Ambassadors, DVD images may still be
> useful
>  > giveaways in regions with less access to bandwidth. I'm not sure what
>  > to do about that.
>  U.S. rural areas? :-D 
Possibly, although I think most people would be able to get to
somewhere were downloading a gigabyte or so wouldn't be a big deal.

>  Are we talking about the DVDs being useful in areas with no internet
>  at all, or just slow internet? 
I was thinking no internet or expensive metered connections. Slow...
eh, I downloaded djgpp over a second-hand 1200 baud modem in the early
90s. Kids today can set it a Fedora download overnight. :)

>  And would an optical specific Live Workstation spin for Ambassadors
>  be a useful work around? 
I think so.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Richard Ryniker]

> an optical specific Live Workstation spin 
Sounds like the proper category: will not block a regular Fedora release,
will not consume test resources for primary Fedora deliverables, and will
provide a focus for those with some stake in optical media.

While lack of community to support an "optical spin" may not mean there
are few who want it, it would mean there are insufficient people who will
invest their talent to create it.

[Chris Murphy]

On Tue, Oct 11, 2016 at 6:47 AM, Richard Ryniker <ryniker(a)alum.mit.edu&gt; wrote:
> >an optical specific Live Workstation spin
>
>  Sounds like the proper category: will not block a regular Fedora release,
>  will not consume test resources for primary Fedora deliverables, and will
>  provide a focus for those with some stake in optical media.
>
>  While lack of community to support an "optical spin" may not mean there
>  are few who want it, it would mean there are insufficient people who will
>  invest their talent to create it. 
All Fedora ISO's start out as ISO 9660, with El Torito extensions
added; and GRUB 2 and ISOLINUX to boot UEFI and BIOS systems
respectively. The ISO 9660 and El Torito parts are probably pretty
statically reliable. The gotcha comes from the ever changing
bootloaders, and there are two of them. Even if we standardized on one
of them, they're in effect two still, because BIOS and UEFI systems
use completely different binary files and boot mechanisms following
the El Torito part that enables discovery of the bootloader. So the
bottom line is, if we want optical to work, it has to be tested.
*shrug* Otherwise there's a chance we run into bugs like this [1]
which is an bug only affecting boot when the ISO image is burned onto
actual physical optical media, the problem doesn't happen with the
same image written to USB sticks.

Bolted onto that however, isohybrid adds an MBR, and then also a GPT
and APM (yes no kidding three partitions) are added. This is how we
boot Macs from optical or USB sticks, and everything else BIOS or UEFI
from USB sticks. This is for x86_64, I'm not sure what other archs'
ISO's look like.

Is there a less fragile way to do all of this? *shrug* It's a very
delicately, and eyebrow raising, balanced boat. Maybe the strongest
argument for thinning the heard of parts on these ISOs is, it'd be
easier to modify a single partition map to bring about persistence to
the USB stick, but that just makes us trade problems we know for
problems we don't know. There's no net increase in stability,
probably.


[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1148087


-- 
Chris Murphy

[Chris Murphy]

On Mon, Oct 10, 2016 at 9:42 AM, Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Fri, Oct 07, 2016 at 01:08:13PM -0700, Adam Williamson wrote:
> > explicit instructions for all three media types we care about (virtual
> > disc attached to a VM, real optical disc in a real machine, real USB
> > stick in a real machine), we ditch the USB test matrices entirely, and
>
>  I think we should plan to drop out real optical disks sometime in the
>  next couple releases. (In line with "'in an ideal world we'd test
>  everything' and 'but we really don't have time'".) I think the
> hardware
>  on which Fedora will run well which cannot boot from USB media is
>  vanishingly small. 
Quitting cold turkey sounds better than switching from active to
passive testing (only block if someone finds a failure), because all
it takes to block release is a last minute "can't boot this" bug.

Maybe it's practical to only support optical boot on either Lives or
netinstalls (and by extension server DVD). I'd say block on
netinstalls, just becase those can fit on either a DVD or CD, and
desktops and servers are more likely to still have optical drives.
Laptops just don't anymore for the most part. Part of why is dropping
DVD playback support in Windows 8 and 10. To get DVD playback support
now you have to buy the software separate from the OS. Licensing fees.

-- 
Chris Murphy

[Matthew Miller]

On Mon, Oct 10, 2016 at 12:57:30PM -0600, Chris Murphy wrote:
>  Maybe it's practical to only support optical boot on either Lives
> or
>  netinstalls (and by extension server DVD). I'd say block on
>  netinstalls, just becase those can fit on either a DVD or CD, and
>  desktops and servers are more likely to still have optical drives. 
That works for me as long as we can figure out how to message it.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader