Fedora Upgrade - release criteria update proposal

[Frantisek Zatloukal]

We have encountered a bug[0] which seemingly “broke” offline updates after
systems were upgraded from an older Fedora to Fedora 29 and had some
multilib packages installed. After the discussion at last week's Release
Retrospective meeting, I am proposing some changes to our blocking
criterions in order to address similar issues in the future:

What we test now

We are currently blocking on upgrading from Fedora n-1 and n-2 releases
only with default package set installed.

What we can test

We can alter our upgrade test cases to also verify that updates after an
upgrade work. The test case might look like this:

   -

   Install Fedora n-1
   -

   Upgrade to Fedora n (updates-testing disabled)
   -

   Enable updates-testing
   -

   Update to latest packages
   -

   Verify that upgrade and update went fine and that the multilib packages
   installed before are still present


Apart from that, we can add (Final Blocking) test case
upgrade_dnf_current_workstation_multilib /
upgrade_dnf_current_minimal_multilib which will test upgrades (and then
updates as described above) with at least one i686 package installed on
x86_64 system. The other (less-generic, but closer to what users probably
do have on their systems) approach would be to test upgrade with steam (or
wine) installed.


What are our opinions about this?

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1642796

[Adam Williamson]

On Sun, 2018-11-18 at 21:48 +0100, Frantisek Zatloukal wrote:
>  We have encountered a bug[0] which seemingly “broke” offline updates
> after
>  systems were upgraded from an older Fedora to Fedora 29 and had some
>  multilib packages installed. After the discussion at last week's Release
>  Retrospective meeting, I am proposing some changes to our blocking
>  criterions in order to address similar issues in the future: 
I don't think the bug had anything to do with upgrades, did it? AIUI it
would also affect a fresh F29 install to which multilib packages had
been added.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Kamil Paral]

On Sun, Nov 18, 2018 at 9:49 PM Frantisek Zatloukal <fzatlouk(a)redhat.com&gt;
wrote:

>  We have encountered a bug[0] which seemingly “broke” offline updates
> after
>  systems were upgraded from an older Fedora to Fedora 29 and had some
>  multilib packages installed. After the discussion at last week's Release
>  Retrospective meeting, I am proposing some changes to our blocking
>  criterions in order to address similar issues in the future:
>
>  What we test now
>
>  We are currently blocking on upgrading from Fedora n-1 and n-2 releases
>  only with default package set installed.
>
>  What we can test
>
>  We can alter our upgrade test cases to also verify that updates after an
>  upgrade work. The test case might look like this:
>
>     -
>
>     Install Fedora n-1
>     -
>
>     Upgrade to Fedora n (updates-testing disabled)
>     -
>
>     Enable updates-testing
>     -
>
>     Update to latest packages
>     -
>
>     Verify that upgrade and update went fine and that the multilib
>     packages installed before are still present
>
>
>  Apart from that, we can add (Final Blocking) test case
>  upgrade_dnf_current_workstation_multilib /
>  upgrade_dnf_current_minimal_multilib which will test upgrades (and then
>  updates as described above) with at least one i686 package installed on
>  x86_64 system. The other (less-generic, but closer to what users probably
>  do have on their systems) approach would be to test upgrade with steam (or
>  wine) installed.
>
>
>  What are our opinions about this?
>
>  [0] https://bugzilla.redhat.com/show_bug.cgi?id=1642796
>  
I think Adam is right and this was not related to just upgraded systems. So
the proposal can be to add a "multilib update must work" criterion (and
optionally add a test case for it). The question is whether it is needed.
If we knew that multilib updates were broken when we discussed that during
the blocker bug meeting, would we have accepted it as a blocker (as
affecting a large portion of our user base), or not? I think we would've
accepted it as a blocker. Updates must work in general, not just for a
default system installation. So if others agree with me on this, no new
criterion might be needed. If others disagree, they'll probably also
disagree with a dedicated criterion for exactly this use case :-) A new
test case can of course be added, sure. Or perhaps suggest an optional step
to the existing system update test case, if we don't want to have a fully
dedicated test case for this.