post go upgrades (re)testing

[Kevin Fenzi]

Hey folks. 

I wonder if we couldn't add in some (re)testing of upgrades after a release
is 'go' but before it's actually released. We hit at least two issues I
am aware of with f34 due to multilib. ;( 

first: 
pipewire.i686 is in the base x86_64 repo and users had/have it installed.

pipewire.i686 is pulled into the x86_64 base repo by mutter (mutter.i686
is there and requires pipewire.i686). 

In any case it is not in the updates repo, so once an update of them
is pushed (which it has been), upgrades fail due to the missing
packages. 

I have modified the updates pungi config to whitelist pipewire for
multilib and it's in f34-updates now. 

second: 
There's still an issue with iptables. 
See https://bugzilla.redhat.com/show_bug.cgi?id=1953178
basically the version in the base repo has a 'iptables' package. 
The one in updates Obsoletes this for a iptables-compat, but yet it's
not doing things correctly as users are getting dnf errors. 

Anyhow, I think it might be good to perhaps schedule some re-resting of
upgrades after the 0 day updates repo is populated to try and catch
these and fix them before release day. 

We can't test this fully before there is an updates repo (I mean we kind
of can with updates-testing, but it's not the exact same repo). 

kevin

[Kamil Paral]

I wonder what the best way to improve the situation would be. There are two
issues as a I see it:
a) Some packages (pipewire.i686) are not in the default installation and
therefore our automated checks don't catch it. The manual checks are often
done with default installations as well, but someone with a real-world
machine can stumble on it.
b) Some packages (I think this is the case of iptables) are in a default
installation, but are in updates-testing, which we ignore during upgrade
testing. On one hand, it makes sense, we want to make sure the stable repo
is functional, and updates-testing often includes broken stuff which could
prevent us from running the upgrade test completely. On the other hand, on
the release day, a lot of packages get pushed from updates-testing to
updates, and it might result in broken upgrades, while we had no knowledge
about it. To complicate it even further, it's a constantly moving target,
i.e. the set of packages which are "pending stable" (to be pushed to
updates when the freeze is over) can change literally every minute.

The true fix here for both cases would be a static test which would check
the installability of packages, and wouldn't allow pushing updates which
don't pass. Something similar to rpmdeplint which we used to run in
Taskotron (rpmdeplint itself had many issues, it would need to get much
improved or written differently, to be able to make it a reliable gating
test). But we don't have it and likely won't have it for a long time.

So what can we do which is easier and at least partly covers the issues?
For a) I think we need to depend on community reporting, because we can
hardly test a random combination of installed packages. But for b), we
could extend our upgrade testing matrices with two variants - one would
test with stable updates, the other would test with updates-testing
enabled. It is not perfect, because any broken update in updates-testing
can obscure other issues present ("dnf --exclude" is not always the
remedy), so we still can miss important problems. We can't force package
maintainers to unpush problematic updates just because we want to test the
rest of the repo. And even if we could, refreshing the repo takes at least
a day, so there's a very slow turnaround. However, it gives us more chance
to spot potential issues and make sure that they are reported and that they
don't get pushed stable (by giving them negative karma). Thoughts?

We can also test upgrades once the updates repo gets populated, as you
mentioned. It's pretty late to detect problems at this stage :-/ , but
better late than never, I guess. I wonder if we could use openQA for this
[1]. Adam?

[1] Which reminds me, we should make sure to run "dnf system-upgrade" with
"--best", to catch these issues.

[Sérgio Basto]

On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
>  Hey folks. 
>  
>  I wonder if we couldn't add in some (re)testing of upgrades after a
>  release
>  is 'go' but before it's actually released. We hit at least two issues I
>  am aware of with f34 due to multilib. ;( 
>   
I'm asking this for years and to not stress and delay to much the
release, I proposed do a second release in this case it would be 34.1


>  first: 
>  pipewire.i686 is in the base x86_64 repo and users had/have it
>  installed.
>  
>  pipewire.i686 is pulled into the x86_64 base repo by mutter
>  (mutter.i686
>  is there and requires pipewire.i686). 
>  
>  In any case it is not in the updates repo, so once an update of them
>  is pushed (which it has been), upgrades fail due to the missing
>  packages. 
>  
>  I have modified the updates pungi config to whitelist pipewire for
>  multilib and it's in f34-updates now. 
>  
>  second: 
>  There's still an issue with iptables. 
>  See https://bugzilla.redhat.com/show_bug.cgi?id=1953178
>  basically the version in the base repo has a 'iptables' package. 
>  The one in updates Obsoletes this for a iptables-compat, but yet it's
>  not doing things correctly as users are getting dnf errors. 
>  
>  Anyhow, I think it might be good to perhaps schedule some re-resting of
>  upgrades after the 0 day updates repo is populated to try and catch
>  these and fix them before release day. 
>  
>  We can't test this fully before there is an updates repo (I mean we
>  kind
>  of can with updates-testing, but it's not the exact same repo). 
>  
>  kevin
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: 
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Kevin Fenzi]

On Wed, Apr 28, 2021 at 10:47:39AM +0100, Sérgio Basto wrote:
>  On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
>  > Hey folks. 
>  > 
>  > I wonder if we couldn't add in some (re)testing of upgrades after a
>  > release
>  > is 'go' but before it's actually released. We hit at least two issues I
>  > am aware of with f34 due to multilib. ;( 
>  > 
>  
>  I'm asking this for years and to not stress and delay to much the
>  release, I proposed do a second release in this case it would be 34.1 
That would be prohibive on resources and there would be a lot of process
that we do now once per 6 months that would need to be done in
days/weeks. :( So no, I dont think thats a good answer unless we can't
avoid it. 

kevin

[Sérgio Basto]

On Wed, 2021-04-28 at 10:14 -0700, Kevin Fenzi wrote:
>  On Wed, Apr 28, 2021 at 10:47:39AM +0100, Sérgio Basto wrote:
>  > On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
>  > > Hey folks. 
>  > > 
>  > > I wonder if we couldn't add in some (re)testing of upgrades after
>  > > a
>  > > release
>  > > is 'go' but before it's actually released. We hit at least two
>  > > issues I
>  > > am aware of with f34 due to multilib. ;( 
>  > > 
>  > 
>  > I'm asking this for years and to not stress and delay to much the
>  > release, I proposed do a second release in this case it would be
>  > 34.1
>  
>  That would be prohibive on resources and there would be a lot of
>  process
>  that we do now once per 6 months that would need to be done in
>  days/weeks. :( So no, I dont think thats a good answer unless we
>  can't
>  avoid it.  
I had an idea and if instead of calling it stable, we call it RC1
(release candidate 1) and only after adding all the updates in the base
version and (re) testing, we finally call it Fedora 34 stable ? 

>  kevin
>  _______________________________________________
>  test mailing list -- test(a)lists.fedoraproject.org
>  To unsubscribe send an email to test-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: 
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: 
>  https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: 
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: 
>  https://pagure.io/fedora-infrastructure 
-- 
Sérgio M. B.

[Kevin Fenzi]

On Tue, May 04, 2021 at 01:25:50AM +0100, Sérgio Basto wrote:
>  
>  I had an idea and if instead of calling it stable, we call it RC1
>  (release candidate 1) and only after adding all the updates in the base
>  version and (re) testing, we finally call it Fedora 34 stable ?  
Yeah, we could add a few days in and require the updates repo, but... 

It makes it really hard in some cases. Like, for example, say we have a
RC1, we make the updates repo, and we find some release blocking bug. We
make an update for it, but it needs to be in the base repo... so that
just 1 update has to compose into the base repo, but all other updates
have to go to updates-testing. ie, we have to do more things and it
becomes more complex. Bodhi isn't setup to handle both populating a
updates/updates testing repos and leting a branched compose compose just
some updates into the base repo. More states could be added of course,
but it's more work. 

kevin

[Adam Williamson]

On Tue, 2021-04-27 at 11:39 -0700, Kevin Fenzi wrote:
>  Hey folks. 
>  
>  I wonder if we couldn't add in some (re)testing of upgrades after a release
>  is 'go' but before it's actually released. We hit at least two issues I
>  am aware of with f34 due to multilib. ;( 
>  
>  first: 
>  pipewire.i686 is in the base x86_64 repo and users had/have it installed.
>  
>  pipewire.i686 is pulled into the x86_64 base repo by mutter (mutter.i686
>  is there and requires pipewire.i686). 
>  
>  In any case it is not in the updates repo, so once an update of them
>  is pushed (which it has been), upgrades fail due to the missing
>  packages. 
>  
>  I have modified the updates pungi config to whitelist pipewire for
>  multilib and it's in f34-updates now. 
>  
>  second: 
>  There's still an issue with iptables. 
>  See https://bugzilla.redhat.com/show_bug.cgi?id=1953178
>  basically the version in the base repo has a 'iptables' package. 
>  The one in updates Obsoletes this for a iptables-compat, but yet it's
>  not doing things correctly as users are getting dnf errors. 
>  
>  Anyhow, I think it might be good to perhaps schedule some re-resting of
>  upgrades after the 0 day updates repo is populated to try and catch
>  these and fix them before release day. 
>  
>  We can't test this fully before there is an updates repo (I mean we kind
>  of can with updates-testing, but it's not the exact same repo).  
We do sort of have this already, because openQA tests all updates, and
one of the tests it runs on them is an upgrade test.

Now, the iptables thing is interesting, because the logs of that test
on an F34 update show the issue:

2021-04-28T04:30:39-0400 WARNING 
 Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and
iptables-libs-1.8.7-6.fc34.x86_64
  - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) = 1.8.7-3.fc34,
but none of the providers can be installed
  - cannot install the best update candidate for package
iptables-libs-1.8.5-6.fc33.x86_64
  - cannot install the best update candidate for package iptables-1.8.5-6.fc33.x86_64

but it's only logged as a WARNING and does not prevent the upgrade
running. So the test passes. It doesn't even need to use `--
allowerasing` so it doesn't register as a soft fail.

For the pipewire issue, we won't catch it with automation because the
package isn't installed in any default F32 or F33 package sets.

Doing some manual testing might be possible, but then we were already a
bit tired from validating RC2 in five hours. People might have wanted a
break. And of course, signing RC2 off on Friday meant there was
virtually no office time for RH employees to do it in...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Kamil Paral]

On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson <adamwill(a)fedoraproject.org&gt;
wrote:

>  We do sort of have this already, because openQA tests all updates,
> and
>  one of the tests it runs on them is an upgrade test.
>
>  Now, the iptables thing is interesting, because the logs of that test
>  on an F34 update show the issue:
>
>  2021-04-28T04:30:39-0400 WARNING
>   Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and
>  iptables-libs-1.8.7-6.fc34.x86_64
>    - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) =
>  1.8.7-3.fc34, but none of the providers can be installed
>    - cannot install the best update candidate for package
>  iptables-libs-1.8.5-6.fc33.x86_64
>    - cannot install the best update candidate for package
>  iptables-1.8.5-6.fc33.x86_64
>
>  but it's only logged as a WARNING and does not prevent the upgrade
>  running. So the test passes. It doesn't even need to use `--
>  allowerasing` so it doesn't register as a soft fail.
>  
I assume that's because iptables from the main (fedora) repo are used
instead. But the upgrade would fail if we added "--best" (I tested it).
Here's a thought. What if we modified the openQA test to first try with
"--best". If there's any error, log it as a softfail, and continue without
it (and if if fails again, log it as a hard error). This way, we can catch
*some* broken updates (it's still possible that one broken update obscures
a second one and we'll not know about the latter - but the benefit is
definitely there).

If we did it this way, what exactly happens with the softfailed result?
Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing
it soon enough?

As for manual testing, what do you think about my previous proposal?
>  we could extend our upgrade testing matrices with two variants - one
would test with stable updates, the other would test with updates-testing
enabled

[Kamil Paral]

Gmail doesn't display it when you put the quote below your text, let me
quote it differently:
As for manual testing, what do you think about my previous proposal?
"we could extend our upgrade testing matrices with two variants - one would
test with stable updates, the other would test with updates-testing enabled"

[Adam Williamson]

On Thu, 2021-04-29 at 10:09 +0200, Kamil Paral wrote:
>  On Wed, Apr 28, 2021 at 6:56 PM Adam Williamson
> <adamwill(a)fedoraproject.org&gt;
>  wrote:
>  
>  > We do sort of have this already, because openQA tests all updates, and
>  > one of the tests it runs on them is an upgrade test.
>  > 
>  > Now, the iptables thing is interesting, because the logs of that test
>  > on an F34 update show the issue:
>  > 
>  > 2021-04-28T04:30:39-0400 WARNING
>  >  Problem: cannot install both iptables-libs-1.8.7-3.fc34.x86_64 and
>  > iptables-libs-1.8.7-6.fc34.x86_64
>  >   - package iptables-1.8.7-3.fc34.x86_64 requires iptables-libs(x86-64) =
>  > 1.8.7-3.fc34, but none of the providers can be installed
>  >   - cannot install the best update candidate for package
>  > iptables-libs-1.8.5-6.fc33.x86_64
>  >   - cannot install the best update candidate for package
>  > iptables-1.8.5-6.fc33.x86_64
>  > 
>  > but it's only logged as a WARNING and does not prevent the upgrade
>  > running. So the test passes. It doesn't even need to use `--
>  > allowerasing` so it doesn't register as a soft fail.
>  > 
>  
>  I assume that's because iptables from the main (fedora) repo are used
>  instead. But the upgrade would fail if we added "--best" (I tested it).
>  Here's a thought. What if we modified the openQA test to first try with
>  "--best". If there's any error, log it as a softfail, and continue without
>  it (and if if fails again, log it as a hard error). This way, we can catch
>  *some* broken updates (it's still possible that one broken update obscures
>  a second one and we'll not know about the latter - but the benefit is
>  definitely there).
>  
>  If we did it this way, what exactly happens with the softfailed result?
>  Does it disable Bodhi's autopush? Or does it entirely depend on QA noticing
>  it soon enough? 
So I looked into this some more yesterday. openQA did in fact catch
this bug, straight up: when the iptables update was initially
submitted, an openQA upgrade test failed on exactly this problem. The
upgrade succeeded, but the test failed because of the last-step check
we do in openQA that, if any package from the update is installed, it
is the version from the update that is installed. Because the upgrade
pulled in the earlier -3 version and not the -6 version from the
update, the test failed.

I mentioned this in Bodhi, even, but I noted that if the update was
pushed stable *before* F34's final release, the bug should disappear,
because the -3 builds would no longer be available as options and dnf
would in that case choose to go with -6 and remove iptables (I tested
this).

Unfortunately, the update was *not* pushed stable before final release,
so the -3 builds are in the frozen 'fedora' repo forever now, and we
have a more or less insoluble problem (probably the best option is just
to put the iptables package back in F34, and re-do this better for
F35).

See the bug report for more details:
https://bugzilla.redhat.com/show_bug.cgi?id=1953178

Given this, I don't think we need to use your idea, since it's
demonstrated that openQA *does* actually catch bugs like this already.
We just need maintainers to pay more attention and follow up better. 

>  
>  As for manual testing, what do you think about my previous proposal?
>  > we could extend our upgrade testing matrices with two variants - one
>  would test with stable updates, the other would test with updates-testing
>  enabled 
Sure, it's possible. It just adds more testing that either a human or a
robot needs to do. And, again, it turns out our current testing did
actually catch this problem just fine. If anything went wrong, it was
the follow-up.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net