Re: Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Fri, 2022-09-02 at 08:37 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > 
>  > Now, because I glued openQA to the critpath because it was handy, there
>  > are two sets of consequences to a package being in critical path:
>  > 
>  > 1. Tighter Bodhi requirements
>  > 2. openQA tests are run, and results gate the update (except Rawhide)
>  > 
>  > So, one of the implicit questions here is, is it OK to keep twinning
>  > these two sets of consequences, or should we split them up? Splitting
>  > them up kinda implies answer 2) from my original mail: "Keep the
>  > current "critical path" concept but define a broader group
>  > of "gated packages" somewhere". Because we would then need some new
>  > concept that isn't "critical path". As I said, that's more *work*
> -
>  > it'd require us to write new code in several places[0]. Even if we
>  > decide it'd be nice to do this, is it nice *enough* to be worth doing
>  > that work?
>  
>  I'd still vote for keeping a single critpath list and using it as
>  "the list of packages that require extra care and testing".
>  
>  As you describe, the original meaning of critpath has shifted, but
>  it's because the way we do updates and QA has also shifted. Doing
>  gating tests for a package seems much more useful than just keeping
>  it longer in 'updates-testing' in hope that somebody discovers an
>  important regresion in the second week. 
Well, there's a caveat there - openQA doesn't test everything. On the
whole we cover quite a lot with the set of tests that gets run on
updates, but there's certainly lots of potential for there to be
important bugs it misses, that a human tester might catch. So I think
there is still a case for the higher karma requirements too.
>  
>  So yeah, I don't think it makes sense to do the extra work to split
>  the concepts. Also because we have way too many concepts and processes
>  in Fedora already. 
On the whole, though, I agree with you. I just don't trust my own
opinion because it's obviously biased by what's convenient for me. :D

>  > If we don't think it's worth doing that work, then
> we're kinda stuck
>  > with openQA glomming onto the critpath definition to decide which
>  > updates to test and gate, because I don't have any other current viable
>  > choices for that, really. And we'd have to figure out a critpath
>  > definition that's as viable as possible for both purposes.
>  > 
>  > 
>  > BTW, one other thought I've had in relation to all this is that we
>  > could enhance the current critpath definition somewhat. Right now, it's
>  > built out of package groups in comps which are kinda topic-separated:
>  > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
>  > But the generated critical path package list is a monolith: it doesn't
>  > distinguish between a package that's on the GNOME critpath and a
>  > package that's on the KDE critpath, you just get a big list of all
>  > critpath packages. It might be nice if we actually did distinguish
>  > between those - the critpath definition could keep track of which
>  > critpath topic(s) a package is included in, and Bodhi could display
>  > that information in the web UI and provide it via the API. That way
>  > manual testers could get a bit more info on why a package is critpath
>  > and what areas to test, and openQA could potentially target its test
>  > runs to conserve resources a bit, though this might require a bit more
>  > coding work on the gating stuff now I think about it.
>  
>  That sounds useful. We only need a volunteer to figure out the details
>  and do the work ;) 
I actually did a huge rewrite of the thing that generates the critpath
data this week, and it probably wouldn't be tooooo much work, honestly.
The most annoying bit would be the Bodhi frontend stuff, but that's
because I'm bad at frontend dev in general. :P But yeah, this is
definitely off in sky-castle land. I'll add it to my ever-growing list
of sky-castle projects to do when I get a couple of years of spare
time...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Fri, 2022-09-02 at 10:26 -0700, Adam Williamson wrote:
>  
>  > > If we don't think it's worth doing that work, then we're kinda
> stuck
>  > > with openQA glomming onto the critpath definition to decide which
>  > > updates to test and gate, because I don't have any other current viable
>  > > choices for that, really. And we'd have to figure out a critpath
>  > > definition that's as viable as possible for both purposes.
>  > > 
>  > > 
>  > > BTW, one other thought I've had in relation to all this is that we
>  > > could enhance the current critpath definition somewhat. Right now, it's
>  > > built out of package groups in comps which are kinda topic-separated:
>  > > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
>  > > But the generated critical path package list is a monolith: it doesn't
>  > > distinguish between a package that's on the GNOME critpath and a
>  > > package that's on the KDE critpath, you just get a big list of all
>  > > critpath packages. It might be nice if we actually did distinguish
>  > > between those - the critpath definition could keep track of which
>  > > critpath topic(s) a package is included in, and Bodhi could display
>  > > that information in the web UI and provide it via the API. That way
>  > > manual testers could get a bit more info on why a package is critpath
>  > > and what areas to test, and openQA could potentially target its test
>  > > runs to conserve resources a bit, though this might require a bit more
>  > > coding work on the gating stuff now I think about it.
>  > 
>  > That sounds useful. We only need a volunteer to figure out the details
>  > and do the work ;)
>  
>  I actually did a huge rewrite of the thing that generates the critpath
>  data this week, and it probably wouldn't be tooooo much work, honestly.
>  The most annoying bit would be the Bodhi frontend stuff, but that's
>  because I'm bad at frontend dev in general. :P But yeah, this is
>  definitely off in sky-castle land. I'll add it to my ever-growing list
>  of sky-castle projects to do when I get a couple of years of spare
>  time... 
So, an update on this whole area!

First off, I'm actually finding the time to do the sky-castle work. The
releng critpath script now outputs critpath data by group:
https://pagure.io/releng/c/621caa542acc142d57f1247e7644846f737f8eee?branc...

Bodhi (git HEAD Bodhi, anyway) can now read data in that format:
https://github.com/fedora-infra/bodhi/pull/4755

and when this PR is merged, it will be able to mark updates as critpath
by groups:
https://github.com/fedora-infra/bodhi/pull/4759

This has the handy bonus effect of enabling us to remove one of our
remaining uses of PDC. Once the Bodhi work is merged, I can send a PR
for the releng ansible repo to define per-group greenwave policies, run
the critpath.py script nightly on the Bodhi boxes, and switch to the
'json' critpath type in Bodhi config, and finally I can then enhance
the openQA scheduler to only run the necessary tests for each update.

Second, since nobody really opposed the idea of extending the critpath
definition slightly, here's a formal proposal to implement that. I want
to edit the wiki critpath page:
https://fedoraproject.org/wiki/Critical_path_package

and change it as follows:

1. Under Background, change "The packages required to sustain these
actions make up the critical path." to:

"The packages required to sustain these actions initially made up the
critical path. Later, it was agreed that the maintainers of Editions,
spins, and labs can also declare packages that provide other key
functionality to be part of the ''critical path'' for that Edition,
spin or lab."

2. Under Actions, change the first two sentences to read:

"Packages required to perform the most fundamental actions on a system
are always considered part of the ''critical path''. Those actions
include: "

Does anyone object to these proposed changes? Thanks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

Thoughts welcome: interface between automated test gating and the "critical path"

[Ben Cotton]

On Mon, Oct 17, 2022 at 8:54 PM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  Does anyone object to these proposed changes? Thanks! 
Ship it!


-- 
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis

Thoughts welcome: interface between automated test gating and the "critical path"

On 10/17/22 8:45 PM, Adam Williamson wrote:
>  On Fri, 2022-09-02 at 10:26 -0700, Adam Williamson wrote:
> >
> >>> If we don't think it's worth doing that work, then we're kinda
> stuck
> >>> with openQA glomming onto the critpath definition to decide which
> >>> updates to test and gate, because I don't have any other current viable
> >>> choices for that, really. And we'd have to figure out a critpath
> >>> definition that's as viable as possible for both purposes.
> >>>
> >>>
> >>> BTW, one other thought I've had in relation to all this is that we
> >>> could enhance the current critpath definition somewhat. Right now, it's
> >>> built out of package groups in comps which are kinda topic-separated:
> >>> there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> >>> But the generated critical path package list is a monolith: it doesn't
> >>> distinguish between a package that's on the GNOME critpath and a
> >>> package that's on the KDE critpath, you just get a big list of all
> >>> critpath packages. It might be nice if we actually did distinguish
> >>> between those - the critpath definition could keep track of which
> >>> critpath topic(s) a package is included in, and Bodhi could display
> >>> that information in the web UI and provide it via the API. That way
> >>> manual testers could get a bit more info on why a package is critpath
> >>> and what areas to test, and openQA could potentially target its test
> >>> runs to conserve resources a bit, though this might require a bit more
> >>> coding work on the gating stuff now I think about it.
> >>
> >> That sounds useful. We only need a volunteer to figure out the details
> >> and do the work ;)
> >
> > I actually did a huge rewrite of the thing that generates the critpath
> > data this week, and it probably wouldn't be tooooo much work, honestly.
> > The most annoying bit would be the Bodhi frontend stuff, but that's
> > because I'm bad at frontend dev in general. :P But yeah, this is
> > definitely off in sky-castle land. I'll add it to my ever-growing list
> > of sky-castle projects to do when I get a couple of years of spare
> > time...
>  
>  So, an update on this whole area!
>  
>  First off, I'm actually finding the time to do the sky-castle work. The
>  releng critpath script now outputs critpath data by group:
>  https://pagure.io/releng/c/621caa542acc142d57f1247e7644846f737f8eee?branc...
>  
>  Bodhi (git HEAD Bodhi, anyway) can now read data in that format:
>  https://github.com/fedora-infra/bodhi/pull/4755
>  
>  and when this PR is merged, it will be able to mark updates as critpath
>  by groups:
>  https://github.com/fedora-infra/bodhi/pull/4759
>  
>  This has the handy bonus effect of enabling us to remove one of our
>  remaining uses of PDC. Once the Bodhi work is merged, I can send a PR
>  for the releng ansible repo to define per-group greenwave policies, run
>  the critpath.py script nightly on the Bodhi boxes, and switch to the
>  'json' critpath type in Bodhi config, and finally I can then enhance
>  the openQA scheduler to only run the necessary tests for each update.
>  
>  Second, since nobody really opposed the idea of extending the critpath
>  definition slightly, here's a formal proposal to implement that. I want
>  to edit the wiki critpath page:
>  https://fedoraproject.org/wiki/Critical_path_package
>  
>  and change it as follows:
>  
>  1. Under Background, change "The packages required to sustain these
>  actions make up the critical path." to:
>  
>  "The packages required to sustain these actions initially made up the
>  critical path. Later, it was agreed that the maintainers of Editions,
>  spins, and labs can also declare packages that provide other key
>  functionality to be part of the ''critical path'' for that Edition,
>  spin or lab."
>  
>  2. Under Actions, change the first two sentences to read:
>  
>  "Packages required to perform the most fundamental actions on a system
>  are always considered part of the ''critical path''. Those actions
>  include: "
>  
>  Does anyone object to these proposed changes? Thanks! 

I think this is a great improvement over what we had.

	Have a Great Day!

	Pat	(tablepc)

Thoughts welcome: interface between automated test gating and the "critical path"

[Adam Williamson]

On Tue, 2022-10-18 at 09:37 -0400, pmkellly(a)frontier.com wrote:
>  > 
>  > Second, since nobody really opposed the idea of extending the critpath
>  > definition slightly, here's a formal proposal to implement that. I want
>  > to edit the wiki critpath page:
>  > https://fedoraproject.org/wiki/Critical_path_package
>  > 
>  > and change it as follows:
>  > 
>  > 1. Under Background, change "The packages required to sustain these
>  > actions make up the critical path." to:
>  > 
>  > "The packages required to sustain these actions initially made up the
>  > critical path. Later, it was agreed that the maintainers of Editions,
>  > spins, and labs can also declare packages that provide other key
>  > functionality to be part of the ''critical path'' for that Edition,
>  > spin or lab."
>  > 
>  > 2. Under Actions, change the first two sentences to read:
>  > 
>  > "Packages required to perform the most fundamental actions on a system
>  > are always considered part of the ''critical path''. Those actions
>  > include: "
>  > 
>  > Does anyone object to these proposed changes? Thanks!
>  
>  
>  I think this is a great improvement over what we had. 
So, I just realized I never actually made these changes! I've just done
so now:
https://fedoraproject.org/w/index.php?title=Critical_path_package&dif...
I also updated the information on how critpath is generated and how you
can view it, as that all changed as part of this too.
-- 
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net