We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
IRC: adamw | Twitter: adamw_ha
#fedora-meeting-2: Workstation WG (2023-05-30)
Meeting started by brainycmurf at 00:58:15 UTC. The full logs are
* Present members: Allan, Michael, Tomas, Kalev, Matthias, Chris, Jens,
Owen, Neal (brainycmurf, 00:58:36)
* Guests: (brainycmurf, 00:58:36)
* Regrets: N (brainycmurf, 00:58:36)
* Missing: (brainycmurf, 00:58:36)
* Secretary: Chris (brainycmurf, 00:58:36)
* Enable full preemption (brainycmurf, 00:58:37)
* LINK: https://pagure.io/fedora-workstation/issue/228 (brainycmurf,
* ACTION: Chris to write a change proposal (brainycmurf, 00:58:55)
* Discussion about setting up a Silverblue SIG (brainycmurf, 00:58:57)
* AGREED: Do we think that the Silverblue SIG should be a subgroup of
the Workstation WG? Probably not necessary, if there's an agreement
that Silverblue shouldn't deviate from Workstation. Also if a few of
us participate in the SIG. (brainycmurf, 00:59:20)
* ACTION: Tomas to post on the thread and some of us to engage with
the SIG discussion (brainycmurf, 00:59:24)
* Automatically uninstalling Anaconda post-install (brainycmurf,
* LINK: https://pagure.io/fedora-workstation/issue/242 (brainycmurf,
* AGREED: Deferred until we have Neal (brainycmurf, 00:59:38)
* Consider limiting journal size (brainycmurf, 00:59:40)
* LINK: https://pagure.io/fedora-workstation/issue/213 (brainycmurf,
* ACTION: Chris will circle back with systemd upstream and write up a
change proposal if it's ready (brainycmurf, 00:59:47)
* Red Hat Bugzilla: GNOME packages are in bad shape (brainycmurf,
* LINK: https://pagure.io/fedora-workstation/issue/131 (brainycmurf,
* AGREED: Start with a smaller subset of packages (maybe the ones
where the upstream is gitlab.gnome.org) (brainycmurf, 00:59:56)
* ACTION: Tomas will talk to Matthew Miller. Allan will summarize in
the ticket. (brainycmurf, 00:59:59)
* Announcements and status updates (brainycmurf, 01:00:01)
* The minutes from last meeting have been posted online:
Meeting ended at 01:00:13 UTC.
* Chris to write a change proposal
* Tomas to post on the thread and some of us to engage with the SIG
* Chris will circle back with systemd upstream and write up a change
proposal if it's ready
* Tomas will talk to Matthew Miller. Allan will summarize in the ticket.
Action Items, by person
* Tomas will talk to Matthew Miller. Allan will summarize in the
* Chris to write a change proposal
* Tomas to post on the thread and some of us to engage with the SIG
* Chris will circle back with systemd upstream and write up a change
proposal if it's ready
People Present (lines said)
* brainycmurf (46)
* zodbot (7)
* Allan (0)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
It's a bit hard to read format this time because zodbot dropped off irc
half way through and I had to manually capture the log from my irc client.
16:01 < kalev> #startmeeting Fedora Flatpak Packaging SIG
16:01 < kalev> #meetingname flatpak-sig
16:01 < kalev> #topic Init process
16:01 < zodbot> Meeting started Mon May 15 14:01:09 2023 UTC.
16:01 < zodbot> This meeting is logged and archived in a public location.
16:01 < zodbot> The chair is kalev. Information about MeetBot at
16:01 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link
16:01 -!- zodbot changed the topic of #fedora-meeting to: (Meeting
topic: Fedora Flatpak Packaging SIG)
16:01 < zodbot> The meeting name has been set to
16:01 < zodbot> The meeting name has been set to 'flatpak-sig'
16:01 -!- zodbot changed the topic of #fedora-meeting to: Init process
(Meeting topic: Fedora Flatpak Packaging SIG)
16:01 <+tpopela[m]> .hello tpopela
16:01 < zodbot> tpopela[m]: tpopela 'Tomas Popela' <tpopela(a)redhat.com>
16:01 <+OwenTaylor[m]> .hello otaylor
16:01 < zodbot> OwenTaylor[m]: otaylor 'Owen Taylor' <otaylor(a)redhat.com>
16:01 < kalev> we should have a lot to discuss today about Owen's proposals
16:02 <+JanGrulich[m]> .hello jgrulich
16:02 < zodbot> JanGrulich[m]: jgrulich 'Jan Grulich' <jgrulich(a)redhat.com>
16:03 < kalev> I think we have almost everyone here now. yselkowitz[m]
usually comes half an hour late I believe.
16:04 < kalev> #topic Flatpaks without Modules
16:04 -!- zodbot changed the topic of #fedora-meeting to: Flatpaks
without Modules (Meeting topic: Fedora Flatpak Packaging SIG)
16:04 < kalev> https://fedoraproject.org/wiki/Changes/FlatpaksWithoutModules
16:04 < kalev> sooo, Owen posted the change proposal and I think it
would be good to discuss this a bit
16:05 < travier> .hello siosm
16:05 < zodbot> travier: siosm 'Timothée Ravier' <travier(a)redhat.com>
16:05 < travier> 👋
16:05 < kalev> I myself think it's a major improvement over the way we
do flatpaks now. Using standard koji tags makes everything so much
easier to understand for most of the Fedora packagers
16:05 < kalev> OwenTaylor[m]: are there any specific points we should
16:06 < kalev> when looking at the proposed tag structure, I noticed two
16:06 <+OwenTaylor[m]> Maybe I first should ask if anybody has questions
- is the general way it's going to work clear to people? I'm sure that
some details will have to be worked out as we go
16:07 < kalev> one is that there's a bit of an inconsisteny between
f39-flatpak-runtime and f39-app tags naming - one has "flatpak" in it
and the other doesn't
16:07 <+OwenTaylor[m]> I filed
16:07 < kalev> ah nice, I missed that
16:08 <+OwenTaylor[m]> I'm fine with the second version in there where
the tags have flatpaks but the %dist are without it. While the 'f38-app'
thing made sense to me at the time, I certainly appreciate that
consistency is good
16:09 <+JanGrulich[m]> I assumed '-app' means like application, it never
occured to me it means just a different prefix
16:10 < kalev> same here :)
16:10 < sberg> +1
16:11 <+OwenTaylor[m]> I think we can just go with f38-flatpak-app.
Anybody have opinions about the two questions at the bottom of the ticket?
16:12 < kalev> regarding the %dist tag names, there's a bit of an
ordering question. Do we need the flatpak NVRs to be higher than
non-flatpak ones? I think it doesn't matter, but not entirely sure
16:12 < kalev> Fedora hasn't moved from fcXX to fXX because of ordering:
f39 sorts lower for rpm ordering than fc39, I believe
16:12 -!- misuto7 [~misuto(a)188.8.131.52] has quit [Remote host closed
16:13 -!- misuto7 [~misuto(a)184.108.40.206] has joined #fedora-meeting
16:13 <+OwenTaylor[m]> Hmmm. I dont' thnk you want to rely on ordering,
then you could accidentally pull in a non-Flatpak build that was newer.
16:14 -!- misuto7 [~misuto(a)220.127.116.11] has quit [Remote host closed
16:14 -!- misuto7 [~misuto(a)18.104.22.168] has joined #fedora-meeting
16:15 <+OwenTaylor[m]> My thought right now is to hav e dnf
of runtime packages),*.fc39app
16:16 < kalev> makes sense to me
16:16 <+OwenTaylor[m]> This is a slight variation on the way it works
right now, where we have a big includepkgs that the runtime packages,
and then includes the particular NVR's from the composed Flatpak modules
16:17 < kalev> the variation being that the *.fc39app packages would be
automatically depsolved and included, without an explicit include list?
16:18 <+OwenTaylor[m]> Yeah - anything in the *.f39app (hmmm, apparently
I can't keep .fc39app vs .f39app straight...)
16:18 <+OwenTaylor[m]> would just be available for installation
16:19 < kalev> I think that's a good simplification and makes it easier
to create flatpaks
16:20 < kalev> anyway, regarding the naming, I think I'd lean towards
.fc39app just for consistency with the rest of Fedora, but I think it
doesn't really matter if we wouldn't rely on .fc39app sorting higher in
rpm ordering than regular packages
16:21 <+OwenTaylor[m]> OK, I might go that way. It's probably a bigger
question "why doesn't this have the c, if that C meant something, why
this isn't it .fruntime39"
16:22 * kalev nods.
16:22 <+OwenTaylor[m]> On a related note, I also filed
track Michael Gruber's suggestion that we use a dist tag to distinguish
the containers in Koji rather than a name like firefox-flatpak
16:23 <+tpopela[m]> .funtime39!
16:23 < kalev> I think that makes a lot of sense
16:23 < kalev> what do you think yourself?
16:24 <+tpopela[m]> Owen Taylor: yes, that would be amazing..
16:24 < kalev> from a packager's point of view, they'd do firefox rpm
builds and firefox flatpak build and then they need to submit all of
that to bodhi
16:25 < kalev> if they share the name, then they could type "firefox" in
the bodhi update list and then bodhi could autocomplete that with all of
the various builds, including the flatpak one
16:25 -!- tdawson_ is now known as tdawson
16:25 <+OwenTaylor[m]> kalev: I think my pros and cons are all based on
what we do internally inside Red Hat. Pro: it's more consistent with the
firefox-container thing that we have for RHEL. Con: I always forget to
type 'firefox-container' into brew and wonder "OK, where are the
containers?" And that's all really irrelevant for Fedora :-)
16:26 * kalev nods.
16:26 <+OwenTaylor[m]> kalev: Hmm, would that work? You are thinking you
could do a single cross-release update with both a Flatpak and an RPM?
or just that they could choose the one they wanted without having to
type something different for the flatpak?
16:28 < kalev> bodhi can split up updates on its own, so when you file
an update that has both the rpm builds and a flatpak build, then bodhi
already knows that it needs to split it up into several updates (but the
packager only needs to fill in the form with update notes once)
16:28 <+yselkowitz[m]> .hello yselkowitz
16:28 < zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz'
16:29 < kalev> hi yselkowitz[m]!
16:30 <+OwenTaylor[m]> kalev: You could of course do that without having
a common name, but I guess having the common name encourages it.
16:30 * kalev nods.
16:30 < kalev> how does the KDE runtime fit into the picture? does it
need to copy all of the tag structure so that there's
f39-flatpak-kde5-app, f39-flatpak-kde6-app etc tags and all of that?
16:32 <+OwenTaylor[m]> One possible downside is that it might then be
more confusing that the release numbers aren't coordinated.
firefox-1.112.0-1.flatpak might contain firefox-1.112.0-4.fc38app or
16:32 <+OwenTaylor[m]> We could go with firefox-1.1112.0-4.1.flatpak -
but then of course that contains arbitrary versions of
mozilla-filesystem and so forth
16:33 <+OwenTaylor[m]> I don't think it's duplicated no. I think it's
all the same tag.
16:34 < kalev> oh, good point about versions being confusion
16:34 < kalev> confusing
16:35 < kalev> re the -candidate tag structure, I'd say that it's not
needed because usually what it does is that it holds the builds until
they are submitted to bodhi, but if they are just temporary artifacts
that are never submitted, then there shouldn't be any need for that
16:35 <+OwenTaylor[m]> We should be able to build KDE content into the
same tags - I dont' think we need different builds of anything for the
KDE runtime and the base runtime or for KDE apps and GNOME apps. If we
did, then ew'd have the same issues with normal Fedora packages
16:36 <+OwenTaylor[m]> Yeah, that's my genreal thinking - if we don't
have a bodhi step to promote, then why would we call the target -candidate
16:37 < kalev> we might need a flatpak specific -override tag though if
we need to override some build that needs to go quickly in a flatpak
runtime build, but for some reason don't want to do a global buildroot
override for that
16:37 < kalev> nice, good if we can avoid duplicating the tag structure
for KDE flatpaks
16:39 <+OwenTaylor[m]> kalev: So, basically we have some security update
or bugfix that we want to build runtimes with before it goes stable?
16:39 <+yselkowitz[m]> but how do you deal with packages which are in
one runtime and not the other but used by both gnome and kde apps?
16:40 <+OwenTaylor[m]> yselkowitz: Hmm, so the case where there is a
.fc39app build of libfoo because it's not in the GNOME runtime, but it
is in the KDE runtime
16:40 <+yselkowitz[m]> exactly
16:40 < kalev> OwenTaylor[m]: yes, for example firefox starts requiring
new nss that's only available in updates-testing, and we want to do a
runtime build that includes new nss before it goes stable
16:43 -!- zodbot [~supybot@fedora/bot/zodbot] has quit [Remote host
closed the connection]
16:43 <+OwenTaylor[m]> kalev: I think I need to write down the tag
inheritance for both the -flaptak-runtime-build and -flatpak-app-build
tags in detail :-)
16:43 -!- than
[~than(a)p200300f7df05c500af381d45f33992c4.dip0.t-ipconnect.de] has quit
[Quit: Konversation terminated!]
16:43 * kalev nods.
16:44 <+OwenTaylor[m]> yselkowitz: I think we definitely need to fix
that at the technology level and not at the "let's duplicate all the
16:44 -!- zodbot [~supybot@fedora/bot/zodbot] has joined #fedora-meeting
16:45 < kalev> one way to fix that might be to merge all of the runtimes
back into one :)
16:45 <+yselkowitz[m]> that would be gigantic, particularly with KF6 on
16:45 <+OwenTaylor[m]> yselkowitz: *Roughly* speaking, to the
includepkgs line, you add excludepkgs=qt6-*-fc38app for your kde builds
16:46 <+yselkowitz[m]> it's not the qtN/kfN packages that worry me, its
the "ordinary" packages which are pulled into one runtime but not the
other, not so easily deliniated
16:47 <+OwenTaylor[m]> yselkowitz: But do you have to compute what the
affected packages are before you write that dnf configuration?
16:48 <+OwenTaylor[m]> yselkowitz:yeah, I just used qt6 as an example, I
appreciate that it can affect innocuous looking leaf packages
16:52 <+OwenTaylor[m]> Note that this also in vague theory affects
things at build time - if things build differently against a -devel
subpackage from .fc38app. In practice, hopefully that doesn't matter.
16:53 <+yselkowitz[m]> if anything we should want to use the /app -devel
instead of the /usr -devel, and I've tried to use buildorder to do that
with my flatpaks so far, but there may be outlying issues
16:54 <+OwenTaylor[m]> yselkowitz: it's weird to be using the /app
-devel instead of the /usr -devel i fthe package is in /usr, though you
might actually *want* that - e.g. a pkgconfig file has
schema-installation-location=/usr/share/mylib or something
16:54 <+yselkowitz[m]> when would that happen??
16:56 < kalev> I guess it could happen if a source package is in both
runtime and also built for /app (maybe because a specific subpackage
wasn't part of the runtime and an app flatpak needed it)
16:56 <+yselkowitz[m]> that doesn't really work well if at all
16:57 < kalev> like, libheif that's part of the runtime and
libheif-pixbuf-loader from the same srpm that's included in eog flatpak.
do you then want /usr -devel or /app -devel ?
16:58 <+OwenTaylor[m]> yselkowitz: OK, let say we have 'libfoo' that
looks for plugins in /usr/lib/foo/plugins, it has a .pc file with a
variable meant to be used by dependent variables plugindir=/app/lib/foo
(or /app/foo if built with prefix=/app). It's in the KDE runtime but not
in the GNOME runtime and the KDE runtime sets the environment variable
16:58 <+yselkowitz[m]> often better to just include pixbuf-loader in
runtime, having packages in both creates all sorts of problems
16:59 < kalev> oh I agree, just trying to come up with a weird corner
16:59 <+yselkowitz[m]> trust me there are lots of corner cases here
16:59 < kalev> I think we need to wrap this up here. QA meeting is
starting in the same channel now
16:59 <+OwenTaylor[m]> then if you bundle an a package providing a
libfoo in a KDE app, it's actually better if it builds against the
libfoo-devel......fc38app rather than the the original one, even though
we're using the libfoo in the runtime.
17:00 < kalev> let's continue in the flatpak channel if there's anything
more to discuss - thanks everybody for coming and a huge thanks to Owen
for getting the module migration starting
17:00 < kalev> #endmeeting