Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
rmdepcheck: https://forge.fedoraproject.org/quality/rmdepcheck
rmdepcheck fails if the update under test, when compared against the current buildroot repository, contains unsatisfiable dependencies, or causes new unsatisfiable dependencies to appear in other packages. So if your update makes some other package uninstallable...that's a fail.
This check has been running on all updates for some time now. We've dealt with various sources of false failures and I'm fairly confident it's pretty reliable by now.
Gating all updates for stable releases on it would mean you could no longer ship updates with detectable dependency issues, unless you waive the failure - which implies explicitly taking responsibility for shipping a broken package, or breaking another package.
I don't want to gate Rawhide or Branched on rmdepcheck yet, because sometimes it's kinda realistically necessary to bump an soname without being able to fix every single dependency, or something. But I think (or rather, Carl George pointed out - thanks, Carl) it's more reasonable for stable releases. After all, the update policy specifically says:
"Updates should be carefully considered with respect to their dependencies. An update that required (or provided) a new Python ABI, for example, would almost certainly not be allowed. ABI changes in general are very strongly discouraged, they force larger update sets on users and they make life difficult for third-party packagers."
EPEL has similar stability expectations.
Implementing this is technically trivial (it just requires adding a new rule to the greenwave policy), but it's obviously a pretty significant change, so I thought I'd propose it here and on Discourse for discussion first. I don't think it's a good fit for the Change process.
You can browse existing rmdepcheck results in resultsdb, if you want to look for false failures. Go to https://resultsdb.fedoraproject.org/results , click the Search button, set the 'testcase' to 'fedora-ci.koji-build.rmdepcheck.functional' and hit Go! That'll give you the last month of results. You can tweak the date range, and also search only for failed results, if you like. It's slightly awkward because currently results are reported per package, so testing an update with 500 packages in it produces 500 results that link back to the same execution. I do want to change it to report per update rather than per package, but I don't think that needs to be done before we start gating.
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
In general I think this is a good idea, and I would also think it would be a good idea for rawhide and branched. Failures could always be waived, and in case of "soname bump rebuild that is incomplete because something doesn't build" that would be a perfectly good reason to do so - and it would make it explicit that the packager who waived the failures took responsibility for that breakage - and provide documentation for it alongside the update. That said, stable releases and EPEL is likely less controversial, so starting with that sounds good too.
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
Fabio
On Mon, Jul 06, 2026 at 11:16:13PM +0200, Fabio Valentini wrote:
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
Yes, I am in favor too.
In general I think this is a good idea, and I would also think it would be a good idea for rawhide and branched. Failures could always be waived, and in case of "soname bump rebuild that is incomplete because something doesn't build" that would be a perfectly good reason to do so - and it would make it explicit that the packager who waived the failures took responsibility for that breakage - and provide documentation for it alongside the update. That said, stable releases and EPEL is likely less controversial, so starting with that sounds good too.
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
I suppose it might be a good idea to get a fesco ack here. But also, I think this should be flexable so if it's enabled and some big problem appears it can be disabled again until thats addressed, etc.
kevin
On Tue, Jul 7, 2026 at 12:56 PM Kevin Fenzi kevin@scrye.com wrote:
On Mon, Jul 06, 2026 at 11:16:13PM +0200, Fabio Valentini wrote:
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
Yes, I am in favor too.
In general I think this is a good idea, and I would also think it would be a good idea for rawhide and branched. Failures could always be waived, and in case of "soname bump rebuild that is incomplete because something doesn't build" that would be a perfectly good reason to do so - and it would make it explicit that the packager who waived the failures took responsibility for that breakage - and provide documentation for it alongside the update. That said, stable releases and EPEL is likely less controversial, so starting with that sounds good too.
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
I suppose it might be a good idea to get a fesco ack here. But also, I think this should be flexable so if it's enabled and some big problem appears it can be disabled again until thats addressed, etc.
I think as a global change, I would prefer it to be submitted as a change to fesco for at least an ack, if not a formal proposal. It is effectively changing the contract for how things work.
On Tue, 2026-07-07 at 09:56 -0700, Kevin Fenzi wrote:
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
I suppose it might be a good idea to get a fesco ack here. But also, I think this should be flexable so if it's enabled and some big problem appears it can be disabled again until thats addressed, etc.
Also, that Change isn't really similar to the current case at all. That Change was really about building the *mechanism* to allow per-package opt-in gating, and it actually also encompassed the on-demand side tag creation work too. So it was a lot of detailed engineering work across multiple key components (Bodhi, greenwave, fedpkg etc.)
This is...not that. This is adding a couple of lines to https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshif... and running the playbook, which I can do in five minutes. And if it all goes pear-shaped, we revert the commit and run the playbook again.
I'll note we did not go through the Change process when enabling gating on openQA tests, either - neither when we initially did it for stable releases, nor when we extended it to Rawhide and Branched. I did the same thing as this, more or less - floated it on the mailing lists and in some chats, waited for feedback, then went ahead and did it. That's been working out fine.
On Tue, 2026-07-07 at 18:27 +0100, Adam Williamson wrote:
On Tue, 2026-07-07 at 09:56 -0700, Kevin Fenzi wrote:
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
I suppose it might be a good idea to get a fesco ack here. But also, I think this should be flexable so if it's enabled and some big problem appears it can be disabled again until thats addressed, etc.
Also, that Change isn't really similar to the current case at all. That Change was really about building the *mechanism* to allow per-package opt-in gating, and it actually also encompassed the on-demand side tag creation work too. So it was a lot of detailed engineering work across multiple key components (Bodhi, greenwave, fedpkg etc.)
This is...not that. This is adding a couple of lines to https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshif... and running the playbook, which I can do in five minutes. And if it all goes pear-shaped, we revert the commit and run the playbook again.
I'll note we did not go through the Change process when enabling gating on openQA tests, either - neither when we initially did it for stable releases, nor when we extended it to Rawhide and Branched. I did the same thing as this, more or less - floated it on the mailing lists and in some chats, waited for feedback, then went ahead and did it. That's been working out fine.
Whoops, forgot to say: I'd be fine with filing a FESCo ticket asking for approval, though.
On Tue, 2026-07-07 at 18:28 +0100, Adam Williamson wrote:
On Tue, 2026-07-07 at 18:27 +0100, Adam Williamson wrote:
On Tue, 2026-07-07 at 09:56 -0700, Kevin Fenzi wrote:
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages%C2%A0). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
I suppose it might be a good idea to get a fesco ack here. But also, I think this should be flexable so if it's enabled and some big problem appears it can be disabled again until thats addressed, etc.
Also, that Change isn't really similar to the current case at all. That Change was really about building the *mechanism* to allow per- package opt-in gating, and it actually also encompassed the on-demand side tag creation work too. So it was a lot of detailed engineering work across multiple key components (Bodhi, greenwave, fedpkg etc.)
This is...not that. This is adding a couple of lines to https://forge.fedoraproject.org/infra/ansible/src/branch/main/roles/openshif... and running the playbook, which I can do in five minutes. And if it all goes pear-shaped, we revert the commit and run the playbook again.
I'll note we did not go through the Change process when enabling gating on openQA tests, either - neither when we initially did it for stable releases, nor when we extended it to Rawhide and Branched. I did the same thing as this, more or less - floated it on the mailing lists and in some chats, waited for feedback, then went ahead and did it. That's been working out fine.
Whoops, forgot to say: I'd be fine with filing a FESCo ticket asking for approval, though.
I'm fine with a lightweight vote (and would vote in favour). As careful as one can be when submitting updates, mistakes happen (recently I had two updates that were in flight simultaneously, and one ended up having a dependency issue caused by the other landing first, but both were fine at the time of submission).
One tweak to this - can we force a re-run before promoting a package from testing->stable as well, because as the example above showed (this is the rpki 0.19.3-1 update vs the quick-xml 0.41 update, FWIW) - testing once might not be enough as the state might have changed in subsequent composes?
Best regards,
On Fri, 2026-07-10 at 18:21 +0100, Michel Lind wrote:
One tweak to this - can we force a re-run before promoting a package from testing->stable as well, because as the example above showed (this is the rpki 0.19.3-1 update vs the quick-xml 0.41 update, FWIW) - testing once might not be enough as the state might have changed in subsequent composes?
It's not trivial to do that, unfortunately, no. I don't think we need to solve that to implement the initial gating, though.
The way we actually do "promotion" for stable releases is that, once a day (usually, IIRC), a Bodhi task running on a timer tags all updates queued for stable at the time, assuming some other last-minute sanity checks pass. (Then there's a compose which causes the tagged packages to actually get sent out; I don't recall if that's part of the same process, or a separate one).
We *could* add a run of rmdepcheck to that process, I guess, but we'd either have to set up a whole new messaging flow to get the existing rmdepcheck pipeline to run it, or set up a new pipeline within the Bodhi task, I think.
I don't believe there's currently a flow where that process sends a message saying it's about to push things stable then can wait for external systems to Do Stuff and get their results before *actually* pushing stable, we'd have to build that.
Or else we'd have to wire a run of rmdepcheck inside of the Bodhi task itself, but that's also some work.
Either way is also more complicated if we want to do anything more complex then a straight up/down vote on the *entire set* of queued packages. Say we run rmdepcheck on the whole set of packages about to be pushed and it fails; are we OK with not pushing any of them? If not, we need to figure out (I guess) the largest set of the candidate packages that can be pushed without a failure, but that's not really trivial and could involve a lot of re-runs, depending on how many packages are queued...
On Fri, Jul 10, 2026 at 07:11:38PM +0100, Adam Williamson wrote:
On Fri, 2026-07-10 at 18:21 +0100, Michel Lind wrote:
One tweak to this - can we force a re-run before promoting a package from testing->stable as well, because as the example above showed (this is the rpki 0.19.3-1 update vs the quick-xml 0.41 update, FWIW) - testing once might not be enough as the state might have changed in subsequent composes?
It's not trivial to do that, unfortunately, no. I don't think we need to solve that to implement the initial gating, though.
The way we actually do "promotion" for stable releases is that, once a day (usually, IIRC), a Bodhi task running on a timer tags all updates queued for stable at the time, assuming some other last-minute sanity checks pass. (Then there's a compose which causes the tagged packages to actually get sent out; I don't recall if that's part of the same process, or a separate one).
My memory could be faulty, but I don't think thats how it's setup. There's a celery task that runs every ~5min and checks all the updates to see if they met stable requirements, if they have it adds the request to stable for them.
Then, once a day there's composes that actually process those requests.
We *could* add a run of rmdepcheck to that process, I guess, but we'd either have to set up a whole new messaging flow to get the existing rmdepcheck pipeline to run it, or set up a new pipeline within the Bodhi task, I think.
I don't believe there's currently a flow where that process sends a message saying it's about to push things stable then can wait for external systems to Do Stuff and get their results before *actually* pushing stable, we'd have to build that.
Yeah, there isn't 100%, but there is a 'bodhi submitted update for stable'. That could of course happen 1minute before the daily compose and there would be no time, but the rest of the day that would be an indicator that it's going stable (probibly).
Or else we'd have to wire a run of rmdepcheck inside of the Bodhi task itself, but that's also some work.
Either way is also more complicated if we want to do anything more complex then a straight up/down vote on the *entire set* of queued packages. Say we run rmdepcheck on the whole set of packages about to be pushed and it fails; are we OK with not pushing any of them? If not, we need to figure out (I guess) the largest set of the candidate packages that can be pushed without a failure, but that's not really trivial and could involve a lot of re-runs, depending on how many packages are queued...
yeah, could be a big problem indeed.
kevin
On Fri, 2026-07-10 at 12:07 -0700, Kevin Fenzi wrote:
On Fri, Jul 10, 2026 at 07:11:38PM +0100, Adam Williamson wrote:
On Fri, 2026-07-10 at 18:21 +0100, Michel Lind wrote:
One tweak to this - can we force a re-run before promoting a package from testing->stable as well, because as the example above showed (this is the rpki 0.19.3-1 update vs the quick-xml 0.41 update, FWIW) - testing once might not be enough as the state might have changed in subsequent composes?
It's not trivial to do that, unfortunately, no. I don't think we need to solve that to implement the initial gating, though.
The way we actually do "promotion" for stable releases is that, once a day (usually, IIRC), a Bodhi task running on a timer tags all updates queued for stable at the time, assuming some other last-minute sanity checks pass. (Then there's a compose which causes the tagged packages to actually get sent out; I don't recall if that's part of the same process, or a separate one).
My memory could be faulty, but I don't think thats how it's setup. There's a celery task that runs every ~5min and checks all the updates to see if they met stable requirements, if they have it adds the request to stable for them.
Right, but that's just the *request* for stable, right? Not the actual *push*. (And the thing you're describing is the *automatic* request mechanism, that requests stable when the update meets one of the autopush thresholds - manual requests are also possible).
Then, once a day there's composes that actually process those requests.
Sure, I guess that's the bit I was remembering/describing. So really there's sort of three points:
1. Update submitted / edited 2. Request for stable happens 3. Stable push happens
right? Right now we are only running rmdepcheck (and all other tests outside of Bodhi's internal sanity checks) at 1.
On Fri, Jul 10, 2026 at 08:13:53PM +0100, Adam Williamson wrote:
Right, but that's just the *request* for stable, right? Not the actual *push*. (And the thing you're describing is the *automatic* request mechanism, that requests stable when the update meets one of the autopush thresholds - manual requests are also possible).
yes.
But even the manual ones emit a message... foo requested update bar to stable.
Then, once a day there's composes that actually process those requests.
Sure, I guess that's the bit I was remembering/describing. So really there's sort of three points:
- Update submitted / edited
- Request for stable happens
- Stable push happens
right? Right now we are only running rmdepcheck (and all other tests outside of Bodhi's internal sanity checks) at 1.
Yep.
I think it shouldn't be too hard to hook into the second one, but of course that hits the issue you mentioned... checking against anything thats requested stable at the same time could result in some failure that you have to just not push anything or have a complex way to solve.
And there's no concept of a waiver for that either.
So, yeah, I would say leaving that for the future(tm) would be best.
kevin
On Tue, Jul 07, 2026 at 09:56:42AM -0700, Kevin Fenzi wrote:
On Mon, Jul 06, 2026 at 11:16:13PM +0200, Fabio Valentini wrote:
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
Yes, I am in favor too.
Yes, let's do this please.
Zbyszek
Dne 06. 07. 26 v 23:16 Fabio Valentini napsal(a):
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
In general I think this is a good idea, and I would also think it would be a good idea for rawhide and branched.
+1 including Rawhide / Branched
Vít
Failures could always be waived, and in case of "soname bump rebuild that is incomplete because something doesn't build" that would be a perfectly good reason to do so - and it would make it explicit that the packager who waived the failures took responsibility for that breakage - and provide documentation for it alongside the update. That said, stable releases and EPEL is likely less controversial, so starting with that sounds good too.
And while I kind of agree that the Change process does not seem to be *great* fit for this, but similar changes *have* gone through the Change process before (like https://fedoraproject.org/wiki/Changes/GatingRawhidePackages ). Should a system-wide change (lower case intentional) like this require at least *some* kind of signoff from FESCo?
Fabio
On Thu, 2026-07-09 at 15:55 +0200, Vít Ondruch wrote:
Dne 06. 07. 26 v 23:16 Fabio Valentini napsal(a):
On Mon, Jul 6, 2026 at 11:05 PM Adam Williamson adamwill@fedoraproject.org wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
... you have my bow! ... and my axe!
In general I think this is a good idea, and I would also think it would be a good idea for rawhide and branched.
+1 including Rawhide / Branched
For Rawhide I have a now-less-Sekrit Plan:
https://forge.fedoraproject.org/releng/tooling/src/branch/main/quality-assur...
that's a script that produces a list of "compose-critical packages", i.e. the packages which are used to build, or are included in, the non- failable images in the latest compose. My plan for Rawhide/Branched is to gate updates only if they break the dependencies of any compose- critical package, rather than gating on *all* failures.
That still requires a bit work, though. I need to get something to expose the compose-critical package data in a convenient way, and it's probably Bodhi, since it's Bodhi that stores and exposes the critical path package list (and I want Bodhi to use compose-critical package info for other reasons too). Then I have to adjust rmdepcheck and/or the CI pipeline to somehow make it convenient to distinguish between a 'ccp' and 'non-ccp' failure...
I suppose we *could* decide it's fine to gate Rawhide on all rmdepcheck failures and just require people to waive cases where they intentionally break other deps? Meh. I dunno.
On Thu, Jul 09, 2026 at 04:27:19PM +0100, Adam Williamson wrote:
For Rawhide I have a now-less-Sekrit Plan:
https://forge.fedoraproject.org/releng/tooling/src/branch/main/quality-assur...
that's a script that produces a list of "compose-critical packages", i.e. the packages which are used to build, or are included in, the non- failable images in the latest compose. My plan for Rawhide/Branched is to gate updates only if they break the dependencies of any compose- critical package, rather than gating on *all* failures.
That still requires a bit work, though. I need to get something to expose the compose-critical package data in a convenient way, and it's probably Bodhi, since it's Bodhi that stores and exposes the critical path package list (and I want Bodhi to use compose-critical package info for other reasons too). Then I have to adjust rmdepcheck and/or the CI pipeline to somehow make it convenient to distinguish between a 'ccp' and 'non-ccp' failure...
I suppose we *could* decide it's fine to gate Rawhide on all rmdepcheck failures and just require people to waive cases where they intentionally break other deps? Meh. I dunno.
Conceptually, that sounds fine to me... if you know you are breaking deps for some reason (thing not ready, but very leaf node, etc) doing a waiver to note that would make sense too.
However, it does mean you (and anyone who watches waivers) will probibly see more that you need to look at and see if they look reasonable.
Is there any easy way to be notified on all waivers?
If we do decide to go that route, we should probibly have a flag day we announce in advance to give people time to know it's coming and when.
kevin
On Thu, Jul 09, 2026 at 10:41:07AM -0700, Kevin Fenzi wrote:
On Thu, Jul 09, 2026 at 04:27:19PM +0100, Adam Williamson wrote:
For Rawhide I have a now-less-Sekrit Plan:
https://forge.fedoraproject.org/releng/tooling/src/branch/main/quality-assur...
that's a script that produces a list of "compose-critical packages", i.e. the packages which are used to build, or are included in, the non- failable images in the latest compose. My plan for Rawhide/Branched is to gate updates only if they break the dependencies of any compose- critical package, rather than gating on *all* failures.
That still requires a bit work, though. I need to get something to expose the compose-critical package data in a convenient way, and it's probably Bodhi, since it's Bodhi that stores and exposes the critical path package list (and I want Bodhi to use compose-critical package info for other reasons too). Then I have to adjust rmdepcheck and/or the CI pipeline to somehow make it convenient to distinguish between a 'ccp' and 'non-ccp' failure...
I suppose we *could* decide it's fine to gate Rawhide on all rmdepcheck failures and just require people to waive cases where they intentionally break other deps? Meh. I dunno.
Conceptually, that sounds fine to me... if you know you are breaking deps for some reason (thing not ready, but very leaf node, etc) doing a waiver to note that would make sense too.
However, it does mean you (and anyone who watches waivers) will probibly see more that you need to look at and see if they look reasonable.
Is there any easy way to be notified on all waivers?
If we do decide to go that route, we should probibly have a flag day we announce in advance to give people time to know it's coming and when.
Rather than having a small set of people who attempt to watch all waivers, IMHO, it would be better if we introduced a "2 person rule" for anything tagged as a critical path package. ie any update that is waiving broken deps should require approval from someone other than the person submitting it before getting into the compose.
With regards, Daniel
On Thu, 2026-07-09 at 10:41 -0700, Kevin Fenzi wrote:
On Thu, Jul 09, 2026 at 04:27:19PM +0100, Adam Williamson wrote:
For Rawhide I have a now-less-Sekrit Plan:
https://forge.fedoraproject.org/releng/tooling/src/branch/main/quality-assur...
that's a script that produces a list of "compose-critical packages", i.e. the packages which are used to build, or are included in, the non- failable images in the latest compose. My plan for Rawhide/Branched is to gate updates only if they break the dependencies of any compose- critical package, rather than gating on *all* failures.
That still requires a bit work, though. I need to get something to expose the compose-critical package data in a convenient way, and it's probably Bodhi, since it's Bodhi that stores and exposes the critical path package list (and I want Bodhi to use compose-critical package info for other reasons too). Then I have to adjust rmdepcheck and/or the CI pipeline to somehow make it convenient to distinguish between a 'ccp' and 'non-ccp' failure...
I suppose we *could* decide it's fine to gate Rawhide on all rmdepcheck failures and just require people to waive cases where they intentionally break other deps? Meh. I dunno.
Conceptually, that sounds fine to me... if you know you are breaking deps for some reason (thing not ready, but very leaf node, etc) doing a waiver to note that would make sense too.
However, it does mean you (and anyone who watches waivers) will probibly see more that you need to look at and see if they look reasonable.
I don't really actively monitor waivers currently. It's more like, I look at broken stuff, and look for a waiver if it seems like it would involve one.
So if we did this...I suspect if we saw a compose fail because of broken dependencies, or some significant package broke and people started complaining, I'd go look for a waiver.
Is there any easy way to be notified on all waivers?
Well, FMN, I guess. There's a message published for each waiver. Other than that, not that I know of.
On Fri, Jul 10, 2026 at 03:59:42PM +0100, Adam Williamson wrote:
I don't really actively monitor waivers currently. It's more like, I look at broken stuff, and look for a waiver if it seems like it would involve one.
So if we did this...I suspect if we saw a compose fail because of broken dependencies, or some significant package broke and people started complaining, I'd go look for a waiver.
ok, fair. I just thought it might be nice to have a way to watch waivers to note things like: how often do people use them? If they use them all the time and there's no problems, I think that would point to some tests being wrong?
Is there any easy way to be notified on all waivers?
Well, FMN, I guess. There's a message published for each waiver. Other than that, not that I know of.
I don't think there's any way to do so in FMN...
Yeah, I guess you could grab those messages, but I don't even see them off hand (but they are rare, so perhaps it's just the last one was a long time ago?)
I wish bodhi could search on waived updates, but it doesn't seem to be able to. ;(
kevin
On Fri, 2026-07-10 at 12:02 -0700, Kevin Fenzi wrote:
Yeah, I guess you could grab those messages, but I don't even see them off hand (but they are rare, so perhaps it's just the last one was a long time ago?)
Yeah, they're pretty rare. They definitely exist, though. :) https://apps.fedoraproject.org/datagrepper/raw?category=waiverdb&delta=1... (I added a 0 compared to the default delta of 172800).
The trouble with this is that another packager might be unresponsive or be unable to fix or simple refuses to fix their package (eg. because of your SONAME update) and now you can't release that to anyone in a stable branch. Even if your package is more core than the other person's blocking package.
On Mon, Jul 06, 2026 at 10:05:11PM +0100, Adam Williamson wrote:
Gating all updates for stable releases on it would mean you could no longer ship updates with detectable dependency issues, unless you waive the failure - which implies explicitly taking responsibility for shipping a broken package, or breaking another package.
Sure, but why take responsibility for someone else's problem!
Having said that I'm not really opposed to this, but I think waiving it shouldn't come with some implication that it's your problem, when it may be another packager's problem.
Rich.
On Tue, 2026-07-07 at 11:11 +0100, Richard W.M. Jones wrote:
The trouble with this is that another packager might be unresponsive or be unable to fix or simple refuses to fix their package (eg. because of your SONAME update) and now you can't release that to anyone in a stable branch. Even if your package is more core than the other person's blocking package.
Well, I see a few issues with that.
One, by policy, soname updates are rarely meant to happen on stable releases at all. See the policy text I cited: "ABI changes in general are very strongly discouraged, they force larger update sets on users and they make life difficult for third-party packagers."
Two, if you *are* for some reason absolutely forced to make an ABI change in a stable release, I think you absolutely should be on the hook for making sure all dependencies are fixed. Even if one of their packagers is non-responsive, you can send a pull request and ask a proven packager to merge it after a sufficient period of inactivity. I don't think it's practical at present to expect this for Branched or Rawhide, which is why I'm not proposing it for those branches, but I think it is reasonable in the *exceptional* and *unusual* case of ABI changes in stable releases.
Three, if for some reason it is absolutely necessary to *both* change an ABI *and* break some dependents of it in a stable release, I think it is not particularly onerous to require you to file a waiver. The filing of the waiver effectively acts as a record of the fact that you were aware of your update breaking dependent packages and that you are willing to defend this as being in some way correct or necessary.
On Mon, Jul 06, 2026 at 10:05:11PM +0100, Adam Williamson wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
rmdepcheck: https://forge.fedoraproject.org/quality/rmdepcheck
rmdepcheck fails if the update under test, when compared against the current buildroot repository, contains unsatisfiable dependencies, or causes new unsatisfiable dependencies to appear in other packages. So if your update makes some other package uninstallable...that's a fail.
This check has been running on all updates for some time now. We've dealt with various sources of false failures and I'm fairly confident it's pretty reliable by now.
Gating all updates for stable releases on it would mean you could no longer ship updates with detectable dependency issues, unless you waive the failure - which implies explicitly taking responsibility for shipping a broken package, or breaking another package.
The best time for Fedora to add gating of deps on stable was 20 years ago. The next best time is now :-) Great to see that we're finally willing to try this on stable.
With regards, Daniel
On Mon, Jul 06, 2026 at 10:05:11PM +0100, Adam Williamson wrote:
You can browse existing rmdepcheck results in resultsdb, if you want to look for false failures. Go to https://resultsdb.fedoraproject.org/results , click the Search button, set the 'testcase' to 'fedora-ci.koji-build.rmdepcheck.functional' and hit Go!
For anyone too lazy to do the clickage:
https://resultsdb.fedoraproject.org/results?&testcases=fedora-ci.koji-bu...
Zbyszek
On Mon, 2026-07-06 at 22:05 +0100, Adam Williamson wrote:
Hi folks!
I want to formally propose an idea that has been kicked around a bit informally lately: gating all stable release updates on the relatively new rmdepcheck reverse dependency static checker. This would include Fedora and EPEL updates.
rmdepcheck: https://forge.fedoraproject.org/quality/rmdepcheck
This was approved by FESCo today and is now in production for Fedora (I'll look at EPEL soon).
The rule applies *immediately* and retrospectively to in-flight updates, so some updates have become gated. Out of interest, here's the list:
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-565e049886 (python3-huggingface-hub update breaks deps of python3-logdetective)
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-a5d6caacd1 (rust- tree-sitter-cli update depends on a rust-tree-sitter update which is not yet stable)
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-db54593a40 (this is a pipeline issue with packages that have no x86_64 build and no noarch packages, that's a problem we'll need to solve: https://github.com/fedora-ci/rmdepcheck-pipeline/issues/11 )
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-c27333de57 - looks like a Fedora CI infra blip (podman failed to run with weird errors about aardvark, https://artifacts.dev.testing-farm.io/b53b7191-fe7b-49a4-b07c-d61cc2c0b576/w... ), passed on re-run
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-fc76d50302 - libcec soname update breaks deps of plasma-bigscreen
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-945fce04dc - same infra blip as FEDORA-2026-c27333de57, passed on re-run
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-3b93aae2d6 - all Fedora CI test results missing for some reason? that's weird. Passed on re-run
* https://bodhi.fedoraproject.org/updates/FEDORA-2025-01673c3772 - so old that rmdepcheck never ran on it, passed when I triggered it just now
* https://bodhi.fedoraproject.org/updates/FEDORA-2026-39ffaedff4 - mingw-libsoup update breaks deps of mingw32-osinfo-db-tools
So...that's an interesting crop. Five cases where we caught a genuine dep issue, two infra blips - we'll need to keep an eye out for those - one update that's just been sitting around too long, and one awkward test implementation issue.
I solved the arch issue in the openQA rmdepcheck pipeline but doing it in the Fedora CI pipeline is a bit more awkward (it's about the package download logic, basically). Cristian and I will bump the priority on this as it's obviously not sustainable to have a gating test that always fails on packages that build no x86_64 or noarch subpackages.
Statistically speaking, that's 9 affected updates out of 422 currently- pending ones for F43 and F44. That's not too bad of an impact I guess.
-
Adam Williamson -
Daniel P. Berrangé -
Fabio Valentini -
Kevin Fenzi -
Michel Lind -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Richard W.M. Jones -
Vít Ondruch -
Zbigniew Jędrzejewski-Szmek