Hi folks! I'm hoping this new feature I've been working on since Flock is now ready for prime time.
It's pretty simple, really. Comment '/openqa test' on a dist-git PR and openQA will test it. It should report the state back to the pull request - you'll get an entry called something like "Fedora openQA tests - scratch build 147755141" that will start in 'pending' state then go to 'success', 'failure' or 'error' once all tests complete. The entry is a link to the openQA summary page where you can see the state of each test (so if you get a 'failure' result you can see specifically which test(s) failed; I don't want to report the state of each test separately to the PR because there can be dozens or hundreds).
Here's a PR where you can see it "in action": https://src.fedoraproject.org/rpms/sqlite/pull-request/64
It looks for the Packit scratch build for the target branch of the PR, so if that build fails, the tests won't run. If the scratch build is not done at the time you post the comment, the tests *should* run once the scratch build succeeds (and any subsequent time a scratch build of the PR for the correct branch succeeds).
If the package is in the critical path, it will run only "appropriate" tests (using the same mapping used for update tests). If the package is not in the critical path, it will just run all the update tests.
There is no multi-package / multi-PR capability at this time; the tests will always run on *only* the scratch build of the PR you commented on. (with the relevant buildroot repo enabled in the system-under-test). If the change in the PR depends on a pending change to another package that has not already appeared in the buildroot, the tests will fail. Sorry.
I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way, I'll try and enhance this system to work with it so related PRs are tested together. This is the primary reason this system is "on- demand" - I don't want to trigger tests automatically for all critpath package PRs, or anything like that, because we know they would sometimes fail because of interdependency issues.
Right now there's no authorization check - if *anyone* comments '/openqa test' on a PR it is "enabled" for testing and tests will run every time a scratch build of the PR completes. Please don't abuse this by spamming it on PRs you're not involved in, or anything; if anyone does that I'll just have to turn the feature off unless/until I can implement some kind of "only accept comments from people with commit access to the package" mechanic, and I'd rather believe I can just trust people. Please only use this mechanism if you're actually going to find the results directly useful.
The tests will trigger on both staging and production openQA instances, but only the production instance should report status back to the PR.
The implementation of this is Pagure-specific, but it's not really that complicated and should hopefully be fairly easy to port to Forgejo when we move dist-git. If anyone's interested, this is the code: https://forge.fedoraproject.org/quality/fedora_openqa/commit/ac5a56f41b2a7fc...
Of course, please let me know if you try it out and it doesn't work, or you notice some kind of bug. Thanks!
Dne 13. 07. 26 v 9:10 odp. Adam Williamson napsal(a):
I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way
Does it make sense to implement it as tmt-plan that can be triggered by Packit?
https://github.com/packit/tmt-plans
e.g. see this example https://github.com/packit/tmt-plans/pull/30/changes
On 2026/07/14 8:19, Miroslav Suchý wrote:
Dne 13. 07. 26 v 9:10 odp. Adam Williamson napsal(a):
I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way
Does it make sense to implement it as tmt-plan that can be triggered by Packit?
https://github.com/packit/tmt-plans
e.g. see this example https://github.com/packit/tmt-plans/pull/30/changes
In principle it could be added there, but: - displaying the results could be very nasty - it would be a bit wasteful to just wrap around another test suite - making it on-demand or filtered to specific packages is something we haven't solved yet
As an alternative, I did consider if we could add it into `packit-service` directly. The only blocker there I think is making it trigger based on events (i.e. register a webhook on openQA to announce when the state change) instead of having a process wait for the state to change.
On Mon, 2026-07-20 at 22:53 +0200, Cristian Le via devel wrote:
On 2026/07/14 8:19, Miroslav Suchý wrote:
Dne 13. 07. 26 v 9:10 odp. Adam Williamson napsal(a):
I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way
Does it make sense to implement it as tmt-plan that can be triggered by Packit?
https://github.com/packit/tmt-plans
e.g. see this example https://github.com/packit/tmt-plans/pull/30/changes
In principle it could be added there, but:
- displaying the results could be very nasty
- it would be a bit wasteful to just wrap around another test suite
- making it on-demand or filtered to specific packages is something we
haven't solved yet
As an alternative, I did consider if we could add it into `packit-service` directly. The only blocker there I think is making it trigger based on events (i.e. register a webhook on openQA to announce when the state change) instead of having a process wait for the state to change.
Note: it seems openQA has webhook support - https://open.qa/docs/#setup-webhook-on-github - but this is the first I've heard of it, and I've never used it. Actual openQA scheduling in Fedora always runs through https://forge.fedoraproject.org/quality/fedora_openqa , which is built around fedora-messaging. Trying to bypass it and use openQA webhooks directly may not work properly as there's stuff the scheduler does (test variables it sets, specifically) which the tests rely on.
On July 20, 2026 4:58:44 PM CDT, Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2026-07-20 at 22:53 +0200, Cristian Le via devel wrote:
In principle it could be added there, but:
- displaying the results could be very nasty
- it would be a bit wasteful to just wrap around another test suite
- making it on-demand or filtered to specific packages is something we
haven't solved yet
As an alternative, I did consider if we could add it into `packit-service` directly. The only blocker there I think is making it trigger based on events (i.e. register a webhook on openQA to announce when the state change) instead of having a process wait for the state to change.
Note: it seems openQA has webhook support - https://open.qa/docs/#setup-webhook-on-github - but this is the first I've heard of it, and I've never used it.
It is also the opposite direction that we would need :). That is registering webhooks on github to trigger openqa, or in our case it would be registering a webhook on packit.
What we have with packit-testing-farm is that packit registers the webhook onto testing-farm so that testing-farm reports when it finishes. If we replace testing-farm with openqa we would need a webhook on openqa.
Actual openQA scheduling in Fedora always runs through https://forge.fedoraproject.org/quality/fedora_openqa , which is built around fedora-messaging. Trying to bypass it and use openQA webhooks directly may not work properly as there's stuff the scheduler does (test variables it sets, specifically) which the tests rely on.
And yeah the more I contemplate, the less of a bad idea it seems to me to just wrap it around a tmt test. It is not much different than the current implementation right?
On Mon, 2026-07-20 at 17:15 -0500, Cristian Le via devel wrote:
On July 20, 2026 4:58:44 PM CDT, Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2026-07-20 at 22:53 +0200, Cristian Le via devel wrote:
In principle it could be added there, but:
- displaying the results could be very nasty
- it would be a bit wasteful to just wrap around another test suite
- making it on-demand or filtered to specific packages is something we
haven't solved yet
As an alternative, I did consider if we could add it into `packit-service` directly. The only blocker there I think is making it trigger based on events (i.e. register a webhook on openQA to announce when the state change) instead of having a process wait for the state to change.
Note: it seems openQA has webhook support - https://open.qa/docs/#setup-webhook-on-github - but this is the first I've heard of it, and I've never used it.
It is also the opposite direction that we would need :). That is registering webhooks on github to trigger openqa, or in our case it would be registering a webhook on packit.
What we have with packit-testing-farm is that packit registers the webhook onto testing-farm so that testing-farm reports when it finishes. If we replace testing-farm with openqa we would need a webhook on openqa.
Actual openQA scheduling in Fedora always runs through https://forge.fedoraproject.org/quality/fedora_openqa , which is built around fedora-messaging. Trying to bypass it and use openQA webhooks directly may not work properly as there's stuff the scheduler does (test variables it sets, specifically) which the tests rely on.
And yeah the more I contemplate, the less of a bad idea it seems to me to just wrap it around a tmt test. It is not much different than the current implementation right?
I still don't understand the idea here, honestly, or the problem it's trying to solve. Could someone explain it to me in small words? :D
On 2026/07/21 0:29, Adam Williamson wrote:
On Mon, 2026-07-20 at 17:15 -0500, Cristian Le via devel wrote:
And yeah the more I contemplate, the less of a bad idea it seems to me to just wrap it around a tmt test. It is not much different than the current implementation right?
I still don't understand the idea here, honestly, or the problem it's trying to solve. Could someone explain it to me in small words? :D
I hear your offer of small words and counter it with long paragraphs 😅.
Going back to the original question
I don't want to invent some kind of system for marking PRs as related myself
Best central place to do this would be in packit and if we can bring the PR-based openQA tests on there we would file it under the "figure out how packit can do that bucket". The other issues would be solved automatically as well.
On the how can this be ported to packit, there are 2 possible ways to tackle this
Wrap OpenQA tests as a tmt test
This would be the easiest to approach if a bit inefficient. Afaiu this is already similar to how the new PR handler works right? I.e. you submit a job via `OpenQAScheduler._update_schedule`, which eventually submits a job to OpenQA, waits for the job to finish and then reports back. This just needs to be replicated from a CLI or a python API
|Make OpenQA another test executor for packit (same as testing-farm currently is)|
This is theoretically more efficient, but it depends on how packit can interact with OpenQA. From what I can tell from navigating the packit-service code, this would not be too difficult to implement.
On Tue, 2026-07-21 at 16:40 +0200, Cristian Le via devel wrote:
On 2026/07/21 0:29, Adam Williamson wrote:
On Mon, 2026-07-20 at 17:15 -0500, Cristian Le via devel wrote:
And yeah the more I contemplate, the less of a bad idea it seems to me to just wrap it around a tmt test. It is not much different than the current implementation right?
I still don't understand the idea here, honestly, or the problem it's trying to solve. Could someone explain it to me in small words? :D
I hear your offer of small words and counter it with long paragraphs 😅.
Going back to the original question
I don't want to invent some kind of system for marking PRs as related myself
Best central place to do this would be in packit and if we can bring the PR-based openQA tests on there we would file it under the "figure out how packit can do that bucket". The other issues would be solved automatically as well.
Ah, OK, so it's that bit. I guess my sorta-expectation there was that, if we're gonna do it, we'll follow how CentOS / RHEL are attempting to do it, which AIUI is basically "magic tags" - a known format for tag names which indicates "these PRs are all part of the same group".
On the how can this be ported to packit, there are 2 possible ways to tackle this
I think there might be three...
Wrap OpenQA tests as a tmt testThis would be the easiest to approach if a bit inefficient. Afaiu this is already similar to how the new PR handler works right? I.e. you submit a job via `OpenQAScheduler._update_schedule`, which eventually submits a job to OpenQA, waits for the job to finish and then reports back. This just needs to be replicated from a CLI or a python API
Well...sort of. It doesn't submit "a job", and it doesn't "wait for the job to finish". In openQA you schedule an artifact, essentially. You schedule the *update*, not the job. What _update_schedule does is tell openQA "this update is ready for testing". openQA then schedules a whole bunch of jobs (based on our test templates in https://forge.fedoraproject.org/quality/os-autoinst-distri-fedora/src/branch... ).
The scheduler does get the complete list of jobs back, so in theory it could then literally sit around and wait for them all, but that's not exactly what happens.
fedora_openqa is a set of multiple fedora-messaging consumers. The scheduler consumer only schedules jobs, and submits "results" of the "these jobs are now scheduled" type. It never submits "completed job" reports. That's done by *other* consumers.
In this case, there is a new consumer - the OpenQADistGitReporter class, https://forge.fedoraproject.org/quality/fedora_openqa/src/branch/main/src/fe... . It listens for openQA "job complete" messages. It then reads the message and does a heuristic:
1. Is this job for a dist-git PR? If not, ignore 2. Was this the *last* job for the PR? (There's a field in the message that tells it this). If not, ignore
If both of those checks 'pass', it goes ahead and "reports the results". It queries openQA for *all* the results for the PR, synthesizes an overall status (passed, failed or error), and sets the PR flag to that status.
The consumers themselves are long-lived, but that lifecycle is owned by fedora-messaging and I don't really worry about it. As the author of the consumers, you write code that is *not* long-lived - code that receives a message and immediately decides on an action. Because the consume operation consumes one message at a time and is blocking, it can't sit around indefinitely waiting for something like a test job to finish. It has to just read the message and more or less immediately decide what to do. (I also find it's generally best to keep the consumers stateless, though you *can* make them have state if you really want to).
|Make OpenQA another test executor for packit (same as testing-farm currently is)|This is theoretically more efficient, but it depends on how packit can interact with OpenQA. From what I can tell from navigating the packit-service code, this would not be too difficult to implement.
It might be possible, sure. That could work. I think it's a bit hard to be sure how it'd look because I know exactly how the openQA side works but I have no idea about packit, and probably vice versa for you :D
However, I think there's also a third option. The current fedora_openqa implementation actually already "depends" on packit, because it uses the packit scratch build. The current implementation already has to deal with the case where someone posts '/openqa test' but the scratch build isn't done yet; we do that by having the scheduler consumer also listen out for flag state change messages, and when it sees a 'packit scratch build complete' message, it checks whether the PR in question has an '/openqa test' comment and if so, it schedules the tests. (This also means we re-run the tests any time there's a new scratch build).
It feels like something like this should also be possible if we come up with some kind of related-PR design based around packit? As long as, in that case, packit signals *somehow or other* (in a way that can be picked up from fedora-messaging, though it doesn't have to be *directly*) that it has finished a set of related scratch builds, we should be able to have the openQA scheduler pick up that signal, schedule the jobs with the correct set of scratch builds, and report the results back to the correct set of PRs...
On Tue, 2026-07-14 at 08:19 +0200, Miroslav Suchý wrote:
Dne 13. 07. 26 v 9:10 odp. Adam Williamson napsal(a):
I don't want to invent some kind of system for marking PRs as related myself; we might want to do it as a project, though, and if we do go that way
Does it make sense to implement it as tmt-plan that can be triggered by Packit?
https://github.com/packit/tmt-plans
e.g. see this example https://github.com/packit/tmt-plans/pull/30/changes
I don't quite understand what tmt-plans has to do with the inter- related PR issue?
-
Adam Williamson -
Cristian Le -
Miroslav Suchý