Dear package maintainers currently using Zuul,
Packit as Fedora dist-git CI has recently reached feature-parity with the Zuul CI, and as the first step in the final phase implementation of the "Packit as dist-git CI" change [1] we are migrating the current Zuul CI users to packit Fedora CI [2] and disabling the Zuul runners on src.fedoraproject.org/rpms/* for the time being.
Our plan is to automatically migrate the Zuul dist-git packages on 2026-01-12 (January 12). Please let us know of any concerns you might have with the migration up until then so we can decide whether we can go forward with it. We will send another reminder of this as a reply to this announcement one week before the migration 2026-01-05 (January 5).
After the packages are migrated we will monitor your feedback [3-6] on this migration and decide if we can go ahead with the Zuul deprecation and disablement or need to resolve any blocking issues first. We could also keep the Zuul CI temporarily running on a small subset of packages if requested.
We are looking forward to your feedback on this matter.
PS: The migration of the other Zuul jobs that are linked to pagure.io [7] will be addressed at a later time as these require custom handling and are tied with the forge migration. We do not have a timeline for this part yet, but we will provide an update as soon as we have a plan for this.
[1]: https://fedoraproject.org/wiki/Changes/PackitDistgitCI [2]: https://github.com/packit/deployment/pull/672 [3]: Fedora CI channel https://matrix.to/#/#fedora-ci:fedoraproject.org [4]: Packit channel https://matrix.to/#/#packit:fedora.im [5]: Packit issues https://github.com/packit/packit-service/issues/new?template=fedora-ci.yml [6]: This email and discussion thread [7]: https://pagure.io/fedora-project-config/blob/master/f/resources/fedora-sourc...
On 12/17/25 18:01, Cristian Le via devel wrote:
Dear package maintainers currently using Zuul,
Packit as Fedora dist-git CI has recently reached feature-parity with the Zuul CI, and as the first step in the final phase implementation of the "Packit as dist-git CI" change [1] we are migrating the current Zuul CI users to packit Fedora CI [2] and disabling the Zuul runners on src.fedoraproject.org/rpms/* for the time being.
Our plan is to automatically migrate the Zuul dist-git packages on 2026-01-12 (January 12). Please let us know of any concerns you might have with the migration up until then so we can decide whether we can go forward with it. We will send another reminder of this as a reply to this announcement one week before the migration 2026-01-05 (January 5).
After the packages are migrated we will monitor your feedback [3-6] on this migration and decide if we can go ahead with the Zuul deprecation and disablement or need to resolve any blocking issues first. We could also keep the Zuul CI temporarily running on a small subset of packages if requested.
We are looking forward to your feedback on this matter.
My concerns about missing zuul-feature parity in no particular order:
- ELN scratch builds were available in Zuul - Zuul started STI/tmt tests once x86_64 build finished -- faster results - Zuul's rpminspect was configurable by rpminspect.toml
Could we postpone the Zuul shutdown until this is resolved or until it breaks completely?
Hi Miro, thanks for sharing your concerns, let me share the status of those.
On 2025/12/18 8:19, Miro Hrončok wrote:
My concerns about missing zuul-feature parity in no particular order:
- ELN scratch builds were available in Zuul
There is a tracker for this [1]. I tried to find a project that is currently enrolled to test the eln, but couldn't find one, and afaict there is support in packit itself, so not quite sure what part is missing there. I pinged the tracking issue to get an update on that.
- Zuul started STI/tmt tests once x86_64 build finished -- faster results
I was considering looking into this. Could use some more details on how to approach this [2]?
- Zuul's rpminspect was configurable by rpminspect.toml
It should have been already supported, but it seems my PR did not move to fix this issue [3]. I will ping to get it merged asap.
Could we postpone the Zuul shutdown until this is resolved or until it breaks completely?
The reasoning for the disabling the Zuul during this time is mostly to save up on build resources (3 scratch builds is not ideal). What do you think about a more limited list of packages that would be kept running zuul?
[1]: https://github.com/packit/packit-service/issues/2822 [2]: https://github.com/packit/packit-service/issues/2919 [3]: https://github.com/fedora-ci/rpminspect-pipeline/pull/102
On 12/18/25 07:30, Cristian Le via devel wrote:
Hi Miro, thanks for sharing your concerns, let me share the status of those.
On 2025/12/18 8:19, Miro Hrončok wrote:
My concerns about missing zuul-feature parity in no particular order:
- ELN scratch builds were available in Zuul
There is a tracker for this [1]. I tried to find a project that is currently enrolled to test the eln, but couldn't find one, and afaict there is support in packit itself, so not quite sure what part is missing there. I pinged the tracking issue to get an update on that.
Note that the linked issues is about CI on the eln branch. I don't know if Zuul supported that, I don't have any projects with an eln branch.
What I had in mind is ELN scratch build for rawhide pull requests. The Zuul jobs are called check-for-eln and eln-rpm-scratch-build.
This prevents accidentally braking the build for the %rhel/%fedora conditionals.
And it makes it easier for testing actual changes of some aspect of the build wrt. such conditionals. However, in such case, I can submit a manual ELN scratchbuild, so the previous point is more important to me.
- Zuul started STI/tmt tests once x86_64 build finished -- faster results
I was considering looking into this. Could use some more details on how to approach this [2]?
Will do, thanks!
- Zuul's rpminspect was configurable by rpminspect.toml
It should have been already supported, but it seems my PR did not move to fix this issue [3]. I will ping to get it merged asap.
I was not aware this is a bug, I assumed it is a missing feature. Nice.
Could we postpone the Zuul shutdown until this is resolved or until it breaks completely?
The reasoning for the disabling the Zuul during this time is mostly to save up on build resources (3 scratch builds is not ideal). What do you think about a more limited list of packages that would be kept running zuul?
We can disable the arched scratchbuilds, as they are not used in the STI/tmt tests and the packit scratchbuild will run them anyway. That way, we save up resources on s390x and ppc64le which is what seems usually quite unavailable nowadays.
Thank you again for working on this.
On 2025/12/18 16:50, Miro Hrončok wrote:
On 12/18/25 07:30, Cristian Le via devel wrote:
Hi Miro, thanks for sharing your concerns, let me share the status of those.
On 2025/12/18 8:19, Miro Hrončok wrote:
My concerns about missing zuul-feature parity in no particular order:
- ELN scratch builds were available in Zuul
There is a tracker for this [1]. I tried to find a project that is currently enrolled to test the eln, but couldn't find one, and afaict there is support in packit itself, so not quite sure what part is missing there. I pinged the tracking issue to get an update on that.
Note that the linked issues is about CI on the eln branch. I don't know if Zuul supported that, I don't have any projects with an eln branch.
What I had in mind is ELN scratch build for rawhide pull requests. The Zuul jobs are called check-for-eln and eln-rpm-scratch-build.
This prevents accidentally braking the build for the %rhel/%fedora conditionals.
And it makes it easier for testing actual changes of some aspect of the build wrt. such conditionals. However, in such case, I can submit a manual ELN scratchbuild, so the previous point is more important to me.
Oh interesting, I missed that aspect. Hmm I am thinking if I could make it as a tmt test since that would minimize the special code that packit would need to carry. Would a mock build also work for that? Seems like we would need another issue for this, would you open it in packit-service and we can check what options we have?
Could we postpone the Zuul shutdown until this is resolved or until it breaks completely?
The reasoning for the disabling the Zuul during this time is mostly to save up on build resources (3 scratch builds is not ideal). What do you think about a more limited list of packages that would be kept running zuul?
We can disable the arched scratchbuilds, as they are not used in the STI/tmt tests and the packit scratchbuild will run them anyway. That way, we save up resources on s390x and ppc64le which is what seems usually quite unavailable nowadays.
Hmm, yeah, that could also be an option to reduce the burden :+1:.
Thank you again for working on this.
Also thank you for your feedback and steering the implementation of this.
On 12/18/25 11:03, Cristian Le via devel wrote:
My concerns about missing zuul-feature parity in no particular order:
- ELN scratch builds were available in Zuul
There is a tracker for this [1]. I tried to find a project that is currently enrolled to test the eln, but couldn't find one, and afaict there is support in packit itself, so not quite sure what part is missing there. I pinged the tracking issue to get an update on that.
Note that the linked issues is about CI on the eln branch. I don't know if Zuul supported that, I don't have any projects with an eln branch.
What I had in mind is ELN scratch build for rawhide pull requests. The Zuul jobs are called check-for-eln and eln-rpm-scratch-build.
This prevents accidentally braking the build for the %rhel/%fedora conditionals.
And it makes it easier for testing actual changes of some aspect of the build wrt. such conditionals. However, in such case, I can submit a manual ELN scratchbuild, so the previous point is more important to me.
Oh interesting, I missed that aspect. Hmm I am thinking if I could make it as a tmt test since that would minimize the special code that packit would need to carry. Would a mock build also work for that? Seems like we would need another issue for this, would you open it in packit-service and we can check what options we have?
I have opened https://github.com/packit/packit-service/issues/2920 and will followup there.
As promised (and 1 day late), this is a reminder of the planned migration of the Zuul users to the Packit as dist-git CI runner.
What has been discussed so far: - Limit the initial Zuul disablement to disable all the non-x86_64 scratch builds [1]. Current plan for this is 2026-01-12 (January 12), together with the onboarding on the Packit as dist-git CI runner - Add the capability for Packit to do eln scratch builds on rawhide PRs (see issue [2] on exact details) and afterwards we could go ahead with the remaining disablement - Be able to run the tmt tests as soon as x86_64 builds are finished [3,4]. We will try to get it done as soon as possible, but would not be a blocker for the zuul disablement otherwise
If you have some more feedback, please go ahead and share it with us :)
[1]: https://pagure.io/fedora-project-config/pull-request/354 [2]: https://github.com/packit/packit-service/issues/2920 [3]: https://github.com/packit/packit-service/issues/2919 [4]: https://github.com/fedora-infra/koji-fedoramessaging/issues/155
On 2025/12/17 18:01, Cristian Le via devel wrote:
Dear package maintainers currently using Zuul,
Packit as Fedora dist-git CI has recently reached feature-parity with the Zuul CI, and as the first step in the final phase implementation of the "Packit as dist-git CI" change [1] we are migrating the current Zuul CI users to packit Fedora CI [2] and disabling the Zuul runners on src.fedoraproject.org/rpms/* for the time being.
Our plan is to automatically migrate the Zuul dist-git packages on 2026-01-12 (January 12). Please let us know of any concerns you might have with the migration up until then so we can decide whether we can go forward with it. We will send another reminder of this as a reply to this announcement one week before the migration 2026-01-05 (January 5).
After the packages are migrated we will monitor your feedback [3-6] on this migration and decide if we can go ahead with the Zuul deprecation and disablement or need to resolve any blocking issues first. We could also keep the Zuul CI temporarily running on a small subset of packages if requested.
We are looking forward to your feedback on this matter.
PS: The migration of the other Zuul jobs that are linked to pagure.io [7] will be addressed at a later time as these require custom handling and are tied with the forge migration. We do not have a timeline for this part yet, but we will provide an update as soon as we have a plan for this.
[3]: Fedora CI channel https://matrix.to/#/#fedora-ci:fedoraproject.org [4]: Packit channel https://matrix.to/#/#packit:fedora.im [5]: Packit issues https://github.com/packit/packit-service/issues/new?template=fedora-ci.yml [6]: This email and discussion thread [7]: https://pagure.io/fedora-project-config/blob/master/f/resources/fedora-sourc...
On 2026/01/06 15:00, Cristian Le via devel wrote:
- Limit the initial Zuul disablement to disable all the non-x86_64
scratch builds [1]. Current plan for this is 2026-01-12 (January 12), together with the onboarding on the Packit as dist-git CI runner
Update to this topic, this change is now live. From now on please use the Packit as dist-git CI enrollment process [1]. A helper script to enroll users similar to the `zuul-config-generator` will be made live shortly. Keep an eye out on the enrollment process page [1] for instructions on how to use it.
Final update for this thread.
We are moving with the disablement of Zuul for the src.fedoraproject.org/projects/rpms/* projects today.
Also as reminder, we will be disabling the Jenkins based Fedora CI (for PRs only) on 2026-02-16 [1].
As usual, feel free to get in touch if this is creating any issues.
[1]: https://discussion.fedoraproject.org/t/packit-as-fedora-dist-git-ci-final-ph...
-
Cristian Le -
Miro Hrončok