Hey folks! Just an update on an issue I know some packagers are running into.
In the last few days, some Fedora CI jobs have been getting stuck and timing out. Packagers will usually experience this as a test result showing as "ABSENT" in Bodhi, e.g. for this update:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce1fb5a0db
the fedora-ci.koji-build.tier0.functional result shows as ABSENT (red line, question mark icon which shows ABSENT if you hover over it).
We don't have any CI tests that are universally gating by default, but some packages have opted in to gating on some CI tests, and some of these have been stuck in updates-testing for a few days due to this problem.
It seems the issue was caused by a kernel 6.8 bug which broke quite a lot of stuff last week - see https://lore.kernel.org/all/6a150ddd-3267-4f89-81bd-6807700c57c1@redhat.com/ . The fix for that bug was in kernel 6.8rc6, which is now in both Fedora 40 and Rawhide. Miroslav Vadkerti is aware of this and is working to try and clean things up (update the base images for Fedora CI and get all the timed-out tests re-run). I don't know what the ETA is, but just wanted to let folks know it's at least being worked on.
On Wed, 2024-02-28 at 09:03 -0800, Adam Williamson wrote:
Hey folks! Just an update on an issue I know some packagers are running into.
In the last few days, some Fedora CI jobs have been getting stuck and timing out. Packagers will usually experience this as a test result showing as "ABSENT" in Bodhi, e.g. for this update:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce1fb5a0db
the fedora-ci.koji-build.tier0.functional result shows as ABSENT (red line, question mark icon which shows ABSENT if you hover over it).
We don't have any CI tests that are universally gating by default, but some packages have opted in to gating on some CI tests, and some of these have been stuck in updates-testing for a few days due to this problem.
It seems the issue was caused by a kernel 6.8 bug which broke quite a lot of stuff last week - see https://lore.kernel.org/all/6a150ddd-3267-4f89-81bd-6807700c57c1@redhat.com/ . The fix for that bug was in kernel 6.8rc6, which is now in both Fedora 40 and Rawhide. Miroslav Vadkerti is aware of this and is working to try and clean things up (update the base images for Fedora CI and get all the timed-out tests re-run). I don't know what the ETA is, but just wanted to let folks know it's at least being worked on.
Update on this: turns out the kernel bug wasn't the main thing, that was causing some test errors but the main problem was that reporting of Fedora CI results to resultsdb simply broke on Saturday and nobody spotted that was the problem till today.
We got that fixed today, and I have hacked up and am now running a script which should report all the results that were missed (hopefully correctly, and hopefully without overriding or duplicating or missing anything). Once it's finished running I'll do a sweep for updates still stuck in updates-testing and see where we are.
-
Adam Williamson