On Wed, 2024-02-28 at 09:03 -0800, Adam Williamson wrote:
Hey folks! Just an update on an issue I know some packagers are
running
into.
In the last few days, some Fedora CI jobs have been getting stuck and
timing out. Packagers will usually experience this as a test result
showing as "ABSENT" in Bodhi, e.g. for this update:
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce1fb5a0db
the fedora-ci.koji-build.tier0.functional result shows as ABSENT (red
line, question mark icon which shows ABSENT if you hover over it).
We don't have any CI tests that are universally gating by default, but
some packages have opted in to gating on some CI tests, and some of
these have been stuck in updates-testing for a few days due to this
problem.
It seems the issue was caused by a kernel 6.8 bug which broke quite a
lot of stuff last week - see
https://lore.kernel.org/all/6a150ddd-3267-4f89-81bd-6807700c57c1@redhat.com/
. The fix for that bug was in kernel 6.8rc6, which is now in both
Fedora 40 and Rawhide. Miroslav Vadkerti is aware of this and is
working to try and clean things up (update the base images for Fedora
CI and get all the timed-out tests re-run). I don't know what the ETA
is, but just wanted to let folks know it's at least being worked on.
Update on this: turns out the kernel bug wasn't the main thing, that
was causing some test errors but the main problem was that reporting of
Fedora CI results to resultsdb simply broke on Saturday and nobody
spotted that was the problem till today.
We got that fixed today, and I have hacked up and am now running a
script which should report all the results that were missed (hopefully
correctly, and hopefully without overriding or duplicating or missing
anything). Once it's finished running I'll do a sweep for updates still
stuck in updates-testing and see where we are.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net