Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
Sorry for the outage.
kevin
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
Regards,
Hans
Hello,
My apologies for not following this thread and for top posting...
But I have 3 builds that have failed with
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
All the builds have built every arch but s390x
So my question should I restart the koji builds? Will I have to change the NVR so since the rest of the archs have built?
Again... my apologies for been a bit obtuse...
steved.
On 5/7/26 6:24 AM, Hans de Goede wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
Regards,
Hans
Hi Steve,
On 7-May-26 12:50, Steve Dickson wrote:
Hello,
My apologies for not following this thread and for top posting...
But I have 3 builds that have failed with
Could not execute build: 502 Server Error: Bad Gateway for url: https://koji.fedoraproject.org/kojihub
All the builds have built every arch but s390x
So my question should I restart the koji builds? Will I have to change the NVR so since the rest of the archs have built?
Again... my apologies for been a bit obtuse...
There is no need to bump the NVR if a build has failed. You can simply do another "fedpkg build" *once* the builders are back online.
Note also the infra team or rel-eng might just automatically resubmit all the failed builds in cases like this. So you should check that has not happened before re-submitting builds.
Regards,
Hans
On 5/7/26 6:24 AM, Hans de Goede wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
Regards,
Hans
On 5/7/26 12:24, Hans de Goede wrote:
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote: [...] First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
+1
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders. [...] For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
FWIW, the tracking ticket for that issue afaics is: https://github.com/fedora-copr/copr/issues/4219
From a quite recent comment there is seems this was recently fixed by a change from nikromen: https://forge.fedoraproject.org/infra/ansible/pulls/3316
But that wasn't confirmed yet, which is why I Bcced nikromen to this mail.
Ciao, Thorsten
On Thu, May 7, 2026 at 6:24 AM Hans de Goede hans@hansg.org wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
We've just had this conversation at the leadership level with Red Hat. According to Josh Miller (CC'd to this email so he can jump in if he wants to), getting more capacity isn't the issue, the problem is that they are currently unable to deploy them in a way that makes the IT staff happy to maintain them (the mainframes are managed by Red Hat IT these days apparently) and the IT staff they had for this effort left two years ago. So right now we're stuck between a rock and a hard place because the ability to deploy new given capacity doesn't exist due to Red Hat internal issues.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Thu, May 07, 2026 at 07:29:52AM -0400, Neal Gompa wrote:
On Thu, May 7, 2026 at 6:24 AM Hans de Goede hans@hansg.org wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
We've just had this conversation at the leadership level with Red Hat. According to Josh Miller (CC'd to this email so he can jump in if he wants to), getting more capacity isn't the issue, the problem is that they are currently unable to deploy them in a way that makes the IT staff happy to maintain them (the mainframes are managed by Red Hat IT these days apparently) and the IT staff they had for this effort left two years ago. So right now we're stuck between a rock and a hard place because the ability to deploy new given capacity doesn't exist due to Red Hat internal issues.
Was there any indication that Red Hat was actively intending to solve this IT staff / hardware problem in a reasonable timeframe ? I fear this is the kind of situation that could be left on the back burner indefinitely, unless someone has made commitments to address it within Red Hat as a priority task.
With regards, Daniel
On Thu, May 7, 2026 at 7:50 AM Daniel P. Berrangé berrange@redhat.com wrote:
On Thu, May 07, 2026 at 07:29:52AM -0400, Neal Gompa wrote:
On Thu, May 7, 2026 at 6:24 AM Hans de Goede hans@hansg.org wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
We've just had this conversation at the leadership level with Red Hat. According to Josh Miller (CC'd to this email so he can jump in if he wants to), getting more capacity isn't the issue, the problem is that they are currently unable to deploy them in a way that makes the IT staff happy to maintain them (the mainframes are managed by Red Hat IT these days apparently) and the IT staff they had for this effort left two years ago. So right now we're stuck between a rock and a hard place because the ability to deploy new given capacity doesn't exist due to Red Hat internal issues.
Was there any indication that Red Hat was actively intending to solve this IT staff / hardware problem in a reasonable timeframe ? I fear this is the kind of situation that could be left on the back burner indefinitely, unless someone has made commitments to address it within Red Hat as a priority task.
Hi Daniel, (and everyone)
Over the past several weeks, this has been escalated to us (Red Hat Engineering leadership) and we're actively working on it internally.
As with most problems, the situation is complex, but it's a high priority and we've made some progress. It will probably take several more weeks before we have an update, and to be fully open... longer before we have a full solution.
We'll keep the Fedora Infrastructure team looped in throughout the process.
-Evan
Hans, thanks for raising this question once again. Today I again asked myself why s390x (the past) is even a primary architecture while RISC-V (the future) is not?
I do understand that a corporate sponsor entity needs to have a test-bed but this raises another questions. The COPR more-than-1-month degradation to QEMU emulation on x86_64 is particularly screaming - at this point we're not really supporting s390x, we're just maintaining the illusion of support.
The counterargument is obviously going to be "but they pay/contribute significantly to Fedora". Well, if someone's contribution includes keeping the infrastructure running and they're consistently failing to do that, then the relationship needs renegotiating rather than letting it silently degrade everyone else's experience. Speaking technically what this renegotiation could actually mean exactly? Probably just demoting s390x from primary to secondary architecture, similar to RISC-V which we should promote instead. That way builds aren't blocked when s390x has issues, and anyone interested could still maintain it as a secondary if they want to.
On Thu, May 7, 2026 at 12:24 PM Hans de Goede hans@hansg.org wrote:
Hi,
On 7-May-26 03:01, Kevin Fenzi via devel-announce wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Folks have been working today to bring it back up, but the fix will require parts that need to be overnight shipped.
Hopefully the machines will be back online tomorrow after the replacement is installed.
Arched Builds submitted now will wait for s390x builders to be back to complete.
You can watch https://www.fedorastatus.org/ and/or https://forge.fedoraproject.org/infra/tickets/issues/13326
for further status.
First of all, thank you Kevin and infrastructure team for all your work on the Fedora infra, the below is not meant as criticism of the Fedora infra team.
It seems that lately there have been various capacity issues wrt s390x and I think sometimes also powerpc builders.
AFAICT Fedora mainly supports these 2 architectures because IBM wants Fedora to supports these 2 architectures.
Yet IBM seems to lately consistently fail to make adequate resources (builders, test-vms for contributors) available for these resources.
For example I believe that copr has been forced to switch to very slow builds for s390x and powerpc using emulation in qemu on x86_64 builders?
Looking at the COPR example I believe the under-resourcing problem has become so big that we (the Fedora project) need to seriously consider if we want to keep supporting s390x and powerpc going forward or if the time has come to drop these ?
Regards,
Hans
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
On Thu, May 07, 2026 at 01:38:33PM +0200, Peter Lemenkov wrote:
Hans, thanks for raising this question once again. Today I again asked myself why s390x (the past) is even a primary architecture while RISC-V (the future) is not?
Well, I can answer the second part: riscv hardware is still not fast enough to be able to keep up with the flow of builds in fedora. If we added it today with the best hardware we could get, it would be slower than s390x by a good deal.
There's a number of people pressing vendors and watching new hardware releases as they happen. Once sufficent hardware exists we will likely add it.
kevin
On 5/7/26 12:26 PM, Kevin Fenzi wrote:
Well, I can answer the second part: riscv hardware is still not fast enough to be able to keep up with the flow of builds in fedora. If we added it today with the best hardware we could get, it would be slower than s390x by a good deal.
Sure, but as you know right now S390 is apparently run in QEMU emulation, so, today, a $100 8-core K1 would probably be competitive.
There's a number of people pressing vendors and watching new hardware releases as they happen. Once sufficent hardware exists we will likely add it.
This is great to hear. I understand that the new RVA23 boards (with K3 SoC, like MilkV Jupiter2, BPi-SM10; and Tenstorrent and SiFive) are quite performant. My own experience with K1 is 20% of a typical x86 desktop; and the buzz about those RVA23 SoCs is that they are several times faster, so about where a typical desktop might be.
On Wed, May 20, 2026 at 12:18:54PM -0400, Przemek Klosowski via devel wrote:
On 5/7/26 12:26 PM, Kevin Fenzi wrote:
Well, I can answer the second part: riscv hardware is still not fast enough to be able to keep up with the flow of builds in fedora. If we added it today with the best hardware we could get, it would be slower than s390x by a good deal.
Sure, but as you know right now S390 is apparently run in QEMU emulation, so, today, a $100 8-core K1 would probably be competitive.
It is not. I can say for sure that fedora s390x builders are kvm vm's on a real s390x lpar.
I am not super sure about copr. I think they did use emulation for a while, but I think they switched back to real hardware.
There's a number of people pressing vendors and watching new hardware releases as they happen. Once sufficent hardware exists we will likely add it.
This is great to hear. I understand that the new RVA23 boards (with K3 SoC, like MilkV Jupiter2, BPi-SM10; and Tenstorrent and SiFive) are quite performant. My own experience with K1 is 20% of a typical x86 desktop; and the buzz about those RVA23 SoCs is that they are several times faster, so about where a typical desktop might be.
It's definitely heading the right direction. ;)
kevin
On Wed, 20 May 2026 at 17:19, Przemek Klosowski via devel devel@lists.fedoraproject.org wrote:
On 5/7/26 12:26 PM, Kevin Fenzi wrote:
Well, I can answer the second part: riscv hardware is still not fast enough to be able to keep up with the flow of builds in fedora. If we added it today with the best hardware we could get, it would be slower than s390x by a good deal.
Sure, but as you know right now S390 is apparently run in QEMU emulation, so, today, a $100 8-core K1 would probably be competitive.
No, that's only for copr, the main official Fedora builds are proper Z-series HW.
There's a number of people pressing vendors and watching new hardware releases as they happen. Once sufficent hardware exists we will likely add it.
This is great to hear. I understand that the new RVA23 boards (with K3 SoC, like MilkV Jupiter2, BPi-SM10; and Tenstorrent and SiFive) are quite performant. My own experience with K1 is 20% of a typical x86 desktop; and the buzz about those RVA23 SoCs is that they are several times faster, so about where a typical desktop might be.
They are quite performant in a single user environment with a single user, put them in a build farm and thrash them for days and it's not the case.
On 5/20/26 12:47 PM, Peter Robinson wrote:
They are quite performant in a single user environment with a single user, put them in a build farm and thrash them for days and it's not the case.
For what it's worth, I stressed a BPI-F3 (8-core K1) and it held up remarkably: no crashes for weeks, stable 60C, drawing ~4W, with a tiny, zero-noise fan. I was impressed.
Bonus--it had a serial console so you can do "pre-boot" stuff almost like on a server-grade ilo/idrac. I wish more motherboards had serial consoles.
On Wed, May 20, 2026 at 7:06 PM Przemek Klosowski via devel devel@lists.fedoraproject.org wrote:
On 5/20/26 12:47 PM, Peter Robinson wrote:
They are quite performant in a single user environment with a single user, put them in a build farm and thrash them for days and it's not the case.
For what it's worth, I stressed a BPI-F3 (8-core K1) and it held up remarkably: no crashes for weeks, stable 60C, drawing ~4W, with a tiny, zero-noise fan. I was impressed.
Bonus--it had a serial console so you can do "pre-boot" stuff almost like on a server-grade ilo/idrac. I wish more motherboards had serial consoles.
This is my personal opinion, but K3 (X100 cores) is still not good enough from a performance point of view.
It will take a few more SoC before it's decent enough, but otherwise experience improves significantly with each new SoC.
On Thu, 2026-05-07 at 13:38 +0200, Peter Lemenkov wrote:
Hans, thanks for raising this question once again. Today I again asked myself why s390x (the past) is even a primary architecture while RISC-V (the future) is not?
s390x is not "the past". It may be unfashionable, but a lot of people still run stuff on mainframes and will continue to do so for the foreseeable future. It's a live, actively developed arch.
Yes, there are specific problems with the IBM/RH backing for s390x in Fedora which, if not resolved, may form a reasonable argument for dropping/downgrading it in Fedora. But calling it "the past" is just off.
riscv is some people's idea of the future, but that's hardly a nailed- on truth (some other people were telling us aarch64 was the future for everyone a few years ago; so far that has not panned out).
More practically speaking, the currently-reasonably-available riscv hardware is incredibly slow (much worse than ppc64 or s390x) and not datacenter-ready. The folks working on riscv enablement are perfectly aware of this, and that's why they have not yet proposed making it a primary arch.
On Thu, 7 May 2026 at 18:20, Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2026-05-07 at 13:38 +0200, Peter Lemenkov wrote:
Hans, thanks for raising this question once again. Today I again asked myself why s390x (the past) is even a primary architecture while RISC-V (the future) is not?
s390x is not "the past". It may be unfashionable, but a lot of people still run stuff on mainframes and will continue to do so for the foreseeable future. It's a live, actively developed arch.
Yes, there are specific problems with the IBM/RH backing for s390x in Fedora which, if not resolved, may form a reasonable argument for dropping/downgrading it in Fedora. But calling it "the past" is just off.
I agree with Adam here, and frankly the issue here isn't anything to do with the s390x builders as an architecture, it seems whoever runs the RH infrastructure (outside of the Fedora team's control) hasn't been monitoring the storage array or the support contract for the 3rd party who provides monitoring has failed, so I don't think the discussion of the architecture is the problem here but the complete and utter failure of RH/IBM to run proper infrastructure. I have no doubt it's affecting other builders internally as well. I look forward to the RCA.
riscv is some people's idea of the future, but that's hardly a nailed- on truth (some other people were telling us aarch64 was the future for everyone a few years ago; so far that has not panned out).
More practically speaking, the currently-reasonably-available riscv hardware is incredibly slow (much worse than ppc64 or s390x) and not datacenter-ready. The folks working on riscv enablement are perfectly aware of this, and that's why they have not yet proposed making it a primary arch.
Yes, it was the same issue that was had when we were dealing with ARMv7 and aarch64, and sadly from my experience there and the related issues in RISC-V world we're still 5 years away from any form of remotely sane enterprise RISC-V HW.
On Thu, May 7, 2026 at 5:20 PM Adam Williamson adamwill@fedoraproject.org wrote:
More practically speaking, the currently-reasonably-available riscv hardware is incredibly slow (much worse than ppc64 or s390x) and not datacenter-ready. The folks working on riscv enablement are perfectly aware of this, and that's why they have not yet proposed making it a primary arch.
While RVA23 datacenter products have seen press releases, I suspect the current fab constraints (thank you, AI demand) are going to push back actual delivery of anything more than a sample quantity of devices for quite some time.
On Thu, May 7, 2026 at 1:03 AM Kevin Fenzi via devel-announce devel-announce@lists.fedoraproject.org wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Such things happen. And I have no doubt that it will get resolved soon enough.
That said, kernel 7.0.4, which fixes dirty frag, is currently waiting for the s390 builder availability to move to deployment.
Can we bypass the s390 build and get kernel 7.0.4 rolled out ASAP for other archs?
Thanks.
On Thu, May 7, 2026 at 1:03 AM Kevin Fenzi via devel-announce devel-announce@lists.fedoraproject.org wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Such things happen. And I have no doubt that it will get resolved soon enough.
That said, kernel 7.0.4, which fixes dirty frag, is currently waiting for the s390 builder availability to move to deployment.
Can we bypass the s390 build and get kernel 7.0.4 rolled out ASAP for other archs?
Thanks.
On Thu, May 7, 2026 at 9:08 PM Gary Buhrmaster gary.buhrmaster@gmail.com wrote:
On Thu, May 7, 2026 at 1:03 AM Kevin Fenzi via devel-announce devel-announce@lists.fedoraproject.org wrote:
Just to notify a wider audience, our s390x builders have been offline today due to a failure of a storage array.
Such things happen. And I have no doubt that it will get resolved soon enough.
That said, kernel 7.0.4, which fixes dirty frag, is currently waiting for the s390 builder availability to move to deployment.
Can we bypass the s390 build and get kernel 7.0.4 rolled out ASAP for other archs?
It is possible, but gets messy. I am staying up in hopes that the parts arrive and we get back online this evening. Some builders are supposed to be set to the secure boot channel when they come up to ensure that kernel jumps to the front of the line. In the meantime, builds are available for all supported Fedora releases on all other arches: Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=145225185 F44: https://koji.fedoraproject.org/koji/taskinfo?taskID=145224859 F43: https://koji.fedoraproject.org/koji/taskinfo?taskID=145224875 F42: https://koji.fedoraproject.org/koji/taskinfo?taskID=145226847
And since the builds aren't "complete" you might need to select the arch you need to get the taskid and use "koji download-task"
As this is also rushing through the rebase faster than I wanted, I will make the F42 build of 6.19.14 available in koji for F43 and F44 once s390 is back up and we have the official build through. I want to make sure there is an option for people in the event of a regression in 7.0.x
Thanks, Justin
Thanks.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://forge.fedoraproject.org/infra/tickets/issues/new
On Fri, May 8, 2026 at 4:16 AM Justin Forbes jmforbes@linuxtx.org wrote:
It is possible, but gets messy. I am staying up in hopes that the parts arrive and we get back online this evening.
Thank you for the "above and beyond" efforts.
In the meantime, builds are available for all supported Fedora releases on all other arches
Yes, I was thinking about mentioning that, and I had already used them, but then forgot to add that, so thank you for sharing what I forgot to do in my haste to get the query out. Of course, not everyone in the community is going to be aware of, or comfortable using, that "side-door" solution (for some reason, not everyone is on the devel email list :-)).
As this is also rushing through the rebase faster than I wanted
Premature embargo release is rarely a good thing (and result). There is, of course, also a mitigation for those that do not need those modules, but a full fix is always preferred.
Thanks, again.
On Fri, 2026-05-08 at 12:59 +0000, Gary Buhrmaster wrote:
On Fri, May 8, 2026 at 4:16 AM Justin Forbes jmforbes@linuxtx.org wrote:
It is possible, but gets messy. I am staying up in hopes that the parts arrive and we get back online this evening.
Thank you for the "above and beyond" efforts.
In the meantime, builds are available for all supported Fedora releases on all other arches
Yes, I was thinking about mentioning that, and I had already used them, but then forgot to add that, so thank you for sharing what I forgot to do in my haste to get the query out. Of course, not everyone in the community is going to be aware of, or comfortable using, that "side-door" solution (for some reason, not everyone is on the devel email list :-)).
e.g. those that require signed builds! IIRC signing happens after the build is considered done.
Best regards,
On Tue, May 12, 2026 at 6:35 AM Michel Lind salimma@fedoraproject.org wrote:
On Fri, 2026-05-08 at 12:59 +0000, Gary Buhrmaster wrote:
On Fri, May 8, 2026 at 4:16 AM Justin Forbes jmforbes@linuxtx.org wrote:
It is possible, but gets messy. I am staying up in hopes that the parts arrive and we get back online this evening.
Thank you for the "above and beyond" efforts.
In the meantime, builds are available for all supported Fedora releases on all other arches
Yes, I was thinking about mentioning that, and I had already used them, but then forgot to add that, so thank you for sharing what I forgot to do in my haste to get the query out. Of course, not everyone in the community is going to be aware of, or comfortable using, that "side-door" solution (for some reason, not everyone is on the devel email list :-)).
e.g. those that require signed builds! IIRC signing happens after the build is considered done.
Secure Boot signing is *part* of the build (for the kernel it is part of the build step). So if Justin doesn't do it, it gets signed with a non-trusted certificate. So he's the only one who can do it. :(
-
Adam Williamson -
Daniel P. Berrangé -
David Abdurachmanov -
Evan McNabb -
Gary Buhrmaster -
Hans de Goede -
Justin Forbes -
Kevin Fenzi -
Kevin Fenzi via devel-announce -
Michel Lind -
Neal Gompa -
Peter Lemenkov -
Peter Robinson -
Przemek Klosowski -
Steve Dickson -
Thorsten Leemhuis