Heads up: Yet another targeted mass rebuild of ~4k Python packages will happen in Fedora 43+
Hello,
Yes, that's right. The Python's "magic numer" has had to be bumped again[1] resulting in us needing to recreate the Python 3.14 .pyc files one more time in both Fedora 43 and 44. Python 3.14.0rc3 is planned to be released on Tuesday, September 16th, 2025. This is also the planned Beta Freeze end for Fedora 43.
To obtain a list of packages, use:
$ repoquery --repo=rawhide --refresh -f *.cpython-314.pyc --source (3743 packages atm)
and respectively:
$ repoquery --repo=fedora --releasever 43 --refresh -f *.cpython-314.pyc --source (3751 packages atm)
Packages with old .pyc files will have slower import times and might report AVC denials in restricted contexts. However, they remain functional, there is no need to panic.
The plan, as of now, is:
1. Wait for Python 3.14.0rc3 release. 2. Wait for the Fedora 43 Beta Freeze to finish. 3. Bump and build all packages in rawhide (except kernel). 4. FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43. 6. Open bugzillas for the remaining packages (failed or skipped rebuild).
The rebuild can be done in any order (i.e. all at once) and does not need to happen in a side tag. I will follow up when things start moving. Let me know if I should alter the plan somehow.
[1] https://github.com/python/cpython/pull/138749
On Thu, Sep 11, 2025 at 03:05:35PM +0200, Karolina Surma via devel-announce wrote:
Hello,
Yes, that's right. The Python's "magic numer" has had to be bumped again[1] resulting in us needing to recreate the Python 3.14 .pyc files one more time in both Fedora 43 and 44. Python 3.14.0rc3 is planned to be released on Tuesday, September 16th, 2025. This is also the planned Beta Freeze end for Fedora 43.
To obtain a list of packages, use:
$ repoquery --repo=rawhide --refresh -f *.cpython-314.pyc --source (3743 packages atm)
and respectively:
$ repoquery --repo=fedora --releasever 43 --refresh -f *.cpython-314.pyc --source (3751 packages atm)
Packages with old .pyc files will have slower import times and might report AVC denials in restricted contexts. However, they remain functional, there is no need to panic.
The plan, as of now, is:
- Wait for Python 3.14.0rc3 release.
- Wait for the Fedora 43 Beta Freeze to finish.
- Bump and build all packages in rawhide (except kernel).
- FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for
Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43.
How are you planning on doing the updates? In groups? one at a time? all at once?
I don't think bodhi can handle ~3700 packages in one update. :(
We could also bypass bodhi and just tag them directly in, but then that bypasses gating so we would need to make sure QE is ok with that.
- Open bugzillas for the remaining packages (failed or skipped rebuild).
The rebuild can be done in any order (i.e. all at once) and does not need to happen in a side tag. I will follow up when things start moving. Let me know if I should alter the plan somehow.
kevin
On Thu, Sep 11, 2025 at 5:44 PM Kevin Fenzi kevin@scrye.com wrote:
On Thu, Sep 11, 2025 at 03:05:35PM +0200, Karolina Surma via devel-announce wrote:
Hello,
Yes, that's right. The Python's "magic numer" has had to be bumped again[1] resulting in us needing to recreate the Python 3.14 .pyc files one more time in both Fedora 43 and 44. Python 3.14.0rc3 is planned to be released on Tuesday, September 16th, 2025. This is also the planned Beta Freeze end for Fedora 43.
To obtain a list of packages, use:
$ repoquery --repo=rawhide --refresh -f *.cpython-314.pyc --source (3743 packages atm)
and respectively:
$ repoquery --repo=fedora --releasever 43 --refresh -f *.cpython-314.pyc --source (3751 packages atm)
Packages with old .pyc files will have slower import times and might report AVC denials in restricted contexts. However, they remain functional, there is no need to panic.
The plan, as of now, is:
- Wait for Python 3.14.0rc3 release.
- Wait for the Fedora 43 Beta Freeze to finish.
- Bump and build all packages in rawhide (except kernel).
- FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for
Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43.
How are you planning on doing the updates? In groups? one at a time? all at once?
I don't think bodhi can handle ~3700 packages in one update. :(
This is something we actually wanted to try for ELN. I feel like I remember a conversation from the recent past where we figured out that the major impediment to doing this had actually been fixed (it was related to HTTP timeouts trying to add all of the packages, I think). I'm pretty sure we'd been given the green light to try this for the ~2500 packages in ELN. I'm CCing Yaakov Selkowitz and Adam Williamson directly who could correct me if I'm misremembering.
So maybe it's worth giving it a try? If it fails, we can always fall back to the direct tagging option.
Karolina: Either way, I think we need to do this in a side-tag, if only to have an easy way to look up what potentially needs tagging. Since we're past the Bodhi activation point, we can't rely on automation to create all of the individual updates (short of submitting them as a single update from a side-tag).
We could also bypass bodhi and just tag them directly in, but then that bypasses gating so we would need to make sure QE is ok with that.
- Open bugzillas for the remaining packages (failed or skipped rebuild).
The rebuild can be done in any order (i.e. all at once) and does not need to happen in a side tag. I will follow up when things start moving. Let me know if I should alter the plan somehow.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Thu, Sep 11, 2025 at 07:36:04PM -0400, Stephen Gallagher wrote:
This is something we actually wanted to try for ELN. I feel like I remember a conversation from the recent past where we figured out that the major impediment to doing this had actually been fixed (it was related to HTTP timeouts trying to add all of the packages, I think). I'm pretty sure we'd been given the green light to try this for the ~2500 packages in ELN. I'm CCing Yaakov Selkowitz and Adam Williamson directly who could correct me if I'm misremembering.
https://pagure.io/releng/issue/12869 we were going to test in staging, but haven't yet done so.
So maybe it's worth giving it a try? If it fails, we can always fall back to the direct tagging option.
I suppose... if it fails it might cause a mess, but if we do decide to do this, please coordinate so we can bring bodhi back if it gets in a bad state.
Karolina: Either way, I think we need to do this in a side-tag, if only to have an easy way to look up what potentially needs tagging. Since we're past the Bodhi activation point, we can't rely on automation to create all of the individual updates (short of submitting them as a single update from a side-tag).
yeah.
kevin
On 9/11/25 19:36, Stephen Gallagher wrote:
On Thu, Sep 11, 2025 at 5:44 PM Kevin Fenzi kevin@scrye.com wrote:
On Thu, Sep 11, 2025 at 03:05:35PM +0200, Karolina Surma via devel-announce wrote:
Hello,
Yes, that's right. The Python's "magic numer" has had to be bumped again[1] resulting in us needing to recreate the Python 3.14 .pyc files one more time in both Fedora 43 and 44. Python 3.14.0rc3 is planned to be released on Tuesday, September 16th, 2025. This is also the planned Beta Freeze end for Fedora 43.
To obtain a list of packages, use:
$ repoquery --repo=rawhide --refresh -f *.cpython-314.pyc --source (3743 packages atm)
and respectively:
$ repoquery --repo=fedora --releasever 43 --refresh -f *.cpython-314.pyc --source (3751 packages atm)
Packages with old .pyc files will have slower import times and might report AVC denials in restricted contexts. However, they remain functional, there is no need to panic.
The plan, as of now, is:
- Wait for Python 3.14.0rc3 release.
- Wait for the Fedora 43 Beta Freeze to finish.
- Bump and build all packages in rawhide (except kernel).
- FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for
Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43.
How are you planning on doing the updates? In groups? one at a time? all at once?
I don't think bodhi can handle ~3700 packages in one update. :(
This is something we actually wanted to try for ELN. I feel like I remember a conversation from the recent past where we figured out that the major impediment to doing this had actually been fixed (it was related to HTTP timeouts trying to add all of the packages, I think). I'm pretty sure we'd been given the green light to try this for the ~2500 packages in ELN. I'm CCing Yaakov Selkowitz and Adam Williamson directly who could correct me if I'm misremembering.
So maybe it's worth giving it a try? If it fails, we can always fall back to the direct tagging option.
Karolina: Either way, I think we need to do this in a side-tag, if only to have an easy way to look up what potentially needs tagging. Since we're past the Bodhi activation point, we can't rely on automation to create all of the individual updates (short of submitting them as a single update from a side-tag).
To clarify, we do want to try a massive update, but *only* from a single side tag. We do NOT recommend trying to make a massive update from individual builds. Because of how bodhi works, we believe there is a major difference in performance between these two possibilities, and that only the former has a decent chance of success.
The plan, as of now, is:
- Wait for Python 3.14.0rc3 release.
- Wait for the Fedora 43 Beta Freeze to finish.
- Bump and build all packages in rawhide (except kernel).
- FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for
Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43.
How are you planning on doing the updates? In groups? one at a time? all at once?
I don't think bodhi can handle ~3700 packages in one update. :(
This is something we actually wanted to try for ELN. I feel like I remember a conversation from the recent past where we figured out that the major impediment to doing this had actually been fixed (it was related to HTTP timeouts trying to add all of the packages, I think). I'm pretty sure we'd been given the green light to try this for the ~2500 packages in ELN. I'm CCing Yaakov Selkowitz and Adam Williamson directly who could correct me if I'm misremembering.
So maybe it's worth giving it a try? If it fails, we can always fall back to the direct tagging option.
I'd much prefer not to proceed in yet untested ways, there will be a lot to handle in a rather short timeframe. A single Bodhi update with all the builds would have more downsides to it: any failed gating would stop the whole update; any updated-in-the-meantime package would require us to bump the one in the side tag and reset the timer to push to stable.
Karolina: Either way, I think we need to do this in a side-tag, if only to have an easy way to look up what potentially needs tagging. Since we're past the Bodhi activation point, we can't rely on automation to create all of the individual updates (short of submitting them as a single update from a side-tag).
Since all the builds will most probably be trigerred by me, a simple koji query filtering my builds after a given timestamp will be a good enough solution to make sure we caught them all.
---
Kevin, to your original question: if we create individual updates for each package, what, except for my work behind the scenes (to determine built packages and trigger an update for them) can go wrong? Can Bodhi handle ~3700 individual updates appearing in the span of ~2 days? If so, we'd most likely go this route. What is the best way forward from your point of view?
Karolina
On Fri, Sep 12, 2025 at 5:05 AM Karolina Surma ksurma@redhat.com wrote:
The plan, as of now, is:
- Wait for Python 3.14.0rc3 release.
- Wait for the Fedora 43 Beta Freeze to finish.
- Bump and build all packages in rawhide (except kernel).
- FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for
Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43.
How are you planning on doing the updates? In groups? one at a time? all at once?
I don't think bodhi can handle ~3700 packages in one update. :(
This is something we actually wanted to try for ELN. I feel like I remember a conversation from the recent past where we figured out that the major impediment to doing this had actually been fixed (it was related to HTTP timeouts trying to add all of the packages, I think). I'm pretty sure we'd been given the green light to try this for the ~2500 packages in ELN. I'm CCing Yaakov Selkowitz and Adam Williamson directly who could correct me if I'm misremembering.
So maybe it's worth giving it a try? If it fails, we can always fall back to the direct tagging option.
I'd much prefer not to proceed in yet untested ways, there will be a lot to handle in a rather short timeframe. A single Bodhi update with all the builds would have more downsides to it: any failed gating would stop the whole update; any updated-in-the-meantime package would require us to bump the one in the side tag and reset the timer to push to stable.
Karolina: Either way, I think we need to do this in a side-tag, if only to have an easy way to look up what potentially needs tagging. Since we're past the Bodhi activation point, we can't rely on automation to create all of the individual updates (short of submitting them as a single update from a side-tag).
Since all the builds will most probably be trigerred by me, a simple koji query filtering my builds after a given timestamp will be a good enough solution to make sure we caught them all.
Kevin, to your original question: if we create individual updates for each package, what, except for my work behind the scenes (to determine built packages and trigger an update for them) can go wrong? Can Bodhi handle ~3700 individual updates appearing in the span of ~2 days? If so, we'd most likely go this route. What is the best way forward from your point of view?
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
To my mind, we have three realistic options here:
Option 1) You perform all of the builds, we identify them and tag them manually into Fedora, bypassing Bodhi entirely. Pros: Low resource usage, can be done with or without a side-tag as preferred. (Though a side-tag means it's easier to keep track of which packages were built specifically for this and it allows maintainers to submit their own builds into the side-tag if needed, rather than relying only on Karolina to build everything.) Cons: It bypasses all testing, so we won't know until the next compose attempt whether things still work.
Option 2) Build everything into a side-tag, we try submitting the side tag as a Bodhi update Pros: The set of changes gets tested (though it probably fails and needs to waive at least a few cases). It exercises a path ELN wants to take and it provides Fedora data on whether Bodhi can handle such a large update set. Even if Bodhi fails, we still have the option to just migrate everything from the side-tag directly into stable. Cons: If the load generated is too high, we could cause a DoS on Bodhi.
Option 3) Don't do this mass-rebuild at all. Pros: No resource usage, no risk to infrastructure. Cons: Low initial end-user performance, as we'll ship Python packages that need to byte-compile every module the first time they are used.
Does anyone see an Option 4 somewhere?
On Fri, Sep 12, 2025 at 11:43:38AM -0400, Stephen Gallagher wrote:
On Fri, Sep 12, 2025 at 5:05 AM Karolina Surma ksurma@redhat.com wrote:
Since all the builds will most probably be trigerred by me, a simple koji query filtering my builds after a given timestamp will be a good enough solution to make sure we caught them all.
Well, we already have a releng script that handles merging side tags correctly. It will check the dest tag and not tag something if there's a newer build already (instead of just blindly tagging it in and overriding a newer maintainer build that was there).
But as long as anything handles these corner cases, ok...
Kevin, to your original question: if we create individual updates for each package, what, except for my work behind the scenes (to determine built packages and trigger an update for them) can go wrong? Can Bodhi handle ~3700 individual updates appearing in the span of ~2 days? If so, we'd most likely go this route. What is the best way forward from your point of view?
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
It will definitely cause ci load... since all of those will need to pass gating tests (if any).
To my mind, we have three realistic options here:
Option 1) You perform all of the builds, we identify them and tag them manually into Fedora, bypassing Bodhi entirely. Pros: Low resource usage, can be done with or without a side-tag as preferred. (Though a side-tag means it's easier to keep track of which packages were built specifically for this and it allows maintainers to submit their own builds into the side-tag if needed, rather than relying only on Karolina to build everything.)
Well, or use a side tag and merge via script. Then if maintainers have a 'newer' build already in, it just doesn't tag the side tag one. No need to tell people to build in a sidetag or anything, they just keep doing what they are doing.
Cons: It bypasses all testing, so we won't know until the next compose attempt whether things still work.
Right, although, I think Adam might have some way to test a side tag seperately in openqa...
Option 2) Build everything into a side-tag, we try submitting the side tag as a Bodhi update Pros: The set of changes gets tested (though it probably fails and needs to waive at least a few cases). It exercises a path ELN wants to take and it provides Fedora data on whether Bodhi can handle such a large update set. Even if Bodhi fails, we still have the option to just migrate everything from the side-tag directly into stable. Cons: If the load generated is too high, we could cause a DoS on Bodhi.
Option 3) Don't do this mass-rebuild at all. Pros: No resource usage, no risk to infrastructure. Cons: Low initial end-user performance, as we'll ship Python packages that need to byte-compile every module the first time they are used.
Right, and many things will be rebuilt before final, but of course not all.
Does anyone see an Option 4 somewhere?
I don't off hand.
kevin
On Fri, 2025-09-12 at 08:59 -0700, Kevin Fenzi wrote:
Cons: It bypasses all testing, so we won't know until the next compose attempt whether things still work.
Right, although, I think Adam might have some way to test a side tag seperately in openqa...
I *do*, but I've never tried it on a 3700 package side tag ;) It *should* work, though, as it doesn't try to download all the packages one at a time, or anything. It just adds the side tag repo to the test config, everywhere we define repos.
Sep 12, 2025 10:44:04 AM Stephen Gallagher sgallagh@redhat.com:
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
To my mind, we have three realistic options here:
For the record, they did the same thing for the rebuild a few weeks ago last time the pyc magic number was bumped and it seemed to work well enough. It was actually 3700 * 2 Bodhi updates (x2 for rawhide and f43). Bodhi ended up catching up and each update got gated separately. The Go SIG and I did something similar for the Go rebuild (400-500 updates for each branch) the same weekend, and that procedure worked well enough for us also. There were some signing infra failures (long-standing issue) and CI infra issues (once Fedora CI team restarted Jenkins, everything caught up without any extra intervention needed) that affected us, but those were unrelated to either rebuild.
Doing one large side tag is untested and is unnecessary when there are no build ordering issues (only Python needs to be updated and then everything else can be built and submitted to Bodhi in any order). Also, as Karolina said, one update that includes everything would block every single build even if only one package fails gating. By doing an update for each package, each gating failure can be handled independently by the package's maintainer. Any solution that involves manually tagging builds and skipping gating entirely is even less ideal and also creates more work for releng.
On Fri, 2025-09-12 at 11:43 -0400, Stephen Gallagher wrote:
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
I'm not really sure this is true. We get ~800 individual updates pretty quickly with the ELN mass rebuilds, and that works fine, AFAIK.
Bodhi is a reasonably sane webapp. Creating an update isn't an *excessively* onerous operation in it. I can't really think why it shouldn't be able to cope with creating 3700 relatively quickly.
I can imagine it might cause signing to get backed up, but then, that's a problem however we do this, isn't it? However you slice it, 3700 packages need signing.
It might back Fedora CI up a bit, I guess, since Fedora CI runs tests on every new update. But a 3700-package megaupdate would cause more or less the same load on Fedora CI I think (as it *mostly* tests at the package level, not the update level). Only bypassing Bodhi entirely would bypass Fedora CI.
openQA should be fine as it only tests critpath stuff, and much fewer than 3700 of the updates will be critpath.
On Fri, Sep 12, 2025 at 10:25:17AM -0700, Adam Williamson wrote:
On Fri, 2025-09-12 at 11:43 -0400, Stephen Gallagher wrote:
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
I'm not really sure this is true. We get ~800 individual updates pretty quickly with the ELN mass rebuilds, and that works fine, AFAIK.
Bodhi is a reasonably sane webapp. Creating an update isn't an *excessively* onerous operation in it. I can't really think why it shouldn't be able to cope with creating 3700 relatively quickly.
I can imagine it might cause signing to get backed up, but then, that's a problem however we do this, isn't it? However you slice it, 3700 packages need signing.
Well, if we do this in a named/releng created side tag we could also setup things to sign as builds complete.
It might back Fedora CI up a bit, I guess, since Fedora CI runs tests on every new update. But a 3700-package megaupdate would cause more or less the same load on Fedora CI I think (as it *mostly* tests at the package level, not the update level). Only bypassing Bodhi entirely would bypass Fedora CI.
openQA should be fine as it only tests critpath stuff, and much fewer than 3700 of the updates will be critpath.
ok, if you think the load will be acceptable then we can give it a try that way. I still think a sidetag would be better (for the signing if nothing else), but either way...
kevin
On Mon, 2025-09-15 at 09:25 -0700, Kevin Fenzi wrote:
On Fri, Sep 12, 2025 at 10:25:17AM -0700, Adam Williamson wrote:
On Fri, 2025-09-12 at 11:43 -0400, Stephen Gallagher wrote:
~3700 individual updates hitting Bodhi over a short period is almost certainly going to cause problems with the load, similar to trying to manually create a single update with all the same packages.
I'm not really sure this is true. We get ~800 individual updates pretty quickly with the ELN mass rebuilds, and that works fine, AFAIK.
Bodhi is a reasonably sane webapp. Creating an update isn't an *excessively* onerous operation in it. I can't really think why it shouldn't be able to cope with creating 3700 relatively quickly.
I can imagine it might cause signing to get backed up, but then, that's a problem however we do this, isn't it? However you slice it, 3700 packages need signing.
Well, if we do this in a named/releng created side tag we could also setup things to sign as builds complete.
It might back Fedora CI up a bit, I guess, since Fedora CI runs tests on every new update. But a 3700-package megaupdate would cause more or less the same load on Fedora CI I think (as it *mostly* tests at the package level, not the update level). Only bypassing Bodhi entirely would bypass Fedora CI.
openQA should be fine as it only tests critpath stuff, and much fewer than 3700 of the updates will be critpath.
ok, if you think the load will be acceptable then we can give it a try that way. I still think a sidetag would be better (for the signing if nothing else), but either way...
Oh, if you think a side tag would be better I'm happy to defer to that :) I just wanted to note I don't think the problems in Bodhi are *quite* as bad as suggested. Of course we never know for sure till we try.
On Mon, Sep 15, 2025 at 09:47:53AM -0700, Adam Williamson wrote:
Oh, if you think a side tag would be better I'm happy to defer to that :) I just wanted to note I don't think the problems in Bodhi are *quite* as bad as suggested. Of course we never know for sure till we try.
It might be a wash... releng side-tag would allow signing as builds finish, but per build bodhi updates would allow for gating to catch anything that was really broken.
kevin
On Mon, 2025-09-15 at 10:06 -0700, Kevin Fenzi wrote:
On Mon, Sep 15, 2025 at 09:47:53AM -0700, Adam Williamson wrote:
Oh, if you think a side tag would be better I'm happy to defer to that :) I just wanted to note I don't think the problems in Bodhi are *quite* as bad as suggested. Of course we never know for sure till we try.
It might be a wash... releng side-tag would allow signing as builds finish, but per build bodhi updates would allow for gating to catch anything that was really broken.
As I mentioned, I can get openQA to test a side tag. It's not automated, but it's a single command that takes me five seconds to type. :D
On Mon, Sep 15, 2025 at 11:51:41AM -0700, Adam Williamson wrote:
On Mon, 2025-09-15 at 10:06 -0700, Kevin Fenzi wrote:
On Mon, Sep 15, 2025 at 09:47:53AM -0700, Adam Williamson wrote:
Oh, if you think a side tag would be better I'm happy to defer to that :) I just wanted to note I don't think the problems in Bodhi are *quite* as bad as suggested. Of course we never know for sure till we try.
It might be a wash... releng side-tag would allow signing as builds finish, but per build bodhi updates would allow for gating to catch anything that was really broken.
As I mentioned, I can get openQA to test a side tag. It's not automated, but it's a single command that takes me five seconds to type. :D
Sure, but then it would be a bit of a wack-a-mole if somethings failed gating. ie, have to fix them each or remove from the sidetag and re-run, etc.
kevin
On 12. 09. 25 11:04, Karolina Surma wrote:
Can Bodhi handle ~3700 individual updates appearing in the span of ~2 days? If so, we'd most likely go this route.
During the Python 3.14.0rc2 pyc rebuild, I spawned ~8k automatic updates (~4k for rawhide and ~4k for f43) and Bodhi handled it alright.
On 9/11/25 15:05, Karolina Surma wrote: <snip>
The plan, as of now, is:
1. Wait for Python 3.14.0rc3 release. 2. Wait for the Fedora 43 Beta Freeze to finish. 3. Bump and build all packages in rawhide (except kernel). 4. FF-merge into f43 and build all packages that match the criteria: a) the rawhide and f43 branches were not different before 3. b) they have no update stuck in bodhi/gating. c) their latest commit hash has been successfully built and shipped for Fedora 43. 5. Open Bodhi updates for packages built for Fedora 43. 6. Open bugzillas for the remaining packages (failed or skipped rebuild).
The rebuild can be done in any order (i.e. all at once) and does not need to happen in a side tag. I will follow up when things start moving. Let me know if I should alter the plan somehow.
After considering the replies to the thread, we decided to go the buildroot override route with individual package updates for F43.
Python 3.14.0rc3 has been released today, I'm now testing the package in Copr and expect to start the actual Fedora rebuild tomorrow.
On Thu, 2025-09-18 at 18:02 +0200, Karolina Surma wrote:
On 9/11/25 15:05, Karolina Surma wrote:
<snip> > > The plan, as of now, is: > > 1. Wait for Python 3.14.0rc3 release. > 2. Wait for the Fedora 43 Beta Freeze to finish. > 3. Bump and build all packages in rawhide (except kernel). > 4. FF-merge into f43 and build all packages that match the criteria: > a) the rawhide and f43 branches were not different before 3. > b) they have no update stuck in bodhi/gating. > c) their latest commit hash has been successfully built and shipped > for Fedora 43. > 5. Open Bodhi updates for packages built for Fedora 43. > 6. Open bugzillas for the remaining packages (failed or skipped rebuild). > > The rebuild can be done in any order (i.e. all at once) and does not > need to happen in a side tag. > I will follow up when things start moving. Let me know if I should alter > the plan somehow. > > [1] https://github.com/python/cpython/pull/138749 > >
After considering the replies to the thread, we decided to go the buildroot override route with individual package updates for F43.
Python 3.14.0rc3 has been released today, I'm now testing the package in Copr and expect to start the actual Fedora rebuild tomorrow.
Live progress report: we seem to be in the midst of the flood now. Bodhi looks like it's coping fine. openQA is about four hours behind at present, which isn't terrible. There've been slightly more flakey failures than normal, especially on aarch64, probably due to the constant load on the workers. aarch64 is probably harder hit because the aarch64 workers have slightly less storage bandwidth than the x86_64 workers (6 disks per worker vs. 8 for x86_64). On the whole this is all within manageable ranges, though.
-
Adam Williamson -
Karolina Surma -
Kevin Fenzi -
Maxwell G -
Miro Hrončok -
Stephen Gallagher -
Yaakov Selkowitz