-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On behalf of the ELN SIG, I'd like to share with you some of our plans for summer 2023. First, as some of you may know, we will be more actively beginning the process of launching CentOS Stream 10. We have made some recent changes to our strategy and schedule and this email will attempt to cover them both in detail. This will be a long email, so strap in!
== tl;dr == * We'll do an ELN mass rebuild to see how many builds will fail * We'll import ELN packages to c10s gradually, starting with just the runtime set. We'll use ELN as the buildroot at the start. * We'll import the buildroot set to c10s when it gets small enough. * You’ll see activity in CentOS Stream 10, but it’s not yet time to get involved. A general availability announcement will follow sometime in the first half of 2024. == tl;dr ==
We plan to perform a targeted mass-rebuild of Fedora ELN in a side-tag beginning on June 26th. We will be bumping the %{dist} value for ELN (currently .eln126) to avoid rpmdev-bumpspec noise. This rebuild will be used as a "canary" for how successful the initial import of CentOS Stream would be if we were to start it on that day. We are making several changes to our original plan as we do this.
First, as the cleanup of unnecessary dependencies in the Fedora ELN has not yet completed, we are revising our plan of performing a mass-import of *all* Fedora ELN content to CentOS Stream 10 in July. Instead, we plan to move to a phased approach wherein we will first import, build and test only the portions of CentOS Stream 10 that we currently have slated for inclusion in the delivered "runtime". As a necessary side-effect of this, it means that we will NOT be transitioning directly to CentOS Stream 10 as a self-hosted environment.
Instead, we will be retaining the Fedora ELN buildroot repository for use as the CentOS Stream 10 buildroot repository for some time (quite possibly until as late as February 6th, when Fedora 40 branches from Rawhide and CentOS Stream ceases syncing builds from Fedora ELN). During this time, CentOS Stream 10 builds will be superseded in the buildroot by Fedora ELN builds. While this sounds counterintuitive, this will actually allow us to take advantage of Fedora ELN to avoid build-ordering problems, such as those from soname bumps. The reason for this is that Fedora ELN, unlike when it in turn is synced from Fedora Rawhide, will functionally contain all of the same intended build attributes as CentOS Stream, such as build macros, compiler baseline flags and specfile conditionals. Since Fedora ELN should already be a preview of what CentOS Stream 10 should look like, we anticipate that there will be far fewer opportunities for build environment differences to rear their heads. Note that there will, of course, be an override option in place should we need to explicitly tag a CentOS Stream 10 build into the buildroot.
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild. Once the branches are created, we will also kick off the mass-build of that package set using Fedora ELN as the buildroot. Given the timing with the Fedora mass-rebuild, we are looking into whether to have our CentOS Stream 10 import "piggyback" off of it by carrying our Rawhide->ELN sync through to the ELN->CS10 sync, or if we would be better off waiting and performing the import manually.
The last bit of the plan is something of a non-announcement. As of right now, we do not have a firm date on when we will be importing the CentOS Stream buildroot-only components into CentOS Stream 10. We're toying with the idea of not doing this at all until we diverge from Fedora 40 (2024-02-06) in order to maximize the time we have to trim down the build dependencies. This way we do not end up importing and retiring dozens or hundreds of unwanted packages into CentOS Stream Gitlab, only to retire them shortly thereafter.
A lot of this is new and somewhat experimental. That's why we have decided on this rather aggressive timeline: if this does not go as smoothly as we hope, we will have ample time to correct before CentOS Stream 10 goes live for wider contribution.
Alright, I guess this isn't quite as long as I expected it was going to be, but it's quite dense. Questions and suggestions are most welcome.
== Some Expected Questions We Have Answers For ==
Q1: Does this mean that CentOS Stream 10 is opening for contribution soon? A1: From the geological scale of enterprise software, absolutely! From the scale of a Fedora cycle, it's still at least six months out. That said, since we're priming the pump in public this time around, you will have a much clearer window into how things get rolling.
Q2: Why are the CentOS Stream 10 builds not going to take priority over the ELN external repo in the buildroot once they succeed at building? A2: Because as we discovered with ELN, it's extremely difficult to get build ordering correct. If we build PackageA in CentOS Stream 10 and then LibraryB on which it depends bumps soname in Fedora ELN and rebuilds both LibraryB and PackageA, in a side-tag, we would need logic to figure out which order things need to be rebuilt in CentOS Stream 10 so that we didn't inadvertently get the timing incorrect. It's much harder to get right than you might expect and it's why in Fedora ELN nowadays we largely "cheat" when a Rawhide side-tag is merged and just tag the Rawhide build into Fedora ELN buildroot instead, since we know it is already built in the correct order, so we just need to rebuild it. I can give you a much longer (neé exhaustively longer) explanation if you want. (You don't.)
== Open Questions == 1) For the "canary" Fedora ELN rebuild, we have two choices on how to select the git hash to be built for each package in the ELN list:
Approach 1 (Rawhide-style): 1. Clone each package 2. Check for the existence of the `eln` branch. a. If the `eln` branch exists, build from the HEAD of that branch into the side-tag. b. Otherwise, build from the HEAD of the `rawhide` branch into the side-tag.
Approach 2 (Conservative-style): 1. Query Koji for the latest-tagged build of each package in Fedora ELN 2. Interrogate the build task of that build for the git hash 3. Build that git hash into the side-tag.
Approach 2 is more heavyweight, relying on a lot of Koji queries back-and-forth, whereas Approach 1 will pick up changes that have appeared in Rawhide since the last build (which is more in line with how Fedora's mass-rebuilds work). I'm personally leaning towards Approach 2, but I'm open to good arguments either way.
If you have read all the way to the end, I hereby award you one gold star for sheer tenacity. ⭐
[1] https://tiny.distro.builders/view-source-package-name-list--view-eln.txt See also https://tiny.distro.builders/view--view-eln.html for a more readable view.
_______________________________________________ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedorapro... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
* Stephen Gallagher:
First, as the cleanup of unnecessary dependencies in the Fedora ELN has not yet completed, we are revising our plan of performing a mass-import of *all* Fedora ELN content to CentOS Stream 10 in July. Instead, we plan to move to a phased approach wherein we will first import, build and test only the portions of CentOS Stream 10 that we currently have slated for inclusion in the delivered "runtime". As a necessary side-effect of this, it means that we will NOT be transitioning directly to CentOS Stream 10 as a self-hosted environment.
A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently porting CentOS 9 Stream to the x86-64-v3 ISA level. I believe there are some failures that are not yet fixed in ELN/upstream, and we'll be sharing our workarounds with rawhide/ELN as appropriate.
Thanks, Florian
On Fri, Jun 16, 2023 at 11:46 AM Florian Weimer fweimer@redhat.com wrote:
- Stephen Gallagher:
First, as the cleanup of unnecessary dependencies in the Fedora ELN has not yet completed, we are revising our plan of performing a mass-import of *all* Fedora ELN content to CentOS Stream 10 in July. Instead, we plan to move to a phased approach wherein we will first import, build and test only the portions of CentOS Stream 10 that we currently have slated for inclusion in the delivered "runtime". As a necessary side-effect of this, it means that we will NOT be transitioning directly to CentOS Stream 10 as a self-hosted environment.
A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently porting CentOS 9 Stream to the x86-64-v3 ISA level. I believe there are some failures that are not yet fixed in ELN/upstream, and we'll be sharing our workarounds with rawhide/ELN as appropriate.
Much appreciated. I know Yaakov Selkowitz has stumbled across at least one v3-specific issue that he's sent your way. We'll do whatever we can to help.
Florian Weimer fweimer@redhat.com writes:
- Stephen Gallagher:
First, as the cleanup of unnecessary dependencies in the Fedora ELN has not yet completed, we are revising our plan of performing a mass-import of *all* Fedora ELN content to CentOS Stream 10 in July. Instead, we plan to move to a phased approach wherein we will first import, build and test only the portions of CentOS Stream 10 that we currently have slated for inclusion in the delivered "runtime". As a necessary side-effect of this, it means that we will NOT be transitioning directly to CentOS Stream 10 as a self-hosted environment.
A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently porting CentOS 9 Stream to the x86-64-v3 ISA level. I believe there are some failures that are not yet fixed in ELN/upstream, and we'll be sharing our workarounds with rawhide/ELN as appropriate.
Just out of curiosity, have you looked at the glibc-hwcaps effort that has been going on the openSUSE side of the world for this? In theory it should allow to have the optimized subpackages for the respective architecture levels in the same repositories and it would not require a full rebuild of the who distribution.
Cheers,
Dan
* Dan Čermák:
A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently porting CentOS 9 Stream to the x86-64-v3 ISA level. I believe there are some failures that are not yet fixed in ELN/upstream, and we'll be sharing our workarounds with rawhide/ELN as appropriate.
Just out of curiosity, have you looked at the glibc-hwcaps effort that has been going on the openSUSE side of the world for this? In theory it should allow to have the optimized subpackages for the respective architecture levels in the same repositories and it would not require a full rebuild of the who distribution.
We are aware, but we want to compare performance numbers of equivalent builds before exploring delivery mechanisms.
Thanks, Florian
On 16. 06. 23 15:39, Stephen Gallagher wrote:
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild.
Will you please, please, please, please import the packages with git history and not via fepdkg srpm -> centpkg import?
On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 06. 23 15:39, Stephen Gallagher wrote:
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild.
Will you please, please, please, please import the packages with git history and not via fepdkg srpm -> centpkg import?
YES!
Stephen Gallagher wrote:
On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 06. 23 15:39, Stephen Gallagher wrote:
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild.
Will you please, please, please, please import the packages with git history and not via fepdkg srpm -> centpkg import?
YES!
Excellent news! As someone who takes git history (maybe a little too) seriously, this is quite welcome. Thanks to all who helped make the change. :)
On Fri, Jun 16, 2023 at 6:08 PM Todd Zullinger tmz@pobox.com wrote:
Stephen Gallagher wrote:
On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok mhroncok@redhat.com
wrote:
On 16. 06. 23 15:39, Stephen Gallagher wrote:
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild.
Will you please, please, please, please import the packages with git
history
and not via fepdkg srpm -> centpkg import?
YES!
Excellent news! As someone who takes git history (maybe a little too) seriously, this is quite welcome. Thanks to all who helped make the change. :)
YES!
Excellent, that is going to make my life a lot easier trying to keep GNOME packages in CentOS Stream 10 up to date (can easily cherry-pick commits)!
Thanks!
On 16. 06. 23 17:59, Stephen Gallagher wrote:
On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok mhroncok@redhat.com wrote:
On 16. 06. 23 15:39, Stephen Gallagher wrote:
This leads me to my next schedule announcement: we will be performing the initial CentOS Stream 10 branch creation in Gitlab for all packages in the Fedora ELN runtime package set[1] during the week of July 19th, coincident with the Fedora 39 mass-rebuild.
Will you please, please, please, please import the packages with git history and not via fepdkg srpm -> centpkg import?
YES!
Thank you so much for this!
On Fri, Jun 16, 2023 at 6:42 AM Stephen Gallagher sgallagh@redhat.com wrote:
== Open Questions ==
- For the "canary" Fedora ELN rebuild, we have two choices on how to
select the git hash to be built for each package in the ELN list:
Approach 1 (Rawhide-style):
- Clone each package
- Check for the existence of the `eln` branch.
a. If the `eln` branch exists, build from the HEAD of that branch into the side-tag. b. Otherwise, build from the HEAD of the `rawhide` branch into the side-tag.
Approach 2 (Conservative-style):
- Query Koji for the latest-tagged build of each package in Fedora ELN
- Interrogate the build task of that build for the git hash
- Build that git hash into the side-tag.
Approach 2 is more heavyweight, relying on a lot of Koji queries back-and-forth, whereas Approach 1 will pick up changes that have appeared in Rawhide since the last build (which is more in line with how Fedora's mass-rebuilds work). I'm personally leaning towards Approach 2, but I'm open to good arguments either way.
Sorry for being late with this, but I thought I'd already sent this.
I'd like to go with Approach 2 (Conservative-style) I've got a python script that grabs the githash for various builds. Although the initial run takes a couple of hours, after that it's fairly quick. It would make it possible to build using the ELN build source githash.
In the long run, I believe it takes less time to get the githash rather than pulling down the git repos. It's also something the can be pre-staged, then just run the script right before you run the mass rebuild, to get the latest.
Troy
-
Dan Čermák -
Florian Weimer -
Kalev Lember -
Miro Hrončok -
Stephen Gallagher -
Todd Zullinger -
Troy Dawson