Fedora ELN Plans for Summer 2023

[Stephen Gallagher]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On behalf of the ELN SIG, I'd like to share with you some of our plans
for summer 2023. First, as some of you may know, we will be more
actively beginning the process of launching CentOS Stream 10. We have
made some recent changes to our strategy and schedule and this email
will attempt to cover them both in detail. This will be a long email,
so strap in!

== tl;dr ==
* We'll do an ELN mass rebuild to see how many builds will fail
* We'll import ELN packages to c10s gradually, starting with just the
runtime set. We'll use ELN as the buildroot at the start.
* We'll import the buildroot set to c10s when it gets small enough.
* You’ll see activity in CentOS Stream 10, but it’s not yet time to
get involved. A general availability announcement will follow sometime
in the first half of 2024.
== tl;dr ==

We plan to perform a targeted mass-rebuild of Fedora ELN in a side-tag
beginning on June 26th. We will be bumping the %{dist} value for ELN
(currently .eln126) to avoid rpmdev-bumpspec noise. This rebuild will
be used as a "canary" for how successful the initial import of CentOS
Stream would be if we were to start it on that day. We are making
several changes to our original plan as we do this.

First, as the cleanup of unnecessary dependencies in the Fedora ELN
has not yet completed, we are revising our plan of performing a
mass-import of *all* Fedora ELN content to CentOS Stream 10 in July.
Instead, we plan to move to a phased approach wherein we will first
import, build and test only the portions of CentOS Stream 10 that we
currently have slated for inclusion in the delivered "runtime". As a
necessary side-effect of this, it means that we will NOT be
transitioning directly to CentOS Stream 10 as a self-hosted
environment.

Instead, we will be retaining the Fedora ELN buildroot repository for
use as the CentOS Stream 10 buildroot repository for some time (quite
possibly until as late as February 6th, when Fedora 40 branches from
Rawhide and CentOS Stream ceases syncing builds from Fedora ELN).
During this time, CentOS Stream 10 builds will be superseded in the
buildroot by Fedora ELN builds. While this sounds counterintuitive,
this will actually allow us to take advantage of Fedora ELN to avoid
build-ordering problems, such as those from soname bumps. The reason
for this is that Fedora ELN, unlike when it in turn is synced from
Fedora Rawhide, will functionally contain all of the same intended
build attributes as CentOS Stream, such as build macros, compiler
baseline flags and specfile conditionals. Since Fedora ELN should
already be a preview of what CentOS Stream 10 should look like, we
anticipate that there will be far fewer opportunities for build
environment differences to rear their heads. Note that there will, of
course, be an override option in place should we need to explicitly
tag a CentOS Stream 10 build into the buildroot.

This leads me to my next schedule announcement: we will be performing
the initial CentOS Stream 10 branch creation in Gitlab for all
packages in the Fedora ELN runtime package set[1]  during the week of
July 19th, coincident with the Fedora 39 mass-rebuild. Once the
branches are created, we will also kick off the mass-build of that
package set using Fedora ELN as the buildroot. Given the timing with
the Fedora mass-rebuild, we are looking into whether to have our
CentOS Stream 10 import "piggyback" off of it by carrying our
Rawhide->ELN sync through to the ELN->CS10 sync, or if we would be
better off waiting and performing the import manually.

The last bit of the plan is something of a non-announcement. As of
right now, we do not have a firm date on when we will be importing the
CentOS Stream buildroot-only components into CentOS Stream 10. We're
toying with the idea of not doing this at all until we diverge from
Fedora 40 (2024-02-06) in order to maximize the time we have to trim
down the build dependencies. This way we do not end up importing and
retiring dozens or hundreds of unwanted packages into CentOS Stream
Gitlab, only to retire them shortly thereafter.

A lot of this is new and somewhat experimental. That's why we have
decided on this rather aggressive timeline: if this does not go as
smoothly as we hope, we will have ample time to correct before CentOS
Stream 10 goes live for wider contribution.

Alright, I guess this isn't quite as long as I expected it was going
to be, but it's quite dense. Questions and suggestions are most
welcome.


== Some Expected Questions We Have Answers For ==

Q1: Does this mean that CentOS Stream 10 is opening for contribution soon?
A1: From the geological scale of enterprise software, absolutely! From
the scale of a Fedora cycle, it's still at least six months out. That
said, since we're priming the pump in public this time around, you
will have a much clearer window into how things get rolling.

Q2: Why are the CentOS Stream 10 builds not going to take priority
over the ELN external repo in the buildroot once they succeed at
building?
A2: Because as we discovered with ELN, it's extremely difficult to get
build ordering correct. If we build PackageA in CentOS Stream 10 and
then LibraryB on which it depends bumps soname in Fedora ELN and
rebuilds both LibraryB and PackageA, in a side-tag, we would need
logic to figure out which order things need to be rebuilt in CentOS
Stream 10 so that we didn't inadvertently get the timing incorrect.
It's much harder to get right than you might expect and it's why in
Fedora ELN nowadays we largely "cheat" when a Rawhide side-tag is
merged and just tag the Rawhide build into Fedora ELN buildroot
instead, since we know it is already built in the correct order, so we
just need to rebuild it. I can give you a much longer (neé
exhaustively longer) explanation if you want. (You don't.)


== Open Questions ==
1) For the "canary" Fedora ELN rebuild, we have two choices on how to
select the git hash to be built for each package in the ELN list:

Approach 1 (Rawhide-style):
1. Clone each package
2. Check for the existence of the `eln` branch.
  a. If the `eln` branch exists, build from the HEAD of that branch
into the side-tag.
  b. Otherwise, build from the HEAD of the `rawhide` branch into the side-tag.

Approach 2 (Conservative-style):
1. Query Koji for the latest-tagged build of each package in Fedora ELN
2. Interrogate the build task of that build for the git hash
3. Build that git hash into the side-tag.

Approach 2 is more heavyweight, relying on a lot of Koji queries
back-and-forth, whereas Approach 1 will pick up changes that have
appeared in Rawhide since the last build (which is more in line with
how Fedora's mass-rebuilds work). I'm personally leaning towards
Approach 2, but I'm open to good arguments either way.


If you have read all the way to the end, I hereby award you one gold
star for sheer tenacity. ⭐



[1] https://tiny.distro.builders/view-source-package-name-list--view-eln.txt
See also https://tiny.distro.builders/view--view-eln.html for a more
readable view.
> -----BEGIN PGP SIGNATURE-----
>
> iQGzBAEBCAAdFiEEhQqd0dvyrMxvxJSRRduFpWgobREFAmSMZhAACgkQRduFpWgo
> bRGlKQv/SfDVJ6Ny+T80fnUK4KsDacxrpYcXYpAf+GPUdeELdMwXIJL+N/23Cg0L
> NoyK0GhKYqJu9SMKYH/1TWEbbxymyOipm/4XLzh59439hRdaogOOE5G0Q3/4wLLH
> C1woO4rJwRDNEgUBxf/EHXHZc63yEevPaVZfjiBjZTVZ913FA6XqNp8BCxoCliWV
> gFp1tIF6wIk3Nn4vcSdnLbEZnQoS/7qnSQJejddxxKq8RykJaEv0KrwxR9guctSg
> VGbmTvlguhH2xReXL+Dx139PBXNuPYz+zB/vAK+Yu1MTsYHT6ysWHRUpRNJIfUbR
> MaiWrxJ0+wldqNvXs4d0kWqtx7qV1DQ8RvUdBml7S+bjgvWyk4+a+NUk0Jjn64CF
> f1eKXPaE3SEDPzM2CP5o7emTA+jzIcLJN2AhgfeycqfuQbeQ/B6S1Bm7LSzQLYeo
> LyL6b1q5JWbZsEH9lu4MFodY6f+1REvVH4hY01FyhzEnZsHl5Vc1279NHE/S6V15
> b+x6/yE4
> =2u3F
> -----END PGP SIGNATURE-----
>  
_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Florian Weimer]

* Stephen Gallagher:

>  First, as the cleanup of unnecessary dependencies in the Fedora ELN
>  has not yet completed, we are revising our plan of performing a
>  mass-import of *all* Fedora ELN content to CentOS Stream 10 in July.
>  Instead, we plan to move to a phased approach wherein we will first
>  import, build and test only the portions of CentOS Stream 10 that we
>  currently have slated for inclusion in the delivered "runtime". As a
>  necessary side-effect of this, it means that we will NOT be
>  transitioning directly to CentOS Stream 10 as a self-hosted
>  environment. 
A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently
porting CentOS 9 Stream to the x86-64-v3 ISA level.  I believe there are
some failures that are not yet fixed in ELN/upstream, and we'll be
sharing our workarounds with rawhide/ELN as appropriate.

Thanks,
Florian

[Stephen Gallagher]

On Fri, Jun 16, 2023 at 11:46 AM Florian Weimer <fweimer(a)redhat.com&gt; wrote:
>  * Stephen Gallagher:
>
>  > First, as the cleanup of unnecessary dependencies in the Fedora ELN
>  > has not yet completed, we are revising our plan of performing a
>  > mass-import of *all* Fedora ELN content to CentOS Stream 10 in July.
>  > Instead, we plan to move to a phased approach wherein we will first
>  > import, build and test only the portions of CentOS Stream 10 that we
>  > currently have slated for inclusion in the delivered "runtime". As a
>  > necessary side-effect of this, it means that we will NOT be
>  > transitioning directly to CentOS Stream 10 as a self-hosted
>  > environment.
>
>  A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently
>  porting CentOS 9 Stream to the x86-64-v3 ISA level.  I believe there are
>  some failures that are not yet fixed in ELN/upstream, and we'll be
>  sharing our workarounds with rawhide/ELN as appropriate. 
Much appreciated. I know Yaakov Selkowitz has stumbled across at least
one v3-specific issue that he's sent your way. We'll do whatever we
can to help.

[Dan Čermák]

Florian Weimer <fweimer(a)redhat.com&gt; writes:

>  * Stephen Gallagher:
>
> > First, as the cleanup of unnecessary dependencies in the Fedora ELN
> > has not yet completed, we are revising our plan of performing a
> > mass-import of *all* Fedora ELN content to CentOS Stream 10 in July.
> > Instead, we plan to move to a phased approach wherein we will first
> > import, build and test only the portions of CentOS Stream 10 that we
> > currently have slated for inclusion in the delivered "runtime". As a
> > necessary side-effect of this, it means that we will NOT be
> > transitioning directly to CentOS Stream 10 as a self-hosted
> > environment.
>
>  A bit related to the CentOS 10 bringup, the CentOS ISA SIG is currently
>  porting CentOS 9 Stream to the x86-64-v3 ISA level.  I believe there are
>  some failures that are not yet fixed in ELN/upstream, and we'll be
>  sharing our workarounds with rawhide/ELN as appropriate. 
Just out of curiosity, have you looked at the glibc-hwcaps effort that
has been going on the openSUSE side of the world for this? In theory it
should allow to have the optimized subpackages for the respective
architecture levels in the same repositories and it would not require a
full rebuild of the who distribution.


Cheers,

Dan

[Florian Weimer]

* Dan Čermák:

> > A bit related to the CentOS 10 bringup, the CentOS ISA SIG is
> currently
> > porting CentOS 9 Stream to the x86-64-v3 ISA level.  I believe there are
> > some failures that are not yet fixed in ELN/upstream, and we'll be
> > sharing our workarounds with rawhide/ELN as appropriate.
>
>  Just out of curiosity, have you looked at the glibc-hwcaps effort that
>  has been going on the openSUSE side of the world for this? In theory it
>  should allow to have the optimized subpackages for the respective
>  architecture levels in the same repositories and it would not require a
>  full rebuild of the who distribution. 
We are aware, but we want to compare performance numbers of equivalent
builds before exploring delivery mechanisms.

Thanks,
Florian

[Miro Hrončok]

On 16. 06. 23 15:39, Stephen Gallagher wrote:
>  This leads me to my next schedule announcement: we will be
> performing
>  the initial CentOS Stream 10 branch creation in Gitlab for all
>  packages in the Fedora ELN runtime package set[1]  during the week of
>  July 19th, coincident with the Fedora 39 mass-rebuild. 
Will you please, please, please, please import the packages with git history 
and not via fepdkg srpm -> centpkg import?

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Stephen Gallagher]

On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 16. 06. 23 15:39, Stephen Gallagher wrote:
>  > This leads me to my next schedule announcement: we will be performing
>  > the initial CentOS Stream 10 branch creation in Gitlab for all
>  > packages in the Fedora ELN runtime package set[1]  during the week of
>  > July 19th, coincident with the Fedora 39 mass-rebuild.
>
>  Will you please, please, please, please import the packages with git history
>  and not via fepdkg srpm -> centpkg import? 
YES!

[Todd Zullinger]

Stephen Gallagher wrote:
>  On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 16. 06. 23 15:39, Stephen Gallagher wrote:
> >> This leads me to my next schedule announcement: we will be performing
> >> the initial CentOS Stream 10 branch creation in Gitlab for all
> >> packages in the Fedora ELN runtime package set[1]  during the week of
> >> July 19th, coincident with the Fedora 39 mass-rebuild.
> >
> > Will you please, please, please, please import the packages with git history
> > and not via fepdkg srpm -> centpkg import?
>  
>  YES! 
Excellent news!  As someone who takes git history (maybe a
little too) seriously, this is quite welcome.  Thanks to
all who helped make the change. :)

-- 
Todd

[Kalev Lember]

On Fri, Jun 16, 2023 at 6:08 PM Todd Zullinger <tmz(a)pobox.com&gt; wrote:

>  Stephen Gallagher wrote:
>  > On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok <mhroncok(a)redhat.com&gt;
>  wrote:
>  >>
>  >> On 16. 06. 23 15:39, Stephen Gallagher wrote:
>  >>> This leads me to my next schedule announcement: we will be performing
>  >>> the initial CentOS Stream 10 branch creation in Gitlab for all
>  >>> packages in the Fedora ELN runtime package set[1]  during the week of
>  >>> July 19th, coincident with the Fedora 39 mass-rebuild.
>  >>
>  >> Will you please, please, please, please import the packages with git
>  history
>  >> and not via fepdkg srpm -> centpkg import?
>  >
>  > YES!
>
>  Excellent news!  As someone who takes git history (maybe a
>  little too) seriously, this is quite welcome.  Thanks to
>  all who helped make the change. :)
>  
YES!

Excellent, that is going to make my life a lot easier trying to keep GNOME
packages in CentOS Stream 10 up to date (can easily cherry-pick commits)!

Thanks!

-- 
Kalev

[Miro Hrončok]

On 16. 06. 23 17:59, Stephen Gallagher wrote:
>  On Fri, Jun 16, 2023 at 11:53 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> >
> > On 16. 06. 23 15:39, Stephen Gallagher wrote:
> >> This leads me to my next schedule announcement: we will be performing
> >> the initial CentOS Stream 10 branch creation in Gitlab for all
> >> packages in the Fedora ELN runtime package set[1]  during the week of
> >> July 19th, coincident with the Fedora 39 mass-rebuild.
> >
> > Will you please, please, please, please import the packages with git history
> > and not via fepdkg srpm -> centpkg import?
>  
>  YES! 
Thank you so much for this!

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Troy Dawson]

On Fri, Jun 16, 2023 at 6:42 AM Stephen Gallagher <sgallagh(a)redhat.com&gt;
wrote:

>  == Open Questions ==
>  1) For the "canary" Fedora ELN rebuild, we have two choices on how to
>  select the git hash to be built for each package in the ELN list:
>
>  Approach 1 (Rawhide-style):
>  1. Clone each package
>  2. Check for the existence of the `eln` branch.
>    a. If the `eln` branch exists, build from the HEAD of that branch
>  into the side-tag.
>    b. Otherwise, build from the HEAD of the `rawhide` branch into the
>  side-tag.
>
>  Approach 2 (Conservative-style):
>  1. Query Koji for the latest-tagged build of each package in Fedora ELN
>  2. Interrogate the build task of that build for the git hash
>  3. Build that git hash into the side-tag.
>
>  Approach 2 is more heavyweight, relying on a lot of Koji queries
>  back-and-forth, whereas Approach 1 will pick up changes that have
>  appeared in Rawhide since the last build (which is more in line with
>  how Fedora's mass-rebuilds work). I'm personally leaning towards
>  Approach 2, but I'm open to good arguments either way.
>  
Sorry for being late with this, but I thought I'd already sent this.

I'd like to go with Approach 2 (Conservative-style)
I've got a python script that grabs the githash for various builds.
Although the initial run takes a couple of hours, after that it's fairly
quick.
It would make it possible to build using the ELN build source githash.

In the long run, I believe it takes less time to get the githash rather
than pulling down the git repos.
It's also something the can be pre-staged, then just run the script right
before you run the mass rebuild, to get the latest.

Troy