[ELN] Creating a process for ELN-specific changes

[Stephen Gallagher]

In nearly all cases, we want the `rawhide` branch in dist-git to
provide the sources used to build packages for ELN. This ensures that
they are kept up to date and minimizes packager effort (since they do
not have to maintain an extra branch).

However, there are some cases where changes desired for ELN (and by
extension, enterprise linux) cannot go into Rawhide. A non-exhaustive
list:
* ELN wants to build without documentation, experimental features or
similar and the maintainer refuses to include `%{rhel}` conditionals
in the spec
* Fedora wants to use the latest version of an upstream, but ELN wants
to stay on LTS releases (e.g. Firefox)
* A package is retired in Rawhide but is needed as a dependency for a
package in ELN (usually as a result of the previous example)
* A package exists solely for use in ELN (e.g. lorax-templates-rhel)

I'm sure there are other good examples, but these are what I could
come up with off the top of my head. If you have other use-cases that
we need to consider, please suggest them.

Maintaining a separate branch for ELN requires us to do the following things:
* Create an `eln` branch for the package
* Exclude the package from the Rawhide auto-rebuild

Maintaining an extra branch is more work for the packager, so it
should be avoided whenever possible. Our goal with ELN is to maximize
the value we provide to enterprise linux while minimizing the
additional load that we put on maintainers.

We may also look into the possibility of extending the auto-rebuilder
to attempt a merge-and-scratch-build from Rawhide to the ELN branch,
to reduce maintainer effort if they opt to maintain their package in
the ELN branch manually.


This is tracked in the ELN SIG as https://github.com/fedora-eln/eln/issues/56

[Vít Ondruch]

Dne 14. 05. 21 v 16:29 Stephen Gallagher napsal(a):
>  In nearly all cases, we want the `rawhide` branch in dist-git to
>  provide the sources used to build packages for ELN. This ensures that
>  they are kept up to date and minimizes packager effort (since they do
>  not have to maintain an extra branch).
>
>  However, there are some cases where changes desired for ELN (and by
>  extension, enterprise linux) cannot go into Rawhide. A non-exhaustive
>  list:
>  * ELN wants to build without documentation, experimental features or
>  similar and the maintainer refuses to include `%{rhel}` conditionals
>  in the spec
>  * Fedora wants to use the latest version of an upstream, but ELN wants
>  to stay on LTS releases (e.g. Firefox)
>  * A package is retired in Rawhide but is needed as a dependency for a
>  package in ELN (usually as a result of the previous example)
>  * A package exists solely for use in ELN (e.g. lorax-templates-rhel)
>
>  I'm sure there are other good examples, but these are what I could
>  come up with off the top of my head. If you have other use-cases that
>  we need to consider, please suggest them.
>
>  Maintaining a separate branch for ELN requires us to do the following things:
>  * Create an `eln` branch for the package
>  * Exclude the package from the Rawhide auto-rebuild 

This is not necessary as long as `git pull --rebase` works.


>  Maintaining an extra branch is more work for the packager, so it
>  should be avoided whenever possible. Our goal with ELN is to maximize
>  the value we provide to enterprise linux while minimizing the
>  additional load that we put on maintainers. 

Just FTR, you underestimate how much work this is going to save. Really. 
I have just merged this PR [1] into ELN. This removes the build 
dependency on texlive and while on it, it also removes dependency on 
rubygem-stringex. You can check yourselves (and I suggest to take a look 
at internal instance of RHEL9 content resolver) how big the dependency 
chain is I assume I'll be able to remove 20+ packages and therefore the 
Fedora maintainers won't be bothered. It will hopefully remove (at least 
in my domain) the build order concerns.

So far, I have modified 4 packages in RHEL9 and therefore was able to 
open requests for removal of ~25 packages. I call it good deal.


>  We may also look into the possibility of extending the auto-rebuilder
>  to attempt a merge-and-scratch-build from Rawhide to the ELN branch,
>  to reduce maintainer effort if they opt to maintain their package in
>  the ELN branch manually.
>
>
>  This is tracked in the ELN SIG as https://github.com/fedora-eln/eln/issues/56 

I am truly happy for this initiative.


Vít


[1] 
https://gitlab.com/redhat/centos-stream/rpms/rubygem-kramdown/-/merge_req...

[Stephen Gallagher]

On Fri, May 14, 2021 at 10:53 AM Vít Ondruch <vondruch(a)redhat.com&gt; wrote:
...
>  > Maintaining a separate branch for ELN requires us to do the
> following things:
>  > * Create an `eln` branch for the package
>  > * Exclude the package from the Rawhide auto-rebuild
>
>
>  This is not necessary as long as `git pull --rebase` works.
>  
I'm not sure what you mean here. What I was saying is that we need to
make sure that the auto-rebuild doesn't attempt to build the Rawhide
content in the ELN buildroot for this package.

>  >
>  > Maintaining an extra branch is more work for the packager, so it
>  > should be avoided whenever possible. Our goal with ELN is to maximize
>  > the value we provide to enterprise linux while minimizing the
>  > additional load that we put on maintainers.
>
>
>  Just FTR, you underestimate how much work this is going to save. Really.
>  I have just merged this PR [1] into ELN. This removes the build
>  dependency on texlive and while on it, it also removes dependency on
>  rubygem-stringex. You can check yourselves (and I suggest to take a look
>  at internal instance of RHEL9 content resolver) how big the dependency
>  chain is I assume I'll be able to remove 20+ packages and therefore the
>  Fedora maintainers won't be bothered. It will hopefully remove (at least
>  in my domain) the build order concerns.
>
>  So far, I have modified 4 packages in RHEL9 and therefore was able to
>  open requests for removal of ~25 packages. I call it good deal. 
Oh, I absolutely understand that this will lead to dependency
trimming. However, such things are *also* possible via
conditionalizing the Rawhide specfile (which remains the recommended
approach, because it means you don't have to maintain a separate
branch).

>  >
>  > We may also look into the possibility of extending the auto-rebuilder
>  > to attempt a merge-and-scratch-build from Rawhide to the ELN branch,
>  > to reduce maintainer effort if they opt to maintain their package in
>  > the ELN branch manually.
>  >
>  >
>  > This is tracked in the ELN SIG as https://github.com/fedora-eln/eln/issues/56
>
>
>  I am truly happy for this initiative.
>

[Miro Hrončok]

On 14. 05. 21 16:29, Stephen Gallagher wrote:
>  In nearly all cases, we want the `rawhide` branch in dist-git to
>  provide the sources used to build packages for ELN. This ensures that
>  they are kept up to date and minimizes packager effort (since they do
>  not have to maintain an extra branch).
>  
>  However, there are some cases where changes desired for ELN (and by
>  extension, enterprise linux) cannot go into Rawhide. A non-exhaustive
>  list:
>  * ELN wants to build without documentation, experimental features or
>  similar and the maintainer refuses to include `%{rhel}` conditionals
>  in the spec
>  * Fedora wants to use the latest version of an upstream, but ELN wants
>  to stay on LTS releases (e.g. Firefox)
>  * A package is retired in Rawhide but is needed as a dependency for a
>  package in ELN (usually as a result of the previous example)
>  * A package exists solely for use in ELN (e.g. lorax-templates-rhel)
>  
>  I'm sure there are other good examples, but these are what I could
>  come up with off the top of my head. If you have other use-cases that
>  we need to consider, please suggest them.
>  
>  Maintaining a separate branch for ELN requires us to do the following things:
>  * Create an `eln` branch for the package
>  * Exclude the package from the Rawhide auto-rebuild
>  
>  Maintaining an extra branch is more work for the packager, so it
>  should be avoided whenever possible. Our goal with ELN is to maximize
>  the value we provide to enterprise linux while minimizing the
>  additional load that we put on maintainers.
>  
>  We may also look into the possibility of extending the auto-rebuilder
>  to attempt a merge-and-scratch-build from Rawhide to the ELN branch,
>  to reduce maintainer effort if they opt to maintain their package in
>  the ELN branch manually.
>  
>  
>  This is tracked in the ELN SIG as https://github.com/fedora-eln/eln/issues/56 
First of all: Thanks. This is what many of us wanted from the beginning 
of ELN and this will allow us to crop many of our unwanted dependencies 
for RHEL 10+, already in ELN.

An automation that cherry-picks (rather than merges) Rawhide changes 
into ELN would be even more helpful. In Python Maint, we have some 
semi-automatic tools ready; let me know in case you want to explore them.


I however do have several concerns to discuss:


1) During the RHEL 9 development cycle, the process was more or less:

      ELN -> Fedora 34 -> CentOS Stream 9

Do we assume similar concept for RHEL 10? I.e.:

      ELN -> Fedora 40-ish -> CentOS Stream 10

If so, every change we make in ELN will be de-facto overridden by the 
sync from Fedora branched and hence making changes in the ELN branch 
will not help our RHEL-related work much. Or am I missing something?


2) Who opts in for the eln branch and who is responsible for keeping it 
up to date? For cases where the Fedora and RHEL maintainers are the same 
people, it is obviously them. But what about other cases? E.g.:

  1. Fedora maintainer of package foo refuses RHEL-only changes.
  2. RHEL maintainer of foo requests an eln branch.
  3. Fedora maintainer frequently updates foo in Rawhide.
  4. RHEL maintainer does not care about foo in Fedora for 2 years.
  5. The package in ELN is much older than the Rawhide one.

I think this could be solved by policy. Something like "If the eln 
branch is neglected (needs to be well defined), the package switches 
back to the rawhide branch."


3) Similarly, assume foo has opted in for an eln branch. The Fedora 
packager maintainer walks away and foo is adapted by another maintainer 
who wants to maintain %if-spaghetti rather than 2 distinct branches. How 
do they opt out? Do we "archive" the eln branch until it is requested 
again? Or do we merge rawhide and eln branches, sot hey have the same 
HEAD and than turn eln into a symbolic reference?

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Matthew Miller]

On Fri, May 14, 2021 at 10:29:11AM -0400, Stephen Gallagher wrote:
>  * Fedora wants to use the latest version of an upstream, but ELN
> wants
>  to stay on LTS releases (e.g. Firefox) 
Can we provide these things as modules and make them available in both?
Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
Linux wanting the option and if the ELN folks are maintaining a package
_anyway_....

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Miro Hrončok]

On 14. 05. 21 16:58, Stephen Gallagher wrote:
>  Oh, I absolutely understand that this will lead to dependency
>  trimming. However, such things are*also*  possible via
>  conditionalizing the Rawhide specfile (which remains the recommended
>  approach, because it means you don't have to maintain a separate
>  branch). 
To be fair, I don't think the reasoning here for why this remains the 
recommended approach is sound. Compere your statement with:

"""Conditionalizing the Rawhide specfile can be used for dependency 
trimming. However, such things are *also*  possible via maintaining a 
separate branch (which remains the recommended approach, because it 
means you don't have conditionale the Rawhide specfile)."""

Technically, when presented with two mutually exclusive options A and B, 
saying "A is a better option, because it avoids option B" does not feel 
like a fair argument.



Disclaimer: I am not writing this email to discuss once again whether 
%ifs or branches are better (we disagree on that one and we both know 
it) nor to discuss which option should be the recommended one (I think 
we cannot have that discussion before we see eln branches working). I 
just wanted to point out the problem in the statement.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Stephen Gallagher]

On Fri, May 14, 2021 at 11:05 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
...
>  First of all: Thanks. This is what many of us wanted from the
> beginning
>  of ELN and this will allow us to crop many of our unwanted dependencies
>  for RHEL 10+, already in ELN.
>
>  An automation that cherry-picks (rather than merges) Rawhide changes
>  into ELN would be even more helpful. In Python Maint, we have some
>  semi-automatic tools ready; let me know in case you want to explore them. 
If you've already got tools that can help with this, I'm all ears!

>  I however do have several concerns to discuss:
>
>
>  1) During the RHEL 9 development cycle, the process was more or less:
>
>        ELN -> Fedora 34 -> CentOS Stream 9
>
>  Do we assume similar concept for RHEL 10? I.e.:
>
>        ELN -> Fedora 40-ish -> CentOS Stream 10
>
>  If so, every change we make in ELN will be de-facto overridden by the
>  sync from Fedora branched and hence making changes in the ELN branch
>  will not help our RHEL-related work much. Or am I missing something? 
Hmm, that is indeed an interesting question. I don't want to give an
off-the-cuff answer (it deserves more thought), but we *will* need to
address this. The time period in question (when Rawhide and the Fedora
we intend to branch from has diverged) is complicated. This time
around, we let ELN stay with Rawhide and used F34 as source for the
sync to EL9 until F34 Final Freeze, but indeed if some packages are
going to be tracked independently via an ELN branch, that gets...
trickier.

Of course, this time around we were also rushing to get the
infrastructure in place for CentOS Stream 9, which will already be
available for EL 10... so maybe the answer here is to just go directly
from ELN into CentOS Stream 10 at the Fedora 40 Branch point? It means
a bit more manual syncing from Fedora during that time period, but
that might not be too painful.

We have a couple years to think this through; thanks for bringing it up.

>  2) Who opts in for the eln branch and who is responsible for keeping it
>  up to date? For cases where the Fedora and RHEL maintainers are the same
>  people, it is obviously them. But what about other cases? E.g.:
>
>    1. Fedora maintainer of package foo refuses RHEL-only changes.
>    2. RHEL maintainer of foo requests an eln branch.
>    3. Fedora maintainer frequently updates foo in Rawhide.
>    4. RHEL maintainer does not care about foo in Fedora for 2 years.
>    5. The package in ELN is much older than the Rawhide one.
>
>  I think this could be solved by policy. Something like "If the eln
>  branch is neglected (needs to be well defined), the package switches
>  back to the rawhide branch." 
I think we can make the policy super-simple: every package on ELN gets
forcibly re-synced to Rawhide on the day we fork to CentOS Stream.
Each new major release reset, we require them to request to diverge
the sync again.

>  3) Similarly, assume foo has opted in for an eln branch. The Fedora
>  packager maintainer walks away and foo is adapted by another maintainer
>  who wants to maintain %if-spaghetti rather than 2 distinct branches. How
>  do they opt out? Do we "archive" the eln branch until it is requested
>  again? Or do we merge rawhide and eln branches, sot hey have the same
>  HEAD and than turn eln into a symbolic reference?
>  
I think archiving the ELN branch is probably the simplest approach
here. All we need to do is flip the auto-rebuild configuration back
over to stop excluding the package and the next Rawhide build will
trigger an ELN build again.

[Stephen Gallagher]

On Fri, May 14, 2021 at 11:16 AM Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Fri, May 14, 2021 at 10:29:11AM -0400, Stephen Gallagher wrote:
>  > * Fedora wants to use the latest version of an upstream, but ELN wants
>  > to stay on LTS releases (e.g. Firefox)
>
>  Can we provide these things as modules and make them available in both?
>  Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
>  Linux wanting the option and if the ELN folks are maintaining a package
>  _anyway_....
>  
That's going to be up to the individual maintainers; I don't want to
sidetrack the ELN discussion too much.

[Stephen Gallagher]

On Fri, May 14, 2021 at 11:18 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 14. 05. 21 16:58, Stephen Gallagher wrote:
>  > Oh, I absolutely understand that this will lead to dependency
>  > trimming. However, such things are*also*  possible via
>  > conditionalizing the Rawhide specfile (which remains the recommended
>  > approach, because it means you don't have to maintain a separate
>  > branch).
>
>  To be fair, I don't think the reasoning here for why this remains the
>  recommended approach is sound. Compere your statement with:
>
>  """Conditionalizing the Rawhide specfile can be used for dependency
>  trimming. However, such things are *also*  possible via maintaining a
>  separate branch (which remains the recommended approach, because it
>  means you don't have conditionale the Rawhide specfile)."""
>
>  Technically, when presented with two mutually exclusive options A and B,
>  saying "A is a better option, because it avoids option B" does not feel
>  like a fair argument. 
Acknowledged; I was unclear on my statement. I said "you don't have to
maintain a separate branch" as shorthand for saying "you don't have to
remember to keep another branch up to date, perform separate builds on
it, etc.". That's still the recommended approach from my perspective,
because it means less risk of bitrot which will fall on the SIG to
repair.

[Matthew Miller]

On Fri, May 14, 2021 at 11:21:26AM -0400, Stephen Gallagher wrote:
>  > Can we provide these things as modules and make them available
> in both?
>  > Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
>  > Linux wanting the option and if the ELN folks are maintaining a package
>  > _anyway_....
>  That's going to be up to the individual maintainers; I don't want to
>  sidetrack the ELN discussion too much. 
OK, so let me generalize: isn't "make it a module" the general answer when
different versions of the package are desired?


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Stephen Gallagher]

On Fri, May 14, 2021 at 11:25 AM Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Fri, May 14, 2021 at 11:21:26AM -0400, Stephen Gallagher wrote:
>  > > Can we provide these things as modules and make them available in both?
>  > > Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
>  > > Linux wanting the option and if the ELN folks are maintaining a package
>  > > _anyway_....
>  > That's going to be up to the individual maintainers; I don't want to
>  > sidetrack the ELN discussion too much.
>
>  OK, so let me generalize: isn't "make it a module" the general answer when
>  different versions of the package are desired? 
ELN's purpose is to be a staging ground for the next major release of
enterprise linux. If this is a package that is intended to be in the
non-modular set for EL N+1, then it needs to be non-modular in ELN (so
we can calculate the dependencies properly). If it is intended to be a
module in EL N+1, then ideally it should be a module in ELN (if we can
figure out how to get that to work properly; there are technical
issues there that make modules built for Fedora not directly
importable to CentOS Stream/RHEL).

[Stephen John Smoogen]

On Fri, 14 May 2021 at 11:25, Matthew Miller <mattdm(a)fedoraproject.org&gt;
wrote:

>  On Fri, May 14, 2021 at 11:21:26AM -0400, Stephen Gallagher wrote:
>  > > Can we provide these things as modules and make them available in both?
>  > > Firefox LTS seems like an ideal candidate, and I can see someone on
>  Fedora
>  > > Linux wanting the option and if the ELN folks are maintaining a package
>  > > _anyway_....
>  > That's going to be up to the individual maintainers; I don't want to
>  > sidetrack the ELN discussion too much.
>
>  OK, so let me generalize: isn't "make it a module" the general answer when
>  different versions of the package are desired?
>
>
>  
I don't know. The maintainer has to want to make it a module and most
maintainers are still waiting for time, energy and the equivalent of 'hand
over hand' training materials to know how to make them without breaking
their existing workflows (or to learn new ones). [The fact that from many
conversations, a number of core maintainers do not even use mock to build
their packages locally says that we are needing to start training at a much
lower level than 'here is how to make a module'.]




-- 
Stephen J Smoogen.
I've seen things you people wouldn't believe. Flame wars in
sci.astro.orion. I have seen SPAM filters overload because of Godwin's Law.
All those moments will be lost in time... like posts on  BBS... time to
reboot.

[Neal Gompa]

On Fri, May 14, 2021 at 11:14 AM Matthew Miller
<mattdm(a)fedoraproject.org&gt; wrote:
>  On Fri, May 14, 2021 at 10:29:11AM -0400, Stephen Gallagher wrote:
>  > * Fedora wants to use the latest version of an upstream, but ELN wants
>  > to stay on LTS releases (e.g. Firefox)
>
>  Can we provide these things as modules and make them available in both?
>  Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
>  Linux wanting the option and if the ELN folks are maintaining a package
>  _anyway_....
>  
Firefox ESR can be packaged separately anyway as a "firefox-esr" package...



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Stephen Gallagher]

On Fri, May 14, 2021 at 12:01 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Fri, May 14, 2021 at 11:14 AM Matthew Miller
>  <mattdm(a)fedoraproject.org&gt; wrote:
>  >
>  > On Fri, May 14, 2021 at 10:29:11AM -0400, Stephen Gallagher wrote:
>  > > * Fedora wants to use the latest version of an upstream, but ELN wants
>  > > to stay on LTS releases (e.g. Firefox)
>  >
>  > Can we provide these things as modules and make them available in both?
>  > Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
>  > Linux wanting the option and if the ELN folks are maintaining a package
>  > _anyway_....
>  >
>
>  Firefox ESR can be packaged separately anyway as a "firefox-esr" package...
>  
Please don't fixate on the use of Firefox as the example.

[Miro Hrončok]

On 14. 05. 21 17:25, Matthew Miller wrote:
>  On Fri, May 14, 2021 at 11:21:26AM -0400, Stephen Gallagher wrote:
> >> Can we provide these things as modules and make them available in both?
> >> Firefox LTS seems like an ideal candidate, and I can see someone on Fedora
> >> Linux wanting the option and if the ELN folks are maintaining a package
> >> _anyway_....
> > That's going to be up to the individual maintainers; I don't want to
> > sidetrack the ELN discussion too much.
>  
>  OK, so let me generalize: isn't "make it a module" the general answer when
>  different versions of the package are desired? 
I wouldn't say so. I'd say "package both versions as separate 
non-modular RPM packages with unique names" is the general answer when 
different versions of the package are desired.

However, the problem here is different. We don't want 2 different 
versions packaged in Rawhide AND 2 different versions packaged in ELN. 
We want to allow package versions in ELN and Rawhide to be different.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Stephen John Smoogen]

On Fri, 14 May 2021 at 12:28, Stephen Gallagher <sgallagh(a)redhat.com&gt; wrote:

>  On Fri, May 14, 2021 at 12:01 PM Neal Gompa
> <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Fri, May 14, 2021 at 11:14 AM Matthew Miller
>  > <mattdm(a)fedoraproject.org&gt; wrote:
>  > >
>  > > On Fri, May 14, 2021 at 10:29:11AM -0400, Stephen Gallagher wrote:
>  > > > * Fedora wants to use the latest version of an upstream, but ELN
>  wants
>  > > > to stay on LTS releases (e.g. Firefox)
>  > >
>  > > Can we provide these things as modules and make them available in both?
>  > > Firefox LTS seems like an ideal candidate, and I can see someone on
>  Fedora
>  > > Linux wanting the option and if the ELN folks are maintaining a package
>  > > _anyway_....
>  > >
>  >
>  > Firefox ESR can be packaged separately anyway as a "firefox-esr"
>  package...
>  >
>
>  Please don't fixate on the use of Firefox as the example.
>  
That is like saying 'dont think about polar bears' to most humans. I will
morph his comment

<LTS package> can be packaged separately anyway as <package-lts>.

or to use Matt's version:

<LTS package> could be packaged separately anyway as <package-module>.

The issue with this suggestion is:
== Who is going to do that work? ==




>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct:
>  https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
>  https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it:
>  https://pagure.io/fedora-infrastructure
>  

-- 
Stephen J Smoogen.
I've seen things you people wouldn't believe. Flame wars in
sci.astro.orion. I have seen SPAM filters overload because of Godwin's Law.
All those moments will be lost in time... like posts on  BBS... time to
reboot.

[Miro Hrončok]

On 14. 05. 21 17:20, Stephen Gallagher wrote:
>  On Fri, May 14, 2021 at 11:05 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
>  ...
> > First of all: Thanks. This is what many of us wanted from the beginning
> > of ELN and this will allow us to crop many of our unwanted dependencies
> > for RHEL 10+, already in ELN.
> >
> > An automation that cherry-picks (rather than merges) Rawhide changes
> > into ELN would be even more helpful. In Python Maint, we have some
> > semi-automatic tools ready; let me know in case you want to explore them.
>  
>  If you've already got tools that can help with this, I'm all ears! 
We use this to cherry pick commits or entire pull requests from 
arbitrary components on src.fp.o to other branches/components/dist-gits:

   https://github.com/fedora-python/ferrypick/blob/master/ferrypick.py

This (highly experimental) script can be used to create various backport 
branches needed to create PRs (it is a wrapper around the script above):

   https://github.com/fedora-python/branchsync

This merge driver can possibly help cherry-pick/merge commits that would 
otherwise fail due to to %chaneglog problems:

https://github.com/encukou/rpm-spec-merge-driver


> > I however do have several concerns to discuss:
> >
> >
> > 1) During the RHEL 9 development cycle, the process was more or less:
> >
> >        ELN -> Fedora 34 -> CentOS Stream 9
> >
> > Do we assume similar concept for RHEL 10? I.e.:
> >
> >        ELN -> Fedora 40-ish -> CentOS Stream 10
> >
> > If so, every change we make in ELN will be de-facto overridden by the
> > sync from Fedora branched and hence making changes in the ELN branch
> > will not help our RHEL-related work much. Or am I missing something?
>  
>  Hmm, that is indeed an interesting question. I don't want to give an
>  off-the-cuff answer (it deserves more thought), but we *will* need to
>  address this. The time period in question (when Rawhide and the Fedora
>  we intend to branch from has diverged) is complicated. This time
>  around, we let ELN stay with Rawhide and used F34 as source for the
>  sync to EL9 until F34 Final Freeze, but indeed if some packages are
>  going to be tracked independently via an ELN branch, that gets...
>  trickier.
>  
>  Of course, this time around we were also rushing to get the
>  infrastructure in place for CentOS Stream 9, which will already be
>  available for EL 10... so maybe the answer here is to just go directly
>  from ELN into CentOS Stream 10 at the Fedora 40 Branch point? It means
>  a bit more manual syncing from Fedora during that time period, but
>  that might not be too painful. 
If the manual sync is easy (no bugzillas flags required), I would say 
that is the way to go. It eliminates one extra "why is the workflow 
changing every week" moment. It also no longer encourages the RHEL 
maintains to push their changes to Fedora branched in the same time 
windows when we try to stabilize it.

>  We have a couple years to think this through; thanks for bringing it
> up. 
I agree that we don't need to to necessarily solve this right now.

> > 2) Who opts in for the eln branch and who is responsible for
> keeping it
> > up to date? For cases where the Fedora and RHEL maintainers are the same
> > people, it is obviously them. But what about other cases? E.g.:
> >
> >    1. Fedora maintainer of package foo refuses RHEL-only changes.
> >    2. RHEL maintainer of foo requests an eln branch.
> >    3. Fedora maintainer frequently updates foo in Rawhide.
> >    4. RHEL maintainer does not care about foo in Fedora for 2 years.
> >    5. The package in ELN is much older than the Rawhide one.
> >
> > I think this could be solved by policy. Something like "If the eln
> > branch is neglected (needs to be well defined), the package switches
> > back to the rawhide branch."
>  
>  I think we can make the policy super-simple: every package on ELN gets
>  forcibly re-synced to Rawhide on the day we fork to CentOS Stream.
>  Each new major release reset, we require them to request to diverge
>  the sync again. 
That sounds really harsh towards the maintainers who actually do keep 
the eln branch up to date all the time. We would wipe their changes 
every 3 years, and they would need to reintroduce them again.

> > 3) Similarly, assume foo has opted in for an eln branch. The
> Fedora
> > packager maintainer walks away and foo is adapted by another maintainer
> > who wants to maintain %if-spaghetti rather than 2 distinct branches. How
> > do they opt out? Do we "archive" the eln branch until it is requested
> > again? Or do we merge rawhide and eln branches, sot hey have the same
> > HEAD and than turn eln into a symbolic reference?
> >
>  
>  I think archiving the ELN branch is probably the simplest approach
>  here. All we need to do is flip the auto-rebuild configuration back
>  over to stop excluding the package and the next Rawhide build will
>  trigger an ELN build again. 
However, when looking at the package source, it should be really obvious 
that the eln branch is no longer used for ELN. Otherwise it will be very 
confusing for people who want to contribute.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 14. 05. 21 16:29, Stephen Gallagher wrote:
>  * Fedora wants to use the latest version of an upstream, but ELN
> wants
>  to stay on LTS releases (e.g. Firefox) 
About this use case:

Is ELN still consumed as an "add-on repo" for Rawhide, or that is no 
longer true? Becasue if it is still the case, this might not work properly.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Stephen Gallagher]

On Fri, May 14, 2021 at 12:54 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
...
>  > Of course, this time around we were also rushing to get the
>  > infrastructure in place for CentOS Stream 9, which will already be
>  > available for EL 10... so maybe the answer here is to just go directly
>  > from ELN into CentOS Stream 10 at the Fedora 40 Branch point? It means
>  > a bit more manual syncing from Fedora during that time period, but
>  > that might not be too painful.
>
>  If the manual sync is easy (no bugzillas flags required), I would say
>  that is the way to go. It eliminates one extra "why is the workflow
>  changing every week" moment. It also no longer encourages the RHEL
>  maintains to push their changes to Fedora branched in the same time
>  windows when we try to stabilize it.
>  
Yeah, I was becoming more confident in that idea the longer I typed. I
think that is probably what we will end up doing.

...
>  > I think we can make the policy super-simple: every package on
> ELN gets
>  > forcibly re-synced to Rawhide on the day we fork to CentOS Stream.
>  > Each new major release reset, we require them to request to diverge
>  > the sync again.
>
>  That sounds really harsh towards the maintainers who actually do keep
>  the eln branch up to date all the time. We would wipe their changes
>  every 3 years, and they would need to reintroduce them again. 
Oh, I didn't mean we would clobber the `eln` branch, we'd just flip
the switch to re-enable Rawhide builds (with a loud announcement).
We'd accept requests at that time to *not* flip the switch for their
project, or they could do so at any time afterwards. This would leave
the `eln` branch exactly as it was; and if they request not to sync,
they can resume from there.

I view this as an easy way to do two things at once:
1) See who is still actively maintaining their project for ELN and
reset it if they aren't.
2) Make it a no-op for anyone who wants to jump back on the Rawhide
train (even if only for some number of months until they freeze it
again).

>  >> 3) Similarly, assume foo has opted in for an eln branch. The
> Fedora
>  >> packager maintainer walks away and foo is adapted by another maintainer
>  >> who wants to maintain %if-spaghetti rather than 2 distinct branches. How
>  >> do they opt out? Do we "archive" the eln branch until it is requested
>  >> again? Or do we merge rawhide and eln branches, sot hey have the same
>  >> HEAD and than turn eln into a symbolic reference?
>  >>
>  >
>  > I think archiving the ELN branch is probably the simplest approach
>  > here. All we need to do is flip the auto-rebuild configuration back
>  > over to stop excluding the package and the next Rawhide build will
>  > trigger an ELN build again.
>
>  However, when looking at the package source, it should be really obvious
>  that the eln branch is no longer used for ELN. Otherwise it will be very
>  confusing for people who want to contribute. 
True... maybe we dead.package the `eln` branch if they don't request
to retain it ahead of time. They can always do a git revert to bring
it back to life later.

[Stephen Gallagher]

On Fri, May 14, 2021 at 1:01 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 14. 05. 21 16:29, Stephen Gallagher wrote:
>  > * Fedora wants to use the latest version of an upstream, but ELN wants
>  > to stay on LTS releases (e.g. Firefox)
>
>  About this use case:
>
>  Is ELN still consumed as an "add-on repo" for Rawhide, or that is no
>  longer true? Becasue if it is still the case, this might not work properly.
>  
That's a really good question that I don't have a solid answer to at
the moment. I'm not sure it would *ultimately* be a major issue, as
it's less likely for ELN to have *newer* packages than Rawhide does
(and if they're older, DNF will basically ignore them unless
specifically requested).

I'd like us to get to a point where ELN is consistently installable,
even if that's not an "official" goal.

[Matthew Miller]

On Fri, May 14, 2021 at 06:38:33PM +0200, Miro Hrončok wrote:
>  I wouldn't say so. I'd say "package both versions as
> separate
>  non-modular RPM packages with unique names" is the general answer
>  when different versions of the package are desired.
>  
>  However, the problem here is different. We don't want 2 different
>  versions packaged in Rawhide AND 2 different versions packaged in
>  ELN. We want to allow package versions in ELN and Rawhide to be
>  different. 
This is exactly a case that modularity is supposed to handle. Two different
versions packaged only one time each, built for both targets automatically,
with one the default on target A and the other the default on target B,
output, but with either version selectable in either target.


-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Zbigniew Jędrzejewski-Szmek]

On Fri, May 14, 2021 at 02:57:14PM -0400, Matthew Miller wrote:
>  On Fri, May 14, 2021 at 06:38:33PM +0200, Miro Hrončok wrote:
>  > I wouldn't say so. I'd say "package both versions as separate
>  > non-modular RPM packages with unique names" is the general answer
>  > when different versions of the package are desired.
>  > 
>  > However, the problem here is different. We don't want 2 different
>  > versions packaged in Rawhide AND 2 different versions packaged in
>  > ELN. We want to allow package versions in ELN and Rawhide to be
>  > different.
>  
>  This is exactly a case that modularity is supposed to handle. Two different
>  versions packaged only one time each, built for both targets automatically,
>  with one the default on target A and the other the default on target B,
>  output, but with either version selectable in either target. 
I think this has been discussed at some point ;----]
but since the topic has been raised:

no, doing modules here is very much the wrong answer. For three reasons:
1) it's more work for the maintainer,
2) it's much harder to consume for other packagers,
3) users are more happy with separate packages.

We don't have firefox-esr in the distro, but we actually have firefox-x11,
and to a large extent doing a different version is similar to doing a build
with different flags:
1) either
      a) the maintainer adds a separate section in %build and then
      %package+%files sections (same version, different build flags)
   or
      b) the maintainer requests a compat package (no review required!), and
      just tweaks the .spec as necessary (different versions).
   Compared with that, a module would require ... a module, and a new workflow
   for the packager and users *and* all the work with a non-modular package,
   since the module is just a wrapper around the actual spec file which
   needs to be maintained in two versions.
2) Other packagers can do Requires:firefox-x11 and be done. With modules,
   they need to modularize, and are generally into a world of pain, e.g. the
   lifetime rules for the module don't follow the release cadence.
3) Users do 'dnf install firefox-x11' and can have both versions present
   (and even running!) at the same time. With modules, you get a learn a new
   workflow but no parallel-installability.

If we want the "main" version to be different, then that's also easy
with packages: a few lines of '%if <whatever> Provides: ...' and one
of the subpackages becomes the default.

tl;dr: let's solve ELN-branching problems, but let's not use that to
reintroduce modules.

Zbyszek

[Neal Gompa]

On Sun, May 16, 2021 at 5:01 AM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Fri, May 14, 2021 at 02:57:14PM -0400, Matthew Miller wrote:
>  > On Fri, May 14, 2021 at 06:38:33PM +0200, Miro Hrončok wrote:
>  > > I wouldn't say so. I'd say "package both versions as separate
>  > > non-modular RPM packages with unique names" is the general answer
>  > > when different versions of the package are desired.
>  > >
>  > > However, the problem here is different. We don't want 2 different
>  > > versions packaged in Rawhide AND 2 different versions packaged in
>  > > ELN. We want to allow package versions in ELN and Rawhide to be
>  > > different.
>  >
>  > This is exactly a case that modularity is supposed to handle. Two different
>  > versions packaged only one time each, built for both targets automatically,
>  > with one the default on target A and the other the default on target B,
>  > output, but with either version selectable in either target.
>
>  I think this has been discussed at some point ;----]
>  but since the topic has been raised:
>
>  no, doing modules here is very much the wrong answer. For three reasons:
>  1) it's more work for the maintainer,
>  2) it's much harder to consume for other packagers,
>  3) users are more happy with separate packages.
>
>  We don't have firefox-esr in the distro, but we actually have firefox-x11,
>  and to a large extent doing a different version is similar to doing a build
>  with different flags:
>  1) either
>        a) the maintainer adds a separate section in %build and then
>        %package+%files sections (same version, different build flags)
>     or
>        b) the maintainer requests a compat package (no review required!), and
>        just tweaks the .spec as necessary (different versions).
>     Compared with that, a module would require ... a module, and a new workflow
>     for the packager and users *and* all the work with a non-modular package,
>     since the module is just a wrapper around the actual spec file which
>     needs to be maintained in two versions.
>  2) Other packagers can do Requires:firefox-x11 and be done. With modules,
>     they need to modularize, and are generally into a world of pain, e.g. the
>     lifetime rules for the module don't follow the release cadence.
>  3) Users do 'dnf install firefox-x11' and can have both versions present
>     (and even running!) at the same time. With modules, you get a learn a new
>     workflow but no parallel-installability.
>
>  If we want the "main" version to be different, then that's also easy
>  with packages: a few lines of '%if <whatever> Provides: ...' and one
>  of the subpackages becomes the default.
>
>  tl;dr: let's solve ELN-branching problems, but let's not use that to
>  reintroduce modules.
>  
I don't think that modules are necessarily universally terrible for
this solution. But that said, we probably don't want Firefox ESR in
ELN anyway, since we'd want to be tracking the latest stuff to make
sure everything is sane in Fedora ELN, and *then* transition the
Firefox package to Firefox ESR post-branching for a CentOS Stream
release.

It's more important that we track the latest content from Rawhide and
have it build in an EL-ish context than to implement all the different
policy freezes that RHEL does in Fedora, since we want to be fluid
enough to do differently in a future RHEL release.




--
真実はいつも一つ！/ Always, there's only one truth!

[Vít Ondruch]

Dne 14. 05. 21 v 16:58 Stephen Gallagher napsal(a):
>  On Fri, May 14, 2021 at 10:53 AM Vít Ondruch
> <vondruch(a)redhat.com&gt; wrote:
>  ...
> >> Maintaining a separate branch for ELN requires us to do the following things:
> >> * Create an `eln` branch for the package
> >> * Exclude the package from the Rawhide auto-rebuild
> >
> > This is not necessary as long as `git pull --rebase` works.
> >
>  I'm not sure what you mean here. What I was saying is that we need to
>  make sure that the auto-rebuild doesn't attempt to build the Rawhide
>  content in the ELN buildroot for this package. 

I think you are considering something like:

~~~

$ git checkout eln

$ git merge rawhide

~~~


I am talking about


~~~

$ git checkout eln

$ git pull --rebase rawhide

~~~


IOW if there is some commit on top of Rawhide, it will be preserved as 
long as it cleanly merges, which will be successful 99 % of time IMO.



Vít

[Vít Ondruch]

Dne 17. 05. 21 v 10:39 Vít Ondruch napsal(a):
>  Dne 14. 05. 21 v 16:58 Stephen Gallagher napsal(a):
> > On Fri, May 14, 2021 at 10:53 AM Vít Ondruch <vondruch(a)redhat.com&gt; 
> > wrote:
> > ...
> >>> Maintaining a separate branch for ELN requires us to do the 
> >>> following things:
> >>> * Create an `eln` branch for the package
> >>> * Exclude the package from the Rawhide auto-rebuild
> >>
> >> This is not necessary as long as `git pull --rebase` works.
> >>
> > I'm not sure what you mean here. What I was saying is that we need to
> > make sure that the auto-rebuild doesn't attempt to build the Rawhide
> > content in the ELN buildroot for this package.
>
>
>  I think you are considering something like:
>
>  ~~~
>
>  $ git checkout eln
>
>  $ git merge rawhide
>
>  ~~~
>
>
>  I am talking about
>
>
>  ~~~
>
>  $ git checkout eln
>
>  $ git pull --rebase rawhide
>
>  ~~~
>
>
>  IOW if there is some commit on top of Rawhide, it will be preserved as 
>  long as it cleanly merges, which will be successful 99 % of time IMO. 

Reading the rest of discussion, maybe there should be two strategies for 
ELN branch:

1) the `pull --rebase` I am talking about

2) complete cut-off for cases such as FF, where the ESR should be 
probably maintained independently, with possible cherry-picks from Rawhide.


Vít


>  Vít
>
>

[Vít Ondruch]

Dne 14. 05. 21 v 16:29 Stephen Gallagher napsal(a):
>  In nearly all cases, we want the `rawhide` branch in dist-git to
>  provide the sources used to build packages for ELN. This ensures that
>  they are kept up to date and minimizes packager effort (since they do
>  not have to maintain an extra branch).
>
>  However, there are some cases where changes desired for ELN (and by
>  extension, enterprise linux) cannot go into Rawhide. A non-exhaustive
>  list:
>  * ELN wants to build without documentation, experimental features or
>  similar and the maintainer refuses to include `%{rhel}` conditionals
>  in the spec
>  * Fedora wants to use the latest version of an upstream, but ELN wants
>  to stay on LTS releases (e.g. Firefox) 

We have also GCC/OpenSSL cases, where we would like ELN to be ahead of 
everything. Better to broke ELN at this stage where it already targets 
RHEL10, then try this in RHEL.


Vít


>  * A package is retired in Rawhide but is needed as a dependency for
> a
>  package in ELN (usually as a result of the previous example)
>  * A package exists solely for use in ELN (e.g. lorax-templates-rhel)
>
>  I'm sure there are other good examples, but these are what I could
>  come up with off the top of my head. If you have other use-cases that
>  we need to consider, please suggest them.
>
>  Maintaining a separate branch for ELN requires us to do the following things:
>  * Create an `eln` branch for the package
>  * Exclude the package from the Rawhide auto-rebuild
>
>  Maintaining an extra branch is more work for the packager, so it
>  should be avoided whenever possible. Our goal with ELN is to maximize
>  the value we provide to enterprise linux while minimizing the
>  additional load that we put on maintainers.
>
>  We may also look into the possibility of extending the auto-rebuilder
>  to attempt a merge-and-scratch-build from Rawhide to the ELN branch,
>  to reduce maintainer effort if they opt to maintain their package in
>  the ELN branch manually.
>
>
>  This is tracked in the ELN SIG as https://github.com/fedora-eln/eln/issues/56
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Zbigniew Jędrzejewski-Szmek]

On Mon, May 17, 2021 at 11:01:39AM +0200, Vít Ondruch wrote:
>  
>  Dne 17. 05. 21 v 10:39 Vít Ondruch napsal(a):
>  >
>  >Dne 14. 05. 21 v 16:58 Stephen Gallagher napsal(a):
>  >>On Fri, May 14, 2021 at 10:53 AM Vít Ondruch
>  >><vondruch(a)redhat.com> wrote:
>  >>...
>  >>>>Maintaining a separate branch for ELN requires us to do the
>  >>>>following things:
>  >>>>* Create an `eln` branch for the package
>  >>>>* Exclude the package from the Rawhide auto-rebuild
>  >>>
>  >>>This is not necessary as long as `git pull --rebase` works.
>  >>>
>  >>I'm not sure what you mean here. What I was saying is that we need to
>  >>make sure that the auto-rebuild doesn't attempt to build the Rawhide
>  >>content in the ELN buildroot for this package.
>  >
>  >
>  >I think you are considering something like:
>  >
>  >~~~
>  >
>  >$ git checkout eln
>  >
>  >$ git merge rawhide
>  >
>  >~~~
>  >
>  >
>  >I am talking about
>  >
>  >
>  >~~~
>  >
>  >$ git checkout eln
>  >
>  >$ git pull --rebase rawhide
>  >
>  >~~~
>  >
>  >
>  >IOW if there is some commit on top of Rawhide, it will be
>  >preserved as long as it cleanly merges, which will be successful
>  >99 % of time IMO.
>  
>  
>  Reading the rest of discussion, maybe there should be two strategies
>  for ELN branch:
>  
>  1) the `pull --rebase` I am talking about
>  
>  2) complete cut-off for cases such as FF, where the ESR should be
>  probably maintained independently, with possible cherry-picks from
>  Rawhide. 
Are rebases (or any other change which does not preserve branch
history) allowed? We don't allow them for normal branches [1],
and I would expect the 'eln' branch to follow the same rule.

[1] https://pagure.io/fesco/issue/2387

Zbyszek

[Vít Ondruch]

Dne 17. 05. 21 v 11:56 Zbigniew Jędrzejewski-Szmek napsal(a):
>  On Mon, May 17, 2021 at 11:01:39AM +0200, Vít Ondruch wrote:
> > Dne 17. 05. 21 v 10:39 Vít Ondruch napsal(a):
> >> Dne 14. 05. 21 v 16:58 Stephen Gallagher napsal(a):
> >>> On Fri, May 14, 2021 at 10:53 AM Vít Ondruch
> >>> <vondruch(a)redhat.com&gt; wrote:
> >>> ...
> >>>>> Maintaining a separate branch for ELN requires us to do the
> >>>>> following things:
> >>>>> * Create an `eln` branch for the package
> >>>>> * Exclude the package from the Rawhide auto-rebuild
> >>>> This is not necessary as long as `git pull --rebase` works.
> >>>>
> >>> I'm not sure what you mean here. What I was saying is that we need to
> >>> make sure that the auto-rebuild doesn't attempt to build the Rawhide
> >>> content in the ELN buildroot for this package.
> >>
> >> I think you are considering something like:
> >>
> >> ~~~
> >>
> >> $ git checkout eln
> >>
> >> $ git merge rawhide
> >>
> >> ~~~
> >>
> >>
> >> I am talking about
> >>
> >>
> >> ~~~
> >>
> >> $ git checkout eln
> >>
> >> $ git pull --rebase rawhide
> >>
> >> ~~~
> >>
> >>
> >> IOW if there is some commit on top of Rawhide, it will be
> >> preserved as long as it cleanly merges, which will be successful
> >> 99 % of time IMO.
> >
> > Reading the rest of discussion, maybe there should be two strategies
> > for ELN branch:
> >
> > 1) the `pull --rebase` I am talking about
> >
> > 2) complete cut-off for cases such as FF, where the ESR should be
> > probably maintained independently, with possible cherry-picks from
> > Rawhide.
>  Are rebases (or any other change which does not preserve branch
>  history) allowed? We don't allow them for normal branches [1],
>  and I would expect the 'eln' branch to follow the same rule. 

Good point, I have not thought about it and you are right.

But I'd be more concerned about user experience pulling the rebased 
branches then about builds. Builds can be tagged after all.


Vít


>  [1] https://pagure.io/fesco/issue/2387
>
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure