----- Original Message -----
From: "Stephen Gallagher" <sgallagh(a)redhat.com>
To: devel-announce(a)lists.fedoraproject.org, "Development discussions related to
Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Tuesday, March 31, 2020 5:31:38 PM
Subject: Fedora 33 System-Wide Change Proposal: ELN Buildroot and Compose V3
I sent out the V2 version of the Change on Friday and then promptly
managed to injure myself and be away from email until today. I've read
through the email threads again this morning and I decided that,
rather than try to address them one by one, I'd try again with a V3
that hopefully answers some of the repeated questions and concerns on
that list.
Please see the newly-updated
https://fedoraproject.org/wiki/Changes/ELN_Buildroot_and_Compose
for more details[1].
[1]
https://fedoraproject.org/w/index.php?title=Changes%2FELN_Buildroot_and_C...
_______________________________________________
devel mailing list -- devel(a)lists.fedoraproject.org
To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Ok I took the time to read all the previous threads and the proposal as well as the
clarifications.
In general I am in favor of the idea, however I concur with my fellow packagers, that the
current proposal is having some serious issues, and I'm very reluctant in agreeing
with the given rationale of various details.
I also feel that I am iterating a lot of the feedback that was given already however I
believe it was neither heard nor taken into account.
So to start:
ELN (Enterprise Linux Next) is going to be that place. What we want
to do is establish a set of RPM conditionals that can be used for the set of packages
that may need to be built differently for a RHEL-like target as compared to a Fedora
one. (For example, Fedora often ships with experimental or less-commonly-used features
enabled for packages that Red Hat would not want to support for Enterprise Linux
customers).
Some packagers are fine with adding conditionals in their SPECs and that is ok. However as
one of the maintainers of the main python interpreter as well as various alternative ones,
and hundreds of libraries, not only throughout Fedora but across the RHEL ecosystem, I can
say that this change vastly complicates my work. As a team we maintain approximately 35+
different branches of various python interpreters and if it was making sense to have
single SPECs by using conditionals, we would pretty much do it without hassle. Different
projects/products have differents needs and different goals (and different branches but
more on that later).
I could just add RHEL and SCL conditionals however not only it would clutter the SPEC file
to insane amounts of unnecessary craft, the only people who would know how to work with
it, would be me and my team. I want to encourage contributions from the community at my
packages, not make them more complex and deter others from contributing.
So in that respect, in Fedora I wouldn't personally add any RHEL (or ELN)
conditionals, as our RHEL work can vary significantly. And it would drive potential
collaborators away if every change visible in Fedora, would also have an effect on a black
box that would be ELN. Why would they even bother if they don't have access to that?
And what's their motivation to even work with those conditionals? Am I supposed, for
example, to reject PR's that could break ELN due to its different compiler flags, but
have no effect on Fedora, from a user and a contributor working to better a package on our
OS?
Also contributors *are* users of Fedora and driving them away, may very well drive a
portion of our userbase away as well.
"""
As a sister effort to this, the OSCI team will also be implementing automated tests that
will run against ELN composes and provide non-blocking feedback to the package maintainers
about potential issues in the Enterprise Linux configuration.
"""
I don't like this part either. Feedback is still feedback blocking or not. Seeing a
big red sign that a change broke ELN is just nagging people to fix issues that might not
even affect them in any significant way. I would like the ELN testing feedback to be
opt-in for packagers who do not maintain something in ELN or RHEL and only the actual RHEL
maintainer to get the notification. If not, that is unnecessary noise.
"""
The reasoning behind this approach is so that we break as little as possible when we
implement this new build and compose. Existing packages that are using conditionals such
as if 0%{?rhel} > 7 will transition over to building "like RHEL"
automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue
to build as normal. There will be some unknown number of packages whose existing
conditionals will need to be updated to account for ELN. At present, we are estimating
that there will be around 200 packages that need such attention. The ELN SIG will be
providing guidance and pull-requests to assist with this.
"""
The number of packages here is very optimistic and frankly quite unreal. Having worked on
mass scale package changes, from Python rebases, to decoupling python2 from RHEL8 I can
assure you this number would much much higher, in my opinion at least. Guidance and pull
requests, while helpful, shouldn't be required for packagers who don't want them,
whatever their reasons might be.
"""
The %{eln} variable will be set primarily to allow for the possibility of needing
conditionals that apply to ELN only and should not carry over to RHEL, but we expect its
use to be exceptionally rare.
"""
I don't understand this part. So the %{eln} macro will be in Fedora and ELN, but not
actually RHEL?
"
Such fixes will be done via a pull request that states a time limit. We want the regular
maintainers to see / comment / commit, but we also don't want things to stall for
months and get forgotten about.
For less trivial fixes, we may provide a pull-request or use other methods to communicate
with the maintainers to determine a path forwards that is acceptable to them.
"""
"""
As long as your package builds in ELN then just maintain your package like normal. If
there is a build failure, the ELN SIG may provide a PR as described above or will discuss
alternative approaches on an individual basis with you. The ELN SIG will not dictate how
you must proceed, but will try to find a solution that you are comfortable with.
"""
It is reasonable to provide a pull request to fix potential issues with ELN. What is not
reasonable, is to impose a time limit and also expect the maintainers to follow up with
that. And what exactly would be the way forward for packagers that do not want those
conditionals in their package, while it also fails to build in ELN, potentially blocking
other packages? Would you merge it as a proven packager, despite possible drawbacks? I can
easily see a game of revertions if that was to happen. So please explain what other
possible approaches there would be in that case, and I hope it is not to actually convince
the packager to accept the conditionals.
"""
This adds no value to the current approach where Red Hat maintainers would manually merge
their changes into the internal build infrastructure. There's no way to automate the
sync from the master branch to the eln branch that wouldn't break and require
maintainer involvement. Attempting to branch only individual packages would introduce
significant complexity in the build process as well, leading to far more opportunity for
bugs. Lastly, even the most diligent of maintainers can forget to sync every change to a
new branch, thus leaving us in a situation where the eln branch has fallen behind and is
no longer providing an accurate view of whether the package is still building or
functioning in that environment.
"""
Automatically synced branches or symbolic references to branches is a great solution to
that, and I really like the idea that Fabio proposed to provide an optional eln branch for
packagers who either do not want to merge the changes, or if the change is too complex to
make sense for Fedora.
"""
Classically, Fedora sees a flurry of activity from Red Hat developers in the run-up to a
new Red Hat Enterprise Linux release. These Red Hat teams frantically attempt to push a
few years' downstream effort up into Fedora before it is branched for the next RHEL
development. It is not uncommon for these teams to disappear downstream again once this
has happened. This harms both Fedora and Red Hat with the lack of upstream involvement.
With ELN, we hope to make Fedora Rawhide a more appealing (and canonical) place for those
developers to work. This should increase the amount of ongoing developer involvement in
Fedora, and also make Fedora a more valuable resource for Red Hat, which can help to
ensure funding and support.
"""
That is true to an extend. I don't see that as a failure with the procedures though,
more like a failure to communicate to those teams/developers, to stress just how important
Fedora is for our downstream work. And honestly I haven't interacted with many
developers who don't think Fedora is important enough, so that point from my
experience is a bit moot, in respect to pushing such an intrusive change. I'm
obviously biased here, but looking, as an example at the state of the Python, Ruby, Perl
ecosystem across Fedora and RHEL, you can see the Fedora work is not only important, but
the main drive for a healthy RHEL ecosystem. And I'm happy to say my management
understands that.
"""
ELN artifacts will be made available for testing and development purposes, but we will not
be shipping any content produced from ELN directly to the general public (such as on the
standard mirror network or via
getfedora.org). This does not necessarily mean that they
will be shipped directly from Koji, merely that they won't come from the standard
locations.
"""
It would be better to provide artifacts, there is no reason to keep this in a black box,
maybe I'd like to spin VM's, maybe containers. Just updating a regular rawhide
installation to eln in order to test a simple fix is not something I would do very
enthusiastically.
Some more thoughts:
Since the current Fedora infrastructure is already stretched thin, especially with
modularity, koschei rebuilds, the CI and so on, has a possible impact on the load been
considered/measured? I know for starters that the tasks will have a lower priority,
however has any assessment been made to get a rough idea on what and if any additional
resources will be required?
I'd also like to point out that all the past system-wide changes, except from maybe
rebases of important packages, have always had detailed descriptions on how they were
going to be implemented and a well laid-out contingency plan. They weren't mere ideas,
and while I understand that not all questions can be answered at this point, starting with
a very high level goal and iterating on it, is not something to be done as a fedora
system-wide change, not at least since some details need to be cemented on the
aforementioned points.
A system-wide change and the fedora devel thread of it, is a place to gather feedback from
the community, incorporate that and move forward from there, not somewhere where blind
trust should be placed on the change owners and accuse people and even elected
representatives of ignoring feedback. It really does seem like projection when placing the
whole conversation into context.
I understand that especially during those trying times, everyone might be a bit on edge
while trying to stay productive and within the deadlines, however it really does seem that
the change owners and the community are talking entirely past each other at this point.
Some time ago, during the discussion of the default modular streams, there was a big
pushback on some aspects of it from the community and the feedback was rejected and/or
ignored in a similar manner that happens throughout those ELN threads. In the end, the
eclipse module did exactly that and it affected a big portion of our userbase:
https://bugzilla.redhat.com/show_bug.cgi?id=1780827
So overall, I'm -1 on the proposal and as things currently stand, I won't be
adding any of those conditionals at the Python SPEC files or any other smaller library
that I maintain. I'll happily reconsider if the feedback is heard and the pain points
have been addressed or are at least planned to be addressed.
P.S. It took me some days to actually draft this mail, as in general those times, emotions
can run wild and I wanted to be objective, if that's even possible. And I understand
that some things might have changed as the thread progressed, so apologies if any of my
points are not valid anymore.
--
Regards,
Charalampos Stratakis
Software Engineer
Python Maintenance Team, Red Hat