[EPEL-devel] [Proposal] Converge EPEL and CBS

[Haïkel]

2015-09-22 21:18 GMT+02:00 Dennis Gilmore <dennis at ausil.us>:
> On Monday, September 21, 2015 08:58:21 PM Haïkel wrote:
>> 2015-09-21 19:46 GMT+02:00 Kevin Fenzi <kevin at scrye.com>:
>> > On Mon, 21 Sep 2015 16:12:07 +0200
>> >
>> > Haïkel <hguemar at fedoraproject.org> wrote:
>> >> Hi,
>> >>
>> >> Since the CentOS acquihire, there was a lot of discussion about
>> >> EPEL's future. Since the FOSDEM meetup between Fedora/CentOS folks,
>> >> there was little progress on that topic
>> >>
>> >> After a discussion with a Smooge, I decided to come with a proposal,
>> >> knowing that
>> >> 1. Fedora wants to keep EPEL within it umbrella
>> >> 2. That CentOS SIGs are in practice rebuilding a lot of EPEL packages
>> >> (or retag them for other SIGs)
>> >> leading to poor maintenance as they don't follow EPEL tickets for all
>> >> their dependencies.
>
> Why is this? would it be enough that the CBS had epel as an external repo that
> can be added to SIG's projects? that way epel and updates can be available to
> build against.
>

Well, that was my first idea, but it ended up in dead-end, both EPEL
and CentOS have their arguments.
This situation is no good for both projects,  I tried to revive the
discussion through a proposal.

The point itself is not the proposal but to solve the problem that
there is no integrated vision between EPEL/CentOS.



>> > Which tickets do you mean here? They are only rebuilding some packages,
>> > but not others or?
>>
>> Any tickets filed against EPEL, for instance, if a bug or CVE is fixed
>> against EPEL package,  CBS rebuilds won't be impacted as there's no
>> way to automate that.
>> Some examples from CentOS Cloud SIG:
>> * RabbitMQ: it's a runtime requirement for OpenStack, we could just
>> reuse EPEL packages but that would mean that Cloud SIG repository
>> wouldn't be self-contained
>> => Nick Coghlan's RepoFunnel would be a solution to mash repositories here.
>> * A hell lot of python build requirements, that have to be rebuilt in
>> CBS, as CBS don't have access to EPEL packages.
> if we build and track in epel and use epel in CBS to build against we should
> be covered.
>
>> For instance, if the EPEL package gets fixed for a CVE, the CBS
>> package may not get fixed (and vice-versa).
>> Moreover, it makes mixing EPEL and CentOS SIGs repositories harder.
>>
>> >> 3. EPEL is not part CentOS plans, and as soon as SIGs will progress,
>> >> *may* turn the former irrelevant
>> >
>> > I suppose, but lots of people use/look to epel for packages, I don't
>> > think that will change to using packages from CentOS sigs overnight.
>>
>> I agree.
>>
>> >> 4. Some EPEL packages are poorly maintained especially on older EL
>> >> releases and/or orphaned
>> >
>> > Sure, just like any large collection of packages.
>>
>> Yes, but all the more a reason to make it easier for CentOS community
>> to participate to this efforts
>
> I am all for making it easier to contribute to EPEL, one thing to consider is
> that to contribute to EPEL you must agree to the FPCA, afaik CentOS does not
> have an equivalent and at the least Legal requires it for Fedora and EPEL
>
>> >> We've reached the point where both EPEL/CBS would greatly benefit to
>> >> join hands.
>> >>
>> >> So I suggest that we consider the following:
>> >> * EPEL will still use Fedora dist-git
>> >> * EPEL builds should be done in CBS to make it easier for SIGs to
>> >> consume it.
>> >
>> > How do EPEL maintainers launch builds in CBS?
>>
>> Through bstrinson centpkg tool as for the tooling aspect
>> (infra-related issues are covered in a later point)
> why would it need to if we are using fedora's dist-git? It seems very very
> messy here.
>
>> > How do builds get signed?
>>
>> That would be left to CentOS core SIG team
> Okay, I would like to know what exactly that means and entails, for one we
> have no way to export the private keys for epel from Fedora's sigul server.
> changing keys would be a pain and require a lot of careful work.
>
>> > How do updates get pushed out to EPEL users? Does CentOS have a bodhi
>>
>> Good question, from my current experience, I get little feedback on my
>> EPEL updates and never got one pushed to stable just through karma.
>
> So what can we do to add extra QA and testing? as that is what is needed to
> get builds pushed via karma. I do not see that magically being fixed.
>
>> > instance?
>> >
>> >> * EPEL will use CentOS repositories instead of mirroring RHEL
>> >> repositories
>> >
>> > CBS seems to not have ppc64... so no more ppc64 EPEL packages?
>>
>> True, if we could get some stats over ppc64 (and any arch unsupported
>> by CentOS), that would help weighting on the decision as for any
>> trade-off.
>
> We are talking about adding ppc64le aarch64 and possibly s390x to epel.  there
> is also the issue that will creap up because of the differences in how RHEL
> and CentOS ship that packages from EPEL will not be installable on RHEL when
> build on CentOS.  This is a RHEL issue, but it is nicer to track by building
> against REHL and not CentOS,  though we also discussed having i686 builds that
> use CentOS to build against
>

Architecture-wise, I believe there are similar plans from CentOS sides
(Jim is actively working on aarch64 port and power port is in
discusion)
Ack for the RHEL/CentOS differences.


>> > Also, this would probibly be some kind of big deal to some people who
>> > like that EPEL is built against rhel. Personally, I don't think it
>> > matters, but it would have to be communicated clearly.
>>
>> (I also saw Karsten reply about it)
>> It needs to be communicated, but considering CentOS good history on
>> that matter, I personally don't think it's big deal, too.
> It is a much bigger deal politically than technically
>
>> >> * Bridging Fedora/CentOS accounting system (CentOS is migrating to
>> >> FAS)  <== we need to see the feasibility of this but that would be
>> >> optimal, that would increase the permeability between our two
>> >> contributors pools which is something, we all want to encourage.
>> >
>> > Bridging in which way? what information would be good to communicate
>> > back and forth?
>>
>> I'm not familiar enough with the FAS/pkgdb architecture, so I will
>> just list some requirements.
>> * ensure that EPEL packagers could rebuild their packages in CBS
>> * ensure that CentOS core SIG could administrate epel-provenpackager group
>>
>> Off course, it could be minimal and may not require syncing FAS
>> instances, in the end.
> I would like to know what you mean here and what you are thinking, regardless
> of if its technically possibly today or not. It may be possible in the future.
>

Ideally identity management federation, we should able Fedora account
with CentOS infrastructure, by granting proper ACLs and vice-versa.
But if that confuses people, let's drop it.

>> >> * Create a EPEL provenpackager group under CentOS core SIG
>> >> supervision, allowing them to appoint people to maintain EPEL
>> >> packages.
>> >
>> > Overriding the existing EPEL maintainers?
>>
>> Yes, as provenpackager could do with most Fedora packages but limited
>> to EPEL branches.
>> I know this may be difficult to give some control to another
>> organization over part of our project. But we need to consider that
>> Fedora/CentOS are part of a larger ecosystem.
>
> This should be implementable.
>

Yes!

>> >> I suggest that we keep the EPEL name to acknowledge EPEL historical
>> >> effort to provide quality additional packages for EL distros.
>> >> Fedora contributors would still be able to contribute to EPEL, and
>> >> CentOS contributors to make it up their standards.
>> >>
>> >> Would that work for you?
>> >
>> > I think there would be a large amount of technical and public relations
>> > work needed to get anything like this off the ground.
>> >
>> > If the problem is that CBS only has a subset of epel builds, perhaps we
>> > could solve that by setting up a script that listens to fedora fedmsgs
>> > and imports epel builds from fedora koji when they are done?
>> >
>> > kevin
>>
>> Yes, my first proposal may have sounded that it's trivial but it's
>> not. On the other hand, this is something achievable and that could
>> benefit for both projects.
>>
>> As any proposal, this needs to be discussed, improved and comes with a
>> lot of trade-offs but if nobody starts the discussion, we'll just go
>> nowhere.
>> All the points you raised are perfectly valid, and needs to be
>> discussed with every stakeholder. Maybe the final solution will be
>> completely different, but that's not what matters.
>> This is a concrete way to build Fedora/CentOS collaboration.
>>
>> If merging cost is too important, then we could discuss alternatives
>> (like EPEL rebuilds automation in CBS), but we need to end the
>> discussion at some point.
>> Anyway, I won't champion any proposal that gets no support from both
>> communities.
>
> There is a massive amount of political play here.  it is not trivial
> technically or politically.  I am all for working out ways to enable CentOS
> contributors to contribute to EPEL. I personally do not think that moving to
> CBS solves any problems, but will create many. I do not know where CBS is
> physically located, but if it was in the same data centre as Fedora's koji
> maybe we could look at sharing some resources, storage and database for koji,
> while providing separate frontends and views in.  though Fedora is at about
> 30T of disk right now, not sure where CentOS is and how much we would need,
> but I am guessing in the order of 60T or more.
>

There is a technical cost to such migration, but I agree that the
biggest blocker is political play (from both sides)
It maybe naive from me, but I'd like us stop thinking as
Fedora/EPEL/CentOS as silos but rather as an integrated ecosystem.
Different target, different usage but same DNA.


> I think that shorter term, CentOS needs to allow epel to be used as an
> external repo, and we need to come up with ways to make it easier for people
> that have traditionally worked in CentOS to contribute to EPEL.
>
> Dennis

That makes sense, but we never reached to an agreement, even a minimal one.
>From my perspective as CentOS SIG member, it's preventing us to grow
healthily as we're relying on random EPEL rebuilds: maybe modified,
maybe not updated.
In the end, it will end up having CentOS rebuilding EPEL from scratch
which means duplicated efforts.

What's important is that we move forward on a topic discussed for
almost two years, not the proposal itself.

H.