Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
The same thing needs to happen in Fedora Copr, with the epel-8-* chroots (side note, in Fedora Copr we use the mock-core-configs package for builds without any deployment specific modifications).
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
This would bring some consequences, namely newly with epel-8* chroots,
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2]) - we will miss some build-time packages that Red Hat is not shipping, at least at the beginning till they are added (to RHEL CRB, or other currently unknown place). - cross-arch compilation can not be used, Red Hat subscriptions don't allow that (using QEMU and rpm --forcearch), [3]
The positive thing is that the default configuration will be much closer to the official EPEL builds (because Fedora Koji EPEL builds are actually done also against RHEL).
For the Fedora Copr builders, we already have the necessary Red Hat subscriptions in hand (will be deployed by the end of the year). So we will only loose the opportunity to build on emulated epel-8-armhfp permanently, and epel-8-s390x temporarily (as we already work on the native s390x support).
Any thoughts? Feedback is needed here.
[1] https://github.com/rpm-software-management/mock/pull/802 [2] https://rpm-software-management.github.io/mock/Feature-rhelchroots [3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847
Pavel
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not have a config for that).
But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config which will differ from Koji or we have to fiddle with entitlements:
https://rpm-software-management.github.io/mock/Feature-rhelchroots
https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cost-r...
Miroslav
On Mon, 2021-11-22 at 15:10 +0100, Miro Hrončok wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________
Hi,
I am in agreement here. EPEL has always used a community distro and I would much prefer it be so and AlmaLinux is the perfect choice for replacing CentOS here. I currently use AlmaLinux for my EPEL builds and believe it would raise fewer issues than a move to using RHEL.
Regards
Phil
On 22/11/2021 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I'm going to retire all my EPEL8 packages as I can't build/test them in mock without any subscriptions.
On 22. 11. 21 15:25, Miroslav Suchý wrote:
But EPEL is built against RHEL (not Alma, not Rocky).
True. As well as it is true today that it is not built against CentOS Linux (and yet we do that in mock).
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
On Mon, Nov 22, 2021 at 8:27 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not have a config for that).
But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config which will differ from Koji or we have to fiddle with entitlements:
https://rpm-software-management.github.io/mock/Feature-rhelchroots
https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cost-r...
Miroslav _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Agreed, for quick local builds it's fine to use epel8-next (c8s) to verify it builds and then submit the koji build to the epel8 (rhel8) target. If the local build works but the koji build doesn't, you likely have a candidate for an official epel8-next koji build.
On Mon, Nov 22, 2021 at 8:37 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 22/11/2021 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I'm going to retire all my EPEL8 packages as I can't build/test them in mock without any subscriptions.
-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
This has always been allowed. EPEL packagers are not required to maintain their packages for the entire corresponding RHEL lifecycle.
https://docs.fedoraproject.org/en-US/package-maintainers/Package_Retirement_...
I will point out that it's trivial to avoid dealing with a subscription by doing koji scratch builds, or by using the existing oraclelinux-epel-8-x86_64 mock chroot, or by submitting equivalent {clone}-epel-8-{arch} chroots to mock-core-configs for your preferred clone. Mass retiring all of your epel8 packages seems like an overreaction to me, but it is your choice. If you do decide to go that route I hope you're welcoming to other maintainers that offer to co-maintainer your packages to be responsible for the epel8 branch going forward. Ideally you would also send an email to epel-devel beforehand to avoid a quick retire/unretire churn for the packages other maintainers are interested in keeping around.
On Mon, Nov 22, 2021 at 10:12 AM Carl George carl@redhat.com wrote:
On Mon, Nov 22, 2021 at 8:37 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 22/11/2021 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I'm going to retire all my EPEL8 packages as I can't build/test them in mock without any subscriptions.
-- Sincerely, Vitaly Zaitsev (vitaly@easycoding.org) _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
This has always been allowed. EPEL packagers are not required to maintain their packages for the entire corresponding RHEL lifecycle.
https://docs.fedoraproject.org/en-US/package-maintainers/Package_Retirement_...
I will point out that it's trivial to avoid dealing with a subscription by doing koji scratch builds, or by using the existing oraclelinux-epel-8-x86_64 mock chroot, or by submitting equivalent {clone}-epel-8-{arch} chroots to mock-core-configs for your preferred clone. Mass retiring all of your epel8 packages seems like an overreaction to me, but it is your choice. If you do decide to go that route I hope you're welcoming to other maintainers that offer to co-maintainer your packages to be responsible for the epel8 branch going forward. Ideally you would also send an email to epel-devel beforehand to avoid a quick retire/unretire churn for the packages other maintainers are interested in keeping around.
Note that I've submitted a PR to switch from CentOS Linux to AlmaLinux[1] for similar reasons (my workflow would be totally broken if I had to deal with the foibles of subscription-manager for building packages).
[1]: https://github.com/rpm-software-management/mock/pull/803
On Mon, 22 Nov 2021 at 10:15, Neal Gompa ngompa13@gmail.com wrote:
On Mon, Nov 22, 2021 at 10:12 AM Carl George carl@redhat.com wrote:
On Mon, Nov 22, 2021 at 8:37 AM Vitaly Zaitsev via devel devel@lists.fedoraproject.org wrote:
On 22/11/2021 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is
I will point out that it's trivial to avoid dealing with a subscription by doing koji scratch builds, or by using the existing oraclelinux-epel-8-x86_64 mock chroot, or by submitting equivalent {clone}-epel-8-{arch} chroots to mock-core-configs for your preferred clone. Mass retiring all of your epel8 packages seems like an overreaction to me, but it is your choice. If you do decide to go that route I hope you're welcoming to other maintainers that offer to co-maintainer your packages to be responsible for the epel8 branch going forward. Ideally you would also send an email to epel-devel beforehand to avoid a quick retire/unretire churn for the packages other maintainers are interested in keeping around.
Note that I've submitted a PR to switch from CentOS Linux to AlmaLinux[1] for similar reasons (my workflow would be totally broken if I had to deal with the foibles of subscription-manager for building packages).
I would personally like to go this route (Alma+EPEL) myself. Yes I work for Red Hat and even if I didn't I could get the copies of the code via the Red Hat Developers program.. but dealing with a subscription manager while trying to build on Fedora 3X etc has not been reliable for me. [And saying that I could use a dedicated RHEL8 virtual machine is only going to work for EL8. You would need to have fedpkg and similar tools in EPEL-9 for the tools to work in EL9.]
-- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren
On Mon, Nov 22, 2021 at 9:01 AM Pavel Raiskup praiskup@redhat.com wrote:
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
The same thing needs to happen in Fedora Copr, with the epel-8-* chroots (side note, in Fedora Copr we use the mock-core-configs package for builds without any deployment specific modifications).
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
This would seem to be an exceptionally bad idea. Red Hat elected to force this CentOS Stream release structure down everyone's throats unannounced, compelling default mock behavior to use the stream is a logical consequence. Trying to outsmart Red Hat on this is going to break EPEL dependencies and keep breaking EPEL for Stream deployments.
This would bring some consequences, namely newly with epel-8* chroots,
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
- we will miss some build-time packages that Red Hat is not shipping, at least at the beginning till they are added (to RHEL CRB, or other currently unknown place).
- cross-arch compilation can not be used, Red Hat subscriptions don't allow that (using QEMU and rpm --forcearch), [3]
Which is precisely why pointing it to the 'stream' release seems the only workable solution.
The positive thing is that the default configuration will be much closer to the official EPEL builds (because Fedora Koji EPEL builds are actually done also against RHEL).
That does seem desirable. Frankly, I don't expect this mid-stream pivot to last more than 2 years: it was tried before with Red Hat 9, and discarded thoroughly for RHEL 2.1. Point releases are too useful for environments that cannot afford CI/CD with the underlying operating system, and don't want to spend the time re-inventing point releases themselves.
For the Fedora Copr builders, we already have the necessary Red Hat subscriptions in hand (will be deployed by the end of the year). So we will only loose the opportunity to build on emulated epel-8-armhfp permanently, and epel-8-s390x temporarily (as we already work on the native s390x support).
Any thoughts? Feedback is needed here.
[1] https://github.com/rpm-software-management/mock/pull/802 [2] https://rpm-software-management.github.io/mock/Feature-rhelchroots [3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847
Pavel
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Mon, Nov 22, 2021 at 11:55 AM Carl George carl@redhat.com wrote:
On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok mhroncok@redhat.com wrote:
On 22. 11. 21 15:00, Pavel Raiskup wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm not aware of a RHEL clone that offers all the architectures that EPEL does. As far as I can tell, the three most popular (Alma, Rocky, Oracle) only offer x86_64 and aarch64 but are missing ppc64le and s390x. That said CentOS Linux 8 doesn't offer s390x either, so we already have this problem, but switch the EPEL mock chroot to one of those clones would make the situation worse by also dropping ppc64le.
I've been informed that AlmaLinux is working on ppc64le support, and it's supposed to be released "soon". Perhaps Jack can provide an update here?
On Mon, Nov 22, 2021 at 2:01 PM Pavel Raiskup praiskup@redhat.com wrote:
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-).
I do not believe anyone should blame you for bringing this up (although, it should be noted, killing the messenger has never, really, gone out of style :-).
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
Any thoughts? Feedback is needed here.
I have a number of cases where I package software via mock that (for various reasons) is never really released into a formal public repo, but for which I would like to be able to just build the rpm for EL<x>. Having to obtain and manage (even a free) RH subscription will be a major PITA for such activities (especially as I suspect I will need a number of different subscriptions for the various different role "hats" that are worn).
I have no great alternatives. However, as EL<x> typically has a binary compatibility promise across its lifecycle, is a viable alternative to have a (perhaps separate) mock config for building be the base release version? With the (typical) exception of the compiler(s) themselves, the feature and security updates are at run time, and not build time (and in my cases, the run time systems will be (mostly) fully updated so that would probably work out ok, for me).
Anyway, thanks for bringing the issue up.
Dne 22. 11. 21 v 17:57 Nico Kadel-Garcia napsal(a):
Which is precisely why pointing it to the 'stream' release seems the only workable solution.
That is EPEL-next
https://fedoraproject.org/wiki/EPEL_Next#Introduction
Miroslav
Pavel Raiskup wrote:
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
I, too, believe that we can come to an acceptable solution. Unfortunately, I do not consider the one you propose to be acceptable, at all.
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
I think there are only 2 reasonable defaults: Rocky Linux or AlmaLinux (just pick whatever of those works better in practice, it should not really matter). In particular:
This would bring some consequences, namely newly with epel-8* chroots,
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
Anything that requires a subscription of any sort, even a free-as-in-beer one, should be a non-starter. The default EPEL config needs to just work and to be free of field-of-use restrictions, including technically enforced policy restrictions breaking use cases such as emulation. So requiring a RHEL developer subscription is not an acceptable solution.
- we will miss some build-time packages that Red Hat is not shipping, at least at the beginning till they are added (to RHEL CRB, or other currently unknown place).
That will definitely be a problem. It has also been a constant source of trouble for EPEL itself, and should be reason enough to switch EPEL to a more cooperative (but binary-compatible) base distro (i.e., Rocky or Alma) altogether, even in Koji, i.e., in particular, even for the official RPMs.
- cross-arch compilation can not be used, Red Hat subscriptions don't allow that (using QEMU and rpm --forcearch), [3]
That, I consider to be a field-of-use restriction and incompatible with the Free Software Definition and the Open Source Definition altogether.
(Other field-of-use restrictions probably come from the wording of the developer subscription license, e.g., I guess it is not allowed to use a RHEL developer subscription mock chroot for other purposes than building packages, is it?)
The positive thing is that the default configuration will be much closer to the official EPEL builds (because Fedora Koji EPEL builds are actually done also against RHEL).
But Koji is really the odd one out, so why not instead fix Koji to work like local mock and Copr rather than the opposite?
For the Fedora Copr builders, we already have the necessary Red Hat subscriptions in hand (will be deployed by the end of the year). So we will only loose the opportunity to build on emulated epel-8-armhfp permanently, and epel-8-s390x temporarily (as we already work on the native s390x support).
So even Copr itself is hit by the field-of-use restriction. (Now, armhfp and s390x might also be unsupported by the rebuilds altogether, but at least it would be a technical restriction and not a legal one, and the technical restriction could be fixed by anyone by just building the rebuild, or even any rebuild, for those architectures.)
Any thoughts? Feedback is needed here.
See above.
Kevin Kofler
On Tue, Nov 23, 2021 at 7:31 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Pavel Raiskup wrote:
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
I, too, believe that we can come to an acceptable solution. Unfortunately, I do not consider the one you propose to be acceptable, at all.
I am proposing (as PR against mock upstream ATM [1]) to switch the default epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible (see the pull request [1]).
I think there are only 2 reasonable defaults: Rocky Linux or AlmaLinux (just pick whatever of those works better in practice, it should not really matter). In particular:
Neither of those have much track history. The easy move is to use CentOS 8 Stream for "mock", at least in the short term.
The alternative is to designate the CentOS switch to streaming deployments as a non-starter, and re-invent point releases on top of CentOS 8 Stream It's likely to create small discrepancies with packages released, and rejected or withdrawn, from CentOs 8 Stream before they get to RHEL 8. Since CentOS 8 Stream is the new beta for RHEL 8, this would seem inevitable.
It's a waste of time and resources to generate and manage the relevant repodata, exacerbated by the multiple unwelcome and randomly overlapping channels of CentOS 8. Multiple snapshots of a repo are also not free in disk space or inodes. But I'm not sure if the more "replicate only" distributions will be long-lived multi-platform enough to rely on for EPEL, and I'd expect screaming from various security and support experts if EPEL relies on anything not directly published by Red Hat. I'm finding myself sad that Scientific Linux withdrew from publishing new releases in deference to CentOS.
Nico Kadel-Garcia wrote:
Neither of those have much track history.
Of course they don't. How can they, when CentOS had stolen their show for years and the sudden "change of directions" was announced only earlier this year? Blame RH for not having given the rebuild community more time to prepare. (Pulling the plug on CentOS 8 7-8 years before its originally announced EOL was a huge surprise, and not a good one.)
The easy move is to use CentOS 8 Stream for "mock", at least in the short term.
With CentOS Stream, you want to use epel-next, not plain epel. And the epel- next configs already use CentOS Stream.
The alternative is to designate the CentOS switch to streaming deployments as a non-starter, and re-invent point releases on top of CentOS 8 Stream It's likely to create small discrepancies with packages released, and rejected or withdrawn, from CentOs 8 Stream before they get to RHEL 8. Since CentOS 8 Stream is the new beta for RHEL 8, this would seem inevitable.
Why would we want to reinvent the wheel instead of using one of the two already working solutions? And how would that not necessarily have even less "track history", considering that Alma and Rocky already exist, whereas your idea has not even be started yet?
It's a waste of time and resources to generate and manage the relevant repodata, exacerbated by the multiple unwelcome and randomly overlapping channels of CentOS 8. Multiple snapshots of a repo are also not free in disk space or inodes.
Of course it is. So do not do that.
But I'm not sure if the more "replicate only" distributions will be long-lived multi-platform enough to rely on for EPEL,
Then you switch to another rebuild. The demand is not going to vanish overnight, so I do not expect them to go away without a replacement, just as CentOS did not go away without a replacement either.
and I'd expect screaming from various security and support experts if EPEL relies on anything not directly published by Red Hat.
Well, for the local mock config, that is exactly what had happened until CentOS got absorbed by Red Hat. And if Red Hat does not leave us with a better option, blame Red Hat.
I'm finding myself sad that Scientific Linux withdrew from publishing new releases in deference to CentOS.
Me too. They too got burned by Red Hat's misleading adoption of CentOS and the following surprise "change of directions", and it is unfortunate that they have decided against resurrecting Scientific Linux given the change. But Scientific Linux was "not directly published by Red Hat" either.
Kevin Kofler
On Tue, Nov 23, 2021 at 9:16 PM Kevin Kofler via devel devel@lists.fedoraproject.org wrote:
Nico Kadel-Garcia wrote:
Neither of those have much track history.
Of course they don't. How can they, when CentOS had stolen their show for years and the sudden "change of directions" was announced only earlier this year? Blame RH for not having given the rebuild community more time to prepare. (Pulling the plug on CentOS 8 7-8 years before its originally announced EOL was a huge surprise, and not a good one.)
The easy move is to use CentOS 8 Stream for "mock", at least in the short term.
With CentOS Stream, you want to use epel-next, not plain epel. And the epel- next configs already use CentOS Stream.
Which is available. It would need to be designated as the "default" for compilation. The changes would be in the default settings for mock-core-configs, nowhere else, and not require other repo changes.
The alternative is to designate the CentOS switch to streaming deployments as a non-starter, and re-invent point releases on top of CentOS 8 Stream It's likely to create small discrepancies with packages released, and rejected or withdrawn, from CentOs 8 Stream before they get to RHEL 8. Since CentOS 8 Stream is the new beta for RHEL 8, this would seem inevitable.
Why would we want to reinvent the wheel instead of using one of the two already working solutions? And how would that not necessarily have even less "track history", considering that Alma and Rocky already exist, whereas your idea has not even be started yet?
I'm not saying this is the best option, but it invelves less change to the build systems to rely on mirrors and base repos that Red Hat does not own. It's feasible as a technology, and it's something that some of us are going to have to do anyway to use CentOS for production deployments. I've been doing something like it since... well, since Red Hat 9 in 2003, and for Fedora, to lock down static internal mirrors for staged rather than streaming updates.
It's a waste of time and resources to generate and manage the relevant repodata, exacerbated by the multiple unwelcome and randomly overlapping channels of CentOS 8. Multiple snapshots of a repo are also not free in disk space or inodes.
Of course it is. So do not do that.
Assess the alternatives before discarding them.
But I'm not sure if the more "replicate only" distributions will be long-lived multi-platform enough to rely on for EPEL,
Then you switch to another rebuild. The demand is not going to vanish overnight, so I do not expect them to go away without a replacement, just as CentOS did not go away without a replacement either.
That is *expensive* to do. It leads to people discarding distributions, which I observe has been happening for CentOS and RHEL. Some previous fans are biting the bullet and switching to Ubuntu to avoid this mess, others are switching to Amazon Linux or simply skipping RHEL 8 and CentOS 8 entirely. Amazon, for example, is not planning to publish a matching "Amazon Linux 3", I've asked them. Unfortunately, Amazon Linux 2 is too divergent from RHEL and CentOS to be usable for EPEL, it's python 3.7 instead of python 3.6.
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
Insofar as "phase lag with new point releases", AlmaLinux made their release 48 hours after Red Hat did with RHEL. So, frankly, I'm not worried there with AlmaLinux.
For bandwidth burdens, mirror networks are designed to alleviate that burden and both have those in place.
-- 真実はいつも一つ！/ Always, there's only one truth!
On Mon, Nov 22, 2021 at 03:10:58PM +0100, Miro Hrončok wrote:
- builds will require a valid Red Hat subscription (the no-cost variant is OK as well, though [2])
I cannot help myself but I consider this very unpleasant for EPEL packagers.
Getting and configuring the subscription was always so unfriendly for me that I've been using EPEL mocks even for my RHEL work. This basically means using EPEL mocks will once again be as complicated as using RHEL.
I'm not opposed to working with our friends over at Alma on this, but I think we _also_ should fix this. Would it help if we made it easy to add a RHEL developer entitlement (the 16 no-cost RHEL license thing) to a Fedora Account (perhaps with a check-box agreement in the account system, like the FPCA), and made it trivial in mock use that?
I mean, seriously, RH should make this easy for Fedora packagers.
On Thu, Nov 25, 2021 at 8:05 AM Miroslav Suchý msuchy@redhat.com wrote:
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
Thanks.
Another idea occurred to me, similar to "D" (use alternatives) and incorporates "A" (delete the current epel-8) for their to be something such as having a "mock-config-epel-default-base-rhel" and "mock-config-epel-default-base-alma" and "mock-config-epel-default-base-rocky" rpms (or whatever the agreed upon names would be) that would create the (raw?) epel-8 symlink, and conflict with each other. Then one is left discussing which of the various "Suggests: " or "Enhances: " specifiers would be appropriate and under what circumstances, perhaps such as a Suggests: (....base-alma if alma-release), and a Suggests: (...base-rocky if rocky-release) ???
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
The difficulty with switching mock to AlmaLinux or Rocky is that there is likely to be significant phase lag with new point releases by Red Hat, and that it will inflict quite a bandwidth burden for all the "mock" setups in the field. Can they scale up to handle that?
Insofar as "phase lag with new point releases", AlmaLinux made their release 48 hours after Red Hat did with RHEL. So, frankly, I'm not worried there with AlmaLinux.
48 hours is pretty danged good. I hesitate to rely on a new OS publisher with so little track record, even if they've been very good in their first year. I'm very curious how well they'll do with RHEL 8.6, without a published CentOS example to compare with. And I'd be very, very wary of the "planning fallacy", where people underestimate the time of future tasks, despite knowledge that previous tasks have sometimes taken far longer.
For bandwidth burdens, mirror networks are designed to alleviate that burden and both have those in place.
Sure, and they're very helpful. But to the one managing such networks, a massive increase in bandwidth use or in the *breadth* of content used for a particular mirrored repo for a particular product is very helpful to plan for and avoid local choke points. I'm especially thinking of the cache on relevant proxy servers.
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
Hello Fedora EPEL maintainers!
First I don't feel comfortable announcing this, I'm not happy about the situation and so I don't want to be the lightning rod :-). But I believe that we can come to acceptable Copr/Mock solution and this needs to be discussed... so here we are.
By the end of the year 2021 we have to fix our default EPEL 8 Mock configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
Nico Kadel-Garcia
On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a): > Hello Fedora EPEL maintainers! > > First I don't feel comfortable announcing this, I'm not happy about the > situation and so I don't want to be the lightning rod :-). But I believe > that we can come to acceptable Copr/Mock solution and this needs to be > discussed... so here we are. > > By the end of the year 2021 we have to fix our default EPEL 8 Mock > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 > goes EOL by then.
I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here.
https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V...
Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
In order to keep older package releases, it would require changes to the compose tool pungi. It would also have to make it so it worked for EPEL versus Fedora. [Fedora Linux releases have grown to the point that many mirrors can barely carry the OS as is.. adding in older packages is out of the question for them.] I do not have numbers on how often packages churn or which ones churn the most.
On Mon, Nov 29, 2021 at 6:42 AM Stephen John Smoogen smooge@gmail.com wrote:
On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia nkadel@gmail.com wrote:
On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia nkadel@gmail.com wrote:
On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý msuchy@redhat.com wrote: > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a): > > Hello Fedora EPEL maintainers! > > > > First I don't feel comfortable announcing this, I'm not happy about the > > situation and so I don't want to be the lightning rod :-). But I believe > > that we can come to acceptable Copr/Mock solution and this needs to be > > discussed... so here we are. > > > > By the end of the year 2021 we have to fix our default EPEL 8 Mock > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8 > > goes EOL by then. > > > I wrote down the possible options and their pros and cons and I done my best to catch all the feedback here. > > https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SGy4V... > > Miroslav
That seems to be a succinct listing. I think you left out my suggestion.of "support people re-inventing point releases for CentOS", which is what major CentOS users will do using internal mirrors. due to concern about unexpected and unwelcome updates of CentOS Stream, while they assess whether AlmaLinux or Rocky are reliable and stable enough to use. It's not an uncommon behavior for EPEL itself, partly because of EPEL's bad habit of deleting RPMs without warning and stripping out all previous releases. That's caused me problems with chromium and firefox when updates were incompatible with contemporary regression testing systems.
It's not a "bad habit", it happens because when packages are retired, keeping the packages there does a disservice to the community by effectively forcing a maintenance burden when there's no maintainer. As for stripping out previous releases, that's just how Pungi and Bodhi do update composes at the moment. Someday that'll be fixed, but then we'd have to come up with a policy on how many because there are storage concerns for mirrors if we kept everything published forever.
It causes problems and confusion for people who need to lock down evisting versions for deployment. And it happens for packages that are not retired, but merely updated. I was bitten by it myself with chromium updates last year. It forces users of EPEL to maintain internal repos, or out of band access to previously accessible RPMs. It's destabilizing and breaks the use of bills-of-material based deployments with complete lists of all desired RPMs.
Storage and bandwidth concerns are legitimate concerns, as is potentially continuing to publish older releases with known vulnerabilities or bugs. But neither Fedora nor RHEL simply discard previously published versions this way, they aggregate new releases. I consider this a longstanding bug for EPEL, and one of the reasons I set up internal mirrors in large deployments.
This is not true. Fedora and EPEL share the same system, and have the same issues. The only difference is that the release repo is frozen in Fedora, so only the updates repo is affected this way. So there's at most two versions of a package at any time.
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
RHEL *does* maintain multiple old versions, but their system is completely different and supports that capability.
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos? I appreciate that it would require thought and expand them noticeably, especially for bulky and frequently updating components like chromium. I admit to not having numbers on how much churn happens there: does anyone have numbers?
In order to keep older package releases, it would require changes to the compose tool pungi. It would also have to make it so it worked for EPEL versus Fedora. [Fedora Linux releases have grown to the point that many mirrors can barely carry the OS as is.. adding in older packages is out of the question for them.] I do not have numbers on how often packages churn or which ones churn the most.
EPEL is big enough that I begged Troy Dawson to untag duplicate copies of KDE Plasma so that I could have enough free space for a second mirror job to work. Between EPEL and EPEL Next for 8, it's pretty huge. I would say with all the EPEL repositories, it matches the size of Fedora Linux release trees. I don't think anyone would be too happy with more. And keep in mind that EPEL trees last for 10 years on the mirror network before being frozen. That's a lot of updates over the years.
If Fedora and EPEL were to have older versions, we'd have to have a dedicated CDN endpoint for them, because mirrors would seriously have trouble taking it.
On 29/11/2021 01:31, Nico Kadel-Garcia wrote:
What would it take to get Fedora, or at least EPEL, to preserve old releases in the default published repos?
Mirror owners won't be happy.
On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
If Fedora and EPEL were to have older versions, we'd have to have a dedicated CDN endpoint for them, because mirrors would seriously have trouble taking it.
How often would such packages be used? If we had a non-default repo available but not enabled by default, it could be optional to mirror and probably still be okay.
On Mon, Nov 29, 2021 at 9:04 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
If Fedora and EPEL were to have older versions, we'd have to have a dedicated CDN endpoint for them, because mirrors would seriously have trouble taking it.
How often would such packages be used? If we had a non-default repo available but not enabled by default, it could be optional to mirror and probably still be okay.
I suspect it would be fairly heavily used. There are significant benefits to having those:
* DeltaRPM would be considerably more useful * "dnf history undo" would actually work
We could make it non-default and tell people that rollbacks require an --enablerepo= option, though.
Having its own mirror module would also give mirrors the ability to opt in, and I wouldn't be surprised if a good chunk of them do.
On Mon, Nov 29, 2021 at 09:11:48AM -0500, Neal Gompa wrote:
On Mon, Nov 29, 2021 at 9:04 AM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
If Fedora and EPEL were to have older versions, we'd have to have a dedicated CDN endpoint for them, because mirrors would seriously have trouble taking it.
How often would such packages be used? If we had a non-default repo available but not enabled by default, it could be optional to mirror and probably still be okay.
So, we actually do have this for fedora updates today. We had to set it up to solve an issue with the updates flow and CoreOS.
See the fedora-updates-archive subpackage of fedora-repos. This is hosted on amazon S3, behind cloudfront.
I suspect it would be fairly heavily used. There are significant benefits to having those:
- DeltaRPM would be considerably more useful
In order to have deltarpms the packages would have to be available at compose time and in the same repo.
- "dnf history undo" would actually work
We could make it non-default and tell people that rollbacks require an --enablerepo= option, though.
This would work for Fedora now with the archive repo above.
I suppose we could look at setting up a similar setup with epel.
kevin
Nico Kadel-Garcia wrote:
You're correct. With the current setup, it's also relatively simple to revert to the "frozen" release, which handles most of the regression situations. And Fedora releases are nowhere near so long-lived as RHEL and EPEL, so it tends to be less of a long-lived problem.
At least in my experience, the frozen version of the update is actually entirely useless in almost all cases because it is way too old. What is really needed in most cases is the previous update. I usually fetch that from my local cache (I enable keepcache always and consider the fact that Fedora disables it by default a critical data loss bug) or, failing that, from Koji. IIRC, the old Fedora Extras used to ship the latest 2 builds of every package, which had addressed that very issue. This was unfortunately dropped with the Core/Extras Merge and everything moving to the composing tools from Core (the current incarnation of which is called Pungi).
Kevin Kofler
-
Carl George -
Gary Buhrmaster -
Kevin Fenzi -
Kevin Kofler -
Matthew Miller -
Miro Hrončok -
Miroslav Suchý -
Neal Gompa -
Nico Kadel-Garcia -
Pavel Raiskup -
Philip Wyett -
Stephen John Smoogen -
Vitaly Zaitsev