Mock/Copr default epel-8-* configuration to be changed

[Pavel Raiskup]

Hello Fedora EPEL maintainers!

First I don't feel comfortable announcing this, I'm not happy about the
situation and so I don't want to be the lightning rod :-).  But I believe
that we can come to acceptable Copr/Mock solution and this needs to be
discussed...  so here we are.

By the end of the year 2021 we have to fix our default EPEL 8 Mock
configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
goes EOL by then.

The same thing needs to happen in Fedora Copr, with the epel-8-* chroots
(side note, in Fedora Copr we use the mock-core-configs package for builds
without any deployment specific modifications).

I am proposing (as PR against mock upstream ATM [1]) to switch the default
epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible
(see the pull request [1]).

This would bring some consequences, namely newly with epel-8* chroots,

- builds will require a valid Red Hat subscription (the no-cost variant is
  OK as well, though [2])
- we will miss some build-time packages that Red Hat is not shipping, at
  least at the beginning till they are added (to RHEL CRB, or other
  currently unknown place).
- cross-arch compilation can not be used, Red Hat subscriptions don't
  allow that (using QEMU and rpm --forcearch), [3]

The positive thing is that the default configuration will be much closer
to the official EPEL builds (because Fedora Koji EPEL builds are actually done
also against RHEL).

For the Fedora Copr builders, we already have the necessary Red Hat
subscriptions in hand (will be deployed by the end of the year).  So we will
only loose the opportunity to build on emulated epel-8-armhfp permanently, and
epel-8-s390x temporarily (as we already work on the native s390x support).

Any thoughts?  Feedback is needed here.

[1] https://github.com/rpm-software-management/mock/pull/802
[2] https://rpm-software-management.github.io/mock/Feature-rhelchroots
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847

Pavel

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Miro Hrončok]

On 22. 11. 21 15:00, Pavel Raiskup wrote:
>  - builds will require a valid Red Hat subscription (the no-cost
> variant is
>     OK as well, though [2]) 
I cannot help myself but I consider this very unpleasant for EPEL packagers.

Getting and configuring the subscription was always so unfriendly for me that 
I've been using EPEL mocks even for my RHEL work. This basically means using 
EPEL mocks will once again be as complicated as using RHEL.

However, enough of my personal views. Since we have not used RHEL for copr/mock 
EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy 
(CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Miroslav Suchý]

Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
>  However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL
> buidlroots until now, but we used 
>  a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing so by
> using e.g. AlmaLinux? 
For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not
have a config for that).

But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config
which will differ from Koji or we 
have to fiddle with entitlements:

https://rpm-software-management.github.io/mock/Feature-rhelchroots

https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cos...

Miroslav

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Miro Hrončok]

On 22. 11. 21 15:25, Miroslav Suchý wrote:
>  But EPEL is built against RHEL (not Alma, not Rocky). 
True. As well as it is true today that it is not built against CentOS Linux 
(and yet we do that in mock).

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Carl George]

On Mon, Nov 22, 2021 at 8:27 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 22. 11. 21 v 15:10 Miro Hrončok napsal(a):
>  >
>  > However, enough of my personal views. Since we have not used RHEL for copr/mock EPEL
> buidlroots until now, but we used
>  > a downstream freely-available RHEL-copy (CentOS Linux), could we not continue doing
> so by using e.g. AlmaLinux?
>
>  For day to day work I would suggest to move to centos-stream + epel-next (hmm, we do not
> have a config for that).
>
>  But EPEL is built against RHEL (not Alma, not Rocky). So we either use default config
> which will differ from Koji or we
>  have to fiddle with entitlements:
>
>  https://rpm-software-management.github.io/mock/Feature-rhelchroots
>
>
> https://developers.redhat.com/blog/2021/02/10/how-to-activate-your-no-cos...
>
>  Miroslav
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Agreed, for quick local builds it's fine to use epel8-next (c8s) to
verify it builds and then submit the koji build to the epel8 (rhel8)
target.  If the local build works but the koji build doesn't, you
likely have a candidate for an official epel8-next koji build.

-- 
Carl George

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Carl George]

On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 22. 11. 21 15:00, Pavel Raiskup wrote:
>  > - builds will require a valid Red Hat subscription (the no-cost variant is
>  >    OK as well, though [2])
>
>  I cannot help myself but I consider this very unpleasant for EPEL packagers.
>
>  Getting and configuring the subscription was always so unfriendly for me that
>  I've been using EPEL mocks even for my RHEL work. This basically means using
>  EPEL mocks will once again be as complicated as using RHEL.
>
>  However, enough of my personal views. Since we have not used RHEL for copr/mock
>  EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy
>  (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
>
>  --
>  Miro Hrončok
>  --
>  Phone: [phone]
>  IRC: mhroncok
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm not aware of a RHEL clone that offers all the architectures that
EPEL does.  As far as I can tell, the three most popular (Alma, Rocky,
Oracle) only offer x86_64 and aarch64 but are missing ppc64le and
s390x.  That said CentOS Linux 8 doesn't offer s390x either, so we
already have this problem, but switch the EPEL mock chroot to one of
those clones would make the situation worse by also dropping ppc64le.

-- 
Carl George

[EPEL-devel]Re:Mock/Coprdefaultepel-8-*configurationtobechanged

[Neal Gompa]

On Mon, Nov 22, 2021 at 11:55 AM Carl George <carl(a)redhat.com&gt; wrote:
>  On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  >
>  > On 22. 11. 21 15:00, Pavel Raiskup wrote:
>  > > - builds will require a valid Red Hat subscription (the no-cost variant is
>  > >    OK as well, though [2])
>  >
>  > I cannot help myself but I consider this very unpleasant for EPEL packagers.
>  >
>  > Getting and configuring the subscription was always so unfriendly for me that
>  > I've been using EPEL mocks even for my RHEL work. This basically means using
>  > EPEL mocks will once again be as complicated as using RHEL.
>  >
>  > However, enough of my personal views. Since we have not used RHEL for copr/mock
>  > EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy
>  > (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
>  >
>  > --
>  > Miro Hrončok
>  > --
>  > Phone: [phone]
>  > IRC: mhroncok
>  > _______________________________________________
>  > devel mailing list -- devel(a)lists.fedoraproject.org
>  > To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  > Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  > List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  > Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
>
>  I'm not aware of a RHEL clone that offers all the architectures that
>  EPEL does.  As far as I can tell, the three most popular (Alma, Rocky,
>  Oracle) only offer x86_64 and aarch64 but are missing ppc64le and
>  s390x.  That said CentOS Linux 8 doesn't offer s390x either, so we
>  already have this problem, but switch the EPEL mock chroot to one of
>  those clones would make the situation worse by also dropping ppc64le.
>  
I've been informed that AlmaLinux is working on ppc64le support, and
it's supposed to be released "soon". Perhaps Jack can provide an
update here?



-- 
真実はいつも一つ！/ Always, there's only one truth!

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Jean-Marc Liger]

Le 22/11/2021 à 15:52, Carl George a écrit :

>  On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > On 22. 11. 21 15:00, Pavel Raiskup wrote:
> >> - builds will require a valid Red Hat subscription (the no-cost variant is
> >>     OK as well, though [2])
> > I cannot help myself but I consider this very unpleasant for EPEL packagers.
> >
> > Getting and configuring the subscription was always so unfriendly for me that
> > I've been using EPEL mocks even for my RHEL work. This basically means using
> > EPEL mocks will once again be as complicated as using RHEL.
> >
> > However, enough of my personal views. Since we have not used RHEL for copr/mock
> > EPEL buidlroots until now, but we used a downstream freely-available RHEL-copy
> > (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
> >
> > --
> > Miro Hrončok
> > --
> > Phone: [phone]
> > IRC: mhroncok
> > _______________________________________________
> >
> >
> > I'm not aware of a RHEL clone that offers all the architectures that
> > EPEL does.  As far as I can tell, the three most popular (Alma, Rocky,
> > Oracle) only offer x86_64 and aarch64 but are missing ppc64le and
> > s390x.  That said CentOS Linux 8 doesn't offer s390x either, so we
> > already have this problem, but switch the EPEL mock chroot to one of
> > those clones would make the situation worse by also dropping ppc64le.
> > 
The success of COPR is based on it's easy way to build packages, 
especially for EPEL,
do whatever you want guys but please at the end, try to not choose the 
bullshito way.

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Jean-Marc Liger]

Le 22/11/2021 à 15:52, Carl George a écrit :

>  On Mon, Nov 22, 2021 at 8:11 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > On 22. 11. 21 15:00, Pavel Raiskup wrote:
> >> - builds will require a valid Red Hat subscription (the no-cost 
> >> variant is
> >> OK as well, though [2])
> > I cannot help myself but I consider this very unpleasant for EPEL 
> > packagers.
> >
> > Getting and configuring the subscription was always so unfriendly for 
> > me that
> > I've been using EPEL mocks even for my RHEL work. This basically 
> > means using
> > EPEL mocks will once again be as complicated as using RHEL.
> >
> > However, enough of my personal views. Since we have not used RHEL for 
> > copr/mock
> > EPEL buidlroots until now, but we used a downstream freely-available 
> > RHEL-copy
> > (CentOS Linux), could we not continue doing so by using e.g. AlmaLinux?
> >
> > --
> > Miro Hrončok
> > --
> > Phone: [phone]
> > IRC: mhroncok
> > _______________________________________________
> >
> >
> > I'm not aware of a RHEL clone that offers all the architectures that
> > EPEL does. As far as I can tell, the three most popular (Alma, Rocky,
> > Oracle) only offer x86_64 and aarch64 but are missing ppc64le and
> > s390x. That said CentOS Linux 8 doesn't offer s390x either, so we
> > already have this problem, but switch the EPEL mock chroot to one of
> > those clones would make the situation worse by also dropping ppc64le.
> > 
The success of COPR is based on it's easy way to build packages, 
especially for EPEL,
do whatever you want guys but please at the end, try to not choose the 
bullshito way.

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Matthew Miller]

On Mon, Nov 22, 2021 at 03:10:58PM +0100, Miro Hrončok wrote:
>  >- builds will require a valid Red Hat subscription (the no-cost
> variant is
>  >   OK as well, though [2])
>  
>  I cannot help myself but I consider this very unpleasant for EPEL packagers.
>  
>  Getting and configuring the subscription was always so unfriendly
>  for me that I've been using EPEL mocks even for my RHEL work. This
>  basically means using EPEL mocks will once again be as complicated
>  as using RHEL. 
I'm not opposed to working with our friends over at Alma on this, but I
think we _also_ should fix this. Would it help if we made it easy to add a
RHEL developer entitlement (the 16 no-cost RHEL license thing) to a Fedora
Account (perhaps with a check-box agreement in the account system, like the
FPCA), and made it trivial in mock use that?

I mean, seriously, RH should make this easy for Fedora packagers.



-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

[Nico Kadel-Garcia]

On Mon, Nov 22, 2021 at 9:01 AM Pavel Raiskup <praiskup(a)redhat.com&gt; wrote:
>  Hello Fedora EPEL maintainers!
>
>  First I don't feel comfortable announcing this, I'm not happy about the
>  situation and so I don't want to be the lightning rod :-).  But I believe
>  that we can come to acceptable Copr/Mock solution and this needs to be
>  discussed...  so here we are.
>
>  By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
>  goes EOL by then.
>
>  The same thing needs to happen in Fedora Copr, with the epel-8-* chroots
>  (side note, in Fedora Copr we use the mock-core-configs package for builds
>  without any deployment specific modifications).
>
>  I am proposing (as PR against mock upstream ATM [1]) to switch the default
>  epel-* configuration from CentOS+EPEL to RHEL+EPEL as soon as possible
>  (see the pull request [1]). 
This would seem to be an exceptionally bad idea. Red Hat elected to
force this CentOS Stream release structure down everyone's throats
unannounced, compelling default mock behavior to use the stream is a
logical consequence. Trying to outsmart Red Hat on this is going to
break EPEL dependencies and keep breaking EPEL for Stream deployments.

>  This would bring some consequences, namely newly with epel-8*
> chroots,
>
>  - builds will require a valid Red Hat subscription (the no-cost variant is
>    OK as well, though [2])
>  - we will miss some build-time packages that Red Hat is not shipping, at
>    least at the beginning till they are added (to RHEL CRB, or other
>    currently unknown place).
>  - cross-arch compilation can not be used, Red Hat subscriptions don't
>    allow that (using QEMU and rpm --forcearch), [3] 
Which is precisely why pointing it to the 'stream' release seems the
only workable solution.

>  The positive thing is that the default configuration will be much
> closer
>  to the official EPEL builds (because Fedora Koji EPEL builds are actually done
>  also against RHEL). 
That does seem desirable. Frankly, I don't expect this mid-stream
pivot to last more than 2 years: it was tried before with Red Hat 9,
and discarded thoroughly for RHEL 2.1. Point releases are too useful
for environments that cannot afford CI/CD with the underlying
operating system, and don't want to spend the time re-inventing point
releases themselves.

> For the Fedora Copr builders, we already have the necessary Red Hat
> subscriptions in hand (will be deployed by the end of the year).  So we will
> only loose the opportunity to build on emulated epel-8-armhfp permanently, and
> epel-8-s390x temporarily (as we already work on the native s390x support).
>
> Any thoughts?  Feedback is needed here.
>
> [1] https://github.com/rpm-software-management/mock/pull/802
> [2] https://rpm-software-management.github.io/mock/Feature-rhelchroots
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1912847
>
> Pavel
>
>
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Miroslav Suchý]

Dne 22. 11. 21 v 17:57 Nico Kadel-Garcia napsal(a):
>  Which is precisely why pointing it to the 'stream' release
> seems the
>  only workable solution. 
That is EPEL-next

https://fedoraproject.org/wiki/EPEL_Next#Introduction

Miroslav

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobe changed

[Miroslav Suchý]

Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  Hello Fedora EPEL maintainers!
>
>  First I don't feel comfortable announcing this, I'm not happy about the
>  situation and so I don't want to be the lightning rod :-).  But I believe
>  that we can come to acceptable Copr/Mock solution and this needs to be
>  discussed...  so here we are.
>
>  By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
>  goes EOL by then. 

I wrote down the possible options and their pros and cons and I done my best to catch all
the feedback here.

https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...

Miroslav

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Nico Kadel-Garcia]

On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > Hello Fedora EPEL maintainers!
>  >
>  > First I don't feel comfortable announcing this, I'm not happy about the
>  > situation and so I don't want to be the lightning rod :-).  But I believe
>  > that we can come to acceptable Copr/Mock solution and this needs to be
>  > discussed...  so here we are.
>  >
>  > By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
>  > goes EOL by then.
>
>
>  I wrote down the possible options and their pros and cons and I done my best to catch all
> the feedback here.
>
>
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>
>  Miroslav 
That seems to be a succinct listing. I think you left out my
suggestion.of "support people re-inventing point releases for CentOS",
which is what major CentOS users will do using internal mirrors. due
to concern about unexpected and unwelcome updates of CentOS Stream,
while they assess whether AlmaLinux or Rocky are reliable and stable
enough to use. It's not an uncommon behavior for EPEL itself, partly
because of EPEL's bad habit of deleting RPMs without warning and
stripping out all previous releases. That's caused me problems with
chromium and firefox when updates were incompatible with contemporary
regression testing systems.

The difficulty with switching mock to AlmaLinux or Rocky is that there
is likely to be significant phase lag with new point releases by Red
Hat, and that it will inflict quite a bandwidth burden for all the
"mock" setups in the field. Can they scale up to handle that?

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Neal Gompa]

On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  >
>  > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > Hello Fedora EPEL maintainers!
>  > >
>  > > First I don't feel comfortable announcing this, I'm not happy about
> the
>  > > situation and so I don't want to be the lightning rod :-).  But I believe
>  > > that we can come to acceptable Copr/Mock solution and this needs to be
>  > > discussed...  so here we are.
>  > >
>  > > By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > goes EOL by then.
>  >
>  >
>  > I wrote down the possible options and their pros and cons and I done my best to
> catch all the feedback here.
>  >
>  >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  >
>  > Miroslav
>
>  That seems to be a succinct listing. I think you left out my
>  suggestion.of "support people re-inventing point releases for CentOS",
>  which is what major CentOS users will do using internal mirrors. due
>  to concern about unexpected and unwelcome updates of CentOS Stream,
>  while they assess whether AlmaLinux or Rocky are reliable and stable
>  enough to use. It's not an uncommon behavior for EPEL itself, partly
>  because of EPEL's bad habit of deleting RPMs without warning and
>  stripping out all previous releases. That's caused me problems with
>  chromium and firefox when updates were incompatible with contemporary
>  regression testing systems.
>  
It's not a "bad habit", it happens because when packages are retired,
keeping the packages there does a disservice to the community by
effectively forcing a maintenance burden when there's no maintainer.
As for stripping out previous releases, that's just how Pungi and
Bodhi do update composes at the moment. Someday that'll be fixed, but
then we'd have to come up with a policy on how many because there are
storage concerns for mirrors if we kept everything published forever.

>  The difficulty with switching mock to AlmaLinux or Rocky is that
> there
>  is likely to be significant phase lag with new point releases by Red
>  Hat, and that it will inflict quite a bandwidth burden for all the
>  "mock" setups in the field. Can they scale up to handle that? 
Insofar as "phase lag with new point releases", AlmaLinux made their
release 48 hours after Red Hat did with RHEL. So, frankly, I'm not
worried there with AlmaLinux.

For bandwidth burdens, mirror networks are designed to alleviate that
burden and both have those in place.




--
真実はいつも一つ！/ Always, there's only one truth!

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Nico Kadel-Garcia]

On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  > >
>  > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > Hello Fedora EPEL maintainers!
>  > > >
>  > > > First I don't feel comfortable announcing this, I'm not happy
> about the
>  > > > situation and so I don't want to be the lightning rod :-).  But I
> believe
>  > > > that we can come to acceptable Copr/Mock solution and this needs to be
>  > > > discussed...  so here we are.
>  > > >
>  > > > By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  > > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > > goes EOL by then.
>  > >
>  > >
>  > > I wrote down the possible options and their pros and cons and I done my best to
> catch all the feedback here.
>  > >
>  > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > >
>  > > Miroslav
>  >
>  > That seems to be a succinct listing. I think you left out my
>  > suggestion.of "support people re-inventing point releases for CentOS",
>  > which is what major CentOS users will do using internal mirrors. due
>  > to concern about unexpected and unwelcome updates of CentOS Stream,
>  > while they assess whether AlmaLinux or Rocky are reliable and stable
>  > enough to use. It's not an uncommon behavior for EPEL itself, partly
>  > because of EPEL's bad habit of deleting RPMs without warning and
>  > stripping out all previous releases. That's caused me problems with
>  > chromium and firefox when updates were incompatible with contemporary
>  > regression testing systems.
>  >
>
>  It's not a "bad habit", it happens because when packages are retired,
>  keeping the packages there does a disservice to the community by
>  effectively forcing a maintenance burden when there's no maintainer.
>  As for stripping out previous releases, that's just how Pungi and
>  Bodhi do update composes at the moment. Someday that'll be fixed, but
>  then we'd have to come up with a policy on how many because there are
>  storage concerns for mirrors if we kept everything published forever. 
It causes problems and confusion for people who need to lock down
evisting versions for deployment. And it happens for packages that are
not retired, but merely updated. I was bitten by it myself with
chromium updates last year. It forces users of EPEL to maintain
internal repos, or out of band access to previously accessible RPMs.
It's destabilizing and breaks the use of bills-of-material based
deployments with complete lists of all desired RPMs.

Storage and bandwidth concerns are legitimate concerns, as is
potentially continuing to publish older releases with known
vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
previously published versions this way, they aggregate new releases. I
consider this a longstanding bug for EPEL, and one of the reasons I
set up internal mirrors in large deployments.

>  > The difficulty with switching mock to AlmaLinux or Rocky is that
> there
>  > is likely to be significant phase lag with new point releases by Red
>  > Hat, and that it will inflict quite a bandwidth burden for all the
>  > "mock" setups in the field. Can they scale up to handle that?
>
>  Insofar as "phase lag with new point releases", AlmaLinux made their
>  release 48 hours after Red Hat did with RHEL. So, frankly, I'm not
>  worried there with AlmaLinux. 
48 hours is pretty danged good. I hesitate to rely on a new OS
publisher with so little track record, even if they've been very good
in their first year. I'm very curious how well they'll do with RHEL
8.6, without a published CentOS example to compare with. And I'd be
very, very wary of the "planning fallacy", where people underestimate
the time of future tasks, despite knowledge that previous tasks have
sometimes taken far longer.

>  For bandwidth burdens, mirror networks are designed to alleviate
> that
>  burden and both have those in place. 
Sure, and they're very helpful. But to the one managing such networks,
a massive increase in bandwidth use or in the *breadth* of content
used for a particular mirrored repo for a particular product is very
helpful to plan for and avoid local choke points. I'm especially
thinking of the cache on relevant proxy servers.

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Neal Gompa]

On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý <msuchy(a)redhat.com&gt;
> wrote:
>  > > >
>  > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > Hello Fedora EPEL maintainers!
>  > > > >
>  > > > > First I don't feel comfortable announcing this, I'm not happy
> about the
>  > > > > situation and so I don't want to be the lightning rod :-).  But I
> believe
>  > > > > that we can come to acceptable Copr/Mock solution and this needs to
> be
>  > > > > discussed...  so here we are.
>  > > > >
>  > > > > By the end of the year 2021 we have to fix our default EPEL 8 Mock
>  > > > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as
> CentOS 8
>  > > > > goes EOL by then.
>  > > >
>  > > >
>  > > > I wrote down the possible options and their pros and cons and I done my
> best to catch all the feedback here.
>  > > >
>  > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > >
>  > > > Miroslav
>  > >
>  > > That seems to be a succinct listing. I think you left out my
>  > > suggestion.of "support people re-inventing point releases for
> CentOS",
>  > > which is what major CentOS users will do using internal mirrors. due
>  > > to concern about unexpected and unwelcome updates of CentOS Stream,
>  > > while they assess whether AlmaLinux or Rocky are reliable and stable
>  > > enough to use. It's not an uncommon behavior for EPEL itself, partly
>  > > because of EPEL's bad habit of deleting RPMs without warning and
>  > > stripping out all previous releases. That's caused me problems with
>  > > chromium and firefox when updates were incompatible with contemporary
>  > > regression testing systems.
>  > >
>  >
>  > It's not a "bad habit", it happens because when packages are retired,
>  > keeping the packages there does a disservice to the community by
>  > effectively forcing a maintenance burden when there's no maintainer.
>  > As for stripping out previous releases, that's just how Pungi and
>  > Bodhi do update composes at the moment. Someday that'll be fixed, but
>  > then we'd have to come up with a policy on how many because there are
>  > storage concerns for mirrors if we kept everything published forever.
>
>  It causes problems and confusion for people who need to lock down
>  evisting versions for deployment. And it happens for packages that are
>  not retired, but merely updated. I was bitten by it myself with
>  chromium updates last year. It forces users of EPEL to maintain
>  internal repos, or out of band access to previously accessible RPMs.
>  It's destabilizing and breaks the use of bills-of-material based
>  deployments with complete lists of all desired RPMs.
>
>  Storage and bandwidth concerns are legitimate concerns, as is
>  potentially continuing to publish older releases with known
>  vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  previously published versions this way, they aggregate new releases. I
>  consider this a longstanding bug for EPEL, and one of the reasons I
>  set up internal mirrors in large deployments.
>  
This is not true. Fedora and EPEL share the same system, and have the
same issues. The only difference is that the release repo is frozen in
Fedora, so only the updates repo is affected this way. So there's at
most two versions of a package at any time.

RHEL *does* maintain multiple old versions, but their system is
completely different and supports that capability.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Nico Kadel-Garcia]

On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt;
> wrote:
>  > > >
>  > > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý <msuchy(a)redhat.com&gt;
> wrote:
>  > > > >
>  > > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > > Hello Fedora EPEL maintainers!
>  > > > > >
>  > > > > > First I don't feel comfortable announcing this, I'm not
> happy about the
>  > > > > > situation and so I don't want to be the lightning rod :-). 
> But I believe
>  > > > > > that we can come to acceptable Copr/Mock solution and this needs
> to be
>  > > > > > discussed...  so here we are.
>  > > > > >
>  > > > > > By the end of the year 2021 we have to fix our default EPEL 8
> Mock
>  > > > > > configuration (mock-core-configs.rpm, /etc/mock/epel-8-*.cfg) as
> CentOS 8
>  > > > > > goes EOL by then.
>  > > > >
>  > > > >
>  > > > > I wrote down the possible options and their pros and cons and I done
> my best to catch all the feedback here.
>  > > > >
>  > > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > > >
>  > > > > Miroslav
>  > > >
>  > > > That seems to be a succinct listing. I think you left out my
>  > > > suggestion.of "support people re-inventing point releases for
> CentOS",
>  > > > which is what major CentOS users will do using internal mirrors. due
>  > > > to concern about unexpected and unwelcome updates of CentOS Stream,
>  > > > while they assess whether AlmaLinux or Rocky are reliable and stable
>  > > > enough to use. It's not an uncommon behavior for EPEL itself, partly
>  > > > because of EPEL's bad habit of deleting RPMs without warning and
>  > > > stripping out all previous releases. That's caused me problems with
>  > > > chromium and firefox when updates were incompatible with contemporary
>  > > > regression testing systems.
>  > > >
>  > >
>  > > It's not a "bad habit", it happens because when packages are
> retired,
>  > > keeping the packages there does a disservice to the community by
>  > > effectively forcing a maintenance burden when there's no maintainer.
>  > > As for stripping out previous releases, that's just how Pungi and
>  > > Bodhi do update composes at the moment. Someday that'll be fixed, but
>  > > then we'd have to come up with a policy on how many because there are
>  > > storage concerns for mirrors if we kept everything published forever.
>  >
>  > It causes problems and confusion for people who need to lock down
>  > evisting versions for deployment. And it happens for packages that are
>  > not retired, but merely updated. I was bitten by it myself with
>  > chromium updates last year. It forces users of EPEL to maintain
>  > internal repos, or out of band access to previously accessible RPMs.
>  > It's destabilizing and breaks the use of bills-of-material based
>  > deployments with complete lists of all desired RPMs.
>  >
>  > Storage and bandwidth concerns are legitimate concerns, as is
>  > potentially continuing to publish older releases with known
>  > vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  > previously published versions this way, they aggregate new releases. I
>  > consider this a longstanding bug for EPEL, and one of the reasons I
>  > set up internal mirrors in large deployments.
>  >
>
>  This is not true. Fedora and EPEL share the same system, and have the
>  same issues. The only difference is that the release repo is frozen in
>  Fedora, so only the updates repo is affected this way. So there's at
>  most two versions of a package at any time. 
You're correct. With the current setup, it's also relatively simple to
revert to the "frozen" release, which handles most of the regression
situations. And Fedora releases are nowhere near so long-lived as RHEL
and EPEL, so it tends to be less of a long-lived problem.

>  RHEL *does* maintain multiple old versions, but their system is
>  completely different and supports that capability. 
What would it take to get Fedora, or at least EPEL, to preserve old
releases in the default published repos? I appreciate that it would
require thought and expand them noticeably, especially for bulky and
frequently updating components like chromium. I admit to not having
numbers on how much churn happens there: does anyone have numbers?

Nico Kadel-Garcia

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Stephen John Smoogen]

On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > > >
>  > > > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt;
> wrote:
>  > > > >
>  > > > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý
> <msuchy(a)redhat.com&gt; wrote:
>  > > > > >
>  > > > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > > > Hello Fedora EPEL maintainers!
>  > > > > > >
>  > > > > > > First I don't feel comfortable announcing this, I'm
> not happy about the
>  > > > > > > situation and so I don't want to be the lightning rod
> :-).  But I believe
>  > > > > > > that we can come to acceptable Copr/Mock solution and this
> needs to be
>  > > > > > > discussed...  so here we are.
>  > > > > > >
>  > > > > > > By the end of the year 2021 we have to fix our default EPEL
> 8 Mock
>  > > > > > > configuration (mock-core-configs.rpm,
> /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > > > > > goes EOL by then.
>  > > > > >
>  > > > > >
>  > > > > > I wrote down the possible options and their pros and cons and I
> done my best to catch all the feedback here.
>  > > > > >
>  > > > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > > > >
>  > > > > > Miroslav
>  > > > >
>  > > > > That seems to be a succinct listing. I think you left out my
>  > > > > suggestion.of "support people re-inventing point releases for
> CentOS",
>  > > > > which is what major CentOS users will do using internal mirrors. due
>  > > > > to concern about unexpected and unwelcome updates of CentOS Stream,
>  > > > > while they assess whether AlmaLinux or Rocky are reliable and stable
>  > > > > enough to use. It's not an uncommon behavior for EPEL itself,
> partly
>  > > > > because of EPEL's bad habit of deleting RPMs without warning and
>  > > > > stripping out all previous releases. That's caused me problems
> with
>  > > > > chromium and firefox when updates were incompatible with
> contemporary
>  > > > > regression testing systems.
>  > > > >
>  > > >
>  > > > It's not a "bad habit", it happens because when packages are
> retired,
>  > > > keeping the packages there does a disservice to the community by
>  > > > effectively forcing a maintenance burden when there's no maintainer.
>  > > > As for stripping out previous releases, that's just how Pungi and
>  > > > Bodhi do update composes at the moment. Someday that'll be fixed, but
>  > > > then we'd have to come up with a policy on how many because there are
>  > > > storage concerns for mirrors if we kept everything published forever.
>  > >
>  > > It causes problems and confusion for people who need to lock down
>  > > evisting versions for deployment. And it happens for packages that are
>  > > not retired, but merely updated. I was bitten by it myself with
>  > > chromium updates last year. It forces users of EPEL to maintain
>  > > internal repos, or out of band access to previously accessible RPMs.
>  > > It's destabilizing and breaks the use of bills-of-material based
>  > > deployments with complete lists of all desired RPMs.
>  > >
>  > > Storage and bandwidth concerns are legitimate concerns, as is
>  > > potentially continuing to publish older releases with known
>  > > vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  > > previously published versions this way, they aggregate new releases. I
>  > > consider this a longstanding bug for EPEL, and one of the reasons I
>  > > set up internal mirrors in large deployments.
>  > >
>  >
>  > This is not true. Fedora and EPEL share the same system, and have the
>  > same issues. The only difference is that the release repo is frozen in
>  > Fedora, so only the updates repo is affected this way. So there's at
>  > most two versions of a package at any time.
>
>  You're correct. With the current setup, it's also relatively simple to
>  revert to the "frozen" release, which handles most of the regression
>  situations. And Fedora releases are nowhere near so long-lived as RHEL
>  and EPEL, so it tends to be less of a long-lived problem.
>
>  > RHEL *does* maintain multiple old versions, but their system is
>  > completely different and supports that capability.
>
>  What would it take to get Fedora, or at least EPEL, to preserve old
>  releases in the default published repos? I appreciate that it would
>  require thought and expand them noticeably, especially for bulky and
>  frequently updating components like chromium. I admit to not having
>  numbers on how much churn happens there: does anyone have numbers? 
In order to keep older package releases, it would require changes to
the compose tool pungi. It would also have to make it so it worked for
EPEL versus Fedora. [Fedora Linux releases have grown to the point
that many mirrors can barely carry the OS as is.. adding in older
packages is out of the question for them.] I do not have numbers on
how often packages churn or which ones churn the most.


-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren

[EPEL-devel]Mock/Coprdefaultepel-8-*configurationtobechanged

[Neal Gompa]

On Mon, Nov 29, 2021 at 6:42 AM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  >
>  > On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt;
> wrote:
>  > > >
>  > > > On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  > > > >
>  > > > > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia
> <nkadel(a)gmail.com&gt; wrote:
>  > > > > >
>  > > > > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý
> <msuchy(a)redhat.com&gt; wrote:
>  > > > > > >
>  > > > > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > > > > Hello Fedora EPEL maintainers!
>  > > > > > > >
>  > > > > > > > First I don't feel comfortable announcing this,
> I'm not happy about the
>  > > > > > > > situation and so I don't want to be the lightning
> rod :-).  But I believe
>  > > > > > > > that we can come to acceptable Copr/Mock solution and
> this needs to be
>  > > > > > > > discussed...  so here we are.
>  > > > > > > >
>  > > > > > > > By the end of the year 2021 we have to fix our default
> EPEL 8 Mock
>  > > > > > > > configuration (mock-core-configs.rpm,
> /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > > > > > > goes EOL by then.
>  > > > > > >
>  > > > > > >
>  > > > > > > I wrote down the possible options and their pros and cons
> and I done my best to catch all the feedback here.
>  > > > > > >
>  > > > > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > > > > >
>  > > > > > > Miroslav
>  > > > > >
>  > > > > > That seems to be a succinct listing. I think you left out my
>  > > > > > suggestion.of "support people re-inventing point releases
> for CentOS",
>  > > > > > which is what major CentOS users will do using internal mirrors.
> due
>  > > > > > to concern about unexpected and unwelcome updates of CentOS
> Stream,
>  > > > > > while they assess whether AlmaLinux or Rocky are reliable and
> stable
>  > > > > > enough to use. It's not an uncommon behavior for EPEL
> itself, partly
>  > > > > > because of EPEL's bad habit of deleting RPMs without warning
> and
>  > > > > > stripping out all previous releases. That's caused me
> problems with
>  > > > > > chromium and firefox when updates were incompatible with
> contemporary
>  > > > > > regression testing systems.
>  > > > > >
>  > > > >
>  > > > > It's not a "bad habit", it happens because when
> packages are retired,
>  > > > > keeping the packages there does a disservice to the community by
>  > > > > effectively forcing a maintenance burden when there's no
> maintainer.
>  > > > > As for stripping out previous releases, that's just how Pungi
> and
>  > > > > Bodhi do update composes at the moment. Someday that'll be fixed,
> but
>  > > > > then we'd have to come up with a policy on how many because there
> are
>  > > > > storage concerns for mirrors if we kept everything published
> forever.
>  > > >
>  > > > It causes problems and confusion for people who need to lock down
>  > > > evisting versions for deployment. And it happens for packages that are
>  > > > not retired, but merely updated. I was bitten by it myself with
>  > > > chromium updates last year. It forces users of EPEL to maintain
>  > > > internal repos, or out of band access to previously accessible RPMs.
>  > > > It's destabilizing and breaks the use of bills-of-material based
>  > > > deployments with complete lists of all desired RPMs.
>  > > >
>  > > > Storage and bandwidth concerns are legitimate concerns, as is
>  > > > potentially continuing to publish older releases with known
>  > > > vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  > > > previously published versions this way, they aggregate new releases. I
>  > > > consider this a longstanding bug for EPEL, and one of the reasons I
>  > > > set up internal mirrors in large deployments.
>  > > >
>  > >
>  > > This is not true. Fedora and EPEL share the same system, and have the
>  > > same issues. The only difference is that the release repo is frozen in
>  > > Fedora, so only the updates repo is affected this way. So there's at
>  > > most two versions of a package at any time.
>  >
>  > You're correct. With the current setup, it's also relatively simple to
>  > revert to the "frozen" release, which handles most of the regression
>  > situations. And Fedora releases are nowhere near so long-lived as RHEL
>  > and EPEL, so it tends to be less of a long-lived problem.
>  >
>  > > RHEL *does* maintain multiple old versions, but their system is
>  > > completely different and supports that capability.
>  >
>  > What would it take to get Fedora, or at least EPEL, to preserve old
>  > releases in the default published repos? I appreciate that it would
>  > require thought and expand them noticeably, especially for bulky and
>  > frequently updating components like chromium. I admit to not having
>  > numbers on how much churn happens there: does anyone have numbers?
>
>  In order to keep older package releases, it would require changes to
>  the compose tool pungi. It would also have to make it so it worked for
>  EPEL versus Fedora. [Fedora Linux releases have grown to the point
>  that many mirrors can barely carry the OS as is.. adding in older
>  packages is out of the question for them.] I do not have numbers on
>  how often packages churn or which ones churn the most.
>  
EPEL is big enough that I begged Troy Dawson to untag duplicate copies
of KDE Plasma so that I could have enough free space for a second
mirror job to work. Between EPEL and EPEL Next for 8, it's pretty
huge. I would say with all the EPEL repositories, it matches the size
of Fedora Linux release trees. I don't think anyone would be too happy
with more. And keep in mind that EPEL trees last for 10 years on the
mirror network before being frozen. That's a lot of updates over the
years.

If Fedora and EPEL were to have older versions, we'd have to have a
dedicated CDN endpoint for them, because mirrors would seriously have
trouble taking it.



-- 
真実はいつも一つ！/ Always, there's only one truth!