Re: Mock/Copr default epel-8-* configuration to be changed

[Stephen John Smoogen]

On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > > >
>  > > > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia <nkadel(a)gmail.com&gt;
> wrote:
>  > > > >
>  > > > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý
> <msuchy(a)redhat.com&gt; wrote:
>  > > > > >
>  > > > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > > > Hello Fedora EPEL maintainers!
>  > > > > > >
>  > > > > > > First I don't feel comfortable announcing this, I'm
> not happy about the
>  > > > > > > situation and so I don't want to be the lightning rod
> :-).  But I believe
>  > > > > > > that we can come to acceptable Copr/Mock solution and this
> needs to be
>  > > > > > > discussed...  so here we are.
>  > > > > > >
>  > > > > > > By the end of the year 2021 we have to fix our default EPEL
> 8 Mock
>  > > > > > > configuration (mock-core-configs.rpm,
> /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > > > > > goes EOL by then.
>  > > > > >
>  > > > > >
>  > > > > > I wrote down the possible options and their pros and cons and I
> done my best to catch all the feedback here.
>  > > > > >
>  > > > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > > > >
>  > > > > > Miroslav
>  > > > >
>  > > > > That seems to be a succinct listing. I think you left out my
>  > > > > suggestion.of "support people re-inventing point releases for
> CentOS",
>  > > > > which is what major CentOS users will do using internal mirrors. due
>  > > > > to concern about unexpected and unwelcome updates of CentOS Stream,
>  > > > > while they assess whether AlmaLinux or Rocky are reliable and stable
>  > > > > enough to use. It's not an uncommon behavior for EPEL itself,
> partly
>  > > > > because of EPEL's bad habit of deleting RPMs without warning and
>  > > > > stripping out all previous releases. That's caused me problems
> with
>  > > > > chromium and firefox when updates were incompatible with
> contemporary
>  > > > > regression testing systems.
>  > > > >
>  > > >
>  > > > It's not a "bad habit", it happens because when packages are
> retired,
>  > > > keeping the packages there does a disservice to the community by
>  > > > effectively forcing a maintenance burden when there's no maintainer.
>  > > > As for stripping out previous releases, that's just how Pungi and
>  > > > Bodhi do update composes at the moment. Someday that'll be fixed, but
>  > > > then we'd have to come up with a policy on how many because there are
>  > > > storage concerns for mirrors if we kept everything published forever.
>  > >
>  > > It causes problems and confusion for people who need to lock down
>  > > evisting versions for deployment. And it happens for packages that are
>  > > not retired, but merely updated. I was bitten by it myself with
>  > > chromium updates last year. It forces users of EPEL to maintain
>  > > internal repos, or out of band access to previously accessible RPMs.
>  > > It's destabilizing and breaks the use of bills-of-material based
>  > > deployments with complete lists of all desired RPMs.
>  > >
>  > > Storage and bandwidth concerns are legitimate concerns, as is
>  > > potentially continuing to publish older releases with known
>  > > vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  > > previously published versions this way, they aggregate new releases. I
>  > > consider this a longstanding bug for EPEL, and one of the reasons I
>  > > set up internal mirrors in large deployments.
>  > >
>  >
>  > This is not true. Fedora and EPEL share the same system, and have the
>  > same issues. The only difference is that the release repo is frozen in
>  > Fedora, so only the updates repo is affected this way. So there's at
>  > most two versions of a package at any time.
>
>  You're correct. With the current setup, it's also relatively simple to
>  revert to the "frozen" release, which handles most of the regression
>  situations. And Fedora releases are nowhere near so long-lived as RHEL
>  and EPEL, so it tends to be less of a long-lived problem.
>
>  > RHEL *does* maintain multiple old versions, but their system is
>  > completely different and supports that capability.
>
>  What would it take to get Fedora, or at least EPEL, to preserve old
>  releases in the default published repos? I appreciate that it would
>  require thought and expand them noticeably, especially for bulky and
>  frequently updating components like chromium. I admit to not having
>  numbers on how much churn happens there: does anyone have numbers? 
In order to keep older package releases, it would require changes to
the compose tool pungi. It would also have to make it so it worked for
EPEL versus Fedora. [Fedora Linux releases have grown to the point
that many mirrors can barely carry the OS as is.. adding in older
packages is out of the question for them.] I do not have numbers on
how often packages churn or which ones churn the most.


-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren

Mock/Copr default epel-8-* configuration to be changed

[Neal Gompa]

On Mon, Nov 29, 2021 at 6:42 AM Stephen John Smoogen <smooge(a)gmail.com&gt; wrote:
>  On Sun, 28 Nov 2021 at 19:32, Nico Kadel-Garcia <nkadel(a)gmail.com&gt; wrote:
>  >
>  > On Sun, Nov 28, 2021 at 7:06 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Thu, Nov 25, 2021 at 2:02 PM Nico Kadel-Garcia <nkadel(a)gmail.com&gt;
> wrote:
>  > > >
>  > > > On Thu, Nov 25, 2021 at 8:26 AM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  > > > >
>  > > > > On Thu, Nov 25, 2021 at 6:19 AM Nico Kadel-Garcia
> <nkadel(a)gmail.com&gt; wrote:
>  > > > > >
>  > > > > > On Thu, Nov 25, 2021 at 3:05 AM Miroslav Suchý
> <msuchy(a)redhat.com&gt; wrote:
>  > > > > > >
>  > > > > > > Dne 22. 11. 21 v 15:00 Pavel Raiskup napsal(a):
>  > > > > > > > Hello Fedora EPEL maintainers!
>  > > > > > > >
>  > > > > > > > First I don't feel comfortable announcing this,
> I'm not happy about the
>  > > > > > > > situation and so I don't want to be the lightning
> rod :-).  But I believe
>  > > > > > > > that we can come to acceptable Copr/Mock solution and
> this needs to be
>  > > > > > > > discussed...  so here we are.
>  > > > > > > >
>  > > > > > > > By the end of the year 2021 we have to fix our default
> EPEL 8 Mock
>  > > > > > > > configuration (mock-core-configs.rpm,
> /etc/mock/epel-8-*.cfg) as CentOS 8
>  > > > > > > > goes EOL by then.
>  > > > > > >
>  > > > > > >
>  > > > > > > I wrote down the possible options and their pros and cons
> and I done my best to catch all the feedback here.
>  > > > > > >
>  > > > > > >
> https://docs.google.com/document/d/1wF7-7_y6Ac_oB-kCFdE6VBWPW8o8zjXd2Z0SG...
>  > > > > > >
>  > > > > > > Miroslav
>  > > > > >
>  > > > > > That seems to be a succinct listing. I think you left out my
>  > > > > > suggestion.of "support people re-inventing point releases
> for CentOS",
>  > > > > > which is what major CentOS users will do using internal mirrors.
> due
>  > > > > > to concern about unexpected and unwelcome updates of CentOS
> Stream,
>  > > > > > while they assess whether AlmaLinux or Rocky are reliable and
> stable
>  > > > > > enough to use. It's not an uncommon behavior for EPEL
> itself, partly
>  > > > > > because of EPEL's bad habit of deleting RPMs without warning
> and
>  > > > > > stripping out all previous releases. That's caused me
> problems with
>  > > > > > chromium and firefox when updates were incompatible with
> contemporary
>  > > > > > regression testing systems.
>  > > > > >
>  > > > >
>  > > > > It's not a "bad habit", it happens because when
> packages are retired,
>  > > > > keeping the packages there does a disservice to the community by
>  > > > > effectively forcing a maintenance burden when there's no
> maintainer.
>  > > > > As for stripping out previous releases, that's just how Pungi
> and
>  > > > > Bodhi do update composes at the moment. Someday that'll be fixed,
> but
>  > > > > then we'd have to come up with a policy on how many because there
> are
>  > > > > storage concerns for mirrors if we kept everything published
> forever.
>  > > >
>  > > > It causes problems and confusion for people who need to lock down
>  > > > evisting versions for deployment. And it happens for packages that are
>  > > > not retired, but merely updated. I was bitten by it myself with
>  > > > chromium updates last year. It forces users of EPEL to maintain
>  > > > internal repos, or out of band access to previously accessible RPMs.
>  > > > It's destabilizing and breaks the use of bills-of-material based
>  > > > deployments with complete lists of all desired RPMs.
>  > > >
>  > > > Storage and bandwidth concerns are legitimate concerns, as is
>  > > > potentially continuing to publish older releases with known
>  > > > vulnerabilities or bugs.  But neither Fedora nor RHEL simply discard
>  > > > previously published versions this way, they aggregate new releases. I
>  > > > consider this a longstanding bug for EPEL, and one of the reasons I
>  > > > set up internal mirrors in large deployments.
>  > > >
>  > >
>  > > This is not true. Fedora and EPEL share the same system, and have the
>  > > same issues. The only difference is that the release repo is frozen in
>  > > Fedora, so only the updates repo is affected this way. So there's at
>  > > most two versions of a package at any time.
>  >
>  > You're correct. With the current setup, it's also relatively simple to
>  > revert to the "frozen" release, which handles most of the regression
>  > situations. And Fedora releases are nowhere near so long-lived as RHEL
>  > and EPEL, so it tends to be less of a long-lived problem.
>  >
>  > > RHEL *does* maintain multiple old versions, but their system is
>  > > completely different and supports that capability.
>  >
>  > What would it take to get Fedora, or at least EPEL, to preserve old
>  > releases in the default published repos? I appreciate that it would
>  > require thought and expand them noticeably, especially for bulky and
>  > frequently updating components like chromium. I admit to not having
>  > numbers on how much churn happens there: does anyone have numbers?
>
>  In order to keep older package releases, it would require changes to
>  the compose tool pungi. It would also have to make it so it worked for
>  EPEL versus Fedora. [Fedora Linux releases have grown to the point
>  that many mirrors can barely carry the OS as is.. adding in older
>  packages is out of the question for them.] I do not have numbers on
>  how often packages churn or which ones churn the most.
>  
EPEL is big enough that I begged Troy Dawson to untag duplicate copies
of KDE Plasma so that I could have enough free space for a second
mirror job to work. Between EPEL and EPEL Next for 8, it's pretty
huge. I would say with all the EPEL repositories, it matches the size
of Fedora Linux release trees. I don't think anyone would be too happy
with more. And keep in mind that EPEL trees last for 10 years on the
mirror network before being frozen. That's a lot of updates over the
years.

If Fedora and EPEL were to have older versions, we'd have to have a
dedicated CDN endpoint for them, because mirrors would seriously have
trouble taking it.



-- 
真実はいつも一つ！/ Always, there's only one truth!

Mock/Copr default epel-8-* configuration to be changed

[Matthew Miller]

On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
>  If Fedora and EPEL were to have older versions, we'd have to have
> a
>  dedicated CDN endpoint for them, because mirrors would seriously have
>  trouble taking it. 
How often would such packages be used? If we had a non-default repo
available but not enabled by default, it could be optional to mirror and
probably still be okay.

-- 
Matthew Miller
<mattdm(a)fedoraproject.org&gt;
Fedora Project Leader

Mock/Copr default epel-8-* configuration to be changed

[Neal Gompa]

On Mon, Nov 29, 2021 at 9:04 AM Matthew Miller <mattdm(a)fedoraproject.org&gt; wrote:
>  On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
>  > If Fedora and EPEL were to have older versions, we'd have to have a
>  > dedicated CDN endpoint for them, because mirrors would seriously have
>  > trouble taking it.
>
>  How often would such packages be used? If we had a non-default repo
>  available but not enabled by default, it could be optional to mirror and
>  probably still be okay.
>  
I suspect it would be fairly heavily used. There are significant
benefits to having those:

* DeltaRPM would be considerably more useful
* "dnf history undo" would actually work

We could make it non-default and tell people that rollbacks require an
--enablerepo= option, though.

Having its own mirror module would also give mirrors the ability to
opt in, and I wouldn't be surprised if a good chunk of them do.


-- 
真実はいつも一つ！/ Always, there's only one truth!

Mock/Copr default epel-8-* configuration to be changed

[Kevin Fenzi]

On Mon, Nov 29, 2021 at 09:11:48AM -0500, Neal Gompa wrote:
>  On Mon, Nov 29, 2021 at 9:04 AM Matthew Miller
> <mattdm(a)fedoraproject.org&gt; wrote:
>  >
>  > On Mon, Nov 29, 2021 at 07:00:30AM -0500, Neal Gompa wrote:
>  > > If Fedora and EPEL were to have older versions, we'd have to have a
>  > > dedicated CDN endpoint for them, because mirrors would seriously have
>  > > trouble taking it.
>  >
>  > How often would such packages be used? If we had a non-default repo
>  > available but not enabled by default, it could be optional to mirror and
>  > probably still be okay. 
So, we actually do have this for fedora updates today. 
We had to set it up to solve an issue with the updates flow and CoreOS.

See the fedora-updates-archive subpackage of fedora-repos. 
This is hosted on amazon S3, behind cloudfront.

>  I suspect it would be fairly heavily used. There are significant
>  benefits to having those:
>  
>  * DeltaRPM would be considerably more useful 
In order to have deltarpms the packages would have to be available at
compose time and in the same repo.

>  * "dnf history undo" would actually work
>  
>  We could make it non-default and tell people that rollbacks require an
>  --enablerepo= option, though. 
This would work for Fedora now with the archive repo above.

I suppose we could look at setting up a similar setup with epel. 

kevin