python-gevent and pytest-cov in el9

[Ken Dreyer]

Hi folks,

The RHEL 9 composes do not have libev-devel and libuv-devel, so we
cannot build python-gevent on EPEL 9 easily.

(It's possible to package the missing -devel packages separately, and
I've been doing this by automatically following the NVR changes in
Stream 9's Koji for several weeks with scripts at
https://github.com/ktdreyer/ceph-el9. My conclusion is that it is so
painful that it's not sustainable to do this for years.)

This means that python-pytest-cov and python-pytest-xdist won't be
available on epel9, since those require gevent.

Several Python packages require python-pytest-cov because upstream
lists it in requirements.txt or tests-requirements.txt. I think we
should just patch these out in Fedora. Even apart from RHEL's
restrictions, it's not a good use of resources to run pytest-cov when
no one reviews coverage reports in the Koji logs, and we'll speed up
builds when mock doesn't have to install this spurious BuildRequires.

Here are a list of packages where I've removed pytest-cov:

https://src.fedoraproject.org/rpms/python-watchdog/pull-request/4
https://src.fedoraproject.org/rpms/python-cheroot/pull-request/15
https://src.fedoraproject.org/rpms/python-portend/pull-request/5
https://src.fedoraproject.org/rpms/python-typing-extensions/pull-request/3

- Ken

[Josh Boyer]

On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt; wrote:
>  Hi folks,
>
>  The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  cannot build python-gevent on EPEL 9 easily. 
https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...

You could request libev-devel in the composes.  I remain confused why
it has to be in the compose though, because libev and it's devel
package are accessible in the CentOS Stream 9 buildroots today.

josh

>  (It's possible to package the missing -devel packages separately,
> and
>  I've been doing this by automatically following the NVR changes in
>  Stream 9's Koji for several weeks with scripts at
>  https://github.com/ktdreyer/ceph-el9. My conclusion is that it is so
>  painful that it's not sustainable to do this for years.)
>
>  This means that python-pytest-cov and python-pytest-xdist won't be
>  available on epel9, since those require gevent.
>
>  Several Python packages require python-pytest-cov because upstream
>  lists it in requirements.txt or tests-requirements.txt. I think we
>  should just patch these out in Fedora. Even apart from RHEL's
>  restrictions, it's not a good use of resources to run pytest-cov when
>  no one reviews coverage reports in the Koji logs, and we'll speed up
>  builds when mock doesn't have to install this spurious BuildRequires.
>
>  Here are a list of packages where I've removed pytest-cov:
>
>  https://src.fedoraproject.org/rpms/python-watchdog/pull-request/4
>  https://src.fedoraproject.org/rpms/python-cheroot/pull-request/15
>  https://src.fedoraproject.org/rpms/python-portend/pull-request/5
>  https://src.fedoraproject.org/rpms/python-typing-extensions/pull-request/3
>
>  - Ken
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Neal Gompa]

On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt; wrote:
>  >
>  > Hi folks,
>  >
>  > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > cannot build python-gevent on EPEL 9 easily.
>
>
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>
>  You could request libev-devel in the composes.  I remain confused why
>  it has to be in the compose though, because libev and it's devel
>  package are accessible in the CentOS Stream 9 buildroots today.
>  
We can't use them in EPEL if they're not in CRB.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Josh Boyer]

On Fri, Sep 24, 2021 at 4:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  >
>  > On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt; wrote:
>  > >
>  > > Hi folks,
>  > >
>  > > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > > cannot build python-gevent on EPEL 9 easily.
>  >
>  >
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>  >
>  > You could request libev-devel in the composes.  I remain confused why
>  > it has to be in the compose though, because libev and it's devel
>  > package are accessible in the CentOS Stream 9 buildroots today.
>  >
>
>  We can't use them in EPEL if they're not in CRB. 
Yes, that's what everyone keeps telling me.  I don't understand why.

josh

[Neal Gompa]

On Fri, Sep 24, 2021 at 4:03 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Fri, Sep 24, 2021 at 4:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > >
>  > > On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt;
> wrote:
>  > > >
>  > > > Hi folks,
>  > > >
>  > > > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > > > cannot build python-gevent on EPEL 9 easily.
>  > >
>  > >
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>  > >
>  > > You could request libev-devel in the composes.  I remain confused why
>  > > it has to be in the compose though, because libev and it's devel
>  > > package are accessible in the CentOS Stream 9 buildroots today.
>  > >
>  >
>  > We can't use them in EPEL if they're not in CRB.
>
>  Yes, that's what everyone keeps telling me.  I don't understand why.
>  
Well, because outside of RHEL, everyone wants remote and local builds
to have access to the same resources and not crush the servers. Since
buildroot stuff isn't going out on the mirror network (otherwise, why
would it be separate from CRB?), it's obvious we shouldn't rely on it
for packages that people should expect to be able to build and rebuild
for RHEL.

And again, by Red Hat's own sword (policy), RHEL doesn't want to ship
everything needed to build stuff, so if EPEL is intended to provide
the requisite community guarantees (reproducibly buildable), we have
to work with what RHEL gives us.




--
真実はいつも一つ！/ Always, there's only one truth!

[Josh Boyer]

On Fri, Sep 24, 2021 at 4:09 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  On Fri, Sep 24, 2021 at 4:03 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  >
>  > On Fri, Sep 24, 2021 at 4:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  > >
>  > > On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > > >
>  > > > On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt;
> wrote:
>  > > > >
>  > > > > Hi folks,
>  > > > >
>  > > > > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > > > > cannot build python-gevent on EPEL 9 easily.
>  > > >
>  > > >
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>  > > >
>  > > > You could request libev-devel in the composes.  I remain confused why
>  > > > it has to be in the compose though, because libev and it's devel
>  > > > package are accessible in the CentOS Stream 9 buildroots today.
>  > > >
>  > >
>  > > We can't use them in EPEL if they're not in CRB.
>  >
>  > Yes, that's what everyone keeps telling me.  I don't understand why.
>  >
>
>  Well, because outside of RHEL, everyone wants remote and local builds
>  to have access to the same resources and not crush the servers. Since
>  buildroot stuff isn't going out on the mirror network (otherwise, why
>  would it be separate from CRB?), it's obvious we shouldn't rely on it
>  for packages that people should expect to be able to build and rebuild
>  for RHEL. 
So you have access to what you want, you have a way to pull it down
and get it locally, but you can't depend on it because... you're
worried a multi-billion dollar company can't pay it's server and CDN
bills?

As to why it's separate from CRB, that's because CRB is a reflection
of what is provided as part of the product.  It's that simple.

>  And again, by Red Hat's own sword (policy), RHEL doesn't want
> to ship
>  everything needed to build stuff, so if EPEL is intended to provide
>  the requisite community guarantees (reproducibly buildable), we have
>  to work with what RHEL gives us. 
I think that is also EPEL falling on EPEL's own sword a bit.  I think
it fails to recognize that building and distributing software can be
separate things.  I can see the need for a developer community to be
able to build, update, and rebuild software it distributes.  Access to
the buildroots facilitates this.  We could even point mock configs at
it, or propose a buildroot repo for it if people are really worried
about "servers".

However, in the context of something like python-gevent, an EPEL *end
user* isn't going to want libuv-devel or libev-devel to be installed
on their system at runtime.  They have no need for it to be available
in a compose.  They only need python-gevent and the requisite runtime
libraries, which are already provided.  I think separating the
personas and thinking about the requirements for each might be worth
doing.

I understand this is a different approach and something that looks
different from the past.  It's been 2+ years since the OS EPEL8 is
based on has shipped and it is taking a different approach than
previous releases.  Every indication we have shows the next major
version will continue this.  I'm worried that sticking with past
policies precludes EPEL from making progress on a project that we all
want to see succeed.

josh

[Troy Dawson]

On Fri, Sep 24, 2021 at 1:03 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:

>  On Fri, Sep 24, 2021 at 4:02 PM Neal Gompa <ngompa13(a)gmail.com&gt;
> wrote:
>  >
>  > On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > >
>  > > On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt;
>  wrote:
>  > > >
>  > > > Hi folks,
>  > > >
>  > > > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > > > cannot build python-gevent on EPEL 9 easily.
>  > >
>  > >
>
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>  > >
>  > > You could request libev-devel in the composes.  I remain confused why
>  > > it has to be in the compose though, because libev and it's devel
>  > > package are accessible in the CentOS Stream 9 buildroots today.
>  > >
>  >
>  > We can't use them in EPEL if they're not in CRB.
>
>  Yes, that's what everyone keeps telling me.  I don't understand why.
>  
EPEL9 will not be built again CentOS Stream 9, it will be built off RHEL
9.  That is the EPEL policy.
Thus, if something is not in RHEL 9, it is not available to be built in
EPEL 9.

It seems pretty straightforward.

Troy

[Carl George]

On Fri, Sep 24, 2021 at 3:05 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  On Fri, Sep 24, 2021 at 4:02 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >
>  > On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
>  > >
>  > > On Fri, Sep 24, 2021 at 3:46 PM Ken Dreyer <ktdreyer(a)ktdreyer.com&gt;
> wrote:
>  > > >
>  > > > Hi folks,
>  > > >
>  > > > The RHEL 9 composes do not have libev-devel and libuv-devel, so we
>  > > > cannot build python-gevent on EPEL 9 easily.
>  > >
>  > >
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
>  > >
>  > > You could request libev-devel in the composes.  I remain confused why
>  > > it has to be in the compose though, because libev and it's devel
>  > > package are accessible in the CentOS Stream 9 buildroots today.
>  > >
>  >
>  > We can't use them in EPEL if they're not in CRB.
>
>  Yes, that's what everyone keeps telling me.  I don't understand why. 
EPEL builds against published RHEL content, not the CentOS Stream
buildroot.  Having a package available in the CentOS Stream buildroot
doesn't make it accessible to EPEL builds.  We can't change EPEL to
use the CentOS Stream buildroot because that will cause some EPEL
packages to not be installable on RHEL.

On a related note, EPEL 9 Next _is_ being set up to build against the
CentOS Stream 9 buildroot.  This works for EPEL Next because it
explicitly targets the next minor release of RHEL (i.e. CentOS
Stream).  This will allow more packages to be built, at the cost of
potentially confusing packagers when their package builds successfully
for EPEL 9 Next but not for EPEL 9.  EPEL 8 Next currently builds
against published CentOS Stream 8 content.  If things go well with
EPEL 9 Next using the CentOS Stream 9 buildroot, EPEL 8 Next may
switch to using the CentOS Stream 8 buildroot in the future.

>  josh
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Ken Dreyer]

On Fri, Sep 24, 2021 at 3:59 PM Josh Boyer <jwboyer(a)redhat.com&gt; wrote:
> https://odcs.stream.centos.org/production/CentOS-Stream-9-20210924.0/comp...
On the one hand, thank you for pointing out that this build is now
available. That's good to know.

On the other hand, this points at the bigger issue that dealing with
the entire problem of missing packages requires a level of scripting
and bookkeeping that is very difficult to keep up when building
layered projects.

>  You could request libev-devel in the composes. 
The reason I did not do that in this case is that pytest-cov is an
optional dependency, and we can just remove it from the Python
packages instead. I'd rather reduce the dependencies on gevent to make
everything faster.

When I looked at gevent in EPEL 8 a month or so ago, it did not look
like many packages depended on it.

>  I remain confused why
>  it has to be in the compose though, because libev and it's devel
>  package are accessible in the CentOS Stream 9 buildroots today. 
We could point at
https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/ ,
but that location will not have GPG-signed builds, and the repo is not
currently in
https://github.com/rpm-software-management/mock/blob/main/mock-core-confi...

- Ken

[Miro Hrončok]

On 24. 09. 21 21:45, Ken Dreyer wrote:
>  This means that python-pytest-cov and python-pytest-xdist won't
> be
>  available on epel9, since those require gevent. 
Ignoring the rest of your email for now, but I don't the gevent dependency does 
not exsist:

$ [dnf] install python3-pytest-cov python3-pytest-xdist
...
Installed:
expat-2.4.1-2.fc35.x86_64
mpdecimal-2.5.1-2.fc35.x86_64
openssl1.1-1:1.1.1l-1.fc36.x86_64
python-pip-wheel-21.2.3-2.fc36.noarch
python-setuptools-wheel-57.4.0-1.fc35.noarch
python3-3.10.0~rc2-2.fc36.x86_64
python3-attrs-21.2.0-4.fc35.noarch
python3-coverage-5.6-0.3b1.fc35.x86_64
python3-execnet-1.9.0-2.fc35.noarch
python3-iniconfig-1.1.1-5.fc35.noarch
python3-libs-3.10.0~rc2-2.fc36.x86_64
python3-packaging-21.0-2.fc35.noarch
python3-pluggy-1.0.0-1.fc36.noarch
python3-py-1.10.0-5.fc35.noarch
python3-pyparsing-2.4.7-9.fc35.noarch
python3-pytest-6.2.5-1.fc36.noarch
python3-pytest-cov-2.12.1-1.fc36.noarch
python3-pytest-forked-1.3.0-4.fc35.noarch
python3-pytest-xdist-2.4.0-1.fc36.noarch
python3-setuptools-57.4.0-1.fc35.noarch
python3-toml-0.10.2-5.fc35.noarch


-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Ken Dreyer]

On Sun, Sep 26, 2021 at 2:25 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 24. 09. 21 21:45, Ken Dreyer wrote:
>  > This means that python-pytest-cov and python-pytest-xdist won't be
>  > available on epel9, since those require gevent.
>
>  Ignoring the rest of your email for now, but I don't the gevent dependency does
>  not exsist: 
You're right, gevent is not a runtime requirement. pytest-xdist
requires execnet, but execnet only BuildRequires gevent for the unit
tests. So we could build execnet on el9 if we disabled %check and the
"execmodel=gevent" feature. Maybe that's fine, since
https://github.com/pytest-dev/execnet is almost unmaintained upstream,
and users have opened unaddressed GitHub tickets in execnet and xdist
regarding gevent hangs.

This is another reason why I want to reduce Fedora's dependence on
pytest-xdist and pytest-cov, if execnet is growing more unstable.

- Ken

To buildroot, or not to buildroot

[Troy Dawson]

This was originally "python-gevent and pytest-cov in el9"
But I wanted to talk about and discuss this paragraph from Carl on it's own.

On Fri, Sep 24, 2021 at 7:43 PM Carl George <carl(a)redhat.com&gt; wrote:

>  On a related note, EPEL 9 Next _is_ being set up to build against
> the
>  CentOS Stream 9 buildroot.  This works for EPEL Next because it
>  explicitly targets the next minor release of RHEL (i.e. CentOS
>  Stream).  This will allow more packages to be built, at the cost of
>  potentially confusing packagers when their package builds successfully
>  for EPEL 9 Next but not for EPEL 9.  EPEL 8 Next currently builds
>  against published CentOS Stream 8 content.  If things go well with
>  EPEL 9 Next using the CentOS Stream 9 buildroot, EPEL 8 Next may
>  switch to using the CentOS Stream 8 buildroot in the future.
>  
Although I'm not against using the CentOS Stream buildroot for epel9-next,
we're already starting to see large sections of KDE that WILL be buildable
against the buildroot but WILL NOT be buildable against plain epel9.[1]

This isn't a matter of "confusing packagers", it's a matter of major
delays
when we switch from epel9-next to epel9.

I'll have to file all those "please move these packages into CRB" bugz
after RHEL9 is out.
I'll also have to request "branch and builds" for all those packages in the
buildroot that are ONLY in the buildroot.
And then request the dependencies for those packages.  And so forth.

And that's just the packages I am working on.  I can't speak for anyone
else.

[possible solution]

If we are going to have the CS buildroots in any of our epel-next repo's,
we need to develop a "missing build-dep" service.

Something that regularly checks to make sure epel9-next packages are built
against packages that will be in epel9.  And if they are not buildable,
let's the package maintainer know.
And let Red Hat know as well.
A nice list that people can easily look at and see which "missing
build-dep" is being used, and by what packages.

Troy

p.s.  I'm sorry if this email sounds accusatory and/or negative.  I've
re-written it twice and it still sounds negative to me.  Possibly cuz I'm
showing the bad things that can/will happen.  But I want to put the
emphasis on the possible solution.  More of a "hay, let's see if this would
work."  So, if someone finds this hurting their feelings, I'm sorry.  At
this point I just need to send it, or I never will.

[1] - https://bugzilla.redhat.com/show_bug.cgi?id=2009831

To buildroot, or not to buildroot

[Miro Hrončok]

On 01. 10. 21 22:11, Troy Dawson wrote:
>  I'll have to file all those "please move these packages into
> CRB" bugz after 
>  RHEL9 is out. 
I wonder whether we should file those before it is out, if at all possible?

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

To buildroot, or not to buildroot

[Neal Gompa]

On Fri, Oct 1, 2021 at 5:39 PM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  On 01. 10. 21 22:11, Troy Dawson wrote:
>  > I'll have to file all those "please move these packages into CRB" bugz
> after
>  > RHEL9 is out.
>
>  I wonder whether we should file those before it is out, if at all possible?
>  
Ideally, yes because it's an extreme pain to deal with them post-GA.
Red Hat's policies around this stuff is such that after 9.0, they can
only be introduced in 6 month intervals (that is, we have to wait for
9.1 for them to show up in EPEL9). We should *never* use the buildroot
repository for EPEL-Next because it massively complicates discovering
this stuff and getting it straightened out immediately. Unlike CentOS
SIGs that only build for CentOS Stream, EPEL needs to work on RHEL,
and Red Hat will *never* ship the buildroot repo for RHEL, because it
opens up the door for things to depend on that content, which is
highly problematic for them.




--
真実はいつも一つ！/ Always, there's only one truth!

To buildroot, or not to buildroot

[Troy Dawson]

*this is worth a discussion in todays EPEL Steering Committee Meeting*

It sounds like the epel9-next is going to startup by building against the
CS buildroot.  Changing it at this time would cause a delay.

Thus we need to write some "verify build deps are released" checker.  I
have an idea of how to do this, so I'm willing to volunteer to write and
run something.

But, it would be good to have some discussion to determine if we want to
keep using the CS buildroot for epel9-next, always.  Or if we want to use
it just as a bootstrap mechanism, and then switch to building just off the
available CentOS Stream repos at some point.

Thoughts?
Should we always use buildroot?  Or just keep up until we're fairly stable?

Troy

To buildroot, or not to buildroot

[Neal Gompa]

On Wed, Oct 6, 2021 at 3:36 PM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
>  *this is worth a discussion in todays EPEL Steering Committee Meeting*
>
>  It sounds like the epel9-next is going to startup by building against the CS buildroot. 
> Changing it at this time would cause a delay.
>
>  Thus we need to write some "verify build deps are released" checker.  I have an
> idea of how to do this, so I'm willing to volunteer to write and run something.
>
>  But, it would be good to have some discussion to determine if we want to keep using the
> CS buildroot for epel9-next, always.  Or if we want to use it just as a bootstrap
> mechanism, and then switch to building just off the available CentOS Stream repos at some
> point.
>
>  Thoughts?
>  Should we always use buildroot?  Or just keep up until we're fairly stable?
>  
We should only use the buildroot repo for as long as we need to. The
*sooner* we can switch to the published content, the better.



-- 
真実はいつも一つ！/ Always, there's only one truth!

To buildroot, or not to buildroot

[Troy Dawson]

On Wed, Oct 6, 2021 at 7:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  On Wed, Oct 6, 2021 at 3:36 PM Troy Dawson <tdawson(a)redhat.com&gt;
> wrote:
>  >
>  > *this is worth a discussion in todays EPEL Steering Committee Meeting*
>  >
>  > It sounds like the epel9-next is going to startup by building against
>  the CS buildroot.  Changing it at this time would cause a delay.
>  >
>  > Thus we need to write some "verify build deps are released" checker.  I
>  have an idea of how to do this, so I'm willing to volunteer to write and
>  run something.
>  >
>  > But, it would be good to have some discussion to determine if we want to
>  keep using the CS buildroot for epel9-next, always.  Or if we want to use
>  it just as a bootstrap mechanism, and then switch to building just off the
>  available CentOS Stream repos at some point.
>  >
>  > Thoughts?
>  > Should we always use buildroot?  Or just keep up until we're fairly
>  stable?
>  >
>
>  We should only use the buildroot repo for as long as we need to. The
>  *sooner* we can switch to the published content, the better.
>  
This was discussed at the EPEL Steering Committee meeting.  Here is the
summary.
Someone correct me if I'm wrong.

epel9-next:
 - starts off building off CS buildroot
 - I will write a "check if all build packages are in the normal repos"
checker, called "will it build" [1]

epel9:
 - Use normal RHEL 9 repos (AppStream, BaseOS, CRB)

Checks/Tests/Future:  (It's a little fuzzy on the timing of these)

 - grobisplitter
 -- see if we really need to use grobisplitter
 -- I'm a little fuzzy on how or when we are going to test this

 - "will it build"
 -- After a period of time, check and see if we are happy with the
combination of CS buildroot + "will it build" ?
 --- Yes - determine if we want to keep it like this even after epel9
 --- No - determine when the best/safest time to switch to normal repos

Troy

[1] - I plan on integrating "will it build" with a "will it install"
checker.  It will be called "will it" or "willit".  I'm sorta
excited about
doing this.  It's something I've wanted to do for a long time and I think
I've finally got it figured out.

To buildroot, or not to buildroot

[Neal Gompa]

On Thu, Oct 7, 2021 at 9:24 AM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
>  On Wed, Oct 6, 2021 at 7:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >
> > On Wed, Oct 6, 2021 at 3:36 PM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
> > >
> > > *this is worth a discussion in todays EPEL Steering Committee Meeting*
> > >
> > > It sounds like the epel9-next is going to startup by building against the CS
> buildroot.  Changing it at this time would cause a delay.
> > >
> > > Thus we need to write some "verify build deps are released" checker. 
> I have an idea of how to do this, so I'm willing to volunteer to write and run
> something.
> > >
> > > But, it would be good to have some discussion to determine if we want to keep
> using the CS buildroot for epel9-next, always.  Or if we want to use it just as a
> bootstrap mechanism, and then switch to building just off the available CentOS Stream
> repos at some point.
> > >
> > > Thoughts?
> > > Should we always use buildroot?  Or just keep up until we're fairly stable?
> > >
> >
> > We should only use the buildroot repo for as long as we need to. The
> > *sooner* we can switch to the published content, the better.
>
>
>  This was discussed at the EPEL Steering Committee meeting.  Here is the summary.
>  Someone correct me if I'm wrong.
>
>  epel9-next:
>   - starts off building off CS buildroot
>   - I will write a "check if all build packages are in the normal repos"
> checker, called "will it build" [1]
>  
How are we going to know whether all the build-time and run-time
packages are in the normal repos? We need to check generated
dependencies too, especially now that it's possible to have dynamic
BuildRequires!

>  epel9:
>   - Use normal RHEL 9 repos (AppStream, BaseOS, CRB)
>
>  Checks/Tests/Future:  (It's a little fuzzy on the timing of these)
>
>   - grobisplitter
>   -- see if we really need to use grobisplitter
>   -- I'm a little fuzzy on how or when we are going to test this
>  
With the retirement of the container-tools default module,
grobisplitter will not be required at all unless we want to use it to
support non-default modules.



--
真実はいつも一つ！/ Always, there's only one truth!

To buildroot, or not to buildroot

[Troy Dawson]

On Thu, Oct 7, 2021 at 6:28 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:

>  On Thu, Oct 7, 2021 at 9:24 AM Troy Dawson <tdawson(a)redhat.com&gt;
> wrote:
>  >
>  >
>  >
>  > On Wed, Oct 6, 2021 at 7:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  >>
>  >> On Wed, Oct 6, 2021 at 3:36 PM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
>  >> >
>  >> > *this is worth a discussion in todays EPEL Steering Committee Meeting*
>  >> >
>  >> > It sounds like the epel9-next is going to startup by building against
>  the CS buildroot.  Changing it at this time would cause a delay.
>  >> >
>  >> > Thus we need to write some "verify build deps are released"
> checker.
>  I have an idea of how to do this, so I'm willing to volunteer to write and
>  run something.
>  >> >
>  >> > But, it would be good to have some discussion to determine if we want
>  to keep using the CS buildroot for epel9-next, always.  Or if we want to
>  use it just as a bootstrap mechanism, and then switch to building just off
>  the available CentOS Stream repos at some point.
>  >> >
>  >> > Thoughts?
>  >> > Should we always use buildroot?  Or just keep up until we're fairly
>  stable?
>  >> >
>  >>
>  >> We should only use the buildroot repo for as long as we need to. The
>  >> *sooner* we can switch to the published content, the better.
>  >
>  >
>  > This was discussed at the EPEL Steering Committee meeting.  Here is the
>  summary.
>  > Someone correct me if I'm wrong.
>  >
>  > epel9-next:
>  >  - starts off building off CS buildroot
>  >  - I will write a "check if all build packages are in the normal repos"
>  checker, called "will it build" [1]
>  >
>
>  How are we going to know whether all the build-time and run-time
>  packages are in the normal repos? We need to check generated
>  dependencies too, especially now that it's possible to have dynamic
>  BuildRequires!
>  
run-time dependencies:
That's always been a problem, even without the buildroot.
But I will also be writing a "will it install" to go along with "will it
build"

build-time dependencies:
Grab the root.log of the package build, and parse it.
This gets around any hidden and dynamic BuildRequires.
I've already written code that does this for Content Resolver, and checked
it against traditional dnf/libsolve dependency generation.
It was 98% equal, and those 2% were on packages where it was possible for
more than one package to be installed for a dependency, and for that, I'd
prefer going with the root.log.

I think I've got everything I need already written, just in three separate
projects.
I really want to pull that code together and make "willit"


>  > epel9:
>  >  - Use normal RHEL 9 repos (AppStream, BaseOS, CRB)
>  >
>  > Checks/Tests/Future:  (It's a little fuzzy on the timing of these)
>  >
>  >  - grobisplitter
>  >  -- see if we really need to use grobisplitter
>  >  -- I'm a little fuzzy on how or when we are going to test this
>  >
>
>  With the retirement of the container-tools default module,
>  grobisplitter will not be required at all unless we want to use it to
>  support non-default modules.
>  
That is the theory, yes, that grobisplitter isn't required.
But nobody was able to say that was for certain.  Thus, it needs to be
tested.

Troy

To buildroot, or not to buildroot

[Neal Gompa]

On Thu, Oct 7, 2021 at 9:44 AM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
>  On Thu, Oct 7, 2021 at 6:28 AM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> >
> > On Thu, Oct 7, 2021 at 9:24 AM Troy Dawson <tdawson(a)redhat.com&gt; wrote:
> > >
> > >
> > >
> > > On Wed, Oct 6, 2021 at 7:39 PM Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
> > >>
> > >> On Wed, Oct 6, 2021 at 3:36 PM Troy Dawson <tdawson(a)redhat.com&gt;
> wrote:
> > >> >
> > >> > *this is worth a discussion in todays EPEL Steering Committee Meeting*
> > >> >
> > >> > It sounds like the epel9-next is going to startup by building against
> the CS buildroot.  Changing it at this time would cause a delay.
> > >> >
> > >> > Thus we need to write some "verify build deps are released"
> checker.  I have an idea of how to do this, so I'm willing to volunteer to write and
> run something.
> > >> >
> > >> > But, it would be good to have some discussion to determine if we want
> to keep using the CS buildroot for epel9-next, always.  Or if we want to use it just as a
> bootstrap mechanism, and then switch to building just off the available CentOS Stream
> repos at some point.
> > >> >
> > >> > Thoughts?
> > >> > Should we always use buildroot?  Or just keep up until we're fairly
> stable?
> > >> >
> > >>
> > >> We should only use the buildroot repo for as long as we need to. The
> > >> *sooner* we can switch to the published content, the better.
> > >
> > >
> > > This was discussed at the EPEL Steering Committee meeting.  Here is the
> summary.
> > > Someone correct me if I'm wrong.
> > >
> > > epel9-next:
> > >  - starts off building off CS buildroot
> > >  - I will write a "check if all build packages are in the normal
> repos" checker, called "will it build" [1]
> > >
> >
> > How are we going to know whether all the build-time and run-time
> > packages are in the normal repos? We need to check generated
> > dependencies too, especially now that it's possible to have dynamic
> > BuildRequires!
>
>
>  run-time dependencies:
>  That's always been a problem, even without the buildroot.
>  But I will also be writing a "will it install" to go along with "will it
> build"
>
>  build-time dependencies:
>  Grab the root.log of the package build, and parse it.
>  This gets around any hidden and dynamic BuildRequires.
>  I've already written code that does this for Content Resolver, and checked it against
> traditional dnf/libsolve dependency generation.
>  It was 98% equal, and those 2% were on packages where it was possible for more than one
> package to be installed for a dependency, and for that, I'd prefer going with the
> root.log.
>
>  I think I've got everything I need already written, just in three separate projects.
>  I really want to pull that code together and make "willit"
>
> >
> > > epel9:
> > >  - Use normal RHEL 9 repos (AppStream, BaseOS, CRB)
> > >
> > > Checks/Tests/Future:  (It's a little fuzzy on the timing of these)
> > >
> > >  - grobisplitter
> > >  -- see if we really need to use grobisplitter
> > >  -- I'm a little fuzzy on how or when we are going to test this
> > >
> >
> > With the retirement of the container-tools default module,
> > grobisplitter will not be required at all unless we want to use it to
> > support non-default modules.
>
>
>  That is the theory, yes, that grobisplitter isn't required.
>  But nobody was able to say that was for certain.  Thus, it needs to be tested.
>  
I've verified this with my internal build infrastructure, so yes, I
know it's not required.

Admittedly, it's not a Koji system, but I'm also not enabling any
modules in my build environment right now. I'm rebuilding content from
Rawhide targeting CentOS Stream 9 to get a list of initial EPEL 9
packages to build for work, which is how some of my requests to add
stuff to CRB have come about[1][2][3].

This can also be verified when using something like mock with
mock-core-configs v36 or higher, because I made the necessary
adjustments to test building on CentOS Stream 9 the same way that
Fedora Koji and the CentOS CBS would.

[1]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...
[2]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...
[3]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...



-- 
真実はいつも一つ！/ Always, there's only one truth!

To buildroot, or not to buildroot

[Stephen John Smoogen]

On Thu, 7 Oct 2021 at 09:52, Neal Gompa <ngompa13(a)gmail.com&gt; wrote:
>  
>
>  
> > That is the theory, yes, that grobisplitter isn't
required.
> > But nobody was able to say that was for certain.  Thus, it needs to be tested.
>
>  
>  
> I've verified this with my
internal build infrastructure, so yes, I
> know it's not required.
>  
> Admittedly, it's not a Koji system, but I'm also
not enabling any
> modules in my build environment right now. I'm rebuilding content from
> Rawhide targeting CentOS Stream 9 to get a list of initial EPEL 9
> packages to build for work, which is how some of my requests to add
> stuff to CRB have come about[1][2][3].

Things have probably improved, but the lesson I learned from EPEL-8
and afterwords was that koji depsolving is weird no matter how set up.
Koji sets up mock environments in a way that will do fine as long as
there are NO modules in the repositories it is looking at. Once a
module, even a non-default module, is there things start to go wonky
because the way that koji depsolves will say that
'foobaz-3.10.1-3+module' is a better solution than
'foobax-3.9.4.<arch>' it will then try to pull that in and boom you
end up with broken builds. You can change the method that koji chooses
packages to be in the buildroot, but the other option ends up trying
to insert things like foobax-3.9.4-i386 into an x86_64  or still does
the modular change but chooses foobar-2 due to some depsolv quirk.

At the moment I think building without grobisplitter will work, but I
am thinking some other solution will need to be made when EL9.x starts
rolling out with modules in it.

>  
> This can also be verified when using something like mock
with
> mock-core-configs v36 or higher, because I made the necessary
> adjustments to test building on CentOS Stream 9 the same way that
> Fedora Koji and the CentOS CBS would.
>  
> [1]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...
> [2]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...
> [3]:
https://gitlab.com/redhat/centos-stream/release-engineering/comps/-/merge...
>  



-- 
Stephen J Smoogen.
I've seen things you people wouldn't believe. Flame wars in
sci.astro.orion. I have seen SPAM filters overload because of Godwin's
Law. All those moments will be lost in time... like posts on a BBS...
time to shutdown -h now.

To buildroot, or not to buildroot

[Andrew C Aitchison]

On Thu, 7 Oct 2021, Stephen John Smoogen wrote:
>  Things have probably improved, but the lesson I learned from EPEL-8
>  and afterwords was that koji depsolving is weird no matter how set up.
>  Koji sets up mock environments in a way that will do fine as long as
>  there are NO modules in the repositories it is looking at. Once a
>  module, even a non-default module, is there things start to go wonky
>  because the way that koji depsolves will say that
>  'foobaz-3.10.1-3+module' is a better solution than
>  'foobax-3.9.4.<arch>' it will then try to pull that in and boom you
>  end up with broken builds. You can change the method that koji chooses
>  packages to be in the buildroot, but the other option ends up trying
>  to insert things like foobax-3.9.4-i386 into an x86_64  or still does
>  the modular change but chooses foobar-2 due to some depsolv quirk. 
Is the foobar/foobaz/foobax intended or are they typos ?
If they are intended, then I think we have a partially ordered set of
packages, ie it isn't always possible to say whether a > b or a < b.

Without knowing anything about koji, I'd say that whilst
foobaz and foobax can provide foobar (and perhaps foobar==3)
they cannot satisfy foobar>=3.9 (unless they are forks or 
reimplementations of foobar-3.9).

-- 
Andrew C. Aitchison					Kendal, UK
 			andrew(a)aitchison.me.uk

To buildroot, or not to buildroot

[Stephen John Smoogen]

On Tue, 19 Oct 2021 at 01:22, Andrew C Aitchison <andrew(a)aitchison.me.uk&gt; wrote:
>  On Thu, 7 Oct 2021, Stephen John Smoogen wrote:
>  > Things have probably improved, but the lesson I learned from EPEL-8
>  > and afterwords was that koji depsolving is weird no matter how set up.
>  > Koji sets up mock environments in a way that will do fine as long as
>  > there are NO modules in the repositories it is looking at. Once a
>  > module, even a non-default module, is there things start to go wonky
>  > because the way that koji depsolves will say that
>  > 'foobaz-3.10.1-3+module' is a better solution than
>  > 'foobax-3.9.4.<arch>' it will then try to pull that in and boom you
>  > end up with broken builds. You can change the method that koji chooses
>  > packages to be in the buildroot, but the other option ends up trying
>  > to insert things like foobax-3.9.4-i386 into an x86_64  or still does
>  > the modular change but chooses foobar-2 due to some depsolv quirk.
>
>  Is the foobar/foobaz/foobax intended or are they typos ?
>  If they are intended, then I think we have a partially ordered set of
>  packages, ie it isn't always possible to say whether a > b or a < b.
>
>  Without knowing anything about koji, I'd say that whilst
>  foobaz and foobax can provide foobar (and perhaps foobar==3)
>  they cannot satisfy foobar>=3.9 (unless they are forks or
>  reimplementations of foobar-3.9).
>  
In the case I was remembering they were forks and say inside them that
they satisfy anything >= 3.9. Normally you would only have one in a
buildroot but you could have alternatives in modules because they each
fixed something specific to that module. If you tried to install both
modules you would get conflicts so only one module was to be installed
at a time.. However, Koji doesn't know that.. one solution method only
knows NEVR, Provides and Requires and another tries to use a bit of
dnf to do resolutions but that seemed to act weirdly also.

In the case where they are all foobar but say foobar-4.0 or foobar4 is
in a non-default module, then koji and dnf might come again to
different conclusions of what is needed to be in the buildroot. The
only mental solution I could come up with was that there would need to
be some tool to put all modules (and bare rpms dependent on them) into
a RHEL-modularity tree and somehow import that into MBS so that it
could deal with them. That would then allow for both EPEL modules to
depend on them, and keep them out of the way for koji regular builds.
The major problems with that was
a) no perl/python/ruby/etc in non-modular builds (all that would be
available is platform-python)
b) it would break Fedora build system in ways because it assumes what
is in its toolkit was built by it and we are faking these.

In any case, it is a conundrum wrapped up in an enigma surrounded by a puzzle.


>  --
>  Andrew C. Aitchison                                     Kendal, UK
>                          andrew(a)aitchison.me.uk
>  _______________________________________________
>  epel-devel mailing list -- epel-devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to epel-devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]...
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure


-- 
Stephen J Smoogen.
I've seen things you people wouldn't believe. Flame wars in
sci.astro.orion. I have seen SPAM filters overload because of Godwin's
Law. All those moments will be lost in time... like posts on a BBS...
time to shutdown -h now.