Can't do EL8 builds due to modular packages

[Mátyás Selmeci]

Hi,

I've been trying to do EL8 builds on our Koji instance
(koji.opensciencegrid.org) but installing software into the
buildroot fails with error messages like

No available modular metadata for modular package 'subversion-1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed on the system

The build machine is CentOS 8 with koji-builder
1.17.0 and mock 1.4.21.  The hub is CentOS 7 with koji 1.17.0.

This is what the build tag looks like:

Tag: dist-el8-build [663]
  Arches: x86_64
  Groups: build, srpm-build
  Required permission: 'admin'
  Tag options:
  Targets that build into this tag:
    dist-el8-build (dist-el8-build, repo#71484: 2020-04-22 17:26:22.270397)
  This tag is a buildroot for one or more targets
  Current repo: repo#71484: 2020-04-22 17:26:22.270397
  Targets that build from this tag:
    dist-el8-build
  External repos:
      9 dist-el8-centos-PowerTools (http://mirror.centos.org/centos-8/8/PowerTools/$arch/os/)
     10 dist-el8-centos-BaseOS (http://mirror.centos.org/centos-8/8/BaseOS/$arch/os/)
     11 dist-el8-centos-AppStream (http://mirror.centos.org/centos-8/8/AppStream/$arch/os/)

(I'm using the main upstream repo just to rule out any mirror issues
on our end.)

I also attached mock_output.log.

How can I fix this?

Thanks,
-Mat

[Tom Stellard]

On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> Hi,
>
> I've been trying to do EL8 builds on our Koji instance
> (koji.opensciencegrid.org) but installing software into the
> buildroot fails with error messages like
>
> No available modular metadata for modular package 'subversion-1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed on the system

I think you need to add enable this option in the yum repo config:

module_hotfixes=1

-Tom
> The build machine is CentOS 8 with koji-builder
> 1.17.0 and mock 1.4.21.  The hub is CentOS 7 with koji 1.17.0.
>
> This is what the build tag looks like:
>
> Tag: dist-el8-build [663]
>   Arches: x86_64
>   Groups: build, srpm-build
>   Required permission: 'admin'
>   Tag options:
>   Targets that build into this tag:
>     dist-el8-build (dist-el8-build, repo#71484: 2020-04-22 17:26:22.270397)
>   This tag is a buildroot for one or more targets
>   Current repo: repo#71484: 2020-04-22 17:26:22.270397
>   Targets that build from this tag:
>     dist-el8-build
>   External repos:
>       9 dist-el8-centos-PowerTools (http://mirror.centos.org/centos-8/8/PowerTools/$arch/os/)
>      10 dist-el8-centos-BaseOS (http://mirror.centos.org/centos-8/8/BaseOS/$arch/os/)
>      11 dist-el8-centos-AppStream (http://mirror.centos.org/centos-8/8/AppStream/$arch/os/)
>
> (I'm using the main upstream repo just to rule out any mirror issues
> on our end.)
>
> I also attached mock_output.log.
>
> How can I fix this?
>
> Thanks,
> -Mat
>
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Mátyás Selmeci]

On 4/22/20 12:46 PM, Tom Stellard wrote:
> On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > Hi,
> >
> > I've been trying to do EL8 builds on our Koji instance
> > (koji.opensciencegrid.org) but installing software into the
> > buildroot fails with error messages like
> >
> > No available modular metadata for modular package 'subversion-1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed on the system
>
> I think you need to add enable this option in the yum repo config:
>
> module_hotfixes=1
>
> -Tom

How do I do that?  I thought koji generated those.  Is it something
like

koji edit-tag dist-el8-build --extra=module_hotfixes=1

?

-Mat

[Tom Stellard]

On 04/22/2020 10:57 AM, Mátyás Selmeci wrote:
> On 4/22/20 12:46 PM, Tom Stellard wrote:
> > On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > > Hi,
> > >
> > > I've been trying to do EL8 builds on our Koji instance
> > > (koji.opensciencegrid.org) but installing software into the
> > > buildroot fails with error messages like
> > >
> > > No available modular metadata for modular package 'subversion-1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed on the system
> >
> > I think you need to add enable this option in the yum repo config:
> >
> > module_hotfixes=1
> >
> > -Tom
>
> How do I do that?  I thought koji generated those.  Is it something
> like
>
> koji edit-tag dist-el8-build --extra=module_hotfixes=1

I'm not sure if there is a way to specify this with koji, but you
might be able to add it to the global yum.conf option in the
/etc/mock/site-defaults.cfg on the builders.

-Tom
> ?
>
> -Mat
> _______________________________________________
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

On Wed, 2020-04-22 at 12:57 -0500, Mátyás Selmeci wrote:
> On 4/22/20 12:46 PM, Tom Stellard wrote:
> > On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > > Hi,
> > >
> > > I've been trying to do EL8 builds on our Koji instance
> > > (koji.opensciencegrid.org) but installing software into the
> > > buildroot fails with error messages like
> > >
> > > No available modular metadata for modular package 'subversion-
> > > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be
> > > installed on the system
> >
> > I think you need to add enable this option in the yum repo config:
> >
> > module_hotfixes=1
> >
> > -Tom
>
> How do I do that?  I thought koji generated those.  Is it something
> like
>
> koji edit-tag dist-el8-build --extra=module_hotfixes=1

koji edit-tag {build tag} --extra=mock.yum.module_hotfixes=1
> ?
>
> -Mat
> _______________________________________________
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Mátyás Selmeci]

On 4/22/20 1:20 PM, [e-mail] wrote:
> On Wed, 2020-04-22 at 12:57 -0500, Mátyás Selmeci wrote:
> > On 4/22/20 12:46 PM, Tom Stellard wrote:
> > > On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > > > Hi,
> > > >
> > > > I've been trying to do EL8 builds on our Koji instance
> > > > (koji.opensciencegrid.org) but installing software into the
> > > > buildroot fails with error messages like
> > > >
> > > > No available modular metadata for modular package 'subversion-
> > > > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be
> > > > installed on the system
> > >
> > > I think you need to add enable this option in the yum repo config:
> > >
> > > module_hotfixes=1
> > >
> > > -Tom
> >
> > How do I do that?  I thought koji generated those.  Is it something
> > like
> >
> > koji edit-tag dist-el8-build --extra=module_hotfixes=1
>
> koji edit-tag {build tag} --extra=mock.yum.module_hotfixes=1

This works when I generate a mock config by hand via
`koji mock-config`, but the change is not present in the mock
configs that get created on the build host.

-Mat

[Mátyás Selmeci]

On 4/22/20 3:53 PM, Mátyás Selmeci wrote:
> On 4/22/20 1:20 PM, [e-mail] wrote:
> > On Wed, 2020-04-22 at 12:57 -0500, Mátyás Selmeci wrote:
> > > On 4/22/20 12:46 PM, Tom Stellard wrote:
> > > > On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > > > > Hi,
> > > > >
> > > > > I've been trying to do EL8 builds on our Koji instance
> > > > > (koji.opensciencegrid.org) but installing software into the
> > > > > buildroot fails with error messages like
> > > > >
> > > > > No available modular metadata for modular package 'subversion-
> > > > > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be
> > > > > installed on the system
> > > >
> > > > I think you need to add enable this option in the yum repo config:
> > > >
> > > > module_hotfixes=1
> > > >
> > > > -Tom
> > >
> > > How do I do that?  I thought koji generated those.  Is it something
> > > like
> > >
> > > koji edit-tag dist-el8-build --extra=module_hotfixes=1
> >
> > koji edit-tag {build tag} --extra=mock.yum.module_hotfixes=1
>
> This works when I generate a mock config by hand via
> `koji mock-config`, but the change is not present in the mock
> configs that get created on the build host.
>
> -Mat

Ah, looks like it got fixed in 1.18.0.

-Mat

[Mátyás Selmeci]

On 4/23/20 12:11 PM, Mátyás Selmeci wrote:
> On 4/22/20 3:53 PM, Mátyás Selmeci wrote:
> > On 4/22/20 1:20 PM, [e-mail] wrote:
> > > On Wed, 2020-04-22 at 12:57 -0500, Mátyás Selmeci wrote:
> > > > On 4/22/20 12:46 PM, Tom Stellard wrote:
> > > > > On 04/22/2020 10:42 AM, Mátyás Selmeci wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I've been trying to do EL8 builds on our Koji instance
> > > > > > (koji.opensciencegrid.org) but installing software into the
> > > > > > buildroot fails with error messages like
> > > > > >
> > > > > > No available modular metadata for modular package 'subversion-
> > > > > > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be
> > > > > > installed on the system
> > > > >
> > > > > I think you need to add enable this option in the yum repo config:
> > > > >
> > > > > module_hotfixes=1
> > > > >
> > > > > -Tom
> > > >
> > > > How do I do that?  I thought koji generated those.  Is it something
> > > > like
> > > >
> > > > koji edit-tag dist-el8-build --extra=module_hotfixes=1
> > >
> > > koji edit-tag {build tag} --extra=mock.yum.module_hotfixes=1
> >
> > This works when I generate a mock config by hand via
> > `koji mock-config`, but the change is not present in the mock
> > configs that get created on the build host.
> >
> > -Mat
>
> Ah, looks like it got fixed in 1.18.0.
>
> -Mat

I ended up backporting that commit to our 1.17.0 build and then making
this change, and it fixed my issue.  (Our testing hub is currently
unavailable so I can't just upgrade yet, but hopefully I will soon.)

Now I have a separate problem:  I can't install some build dependencies
because they appear to be modular.  I'm seeing errors from yum-builddep
like

DEBUG util.py:596:   Problem 22: conflicting requests
DEBUG util.py:596:    - package libtool-ltdl-devel-2.4.6-25.el8.i686 requires automake = 1.16.1, but none of the providers can be installed
DEBUG util.py:596:    - package libtool-ltdl-devel-2.4.6-25.el8.x86_64 requires automake = 1.16.1, but none of the providers can be installed
DEBUG util.py:596:    - package automake-1.16.1-6.el8.noarch requires perl(Thread::Queue), but none of the providers can be installed
DEBUG util.py:596:    - nothing provides perl(:MODULE_COMPAT_5.24.4) needed by perl-Thread-Queue-3.13-2.module_el8.1.0+229+cd132df8.noarch
DEBUG util.py:598:  (try to add '--skip-broken' to skip uninstallable packages)

(full task is here if it helps:
https://koji.opensciencegrid.org/koji/taskinfo?taskID=313860 )

Looks like the dependency it wants is perl 5.24.  I looked at the
external repository (CentOS-AppStream) and there is a modular perl-5.24
RPM and a non-modular perl-5.26 RPM.  Is one shadowing the other?
I added --nobest to the yum-builddep invocation but it didn't help.

How can I get around this?

Thanks,
-Mat

[Patrick Riehecky]

With Koji, I've been targeting 8.0 .  It provides a dnf/yum that does
not enforce this constraint.

Pat

On Wed, 2020-04-22 at 12:42 -0500, Mátyás Selmeci wrote:
> Hi,
>
> I've been trying to do EL8 builds on our Koji instance
> (koji.opensciencegrid.org) but installing software into the
> buildroot fails with error messages like
>
> No available modular metadata for modular package 'subversion-
> 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed
> on the system
>
> The build machine is CentOS 8 with koji-builder
> 1.17.0 and mock 1.4.21.  The hub is CentOS 7 with koji 1.17.0.
>
> This is what the build tag looks like:
>
> Tag: dist-el8-build [663]
>   Arches: x86_64
>   Groups: build, srpm-build
>   Required permission: 'admin'
>   Tag options:
>   Targets that build into this tag:
>     dist-el8-build (dist-el8-build, repo#71484: 2020-04-22
> 17:26:22.270397)
>   This tag is a buildroot for one or more targets
>   Current repo: repo#71484: 2020-04-22 17:26:22.270397
>   Targets that build from this tag:
>     dist-el8-build
>   External repos:
>       9 dist-el8-centos-PowerTools (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> )
>      10 dist-el8-centos-BaseOS (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> )
>      11 dist-el8-centos-AppStream (
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> )
>
> (I'm using the main upstream repo just to rule out any mirror issues
> on our end.)
>
> I also attached mock_output.log.
>
> How can I fix this?
>
> Thanks,
> -Mat
>
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]
> Fedora Code of Conduct: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_...
>
> List Guidelines: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_...
>
> List Archives: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedorahosted.org_...

[Mátyás Selmeci]

That doesn't seem to work for me.  I thought mock used the dnf/yum
from outside the chroot to install packages inside -- does that mean
the build host itself should be 8.0?

-Mat

On 4/22/20 12:53 PM, Patrick Riehecky wrote:
> With Koji, I've been targeting 8.0 .  It provides a dnf/yum that does
> not enforce this constraint.
>
> Pat
>
> On Wed, 2020-04-22 at 12:42 -0500, Mátyás Selmeci wrote:
> > Hi,
> >
> > I've been trying to do EL8 builds on our Koji instance
> > (koji.opensciencegrid.org) but installing software into the
> > buildroot fails with error messages like
> >
> > No available modular metadata for modular package 'subversion-
> > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be installed
> > on the system
> >
> > The build machine is CentOS 8 with koji-builder
> > 1.17.0 and mock 1.4.21.  The hub is CentOS 7 with koji 1.17.0.
> >
> > This is what the build tag looks like:
> >
> > Tag: dist-el8-build [663]
> >   Arches: x86_64
> >   Groups: build, srpm-build
> >   Required permission: 'admin'
> >   Tag options:
> >   Targets that build into this tag:
> >     dist-el8-build (dist-el8-build, repo#71484: 2020-04-22
> > 17:26:22.270397)
> >   This tag is a buildroot for one or more targets
> >   Current repo: repo#71484: 2020-04-22 17:26:22.270397
> >   Targets that build from this tag:
> >     dist-el8-build
> >   External repos:
> >       9 dist-el8-centos-PowerTools (
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > )
> >      10 dist-el8-centos-BaseOS (
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > )
> >      11 dist-el8-centos-AppStream (
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > )
> >
> > (I'm using the main upstream repo just to rule out any mirror issues
> > on our end.)
> >
> > I also attached mock_output.log.
> >
> > How can I fix this?
> >
> > Thanks,
> > -Mat
> >
> > koji-devel mailing list -- [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]
> > Fedora Code of Conduct: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_...
> >
> > List Guidelines: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_...
> >
> > List Archives: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedorahosted.org_...
>
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Patrick Riehecky]

Alas, my koji build host is an SL7 box where I'm using the mock
bootstrap chroot.

I'd recommend going that route, but be aware of 
https://pagure.io/koji/issue/1988

Pat

On Wed, 2020-04-22 at 17:08 -0500, Mátyás Selmeci wrote:
> That doesn't seem to work for me.  I thought mock used the dnf/yum
> from outside the chroot to install packages inside -- does that mean
> the build host itself should be 8.0?
>
> -Mat
>
> On 4/22/20 12:53 PM, Patrick Riehecky wrote:
> > With Koji, I've been targeting 8.0 .  It provides a dnf/yum that
> > does
> > not enforce this constraint.
> >
> > Pat
> >
> > On Wed, 2020-04-22 at 12:42 -0500, Mátyás Selmeci wrote:
> > > Hi,
> > >
> > > I've been trying to do EL8 builds on our Koji instance
> > > (koji.opensciencegrid.org) but installing software into the
> > > buildroot fails with error messages like
> > >
> > > No available modular metadata for modular package 'subversion-
> > > 1.10.2-1.module_el8.0.0+45+75bba4f4.x86_64', it cannot be
> > > installed
> > > on the system
> > >
> > > The build machine is CentOS 8 with koji-builder
> > > 1.17.0 and mock 1.4.21.  The hub is CentOS 7 with koji 1.17.0.
> > >
> > > This is what the build tag looks like:
> > >
> > > Tag: dist-el8-build [663]
> > >   Arches: x86_64
> > >   Groups: build, srpm-build
> > >   Required permission: 'admin'
> > >   Tag options:
> > >   Targets that build into this tag:
> > >     dist-el8-build (dist-el8-build, repo#71484: 2020-04-22
> > > 17:26:22.270397)
> > >   This tag is a buildroot for one or more targets
> > >   Current repo: repo#71484: 2020-04-22 17:26:22.270397
> > >   Targets that build from this tag:
> > >     dist-el8-build
> > >   External repos:
> > >       9 dist-el8-centos-PowerTools (
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > > )
> > >      10 dist-el8-centos-BaseOS (
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > > )
> > >      11 dist-el8-centos-AppStream (
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__mirror.centos.org_centos...
> > > )
> > >
> > > (I'm using the main upstream repo just to rule out any mirror
> > > issues
> > > on our end.)
> > >
> > > I also attached mock_output.log.
> > >
> > > How can I fix this?
> > >
> > > Thanks,
> > > -Mat
> > >
> > > koji-devel mailing list -- [e-mail]
> > > To unsubscribe send an email to 
> > > [e-mail]
> > > Fedora Code of Conduct: 
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_...
> > >
> > > List Guidelines: 
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_...
> > >
> > > List Archives: 
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedorahosted.org_...
> >
> > koji-devel mailing list -- [e-mail]
> > To unsubscribe send an email to 
> > [e-mail]
> > Fedora Code of Conduct: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_...
> >
> > List Guidelines: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_...
> >
> > List Archives: 
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedorahosted.org_...
>
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to 
> [e-mail]
> Fedora Code of Conduct: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_...
>
> List Guidelines: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_...
>
> List Archives: 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedorahosted.org_...