In which order does ELN build packages, what build root is it using?

[Mark Wielaard]

Hi,

I couldn't build elfutils because of an annobin bug that showed up on
ppc64le. Nick was nice enough to fix it and push a new annobin version:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-c78fce452e
So I could build elfutils again:
https://bodhi.fedoraproject.org/updates/FEDORA-2020-06dafb46c1
But now I am getting notices about the ELN build failures:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1613859
Which is clearly because it doesn't have the new annobin package.

Is this something to worry about? Will ELN resync and re-try the build
automatically? Or do I somehow need to take action to get ELN buildroot
synced with fedora/rawhide again?

Thanks,

Mark

[Miro Hrončok]

On 21. 09. 20 15:40, Mark Wielaard wrote:
> Hi,
>
> I couldn't build elfutils because of an annobin bug that showed up on
> ppc64le. Nick was nice enough to fix it and push a new annobin version:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-c78fce452e
> So I could build elfutils again:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-06dafb46c1
> But now I am getting notices about the ELN build failures:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1613859
> Which is clearly because it doesn't have the new annobin package.

In this case, the build fails, which is the better outcome.

However recently, I've rebuild redhat-rpm-config with this change:

https://src.fedoraproject.org/rpms/redhat-rpm-config/c/0d621460

And later I've built python3.9 with the new macro to actually deliver the change.

In order to make it work, I've run:

$ koji wait-repo f34-build --build=redhat-rpm-config-172-1.fc34
   $ koji wait-repo eln-build --build=redhat-rpm-config-172-1.eln103

before building python3.9 in rawhide.

But I wondered: In case I would have not run the ELN waitrepo, would ELN 
possibly rebuild python3.9 before the new redhat-rpm-config was available in the 
ELN buildroot?

[Stephen Gallagher]

On Mon, Sep 21, 2020 at 9:55 AM Miro Hrončok  wrote:
> On 21. 09. 20 15:40, Mark Wielaard wrote:
> > Hi,
> >
> > I couldn't build elfutils because of an annobin bug that showed up on
> > ppc64le. Nick was nice enough to fix it and push a new annobin version:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-c78fce452e
> > So I could build elfutils again:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-06dafb46c1
> > But now I am getting notices about the ELN build failures:
> > https://koji.fedoraproject.org/koji/buildinfo?buildID=1613859
> > Which is clearly because it doesn't have the new annobin package.
>
> In this case, the build fails, which is the better outcome.
>
> However recently, I've rebuild redhat-rpm-config with this change:
>
> https://src.fedoraproject.org/rpms/redhat-rpm-config/c/0d621460
>
> And later I've built python3.9 with the new macro to actually deliver the change.
>
> In order to make it work, I've run:
>
> $ koji wait-repo f34-build --build=redhat-rpm-config-172-1.fc34
>    $ koji wait-repo eln-build --build=redhat-rpm-config-172-1.eln103
>
> before building python3.9 in rawhide.
>
> But I wondered: In case I would have not run the ELN waitrepo, would ELN
> possibly rebuild python3.9 before the new redhat-rpm-config was available in the
> ELN buildroot?

Yes, this race is possible if the time it takes ELN to complete the
rebuild of the first package is longer than the time it takes the
second package to build and then also trigger the ELN build. It's
unlikely to occur often, but it is a possibility. I'm not sure how to
avoid it since without a side-tag we can't know which builds might be
dependent on one another.

For those grouped builds that use a side-tag, we're going to be
monitoring those and rebuilding them in the same order as they are
submitted. We can also add a waitrepo to our behavior to ensure that
we don't start newer builds before the prior ones succeed.

Thanks for bringing this up!

[Orion Poplawski]

On 9/22/20 1:19 PM, Stephen Gallagher wrote:
> On Mon, Sep 21, 2020 at 9:55 AM Miro Hrončok  wrote:
> > On 21. 09. 20 15:40, Mark Wielaard wrote:
> > > Hi,
> > >
> > > I couldn't build elfutils because of an annobin bug that showed up on
> > > ppc64le. Nick was nice enough to fix it and push a new annobin version:
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-c78fce452e
> > > So I could build elfutils again:
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-06dafb46c1
> > > But now I am getting notices about the ELN build failures:
> > > https://koji.fedoraproject.org/koji/buildinfo?buildID=1613859
> > > Which is clearly because it doesn't have the new annobin package.
> >
> > In this case, the build fails, which is the better outcome.
> >
> > However recently, I've rebuild redhat-rpm-config with this change:
> > https://src.fedoraproject.org/rpms/redhat-rpm-config/c/0d621460
> >
> > And later I've built python3.9 with the new macro to actually deliver the change.
> >
> > In order to make it work, I've run:
> > $ koji wait-repo f34-build --build=redhat-rpm-config-172-1.fc34
> > $ koji wait-repo eln-build --build=redhat-rpm-config-172-1.eln103
> >
> > before building python3.9 in rawhide.
> >
> > But I wondered: In case I would have not run the ELN waitrepo, would ELN
> > possibly rebuild python3.9 before the new redhat-rpm-config was available in the
> > ELN buildroot?
>
> Yes, this race is possible if the time it takes ELN to complete the
> rebuild of the first package is longer than the time it takes the
> second package to build and then also trigger the ELN build. It's
> unlikely to occur often, but it is a possibility. I'm not sure how to
> avoid it since without a side-tag we can't know which builds might be
> dependent on one another.
>
> For those grouped builds that use a side-tag, we're going to be
> monitoring those and rebuilding them in the same order as they are
> submitted. We can also add a waitrepo to our behavior to ensure that
> we don't start newer builds before the prior ones succeed.
>
> Thanks for bringing this up!

I think I have another build ordering issue.  Recently libevent had a 
soname bump and deps were built in a side tag.  However, in ELN 
openmpi-4.0.5-2.eln103 (which was the release bump for the libevent 
rebuild) was built before libevent-2.1.12-2.eln103 was.  Now I'm getting 
spammed every couple of hours with netcdf and vtk ELN build failures due 
to broken deps on libevent (and others).

Are we now stuck with bumping the release for openmpi again for the ELN 
build?

Orion

[Stephen Gallagher]

> I think I have another build ordering issue.  Recently libevent had a
> soname bump and deps were built in a side tag.  However, in ELN
> openmpi-4.0.5-2.eln103 (which was the release bump for the libevent
> rebuild) was built before libevent-2.1.12-2.eln103 was.  Now I'm getting
> spammed every couple of hours with netcdf and vtk ELN build failures due
> to broken deps on libevent (and others).
>
> Are we now stuck with bumping the release for openmpi again for the ELN
> build?

We are aware of this and will take care of the rebuild. No action is
required from you at this time. This will be fixed today.

We have not yet landed the side-tag tracking work, so we didn't catch
this. Sorry about the spam.

[Sérgio Basto]

On Fri, 2020-10-02 at 10:50 -0400, Stephen Gallagher wrote:
> > I think I have another build ordering issue.  Recently libevent had
> > a
> > soname bump and deps were built in a side tag.  However, in ELN
> > openmpi-4.0.5-2.eln103 (which was the release bump for the libevent
> > rebuild) was built before libevent-2.1.12-2.eln103 was.  Now I'm
> > getting
> > spammed every couple of hours with netcdf and vtk ELN build
> > failures due
> > to broken deps on libevent (and others).
> >
> > Are we now stuck with bumping the release for openmpi again for the
> > ELN
> > build?
>
> We are aware of this and will take care of the rebuild. No action is
> required from you at this time. This will be fixed today.
>
> We have not yet landed the side-tag tracking work, so we didn't catch
> this. Sorry about the spam.

BTW and since Orion also takes care of VTK package, why or how this
happen [1]  ? why or how vtk-devel requires things that doesn't exist
(yet) in fedora-eln ?

[1]
mock -r fedora-eln-x86_64 --install vtk-devel

- nothing provides libjawt.so(SUNWprivate_1.1)(64bit) needed by vtk-
devel-8.2.0-17.eln103.x86_64

- nothing provides mysql-devel(x86-64) needed by vtk-devel-8.2.0-
17.eln103.x86_64

--
Sérgio M. B.

[Stephen Gallagher]

On Fri, Oct 2, 2020 at 4:56 PM Sérgio Basto  wrote:
...
> BTW and since Orion also takes care of VTK package, why or how this
> happen [1]  ? why or how vtk-devel requires things that doesn't exist
> (yet) in fedora-eln ?
>
> [1]
> mock -r fedora-eln-x86_64 --install vtk-devel
> nothing provides libjawt.so(SUNWprivate_1.1)(64bit) needed by vtk-
>
> devel-8.2.0-17.eln103.x86_64

Java is FTBFS in ELN right now; we're looking into it.
> nothing provides mysql-devel(x86-64) needed by vtk-devel-8.2.0-
>
> 17.eln103.x86_64

This might actually be a packaging mistake on the vtk side; it used to
be that MariaDB had a virtual `Provides: mysql[-devel]` in it, but
that is no longer the case.

[Orion Poplawski]

On 10/2/20 2:55 PM, Sérgio Basto wrote:
> BTW and since Orion also takes care of VTK package, why or how this
> happen [1]  ? why or how vtk-devel requires things that doesn't exist
> (yet) in fedora-eln ?
>
> [1]
> mock -r fedora-eln-x86_64 --install vtk-devel
> nothing provides libjawt.so(SUNWprivate_1.1)(64bit) needed by vtk-
>
> devel-8.2.0-17.eln103.x86_64
> nothing provides mysql-devel(x86-64) needed by vtk-devel-8.2.0-
>
> 17.eln103.x86_64
>
> --
> Sérgio M. B.

Because not all instances of mysql-devel were changed to mariadb-devel. 
I've pushed the change and it should get picked up by the mass rebuild.

Sorry to drop this for so long.