LTO and the F33 mass rebuild

[Jeff Law]

So I've done two passes over the F33 build failures here:

https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html

For any package which was failing and LTO appeared to be a factor, I've assigned
the package's associated FTBFS BZ to me to address the LTO component.  Some of
those I've already fixed and either closed the BZ or handed it back to the
default assignee for non-LTO issues that need to be resolved.

Thus if you have a package on the failing packages list and its still failing,
it's unlikely to be an LTO issue.  I don't mind if you reach out, but start from
the assumption LTO isn't the triggering event.  You can use %define _lto_cflags
%{nil} in your .spec file to disable LTO for testing purposes.

Having said that, I do expect some LTO issues to still pop up.  For example it's
likely we'll continue to see the cmake issues get resolved in various packages
and I wouldn't be surprised if after fixing a package to work with the new cmake
macros that a small number will trip LTO issues.  I'm obviously here to help with
those too.

It's also possible there are packages that are failing and which are not on that
list.  One was passed along to me yesterday (libaio).  Again, happy to help with
analyzing those too.

Anyway, the immediate actions are to find a near term resolution for the LTO
issues which I've assigned to myself in BZ.  There aren't that many and I'm
confident we'll be able to close them out in a reasonable timeframe.

After that I'll be reviewing all the opt-outs to see which we can move forward,
which require upstream GCC bug reports, etc.

Jeff

[Gary Buhrmaster]

On Sat, Aug 8, 2020 at 10:03 PM Jeff Law  wrote:
> So I've done two passes over the F33 build failures here:

Thank you for taking ownership and responsibility
for those things that you can control.  It is a model
for other change owners to attempt to achieve.

[Fabio Valentini]

On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> So I've done two passes over the F33 build failures here:
>
> https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html

Hi Jeff,

Thanks for looking at the failures, it's really appreciated!

Be aware that the f33-failures.html page is not complete though, since
packages for which there was an attempted rebuild *after* the mass
rebuild are removed from the list, whether the builds succeeded or
not. So it's possible that you missed builds that fail for LTO-related
issues, just because they're no longer showing up in the list. The
list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
helpful here. And the f33-need-rebuild list [1] should give you a
complete picture of everything that has not successfully been rebuilt
for f33 yet.

Fabio

[0]: https://bugzilla.redhat.com/show_bug.cgi?id=1803234
[1]: https://kojipkgs.fedoraproject.org/mass-rebuild/f33-need-rebuild.html

[Jeff Law]

On Sun, 2020-08-09 at 12:02 +0200, Fabio Valentini wrote:
> On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> > So I've done two passes over the F33 build failures here:
> >
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
>
> Hi Jeff,
>
> Thanks for looking at the failures, it's really appreciated!
>
> Be aware that the f33-failures.html page is not complete though, since
> packages for which there was an attempted rebuild *after* the mass
> rebuild are removed from the list, whether the builds succeeded or
> not. So it's possible that you missed builds that fail for LTO-related
> issues, just because they're no longer showing up in the list. The
> list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
> helpful here. And the f33-need-rebuild list [1] should give you a
> complete picture of everything that has not successfully been rebuilt
> for f33 yet.

ACK.  I'm poking at the f33-need-rebuild.html list a bit.  THere's a lot of noise
in there -- things that haven't been built in a long time.  Anyway, I'll keep
poking around.

It would be helpful if there was a clean way to download failed log files and
such in batches so that I could run tools on them to look for common things that
don't need my attention (like all the cmake failures).  As it stands I have to do
an insane amount of clickies to get to some basic data.

jeff
>

[Kevin Fenzi]

On Sun, Aug 09, 2020 at 10:06:49PM -0600, Jeff Law wrote:
> On Sun, 2020-08-09 at 12:02 +0200, Fabio Valentini wrote:
> > On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> > > So I've done two passes over the F33 build failures here:
> > >
> > > https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
> >
> > Hi Jeff,
> >
> > Thanks for looking at the failures, it's really appreciated!
> >
> > Be aware that the f33-failures.html page is not complete though, since
> > packages for which there was an attempted rebuild *after* the mass
> > rebuild are removed from the list, whether the builds succeeded or
> > not. So it's possible that you missed builds that fail for LTO-related
> > issues, just because they're no longer showing up in the list. The
> > list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
> > helpful here. And the f33-need-rebuild list [1] should give you a
> > complete picture of everything that has not successfully been rebuilt
> > for f33 yet.
>
> ACK.  I'm poking at the f33-need-rebuild.html list a bit.  THere's a lot of noise
> in there -- things that haven't been built in a long time.  Anyway, I'll keep
> poking around.
>
> It would be helpful if there was a clean way to download failed log files and
> such in batches so that I could run tools on them to look for common things that
> don't need my attention (like all the cmake failures).  As it stands I have to do
> an insane amount of clickies to get to some basic data.

It's been proposed to enable the koji 'save-failed-tree' plugin:
https://pagure.io/releng/issue/8243

It might help for this sort of thing, as it lets you get a complete tar
of the entire tree.

I'll look into it more after we have staging koji back...

kevin

[Vít Ondruch]

Dne 11. 08. 20 v 18:43 Kevin Fenzi napsal(a):
> On Sun, Aug 09, 2020 at 10:06:49PM -0600, Jeff Law wrote:
> > On Sun, 2020-08-09 at 12:02 +0200, Fabio Valentini wrote:
> > > On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> > > > So I've done two passes over the F33 build failures here:
> > > >
> > > > https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
> > >
> > > Hi Jeff,
> > >
> > > Thanks for looking at the failures, it's really appreciated!
> > >
> > > Be aware that the f33-failures.html page is not complete though, since
> > > packages for which there was an attempted rebuild *after* the mass
> > > rebuild are removed from the list, whether the builds succeeded or
> > > not. So it's possible that you missed builds that fail for LTO-related
> > > issues, just because they're no longer showing up in the list. The
> > > list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
> > > helpful here. And the f33-need-rebuild list [1] should give you a
> > > complete picture of everything that has not successfully been rebuilt
> > > for f33 yet.
> >
> > ACK.  I'm poking at the f33-need-rebuild.html list a bit.  THere's a lot of noise
> > in there -- things that haven't been built in a long time.  Anyway, I'll keep
> > poking around.
> >
> > It would be helpful if there was a clean way to download failed log files and
> > such in batches so that I could run tools on them to look for common things that
> > don't need my attention (like all the cmake failures).  As it stands I have to do
> > an insane amount of clickies to get to some basic data.
>
> It's been proposed to enable the koji 'save-failed-tree' plugin:
> https://pagure.io/releng/issue/8243

It could be so much better for FTBFS bugs after mass rebuild ...

Vít
> It might help for this sort of thing, as it lets you get a complete tar
> of the entire tree.
>
> I'll look into it more after we have staging koji back...
>
> kevin
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Jeff Law]

On Sun, 2020-08-09 at 12:02 +0200, Fabio Valentini wrote:
> On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> > So I've done two passes over the F33 build failures here:
> >
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
>
> Hi Jeff,
>
> Thanks for looking at the failures, it's really appreciated!
>
> Be aware that the f33-failures.html page is not complete though, since
> packages for which there was an attempted rebuild *after* the mass
> rebuild are removed from the list, whether the builds succeeded or
> not. So it's possible that you missed builds that fail for LTO-related
> issues, just because they're no longer showing up in the list. The
> list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
> helpful here. And the f33-need-rebuild list [1] should give you a
> complete picture of everything that has not successfully been rebuilt
> for f33 yet.
>
> Fabio

OK.  I've scanned through f33-need-rebuild.html and have a few dozen things to
look at in more depth tomorrow.

jeff
>

[Jeff Law]

On Sun, 2020-08-09 at 12:02 +0200, Fabio Valentini wrote:
> On Sun, Aug 9, 2020 at 12:03 AM Jeff Law  wrote:
> > So I've done two passes over the F33 build failures here:
> >
> > https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
>
> Hi Jeff,
>
> Thanks for looking at the failures, it's really appreciated!
>
> Be aware that the f33-failures.html page is not complete though, since
> packages for which there was an attempted rebuild *after* the mass
> rebuild are removed from the list, whether the builds succeeded or
> not. So it's possible that you missed builds that fail for LTO-related
> issues, just because they're no longer showing up in the list. The
> list of bugs blocking the F33FTBFS bug in bugzilla [0] might be
> helpful here. And the f33-need-rebuild list [1] should give you a
> complete picture of everything that has not successfully been rebuilt
> for f33 yet.
>
> Fabio

I've gone through the f33-need-rebuild page and taken care of a variety of
things.  I've got a half-dozen packages left (openblas, arm-none-eabi-gcc-cs,
cross-gcc gdb and radamsa).  Otherwise if I haven't taken the BZ, then I don't
currently see the failure as being LTO related.

Jeff
>

[Richard Shaw]

Looks like js8call is failing due to LTO...

/usr/bin/ld: /tmp/js8call.Y0yEHy.ltrans36.ltrans.o: in function
`EqualizationToolsDialog::~EqualizationToolsDialog()':
/builddir/build/BUILD/js8call-2.2.0/EqualizationToolsDialog.hpp:12:
undefined reference to `pimplEqualizationToolsDialog::impl::~pimpl()'
/usr/bin/ld: /tmp/js8call.Y0yEHy.ltrans36.ltrans.o: in function
`EqualizationToolsDialog::~EqualizationToolsDialog()':
/builddir/build/BUILD/js8call-2.2.0/EqualizationToolsDialog.hpp:12:
undefined reference to `pimplEqualizationToolsDialog::impl::~pimpl()'
collect2: error: ld returned 1 exit status

Disabling lto does indeed fix the build (finished while I was writing this).

Thanks,
Richard

[Jeff Law]

On Tue, 2020-08-11 at 06:34 -0500, Richard Shaw wrote:
> Looks like js8call is failing due to LTO...
>
> /usr/bin/ld: /tmp/js8call.Y0yEHy.ltrans36.ltrans.o: in function `EqualizationToolsDialog::~EqualizationToolsDialog()':
> /builddir/build/BUILD/js8call-2.2.0/EqualizationToolsDialog.hpp:12: undefined reference to `pimplEqualizationToolsDialog::impl::~pimpl()'
> /usr/bin/ld: /tmp/js8call.Y0yEHy.ltrans36.ltrans.o: in function `EqualizationToolsDialog::~EqualizationToolsDialog()':
> /builddir/build/BUILD/js8call-2.2.0/EqualizationToolsDialog.hpp:12: undefined reference to `pimplEqualizationToolsDialog::impl::~pimpl()'
> collect2: error: ld returned 1 exit status
>
> Disabling lto does indeed fix the build (finished while I was writing this).

Thanks.  For some reason js8call wasn't on the failing webpage.  Thanks for
taking care of it!

jeff

[Miro Hrončok]

On 09. 08. 20 0:02, Jeff Law wrote:
> For any package which was failing and LTO appeared to be a factor, I've assigned
> the package's associated FTBFS BZ to me to address the LTO component.  Some of
> those I've already fixed and either closed the BZ or handed it back to the
> default assignee for non-LTO issues that need to be resolved.
>
> Thus if you have a package on the failing packages list and its still failing,
> it's unlikely to be an LTO issue.  I don't mind if you reach out, but start from
> the assumption LTO isn't the triggering event.  You can use %define _lto_cflags
> %{nil} in your .spec file to disable LTO for testing purposes.

Thank you Jeff so very much for doing this!

I've assigned https://bugzilla.redhat.com/show_bug.cgi?id=1865257 to you as well 
in the spirit of what you've said. The failure is workarounded with %global 
_lto_cflags %{nil} now.

[Jeff Law]

On Mon, 2020-08-10 at 09:59 +0200, Miro Hrončok wrote:
> On 09. 08. 20 0:02, Jeff Law wrote:
> > For any package which was failing and LTO appeared to be a factor, I've assigned
> > the package's associated FTBFS BZ to me to address the LTO component.  Some of
> > those I've already fixed and either closed the BZ or handed it back to the
> > default assignee for non-LTO issues that need to be resolved.
> >
> > Thus if you have a package on the failing packages list and its still failing,
> > it's unlikely to be an LTO issue.  I don't mind if you reach out, but start from
> > the assumption LTO isn't the triggering event.  You can use %define _lto_cflags
> > %{nil} in your .spec file to disable LTO for testing purposes.
>
> Thank you Jeff so very much for doing this!
>
> I've assigned https://bugzilla.redhat.com/show_bug.cgi?id=1865257 to you as well 
> in the spirit of what you've said. The failure is workarounded with %global 
> _lto_cflags %{nil} now.

Thanks.
jeff