Mass rebuild failures only on ppc64le: glibc problem?

[Jason L Tibbitts III]

I noticed one of my packages failed to build because the test suite
failed, but only on ppc64le.

builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
loading shared libraries:
/builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
expected localentry:0 `sasl_client_new'

I have no idea what that means, and a search for "expected localentry:0"
turned up only the patch where that message was added to glibc.

I did some other random checks and found a few other packages with
ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
getdp (and I stopped looking after that).  These all failed in their
test suites with an error from the linker containing "expected
localentry:0".

Does anyone have any idea what's going wrong?

 - J<

[Jason L Tibbitts III]

I'm also seeing builds failing early in %build.  It appears that libcurl
has a similar problem and so cmake won't run:

/usr/bin/cmake: error while loading shared libraries:
  /lib64/libcurl.so.4: expected localentry:0 `pthread_mutex_destroy'
  RPM build errors:

 - J<

[Josh Stone]

On 07/26/2017 04:21 PM, Jason L Tibbitts III wrote:
>  I noticed one of my packages failed to build because the test suite
>  failed, but only on ppc64le.
>  
>  builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
>  loading shared libraries:
>  /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
>  expected localentry:0 `sasl_client_new'
>  
>  I have no idea what that means, and a search for "expected localentry:0"
>  turned up only the patch where that message was added to glibc.
>  
>  I did some other random checks and found a few other packages with
>  ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
>  getdp (and I stopped looking after that).  These all failed in their
>  test suites with an error from the linker containing "expected
>  localentry:0".
>  
>  Does anyone have any idea what's going wrong? 
I see this with Rust too, only when using binutils 2.29.
https://sourceware.org/ml/binutils/2017-07/msg00272.html

[Josh Stone]

On 07/26/2017 06:23 PM, Josh Stone wrote:
>  On 07/26/2017 04:21 PM, Jason L Tibbitts III wrote:
> > I noticed one of my packages failed to build because the test suite
> > failed, but only on ppc64le.
> >
> > builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
> > loading shared libraries:
> > /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
> > expected localentry:0 `sasl_client_new'
> >
> > I have no idea what that means, and a search for "expected localentry:0"
> > turned up only the patch where that message was added to glibc.
> >
> > I did some other random checks and found a few other packages with
> > ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
> > getdp (and I stopped looking after that).  These all failed in their
> > test suites with an error from the linker containing "expected
> > localentry:0".
> >
> > Does anyone have any idea what's going wrong?
>  
>  I see this with Rust too, only when using binutils 2.29.
>  https://sourceware.org/ml/binutils/2017-07/msg00272.html 
I filed: https://bugzilla.redhat.com/show_bug.cgi?id=1475636

[Simone Caronni]

Hello,

On Thu, Jul 27, 2017 at 6:07 AM, Josh Stone <jistone(a)redhat.com&gt; wrote:

>  On 07/26/2017 06:23 PM, Josh Stone wrote:
>  >> I did some other random checks and found a few other packages with
>  >> ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
>  >> getdp (and I stopped looking after that).  These all failed in their
>  >> test suites with an error from the linker containing "expected
>  >> localentry:0". 

Same same. Trying to fix the latest reported CVEs on FreeRDP:

/usr/bin/cmake: error while loading shared libraries: /lib64/libcurl.so.4:
expected localentry:0 `pthread_mutex_destroy'



-- 
You cannot discover new oceans unless you have the courage to lose sight of
the shore (R. W. Emerson).

http://xkcd.com/229/
http://negativo17.org/

[Florian Weimer]

On 07/27/2017 03:23 AM, Josh Stone wrote:
>  On 07/26/2017 04:21 PM, Jason L Tibbitts III wrote:
> > I noticed one of my packages failed to build because the test suite
> > failed, but only on ppc64le.
> >
> > builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
> > loading shared libraries:
> > /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
> > expected localentry:0 `sasl_client_new'
> >
> > I have no idea what that means, and a search for "expected localentry:0"
> > turned up only the patch where that message was added to glibc.
> >
> > I did some other random checks and found a few other packages with
> > ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
> > getdp (and I stopped looking after that).  These all failed in their
> > test suites with an error from the linker containing "expected
> > localentry:0".
> >
> > Does anyone have any idea what's going wrong?
>  
>  I see this with Rust too, only when using binutils 2.29.
>  https://sourceware.org/ml/binutils/2017-07/msg00272.html 
rust-1.19.0-3.fc27 built successfully.

Thanks,
Florian

[Antonio Trande]

On 07/27/2017 01:21 AM, Jason L Tibbitts III wrote:
>  I noticed one of my packages failed to build because the test suite
>  failed, but only on ppc64le.
>  
>  builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
>  loading shared libraries:
>  /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
>  expected localentry:0 `sasl_client_new'
>  
>  I have no idea what that means, and a search for "expected localentry:0"
>  turned up only the patch where that message was added to glibc.
>  
>  I did some other random checks and found a few other packages with
>  ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
>  getdp (and I stopped looking after that).  These all failed in their
>  test suites with an error from the linker containing "expected
>  localentry:0".
>  
>  Does anyone have any idea what's going wrong?
>  
>   - J<
>  _______________________________________________ 
Same problem with MPI libraries of MUMPS just on PPC64le:

+ export
LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
+
LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
+ ./dsimpletest
./dsimpletest: error while loading shared libraries:
../lib/libmumps_common-5.1.1.so: expected localentry:0 `pthread_cond_init'


-- 
--
Antonio Trande
sagitter AT fedoraproject dot org
See my vCard.

[Paul Howarth]

On 2017-07-27 10:15, Antonio Trande wrote:
>  On 07/27/2017 01:21 AM, Jason L Tibbitts III wrote:
> > I noticed one of my packages failed to build because the test suite
> > failed, but only on ppc64le.
> > 
> > builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error 
> > while
> > loading shared libraries:
> > /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
> > expected localentry:0 `sasl_client_new'
> > 
> > I have no idea what that means, and a search for "expected 
> > localentry:0"
> > turned up only the patch where that message was added to glibc.
> > 
> > I did some other random checks and found a few other packages with
> > ppc64le-only build failures: freeipa, gammu, gap-pkg-float, gambas3,
> > getdp (and I stopped looking after that).  These all failed in their
> > test suites with an error from the linker containing "expected
> > localentry:0".
> > 
> > Does anyone have any idea what's going wrong?
> > 
> >  - J<
> > _______________________________________________
>  
>  Same problem with MPI libraries of MUMPS just on PPC64le:
>  
>  + export
>
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
>  +
>
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
>  + ./dsimpletest
>  ./dsimpletest: error while loading shared libraries:
>  ../lib/libmumps_common-5.1.1.so: expected localentry:0 
>  `pthread_cond_init' 
Same problem with appstream-util:

Executing(%check): /bin/sh -e /var/tmp/rpm-tmp.3sh3cm
+ umask 022
+ cd /builddir/build/BUILD
+ cd gtkwave-3.3.82
+ appstream-util validate-relax --nonet 
/builddir/build/BUILDROOT/gtkwave-3.3.82-2.fc27.ppc64le/usr/share/appdata/gtkwave.appdata.xml
appstream-util: error while loading shared libraries: /lib64/librt.so.1: 
expected localentry:0 `pthread_attr_setdetachstate'
error: Bad exit status from /var/tmp/rpm-tmp.3sh3cm (%check)

Paul.

[Florian Weimer]

On 07/27/2017 11:15 AM, Antonio Trande wrote:

>  Same problem with MPI libraries of MUMPS just on PPC64le:
>  
>  + export
>
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
>  +
>
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
>  + ./dsimpletest
>  ./dsimpletest: error while loading shared libraries:
>  ../lib/libmumps_common-5.1.1.so: expected localentry:0 `pthread_cond_init' 
MUMPS is building again on ppc64le.

Thanks,
Florian

[Antonio Trande]

On 07/30/2017 03:19 PM, Florian Weimer wrote:
>  On 07/27/2017 11:15 AM, Antonio Trande wrote:
>  
> > Same problem with MPI libraries of MUMPS just on PPC64le:
> >
> > + export
> >
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
> > +
> >
> LD_LIBRARY_PATH=/builddir/build/BUILD/MUMPS_5.1.1/MUMPS-5.1.1-openmpi/examples:../lib:/usr/lib64/openmpi/lib
> > + ./dsimpletest
> > ./dsimpletest: error while loading shared libraries:
> > ../lib/libmumps_common-5.1.1.so: expected localentry:0 `pthread_cond_init'
>  
>  MUMPS is building again on ppc64le.
>  
>  Thanks,
>  Florian
>   
Thank you Florian.
Is Rawhide now ready for building?
I have other packages that need a rebuild.

-- 
--
Antonio Trande
sagitter AT fedoraproject dot org
See my vCard.

[Florian Weimer]

On 07/30/2017 03:42 PM, Antonio Trande wrote:

>  Is Rawhide now ready for building?
>  I have other packages that need a rebuild. 
It depends.  There are still 48 packages which could show similar
breakage, and python3 and mariadb are among them.  In addition, there
are some broken builds which have not yet been tagged into rawhide, and
I don't have a good way of finding them.

Depending on what you want to build, I'd suggest to wait until the
python3/samba/ceph/sssd package cluster is fixed.

Thanks,
Florian

[Tom Hughes]

On 30/07/17 15:03, Florian Weimer wrote:
>  On 07/30/2017 03:42 PM, Antonio Trande wrote:
>  
> > Is Rawhide now ready for building?
> > I have other packages that need a rebuild.
>  
>  It depends.  There are still 48 packages which could show similar
>  breakage, and python3 and mariadb are among them.  In addition, there
>  are some broken builds which have not yet been tagged into rawhide, and
>  I don't have a good way of finding them. 
I have a rebuild of mapnik (which had failed due to this) running now.

None of this is actually in rawhide yet anyway is it? It's in the 
f27-rebuild side tag?

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Florian Weimer]

On 07/30/2017 04:22 PM, Tom Hughes wrote:
>  On 30/07/17 15:03, Florian Weimer wrote:
> > On 07/30/2017 03:42 PM, Antonio Trande wrote:
> >
> >> Is Rawhide now ready for building?
> >> I have other packages that need a rebuild.
> >
> > It depends.  There are still 48 packages which could show similar
> > breakage, and python3 and mariadb are among them.  In addition, there
> > are some broken builds which have not yet been tagged into rawhide, and
> > I don't have a good way of finding them.
>  
>  I have a rebuild of mapnik (which had failed due to this) running now. 
Based on a quick look at the build dependencies, it should work now.

>  None of this is actually in rawhide yet anyway is it? It's in
> the
>  f27-rebuild side tag? 
binutils 2.29 was uploaded to rawhide and used for the mass rebuild.
Some results of the mass rebuild have already been tagged into rawhide.
So the problem affects rawhide, not just the mass rebuild.

Thanks,
Florian

[Tom Hughes]

On 30/07/17 15:34, Florian Weimer wrote:
>  On 07/30/2017 04:22 PM, Tom Hughes wrote:
> > On 30/07/17 15:03, Florian Weimer wrote:
> >> On 07/30/2017 03:42 PM, Antonio Trande wrote:
> >>
> >>> Is Rawhide now ready for building?
> >>> I have other packages that need a rebuild.
> >>
> >> It depends.  There are still 48 packages which could show similar
> >> breakage, and python3 and mariadb are among them.  In addition, there
> >> are some broken builds which have not yet been tagged into rawhide, and
> >> I don't have a good way of finding them.
> >
> > I have a rebuild of mapnik (which had failed due to this) running now.
>  
>  Based on a quick look at the build dependencies, it should work now. 
Indeed the ppc64le build has completed sucessfully. Just waiting on the 
arm builds...

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Florian Weimer]

I think I have a workaround, but I'm blocked by this koji CLI bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1472106

It seems I can't do any builds through the koji CLI.

Thanks,
Florian

[Tom Hughes]

On 27/07/17 11:16, Florian Weimer wrote:

>  I think I have a workaround, but I'm blocked by this koji CLI
> bug:
>  
>     https://bugzilla.redhat.com/show_bug.cgi?id=1472106
>  
>  It seems I can't do any builds through the koji CLI. 
I think on IRC yesterday it was decided that message was code for "your 
kerberos ticket has expired".

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Florian Weimer]

On 07/27/2017 12:39 PM, Tom Hughes wrote:
>  On 27/07/17 11:16, Florian Weimer wrote:
>  
> > I think I have a workaround, but I'm blocked by this koji CLI bug:
> >
> >    https://bugzilla.redhat.com/show_bug.cgi?id=1472106
> >
> > It seems I can't do any builds through the koji CLI.
>  
>  I think on IRC yesterday it was decided that message was code for "your
>  kerberos ticket has expired". 
That's not a valid explanation in my case.  I freshly obtained the
ticket before attempting a build.

Submitting builds indirectly through fedpkg seems to work for me, though.

Thanks,
Florian

[Jason L Tibbitts III]

> >>>> "FW" == Florian Weimer
> <fweimer(a)redhat.com&gt; writes: 
FW> I think I have a workaround, but I'm blocked by this koji CLI bug:

You can always ask someone else to do a build for you; just let me know
if you need that done.

I don't see any recent koji jobs submitted by you, nor any recent glibc
build besides the mass rebuild one (which I believe failed due to
debuginfo generation issues) so just let me know if you need me to kick
off a glibc build or a build of any other package.

 - J<

[Florian Weimer]

On 07/27/2017 02:31 PM, Jason L Tibbitts III wrote:

>  I don't see any recent koji jobs submitted by you, nor any recent
> glibc
>  build besides the mass rebuild one (which I believe failed due to
>  debuginfo generation issues) so just let me know if you need me to kick
>  off a glibc build or a build of any other package. 
I'm doing scratch builds, and I think they are kind of difficult to see
in Koji.

Thanks,
Florian

[Jason L Tibbitts III]

> >>>> "FW" == Florian Weimer
> <fweimer(a)redhat.com&gt; writes: 
FW> I'm doing scratch builds, and I think they are kind of difficult to
FW> see in Koji.

Yeah, you can see them in the task list when they're active.  I see the
one you're running now.

Still, if you are running into problems and need me or anyone else to
kick off a build, please don't hesitate to ask.

 - J<

[Florian Weimer]

The workaround recommended by binutils upstream (--no-plt-localentry)
does not work, and a glibc built with that still has the problem.

I have not rebuilt dependent packages (cmake, curl) with the flag.  If
*those* are mislinked, we really have a problem.  There's no word on the
root cause yet.

Thanks,
Florian

[Florian Weimer]

Status update:

Nick has pushed binutils-2.29-3.fc27, which disables the binutils change
by default.  (There is still an ld option to enable it.  Hopefully, no
packages use that yet.)

Just rebuilding glibc is insufficient as a fix.  The localentry
optimization is so sensitive to changes that glibc-using packages which
have been linked in a particular way have to be rebuilt.

I have patched symboldb to record the bits which ld uses to mark symbols
as localentry symbol references (which is the same information that the
glibc dynamic linker uses to generate those pesky errors).  A new Fedora
important is currently running.  Afterwards, I will use this data to do
targeted manual rebuilds of core packages which are affected by the
internal localentry optimization incompatibility of glibc.

Rebuilds are currently blocked on a new buildroot with some RPM
debuginfo generation fixes.  Rumor has it that there is an unrelated
issue which prevents buildroot updates.

releng has agreed to a second mass rebuild (for non-noarch packages),
but Paul Frields asked for it to be delayed, due to other tasks, such as
the pagure dist-git migration.  I think we can accommodate this as long
as there is not too much update churn in rawhide before the new mass
rebuild, triggering the new ABI incompatibilities.

Florian

[Florian Weimer]

On 07/28/2017 10:58 PM, Florian Weimer wrote:
>  Rebuilds are currently blocked on a new buildroot with some RPM
>  debuginfo generation fixes.  Rumor has it that there is an unrelated
>  issue which prevents buildroot updates. 
This appears to be an ongoing issue.  I cannot rebuild libcurl until its
dependencies are rebuilt (hopefully done) and have landed in the build
root (still waiting), so cmake remains unfixed for now, I'm afraid.

Florian

[Florian Weimer]

On 07/29/2017 09:08 PM, Florian Weimer wrote:
>  On 07/28/2017 10:58 PM, Florian Weimer wrote:
> > Rebuilds are currently blocked on a new buildroot with some RPM
> > debuginfo generation fixes.  Rumor has it that there is an unrelated
> > issue which prevents buildroot updates.
>  
>  This appears to be an ongoing issue.  I cannot rebuild libcurl until its
>  dependencies are rebuilt (hopefully done) and have landed in the build
>  root (still waiting), so cmake remains unfixed for now, I'm afraid. 
There's a rogue glibc override tag in place which blocks further cleanup
work, see the separate “f27-override tag for glibc-2.25.90-29.fc27” thread.

Once that is resolved and the buildroot regenerated, I think we are
close to getting libcurl (and thus cmake) working again.

Florian

[Kevin Fenzi]

On 07/29/2017 12:08 PM, Florian Weimer wrote:
>  On 07/28/2017 10:58 PM, Florian Weimer wrote:
> > Rebuilds are currently blocked on a new buildroot with some RPM
> > debuginfo generation fixes.  Rumor has it that there is an unrelated
> > issue which prevents buildroot updates.
>  
>  This appears to be an ongoing issue.  I cannot rebuild libcurl until its
>  dependencies are rebuilt (hopefully done) and have landed in the build
>  root (still waiting), so cmake remains unfixed for now, I'm afraid. 
I'm looking into why the buildroot is so slow to populate. ;(

kevin

[Florian Weimer]

On 07/30/2017 10:01 PM, Kevin Fenzi wrote:
>  On 07/29/2017 12:08 PM, Florian Weimer wrote:
> > On 07/28/2017 10:58 PM, Florian Weimer wrote:
> >> Rebuilds are currently blocked on a new buildroot with some RPM
> >> debuginfo generation fixes.  Rumor has it that there is an unrelated
> >> issue which prevents buildroot updates.
> >
> > This appears to be an ongoing issue.  I cannot rebuild libcurl until its
> > dependencies are rebuilt (hopefully done) and have landed in the build
> > root (still waiting), so cmake remains unfixed for now, I'm afraid.
>  
>  I'm looking into why the buildroot is so slow to populate. ;( 
From my perspective, this looks much better now.  At least my own builds
enter the f27-build buildroot in a fairly timely fashion now.

The rebuilds I deemed most critical are still not done, but we are down
to just 12 packages.  java-1.8.0-openjdk and ceph are the current
bottlenecks.  We just have to exercise patience while they build.

Florian

[Kevin Fenzi]

On 07/30/2017 01:09 PM, Florian Weimer wrote:
>  On 07/30/2017 10:01 PM, Kevin Fenzi wrote:
> > On 07/29/2017 12:08 PM, Florian Weimer wrote:
> >> On 07/28/2017 10:58 PM, Florian Weimer wrote:
> >>> Rebuilds are currently blocked on a new buildroot with some RPM
> >>> debuginfo generation fixes.  Rumor has it that there is an unrelated
> >>> issue which prevents buildroot updates.
> >>
> >> This appears to be an ongoing issue.  I cannot rebuild libcurl until its
> >> dependencies are rebuilt (hopefully done) and have landed in the build
> >> root (still waiting), so cmake remains unfixed for now, I'm afraid.
> >
> > I'm looking into why the buildroot is so slow to populate. ;(
>  
> >From my perspective, this looks much better now.  At least my own builds
>  enter the f27-build buildroot in a fairly timely fashion now.
>  
>  The rebuilds I deemed most critical are still not done, but we are down
>  to just 12 packages.  java-1.8.0-openjdk and ceph are the current
>  bottlenecks.  We just have to exercise patience while they build. 
yep. It looks pretty normal now... I think the other day it was still
processing the backlog from the mass rebuild (the signing backlog).

kevin

[Florian Weimer]

On 07/27/2017 01:21 AM, Jason L Tibbitts III wrote:
>  I noticed one of my packages failed to build because the test suite
>  failed, but only on ppc64le.
>  
>  builddir/build/BUILD/cyrus-imapd-3.0.2/cunit/.libs/lt-unit: error while
>  loading shared libraries:
>  /builddir/build/BUILD/cyrus-imapd-3.0.2/imap/.libs/libcyrus_imap.so.0:
>  expected localentry:0 `sasl_client_new' 
cyrus-imapd builds again on ppc64le.  Among other things, I had to
rebuild perl-Time-HiRes because its build had not detected the
availability of clock_gettime. :-/

Thanks,
Florian

[Florian Weimer]

I rebuilt all immediately problematic packages which use pthread_ and
clock_ functions and had the ppc64le optimization, except kernel (which
will be fixed by the upgrade to kernel-4.13.0-0.rc3.git0.1.fc27) and
community-mysql (which ran into an apparently known issue related to
unstable test cases and failed to build on x86-64, not ppc64le-specific).

For ceph, I reenabled ppc64le support.

For python3, I had to disable split debuginfo to get it to build, based
on guidance from the RPM debuginfo folks (thanks Igor and Mark).

Beyond the minimal rebuild set, I attempted further rebuilds to avoid
subsequent breakage due to library updates.  I think we now have a
package set that is fairly risk-free as far as further uploads and
rebuilds are concerned (for example, I bootstrapped binutils, so that it
would not impacted by future glibc or gcc changes).  I did not rebuild
everything affected because I soon ran into hard-to-debug unrelated
build failures (e.g., for couchdb).  Some issues I fixed (MariaDB
breakage of lua-sql, s390x build failure of evince, highlight
/usr/share/doc dependency extraction issue).

Nick uploaded a new binutils version with the upstream patches, but as
far as I can tell, the stop-gap patch was sufficient.

We still need to do a new mass rebuild because the issue affected
feature detection.  For example perl-Time-HiRes lost its clock_gettime
support because a run-time check at build time failed.  I discovered
this only as the result of a build failure; such issues are really hard
to find by other means if everything automatically adjusts to the
missing feature.

Thanks,
Florian

[Florian Weimer]

On 07/31/2017 05:08 PM, Florian Weimer wrote:
>  I rebuilt all immediately problematic packages which use pthread_
> and
>  clock_ functions and had the ppc64le optimization, except kernel (which
>  will be fixed by the upgrade to kernel-4.13.0-0.rc3.git0.1.fc27) and
>  community-mysql (which ran into an apparently known issue related to
>  unstable test cases and failed to build on x86-64, not ppc64le-specific).
>  
>  For ceph, I reenabled ppc64le support.
>  
>  For python3, I had to disable split debuginfo to get it to build, based
>  on guidance from the RPM debuginfo folks (thanks Igor and Mark).
>  
>  Beyond the minimal rebuild set, I attempted further rebuilds to avoid
>  subsequent breakage due to library updates.  I think we now have a
>  package set that is fairly risk-free as far as further uploads and
>  rebuilds are concerned (for example, I bootstrapped binutils, so that it
>  would not impacted by future glibc or gcc changes).  I did not rebuild
>  everything affected because I soon ran into hard-to-debug unrelated
>  build failures (e.g., for couchdb).  Some issues I fixed (MariaDB
>  breakage of lua-sql, s390x build failure of evince, highlight
>  /usr/share/doc dependency extraction issue).
>  
>  Nick uploaded a new binutils version with the upstream patches, but as
>  far as I can tell, the stop-gap patch was sufficient.
>  
>  We still need to do a new mass rebuild because the issue affected
>  feature detection.  For example perl-Time-HiRes lost its clock_gettime
>  support because a run-time check at build time failed.  I discovered
>  this only as the result of a build failure; such issues are really hard
>  to find by other means if everything automatically adjusts to the
>  missing feature. 
Mmph, I forgot to mention two more things:

If you still see localentry:0 build failures after this, please send me
mail with a link to the build log.  This should not happen.

And this is the final update for this issue because I consider it resolved.

Thanks,
Florian

[Honggang LI]

On Mon, Jul 31, 2017 at 05:12:41PM +0200, Florian Weimer wrote:
>  
>  Mmph, I forgot to mention two more things:
>  
>  If you still see localentry:0 build failures after this, please send me
>  mail with a link to the build log.  This should not happen.
>  
hi,
 It seems issue persists for rdma-core ppc64le rebuilding. Please see:
https://koji.fedoraproject.org/koji/taskinfo?taskID=20826757

> And this is the final update for this issue because I consider it resolved.
> 
> Thanks,
> Florian
> _______________________________________________
> devel mailing list -- devel(a)lists.fedoraproject.org
> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org

[Florian Weimer]

On 08/01/2017 02:56 AM, Honggang LI wrote:
>  On Mon, Jul 31, 2017 at 05:12:41PM +0200, Florian Weimer wrote:
> >
> > Mmph, I forgot to mention two more things:
> >
> > If you still see localentry:0 build failures after this, please send me
> > mail with a link to the build log.  This should not happen.
> >
>  
>  hi,
>   It seems issue persists for rdma-core ppc64le rebuilding. Please see:
>  https://koji.fedoraproject.org/koji/taskinfo?taskID=20826757 
That was an old build.  It builds fine now:

  https://koji.fedoraproject.org/koji/taskinfo?taskID=20936388

Thanks,
Florian