Investigation of the F23 mass rebuild

[Adam Jackson]

Following up on the hardened cflags change in F23, I wanted to gather
some statistics on the actual impact: what the most impacted packages
and apps are, what the typical overhead is like, etc. The results
are... unpleasant, but not so much because of the hardening change
itself. I started by grabbing the x86_64 packages of everything koji
believes is in F23, unpacking them all, and then removing every file
that wasn't a dynamic ELF object. From this set, some observations.

Of something like 14785 (binary) packages containing dynamic ELF
objects, 541 have not successfully built in F23, which you can tell
because their dist tag is wrong. This includes 443 from F22, 47 from
F21, 44 from F20 (!) and 7 from F19 (!!). Many of these seem to be from
gcc getting increasingly strict about what qualifies as legal C++. Most
of these are fairly low impact for most systems, but it is worrying
that these failures are sticking around for so long.

Of approximately 56000 packaged dynamic ELF objects, around 16000 are
still not linked with -z now. Of these, 7 are from F19 builds, 95 from
F20, 248 from F21, and 3987 from F22, and the remaining 12037 from F23.

Those 12037 binaries from successful f23 builds (not necessarily from
builds with the hardening change in place) come from 3038 binary
packages. 2041 of those packages provide exactly one non-now binary;
this is somewhat expected given our library packaging guidelines, but
it means it's probably going to be difficult to make big dents in the
problem.

There are 173 non-now binaries installed under /usr/share. 68 of those
are ircd-ratbox, and 56 are rubygem-gherkin. 7 are aircrack-ng, which
installs them into freaking /usr/share/doc! Come on, people.

There are 1378 non-now libraries directly under /usr/lib64 (presumably
just %{_libdir} really). Of these, probably most worrying (according to
my personal sense of outrage) are from: boost-*, bzip2-libs, cups-libs,
evolution-data-server, festival, fftw, giflib, gnutls, libdb and
libdb4, libgo, libicu, libpurple, libselinux, nspr, nss, openssl-libs,
postgresql-libs, rpm-build-libs and rpm-libs, sendmail-milter, tog
-pegasus-libs, unixODBC, xen-libs, xmlrpc-c, and zlib.

There are 5223 non-now binaries in /usr/bin or /usr/sbin. Most of these
are instances of binutils or gcc for various targets; I think it might be nice to fix that, but not urgent for security. Outside that, of 
packages not mentioned in the libs paragraph above, the most worrying
include: aircrack-ng, amanda, aoetools, bogofilter, ceph, clamav,
crypto-utils, docker, exim, keyutils, lsof, mimedefang, nacl, nfs
-utils, perl, spamassassin, wget, and wireshark.

There are 16615 executables in /usr/bin and /usr/sbin.  Of these, 5166
are not PIEs.

---

At this point I gave up trying to get real metrics on the impact of
hardening, because it's clear it's not been done. Since the change was
done by changing the rpm build macros, I think we can conclude that the
build macros aren't being applied. Granted, packages can disable the
hardened build macros, but the packages I've called out above aren't
trying to disable them, or at least not doing so with %undefine.

Beyond that, the fact that we have such blatant packaging errors, and
that nearly 4% of our binary packages haven't rebuilt in F23, is quite
worrisome.

- ajax