Investigation of the F23 mass rebuild

[Adam Jackson]

On Thu, 2015-07-02 at 16:24 +0100, Jonathan Underwood wrote:
> On 2 July 2015 at 15:49, Adam Jackson <ajax at redhat.com> wrote:
> > Following up on the hardened cflags change in F23, I wanted to gather
> > some statistics on the actual impact: what the most impacted packages
> > and apps are, what the typical overhead is like, etc. The results
> > are... unpleasant,
> 
> [snip]
> 
> Impressive data mining.. for those following along for educational
> purposes, care to share the scripts you used for this someplace?

I didn't really write down the shell pipelines I used as scripts up
front. But it was something along the lines of:

# grab stuff from rawhide
% koji -q list-tagged-pkgs f23 | awk '{ print $1 }' | \
>     xargs -n1 -P4 koji download-build --arch x86_64

# unpack
% echo *.rpm | xargs -n1 -P4 rpmdev-extract

# grind away everything that isn't an elf file
% cat > elf-p
#!/bin/sh
file -b "$1" | grep -q ELF
^D
% chmod u+x elf-p
% find . -type f \( -exec elf-p {} \; -o -delete \)

# nuke everything that's not a dynamic ELF object
% find . -name \*.o -delete

# Check if a binary was not linked with -z now
% find . -type f | while read i ; do
>    eu-readelf -d "$i" | grep -q BIND_NOW || echo $i
> done

# Find the ten packages with the most non-now objects
% !! | cut -f1 -d/ | sort | uniq -c | sort -nk1 | tail -10

etc.  I'm getting the details wrong there regarding paths containing
spaces, but it turns out there are any elf files in such paths.  Also
this is probably a lot faster if you trim each package as you unpack it
rather than force it all out to disk.

What I _am_ going to need to really write is some tools to inspect
entire loaded object trees for relocation cost and useless linkage, at
which point shell is clearly the wrong language to be using.

- ajax