Build fails in koji, but not anywhere else ... how to debug?

[Jim Meyering]

Tom Lane wrote:
> I've been pursuing, with increasing frustration, the seemingly simple
> goal of getting mysql to rebuild in rawhide since the mass rebuild.
> It failed in the mass rebuild (on the same source code which had worked
> fine a few weeks before), and has failed multiple attempts since then,
> for example
> http://koji.fedoraproject.org/koji/getfile?taskID=1596536&name=build.log
> http://koji.fedoraproject.org/koji/getfile?taskID=1606630&name=build.log
> http://koji.fedoraproject.org/koji/getfile?taskID=1606834&name=build.log
> The symptoms are not consistent, although there is some repetition;
> for example the first and third runs above encountered the same failure.
> It's usually the x86_64 build that dies, but I think that may just be
> a reflection of the x86_64 builder being faster than the others.
>
> What is frustrating me is that I can't reproduce the problem outside
> koji where I might have a shot at debugging it.  It works fine in F-10,
> F-11, rawhide, rawhide-via-mock, and everything else I've tried on my
> own machines.  I've tried to try it on RHTS machines, but since neither
> F-11 nor rawhide install successfully on those machines, that attempt
> didn't get far either.  (And shouldn't somebody be paying closer
> attention to that?)
>
> I'm starting to wonder about corrupted ccache on the koji machines,
> although unless they all share a common cache that theory doesn't seem
> to hold much water.  I wonder if anyone else has a theory, or at least a
> suggestion how to debug this problem?

Hi Tom,

Sorry if this is too obvious, but have you already tried
running the failing test via valgrind?

Also, it's good to set these variables in your environment,
not just when testing, but for day to day usage:

    # The 171 is arbitrary.  any value in 1..255 usually works as well
    MALLOC_PERTURB_=171 MALLOC_CHECK_=3

In fact, if you're developing and *don't* yet always set those in your
environment, add these lines to your shell startup files right away:

    export MALLOC_CHECK_=3
    # http://udrepper.livejournal.com/11429.html
    export MALLOC_PERTURB_=$(($RANDOM % 255 + 1))