pairing static analysis results with known crashes

[Martin Milata]

Hi devel,

it was suggested to me to repost here two messages originally sent to
firehose-devel list. Hopefully someone might find them interesting.

Martin

----- Forwarded message from Martin Milata <mmilata at redhat.com> -----

Date: Fri, 18 Oct 2013 15:14:57 +0200
To: firehose-devel at lists.fedoraproject.org
Subject: correlating static analysis results with known crashes

Hello,

I've implemented a proof-of-concept of an analysis that tries to pair
static analysis results with known crashes based on the source code
locations, as outlined in [1].

The code extends David Malcolm's mock-with-analysis and is available at
[2]. The machinery for generating static analysis results is unchanged
apart from a few fixes needed for it to run on Fedora 19. The script
make-simple-report.py was extended to accept second argument with crash
reports from FAF server, and the matching crashes are referenced in the
generated reports. There is currently no way to obtain the file with
crashes automatically, I got it from the server administrator.

I ran the analysis on following packages:
 * tracker-0.16.2-1.fc19
 * evolution-3.6.4-3.fc18
 * gnome-shell-3.6.3.1-1.fc18
 * nautilus-3.6.3-4.fc18
 * python-2.7.3-13.fc18
 * rhythmbox-2.98-4.fc18

Tracker was chosen arbitrarily, the rest of the builds are those that
have the highest number of distinct crashes. The results can be viewed
at [3] and given that they were obtained from packages with the highest
number of collected crashes, they don't seem to be very encouraging.
There are only three [4,5,6] matches that are not obvious false
positives. All the data needed to reproduce this should be available at
[7].

There are two main causes of false positives:
 * The code considers all static analysis results, not only those from
   tests for behaviour that would result in a crash at runtime.
 * It considers all stack frames in a crash, not just the topmost one.

As a side note, all three matches come from the clang static analyzer,
which for some reason fails for quite a lot of source files.

What do you think?

Thanks,
Martin


[1] https://lists.fedoraproject.org/pipermail/firehose-devel/2013-October/000065.html
[2] https://github.com/mmilata/mock-with-analysis/tree/crash-correlation
[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/
[4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5223
[5] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5848
[6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97.html#file-71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97-line-1171
[7] http://mmilata.fedorapeople.org/firehose-crash-correlation.tar.xz

----- End forwarded message -----
----- Forwarded message from Martin Milata <mmilata at redhat.com> -----

Date: Tue, 22 Oct 2013 12:26:06 +0200
To: firehose-devel at lists.fedoraproject.org
Subject: Re: correlating static analysis results with known crashes

I uploaded the clang-analyzer-generated html reports for the three
"interesting" cases that the script found and took a further look at
them.

* nautilus 1 [1], clang-analyzer report [2]

The trace from the static analyzer consists of
nautilus_file_operations_copy_move calling nautilus_file_operations_move
which then segfaults. This agrees with the backtraces. Unfortunately
there is no BZ ticket associated probably due to too few people affected
by this bug

* nautilus 2 [3], clang-analyzer report [4]

Only nautilus_file_operations_copy_move is in the static analyzer trace.
There's bugzilla ticket [5] with full backtrace corresponding to this
problem.

* python [6], clang-analyzer report [7]

The trace consists of PyObject_Unicode calling PyObject_GetAttr, which
is not the case of the linked backtrace, making this pair a false
positive. The trace from clang-analyzer describes a real bug though, one
that has been already fixed [8][9].

Didn't know clang-analyzer can do inter-procedural analysis, that's
nice.


[1] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5223
[2] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-build/report-eEHeBD.html#Path1

[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sources/a401071da79df10a29243dc6aaba37466d070c25.html#file-a401071da79df10a29243dc6aaba37466d070c25-line-5848
[4] http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-build/report-YLMHRs.html#Path1
[5] https://bugzilla.redhat.com/show_bug.cgi?id=860109

[6] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources/71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97.html#file-71ff831e4d3c0af53bfbd0ed28f5aef3483d2b97-line-1171
[7] http://mmilata.fedorapeople.org/firehose-crash-correlation/python/scan-build/report-cp0FYq.html#Path1
[8] http://bugs.python.org/issue16839
[9] http://hg.python.org/cpython/rev/0012d4f0ca59

----- End forwarded message -----