correlating static analysis results with known crashes

[Martin Milata]

Hello,

I've implemented a proof-of-concept of an analysis that tries to pair
static analysis results with known crashes based on the source code
locations, as outlined in [1].

The code extends David Malcolm's mock-with-analysis and is available at
[2]. The machinery for generating static analysis results is unchanged
apart from a few fixes needed for it to run on Fedora 19. The script
make-simple-report.py was extended to accept second argument with crash
reports from FAF server, and the matching crashes are referenced in the
generated reports. There is currently no way to obtain the file with
crashes automatically, I got it from the server administrator.

I ran the analysis on following packages:
 * tracker-0.16.2-1.fc19
 * evolution-3.6.4-3.fc18
 * gnome-shell-3.6.3.1-1.fc18
 * nautilus-3.6.3-4.fc18
 * python-2.7.3-13.fc18
 * rhythmbox-2.98-4.fc18

Tracker was chosen arbitrarily, the rest of the builds are those that
have the highest number of distinct crashes. The results can be viewed
at [3] and given that they were obtained from packages with the highest
number of collected crashes, they don't seem to be very encouraging.
There are only three [4,5,6] matches that are not obvious false
positives. All the data needed to reproduce this should be available at
[7].

There are two main causes of false positives:
 * The code considers all static analysis results, not only those from
   tests for behaviour that would result in a crash at runtime.
 * It considers all stack frames in a crash, not just the topmost one.

As a side note, all three matches come from the clang static analyzer,
which for some reason fails for quite a lot of source files.

What do you think?

Thanks,
Martin


[1] https://lists.fedoraproject.org/pipermail/firehose-devel/2013-October/000...
[2] https://github.com/mmilata/mock-with-analysis/tree/crash-correlation
[3] http://mmilata.fedorapeople.org/firehose-crash-correlation/
[4]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc...
[5]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc...
[6]
http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources...
[7] http://mmilata.fedorapeople.org/firehose-crash-correlation.tar.xz

[Martin Milata]

I uploaded the clang-analyzer-generated html reports for the three
"interesting" cases that the script found and took a further look at
them.

* nautilus 1 [1], clang-analyzer report [2]

The trace from the static analyzer consists of
nautilus_file_operations_copy_move calling nautilus_file_operations_move
which then segfaults. This agrees with the backtraces. Unfortunately
there is no BZ ticket associated probably due to too few people affected
by this bug

* nautilus 2 [3], clang-analyzer report [4]

Only nautilus_file_operations_copy_move is in the static analyzer trace.
There's bugzilla ticket [5] with full backtrace corresponding to this
problem.

* python [6], clang-analyzer report [7]

The trace consists of PyObject_Unicode calling PyObject_GetAttr, which
is not the case of the linked backtrace, making this pair a false
positive. The trace from clang-analyzer describes a real bug though, one
that has been already fixed [8][9].

Didn't know clang-analyzer can do inter-procedural analysis, that's
nice.


[1]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc...
[2]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-...

[3]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/sourc...
[4]
http://mmilata.fedorapeople.org/firehose-crash-correlation/nautilus/scan-...
[5] https://bugzilla.redhat.com/show_bug.cgi?id=860109

[6]
http://mmilata.fedorapeople.org/firehose-crash-correlation/python/sources...
[7]
http://mmilata.fedorapeople.org/firehose-crash-correlation/python/scan-bu...
[8] http://bugs.python.org/issue16839
[9] http://hg.python.org/cpython/rev/0012d4f0ca59

[Paul Tagliamonte]

On Tue, Oct 22, 2013 at 12:26:06PM +0200, Martin Milata wrote:
>  I uploaded the clang-analyzer-generated html reports for the three
>  "interesting" cases that the script found and took a further look at
>  them.
>  
>  * nautilus 1 [1], clang-analyzer report [2]
>  
>  The trace from the static analyzer consists of
>  nautilus_file_operations_copy_move calling nautilus_file_operations_move
>  which then segfaults. This agrees with the backtraces. Unfortunately
>  there is no BZ ticket associated probably due to too few people affected
>  by this bug
>  
>  * nautilus 2 [3], clang-analyzer report [4]
>  
>  Only nautilus_file_operations_copy_move is in the static analyzer trace.
>  There's bugzilla ticket [5] with full backtrace corresponding to this
>  problem.
>  
>  * python [6], clang-analyzer report [7]
>  
>  The trace consists of PyObject_Unicode calling PyObject_GetAttr, which
>  is not the case of the linked backtrace, making this pair a false
>  positive. The trace from clang-analyzer describes a real bug though, one
>  that has been already fixed [8][9].
>  
>  Didn't know clang-analyzer can do inter-procedural analysis, that's
>  nice. 
Thrilling stuff, nice work!

I'll soon have a corpus of checks being run against Debian packages,
I'll be sure to forward you data points (if y'all have the same
source/version pair in Fedoraland)

Keep up the great work,
  Paul

-- 
 .''`.  Paul Tagliamonte <paultag(a)debian.org&gt;
: :'  : Proud Debian Developer
`. `'`  4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87
 `-     http://people.debian.org/~paultag

[Martin Milata]

On Tue, Oct 22, 2013 at 11:27:45 -0400, Paul Tagliamonte wrote:
>  Thrilling stuff, nice work!
>  
>  I'll soon have a corpus of checks being run against Debian packages,
>  I'll be sure to forward you data points (if y'all have the same
>  source/version pair in Fedoraland) 
Thanks! It would be interesting to run the analysis on those, though the
possible differences between Debian and Fedora sources could pose a
problem. E.g. a patch in one package and not the other might cause the
line numbers to disagree. I have no idea how often this is the case.

Martin

[David Malcolm]

On Tue, 2013-10-29 at 16:05 +0100, Martin Milata wrote:
>  On Tue, Oct 22, 2013 at 11:27:45 -0400, Paul Tagliamonte wrote:
>  > Thrilling stuff, nice work!
>  > 
>  > I'll soon have a corpus of checks being run against Debian packages,
>  > I'll be sure to forward you data points (if y'all have the same
>  > source/version pair in Fedoraland)
>  
>  Thanks! It would be interesting to run the analysis on those, though the
>  possible differences between Debian and Fedora sources could pose a
>  problem. E.g. a patch in one package and not the other might cause the
>  line numbers to disagree. I have no idea how often this is the case. 
That in itself might be something we could track using firehose,
perhaps?   i.e. have an <info> element that says that the code is
patched downstream by a particular distribution.  Then the UI can render
those elements (though which version of the source would you render in
such a situation), and one can run a query showing patches across
multiple distros and packages.

(Not sure if this is a good idea, but I thought I'd share it)

Dave

[Stefano Zacchiroli]

On Tue, Oct 29, 2013 at 11:10:32AM -0400, David Malcolm wrote:
>  That in itself might be something we could track using firehose,
>  perhaps?  i.e. have an <info> element that says that the code is
>  patched downstream by a particular distribution.  Then the UI can
>  render those elements (though which version of the source would you
>  render in such a situation), and one can run a query showing patches
>  across multiple distros and packages. 
Uhm, so the broader need here is the ability to correlate different
distro-specific versions with one another or, in fact, to the respective
upstream version. We can do that via external databases, but it would
add a pretty heavy infrastructure dependency.

IMO it would be better to pursue one of the following two solutions (or
even both):

- add a new sub-element to <sut> which mentions the *upstream* version;
  once we have that we can correlate reports from different distros via
  the upstream version (if there are significant differences, that
  should come from the distro-specific patching)

- add a new <context> or <excerpt> sub-element to failure/info/etc. that
  can be used to add snippets of code around the location the static
  analysis tool is pointing at. The idea of this is the same of contexts
  for textual diffs: by comparing them we will be able to understand if
  we're talking about the same code or, due to patched, significantly
  different parts of it.

  Of course it's an approximated solution, as the failure might descend
  from patches far far away in the code base, but if it works for diff,
  I think it'd be good enough for us as well. (And if we also have the
  upstream version, we can always lookup the distro-specific patches by
  external means and compare those.)

Just my 0.02 EUR,
Cheers.
-- 
Stefano Zacchiroli  . . . . . . .  zack(a)upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Former Debian Project Leader  . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »