Dealing with static code analysis in Fedora

[David Malcolm]

On Wed, 2012-12-12 at 16:46 +0100, Michal Toman wrote:
> Hi Dave!
> 
> We have started a similar project in ABRT some year ago. The original 
> purpose was to automatically determine some crash characteristics 
> (security impact, unchecked user input etc.). The overall idea is 
> rebuilding the given package with some compiler plugin and go through 
> the AST based on coredump's stack.

Sounds interesting.  Is the idea to capture some higher-level
information about the code?

If I'm understanding things right, your code is looking at a coredump,
which is a snapshot in time of the state of a process (the moment of
death, in fact).  For example, each thread will have the program counter
at a specific location.

What I'm trying to track is a predicted path through the code
(potentially with state annotations): "first the program counter is at
foo, then it's at bar, and then at baz a crash happens".

> We have built an infrastructure working on top of ABRT Server[1]. This 
> is because of its backend synchronizing with several Fedora services 
> (koji, pkgdb, bugzilla) storing all the data in one database and thus 
> ability to make fast cross-service queries. The "storage" (that's how we 
> call it) is more complex - it uses DB for relational data and saves 
> large files (like RPMs) on the FS.

I had a go at reading through the faf sources.  Am I right in thinking
that the "storage" you refer to is the collection of Django apps within
the "pyfaf" subdirectory:
  http://git.fedorahosted.org/cgit/faf.git/tree/pyfaf

> The storage contains (almost) all packages built in koji. We were trying 
> to use mock & yum for a long time, but we were not able to make it work 
> correctly with such a complex repo. That's why we have created our own 
> dependency solver based on libsolv (using the DB as metadata source) and 
> use rpm directly to install the resulting set. This resulted into a fast 
> chroot installer able to interrupt build process in many phases.

> Some guys working on static analysis at Masaryk University, Brno, CZ 
> showed interest in the project. They are working on various clang 
> plugins for static analysis, but have no real input data. We agreed to 
> implement a service rebuilding Fedora packages with LLVM/Clang and 
> providing the output (LLVM bitcode files). They will provide the 
> analysis tools.
> 
> This showed up as a non-trivial problem, because a simple CC=clang is 
> not enough :). The thing is however implemented and I've personally 
> rebuilt Fedora a few times, leaving me with some 60% of packages 
> successfully processed and a few ideas for improvement. The service has 
> not yet been deployed in Fedora infrastructure, because we 1) don't 
> consider it ready 2) lack hardware capacity.

As noted in another post, when I tried running all of Fedora through my
gcc-with-cpychecker, I first tried setting CC, but plenty of things
don't use make.  I then tried setting the rpm build flags, but plenty of
things ignore them.  I finally hit on the idea of moving /usr/bin/gcc in
the chroot to /usr/bin/the-real-gcc, and replacing /usr/bin/gcc with a
script to invoke the-real-gcc with the relevant flags to inject my gcc
plugin, with similar hacks for g++ and c++.  Heh, it's just a temporary
chroot :)  

See
http://git.fedorahosted.org/cgit/gcc-python-plugin.git/tree/misc/fedora/mass-rebuild.py
for the gory details.  

It's a nasty nasty hack... but it works.  Presumably you could do
something similar for your LLVM work.

> This is more or less the current state. We are open to any discussion, 
> new ideas, use cases or patches :).

If I understand your work, you're attempting to gather LLVM bitcode for
all of the sources in Fedora, so that analysis tools can be readily run
on them.  You're also dealing with the issue of tracking "what is
Fedora" within a database, which is no mean feat (this was where I
stumbled in my old "rpmgrok" tool).

I'm approaching this from a different angle: my analysis tools are run
directly from within GCC during a build, and the results are stored
somewhere.

The issue I want to solve is that of managing false positives.  Any
static code analysis tool is by necessity just an approximation of the
actual behavior of the code, and so will have false alarms.  We need a
way of keeping those false alarms to an acceptable level whilst gleaning
useful data from the analysis.  So my proposal is less about: "here's a
big collection of data you can run tests on", and more about: here's a
big body of test *results* of unknown quality":  what can we do with it
so that tests can be improved to the point where package maintainers
might want to care about the results, and at that point, management of
said results.

Or, from an other perspective, the use cases I'm interested are:

  * Anna the Analysis Tool Author: Anna has written a static analysis
tool that detects a particular antipattern in, say, C++ code.  It works
for her example code, but she wants to try running it on all of Fedora.
She expects her tool to crash and to have lots of false positives, etc,
but that after poring over these results she will be able to make a
later version of the checker be significantly more robust and useful.

  * Ferdinand the Fedora Package Maintainer: Ferdinand maintains 34
packages within Fedora.  He's interested in static analysis in as much
as it can proactively find bugs, but doesn't want to deal with hundred
of false alarms, so he only wants to bother with checkers that have
graduated from "experimental" to "useful" status.

  * Uwe the Upstream Maintainer: Uwe is the original author of one of
Ferdinand's packages.  Like Ferdinand he's interested in proactive
bug-finding.  Having a useful service run and hosted by Fedora might
encourage Uwe to use and recommend Fedora.

Hope that makes sense

> 
> Sources: http://git.fedorahosted.org/cgit/faf.git/
> RFE: https://fedorahosted.org/abrt/newticket?component=faf
> 
> Michal & ABRT
> 
> [1] http://abrt.fedoraproject.org
> 
> On 2012-12-11 22:52, David Malcolm  wrote:
> > A while back I ran my static checker on all of the Python extension
> > modules in Fedora 17:
> >    http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts
> >
> > I wrote various scripts to build the packages in a mock environment that
> > injects my checker into gcc, then wrote various scripts to triage the
> > results.  I then filed bugs by hand for the most important results,
> > writing some more scripts along the way to make the process easier.
> >
> > This led to some valuable bug fixes, but the mechanism for running the
> > analysis was very ad hoc and doesn't scale.
> >
> > In particular, we don't yet have an automated way of rerunning the
> > tests, whilst using the old results as a baseline.  For example it would
> > be most useful if only new problems could be reported, and if the system
> > (whatever it is) remembered when a report has been marked as a true bug
> > or as a false positive.  Similarly, there's no automated way of saying
> > "this particular test is bogus; ignore it for now".
> >
> > I'm wondering if there's a Free Software system for doing this kind of
> > thing, and if not, I'm thinking of building it.
> >
> > What I have in mind is a web app backed by a database (perhaps
> > "checker.fedoraproject.org" ?)
> >
> > We'd be able to run all of the code in Fedora through static analysis
> > tools, and slurp the results into the database: primarily my
> > "cpychecker" work, but we could also run the clang analyzer etc.  I've
> > also been working on another as-yet-unreleased static analysis tool for
> > which I'd want a db for the results.  What I have working is a way to
> > inject an analysis payload into gcc within a mock build, which dumps
> > JSON report files into the chroot without disturbing the "real" build.
> > The idea is then to gather up the JSON files and insert the report data
> > into the db, tagging it with version information.
> >
> > There are two dimensions to the version information:
> >   (A) the version of the software under analysis
> >           (name-version-release.arch)
> >   (B) the version of the tool doing the analysis
> >
> > We could use (B) within the system to handle the release cycle of a
> > static analysis tool.  Initially, any such analysis tools would be
> > regarded as "experimental", and package maintainers could happily ignore
> > the results of such a tool.  The maintainer of an analysis tool could
> > work on bug fixes and heuristics to get the signal:noise ratio of the
> > tool up to an acceptable level, and then the status of the analysis tool
> > could be upgraded to an "alpha" level or beyond.
> >
> > Functional Requirements:
> >    * a collection of "reports" (not bugs):
> >      * interprocedural control flow, potentially across multiple source
> >        files (potentially with annotations, such as value of variables,
> >        call stack?)
> >        * syntax highlighting
> >        * capturing of all relevant source (potentially with headers as
> >          well?)
> >        * visualization of control flow so that you can see the path
> >          through the code that leads to the error
> >      * support for my cpychecker analysis
> >      * support for an as-yet-unreleased interprocedural static analysis
> >        tool I've been working on
> >      * support for reports from the clang static analyzer
> >      * ability to mark a report as:
> >        * a true bug (and a way to act on it, e.g. escalate to bugzilla or
> >          to the relevant upstream tracker)
> >        * a false positive (and a way for the analysis maintainer to act
> >          on it)
> >        * other bug associations with a report? (e.g. if the wording from
> >          the tool's message could be improved)
> >        * ability to have a "conversation" about a report within the UI as
> >          a series of comments (similar to bugzilla).
> >      * automated report matching between successive runs, so that the
> >        markings can be inherited
> >      * scriptable triage, so that we can write scripts that mark all
> >        reports matching a certain pattern e.g. as being bogus, as being
> >        security sensitive, etc
> >      * potentially: debug data (from the analysis tool) associated with a
> >        report, so that the maintainers of the tool can analyze a false
> >        positive
> >      * ability to store crash results where some code broke a static
> >        analysis tool, so that the tool can be fixed
> >    * association between reports and builds
> >    * association between builds and source packages
> >    * association between packages and people, so that you can see what
> >      reports are associated with you (perhaps via the pkgdb?)
> >    * prioritization of reports to be generated by the tool
> >    * association between reports and tools (and tool versions)
> >    * "quality marking" of tool versions, so that we can ignore "alpha"
> >      versions of tools and handle phasing in of a new static analysis
> >      tool without spamming everyone
> >    * ability to view the signal:noise ratio of a version of a tool
> >
> > Nonfunctional requirements:
> >    * Free Software
> >    * sanely deployable within Fedora infrastructure
> >    * sane code, since we're likely to want to extend it (fwiw I'd be most
> >      comfortable with a Python implementation).
> >    * able to scale to running all of Fedora through multiple tools
> >      repeatedly
> >    * many simultaneous users
> >    * will want an authentication system so that we can associate comments
> >      with users.  Eventually we may want a way of embargoing
> >      security-sensitive bugs found by the tool so that they're only
> >      visible by a trusted cabal.
> >    * authentication system to support FAS, but not require it, in case
> >      other people want to deploy such a tool.  Maybe OpenID?
> >
> > Implementation ideas:
> >    * as well as a relational database for the usual things, perhaps a
> > lookaside of source files stored gzipped, with content-addressed storage
> > e.g. "0fcb0d45a6353e150e26f1fa54d11d7be86726b6" stored gzipped as:
> >      objects/0f/cb0d45a6353e150e26f1fa54d11d7be86726b6
> > (yes, this looks a lot like git)
> >
> > Thoughts?  Does such a thing already exist?
> >
> > It might be fun to hack on this at the next FUDcon.
> >
> > Dave
> >