JSON serialization
by David Malcolm
I've implemented JSON support for Firehose - as of:
https://github.com/fedora-static-analysis/firehose/commit/453f1944552bdc1...
everything has a to_json and from_json method, and the unit tests verify
that everything roundtrips cleanly from Python objects through JSON back
to Python objects.
I've been using this to slurp large numbers of results into a MongoDB
database, similar to the "storz" work that Paul posted here. I'm not
sure how much I like this approach - I'm much more familiar with SQL, so
that aspect is experimental for now.
[I much prefer XML as the serialization format, given that we can
validate files against the schema]
Dave
11 years, 2 months
[PATCH 1/2] mock-with-analysis: do not hard-wire rpm top directory
by Kamil Dudka
---
mock-with-analysis | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/mock-with-analysis b/mock-with-analysis
index 9964c45..9390fb0 100755
--- a/mock-with-analysis
+++ b/mock-with-analysis
@@ -96,9 +96,12 @@ def setup_chroot(mockcfg, srpmpath):
mockcfg.cmd(['--install', 'cppcheck'])
mockcfg.cmd(['--install', 'clang-analyzer'])
+ # Read the rpm top directory (e.g. /home/david/rpmbuild)
+ rpmtopdir = Popen(["rpm", "--eval", "%{_topdir}"], stdout=PIPE).communicate()[0].rstrip()
+
# Install the pre-built gcc-python-plugin (from the firehose branch):
mockcfg.cmd(['--install',
- '/home/david/rpmbuild/RPMS/x86_64/gcc-python2-plugin-0.11.firehose-1.fc17.x86_64.rpm'])
+ rpmtopdir + '/RPMS/x86_64/gcc-python2-plugin-0.11.firehose-1.fc17.x86_64.rpm'])
# Copy up latest version of the libcpychecker code from a working copy
# overriding the copy from the pre-built plugin:
@@ -113,7 +116,7 @@ def setup_chroot(mockcfg, srpmpath):
# FIXME: package these and get them into Fedora:
mockcfg.cmd(['--install', '../firehose/dist/firehose-0.0.1-1.noarch.rpm'])
mockcfg.cmd(['--install', '../gccinvocation/dist/gccinvocation-0.0.1-1.noarch.rpm'])
- mockcfg.cmd(['--install', '/home/david/rpmbuild/RPMS/x86_64/python-subprocess32-3.2.3-1.fc17.x86_64.rpm'])
+ mockcfg.cmd(['--install', rpmtopdir + '/RPMS/x86_64/python-subprocess32-3.2.3-1.fc17.x86_64.rpm'])
# ^^^ review request for python-subprocess32:
# https://bugzilla.redhat.com/show_bug.cgi?id=910891
--
1.7.1
11 years, 2 months
Storing results in a DB
by Paul Tagliamonte
Hello, Firehosen,
So, forewarning: This is all very very hackey and gross.
I wrote a small native-object converter[1], which uses __slots__ to
populate a dict of dicts (it looses a bit of data, but is otherwise
rad), in order to store the entire Firehose report in a MongoDB
document datastore.
I've only loaded Debian's Chromium build-logs, but I plan on doing a
full run this weekend, if all goes well.
> db.results.find().count()
211
> db.results.find({"results.location.file.givenpath":
"third_party/libphonenumber/src/phonenumbers/utf/unicodetext.cc"}).count()
51
> db.results.find({"metadata.sut.buildarch": "armhf"}).count()
22
Two things:
1) This is gross. I don't like how I'm converting to Pythonic objects.
Who's +1 / -1 on adding this as a top-level method like to_xml &
from_xml ?
2) I like Mongo, a lot. Would anyone mind if I supported it more? I can
work with PostgreSQL or another relational DB, but I've grown to
love MongoDB the more I use it :)
Cheers,
Paul
[1]: https://github.com/paultag/storz/blob/master/storz/decompress.py
https://github.com/paultag/storz/blob/master/storz/store.py
--
.''`. Paul Tagliamonte <paultag(a)debian.org>
: :' : Proud Debian Developer
`. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87
`- http://people.debian.org/~paultag
11 years, 2 months
firehose: Please add a configuration sub-element to the generator
by Florian Weimer
Some tools produce vastly different output based on configuration
settings. It would be nice to have some indication about the
configuration used.
It might also make sense to differentiate between the actual tool and
the generator which translates tool-specific output to the firehose
representation.
--
Florian Weimer / Red Hat Product Security Team
11 years, 2 months
Adding an <info> tag?
by David Malcolm
Currently we have two kinds of result within an <analysis>:
* <issue> elements, representing a warning from an analysis code about
a possible problem with the software-under-test (SUT)
* <failure> elements, representing a report about a failure of an
analysis tool to achieve full coverage of the SUT (e.g. the tool crashed
or timed out)
I'm thinking it might be useful to be able for an analyzer to record
arbitrary extra information about the software-under-test, where it
isn't an issue as such, but for custom data-gathering about the SUT, so
I'm considering adding an <info> element as a third type of result, with
an optional <location> and allowing storing <custom-fields>.
The motivating use-case for me is in my mass-analysis of Python
extension modules: I though it might be interesting to gather some stats
on uses of the CPython API, and it's not a bit stretch to add this to
cpychecker's firehose output. Perhaps one could use this to build an
lxr-style indexing tool also.
Thoughts?
Dave
11 years, 2 months
