gcc-python-plugin March 2012

gcc-python-plugin@lists.fedorahosted.org

8 participants
9 discussions

The Include Macro
by Nicholas Pasternack 08 May '12

08 May '12

The following header does not work on all UNIX machines or machines running multiple versions of python: #include <Python.h> Depending on what version of Python you use, the header needs to change to: #include <python(version).(release)/Python.h> For instance, if you have python 2.7, the include line should read: #include <python2.7/Python.h> To the best of my knowledge, there are no Macros from the gcc that define the Python version the user is running. However, I know the following will be compatible with Mac computers: #ifdef __APPLE__ #include <python2.6/Python.h> #else #include <Python.h> #endif I think there should be a short explanatory note in the README describing how to change the include line; if the user has multiple versions of Python, the code will not compile unless you specifically state what version of Python to use in the include line. Nick

2 16

Psycopg analysis: final comments
by Daniele Varrazzo 30 Mar '12

30 Mar '12

Hello David, We have released psycopg 2.4.5 <http://initd.org/psycopg/articles/2012/03/29/psycopg-245-released/> which, apart from a few bug fixes reported by the users, has had the entire code base cleaned up using the static checker. I haven't managed to run it without false positive, which was probably too high as a target. Some reference counts are not going to be set to zero, just because they are not: it is the case of static variables, for instance we use the python string "NULL" over and over in the library so we just keep a reference to it. Anyway, the checker has allowed me to clean up 47 distinct occurrences of missing checks for errors on return values (thus likely to segfault downstream due to null dereferencing), 14 refcount errors (mostly related to global objects initializations, so not live memory leak) and four wrong function arguments check, three of which exposed to python and leading to a segfault if the wrong object was used. I'm happy it hasn't found any major disaster, our codebase wasn't that bad, but it's great it has made the module more robust in the face of the unexpected (that is, in case of memory error it won't be our module to segfault hiding the real cause of the memory error). So, thank you very much for the tool and the assistance in using it. I'll look forward to try and contribute something else to your project. -- Daniele

2 1

Ideas for getting involved
by David Malcolm 28 Mar '12

28 Mar '12

I've added a new section to the docs "Getting Involved", and added a list of ideas on some possible uses of the plugin; see: http://readthedocs.org/docs/gcc-python-plugin/en/latest/getting-involved.ht… (some of these were mine, others were posted to this list). If anyone is interested in tackling any of these, that would be great! (please let this list know if you do). Note that some of the ideas are likely to be insanely hard^W^Wsomewhat challenging, but I've marked those accordingly! Of course, there's still plenty of work to be done on my own motivating use-case for the plugin: reference-count checking of CPython extension code. Have fun Dave

2 2

Trying out the new HTML visualization on a complicated example
by David Malcolm 27 Mar '12

27 Mar '12

Buck Golemon and others have been working on a new HTML visualization for bug reports from gcc-with-cpychecker: https://fedorahosted.org/pipermail/gcc-python-plugin/2012-March/000210.html You can see an example of the new visualization here: http://fedorapeople.org/~dmalcolm/gcc-python-plugin/2012-03-19/example/exam… I had a go at running the new HTML visualization on a historical example of a real-world bug that he had run into: a nasty reference-counting bug in some complicated code in pylibmc: python bindings for a library for connecting to memcached. The bug can be seen here: https://github.com/lericson/pylibmc/blob/bc1787c3e74ddebb44736fc4eaf8333a2e… Series of events: Everything succeeds up to line 1408. 1. Successfully allocate a `retval` dict #1408 2. _PylibMC_parse_memcached_result fails and returns NULL #1425 3. jump to unpack_error #1427 4. Decrement refcount of retval #1439 5. Return a dict (retval) with zero refcount #1460 6. Bug report: https://github.com/lericson/pylibmc/issues/68 When the refcount hits zero at step 4, the dict is deallocated, so under the above circumstances the function is returning a pointer to deallocated memory. The upstream fix was here: https://github.com/lericson/pylibmc/commit/c1e8725de18d583a90ff61194a882258… In my personal view, the code being analyzed is over-complicated (160 lines), and the function could use some refactoring, but it makes for a good test-case for the checker: how well does it cope with this kind of long and complicated function? Can we generate a useful, readable error report that identifies the issue concisely and clearly? I had a go at using the new visualization code on this; the results can be seen here: http://fedorapeople.org/~dmalcolm/gcc-python-plugin/2012-03-26/pylibmc/exam… (the line numbering is different to the above: I deleted all of the other functions from the source file). Works for me with Chromium 17 and with Firefox 10. For comparison, the old visualization code generated this report: http://fedorapeople.org/~dmalcolm/gcc-python-plugin/2012-03-26/pylibmc/pyli… This is actually considering a slightly different path to the one described above: it's considering the path when PyDict_SetItem fails: if (rc != 0) goto unpack_error; but the underlying issue appears to be the same: Py_DECREF(retval) of the only reference to retval, followed by return retval; I think the new visualization is definitely an improvement - the visualization of the control flow through the function is much clearer - shown in the left-hand columns, rather than splattered on top of the source code. But there's still plenty of ways it could be improved: somehow it should highlight which object we're talking about, where it gets allocated, and where it gets deleted: much of that is currently hidden within less interesting textual descriptions of the control flow and the decisions the checker is making (all of those "taking True path" and "taking False path" strings etc). So what should this look like? Dave

1 1

Ideas for improving the static analyzer's HTML reports
by David Malcolm 21 Mar '12

21 Mar '12

[CCing Buck (see below) as he's not on this list] The cpychecker static analyzer generates HTML reports for the errors it finds, trying to give detailed information to a programmer on the bugs in their code. You can see some examples of the current output here: http://preview.tinyurl.com/7h84nad http://preview.tinyurl.com/8y37vfh http://preview.tinyurl.com/7hy43f3 It's crudely functional, but there's plenty of room for improvement: the reports are much harder to read than they ought to be. They're also ugly. Inspired by a talk on UI design at PyCon [1] by one of Django's co-BDFLs, I had a go at sketching some possible improvements to the HTML output that the cpychecker static analyzer generates. My crude sketches can be seen here: http://fedorapeople.org/~dmalcolm/gcc-python-plugin/ui-ideas/ I hope they're legible! (sorry about my handwriting!) navigation-1.png through navigation-3.png: various ideas on how to move the annotation text from its current location inline within the source code, e.g. to a scrolling sidebar, so that the flow of the code is clearer, but the analysis information is still available (and hopefully easier to follow). refcount-values-1.png through refcount-values-3.png: ideas on visualizing a reference-count value, perhaps highlight difference between the ob_refcnt value of an object vs the actual number of pointers pointing at it (it's when these get out of sync that we have a problem). viewing-values-1.png: idea for allowing inspection of the analyzer's state at a particular location within a function: mouseover a reference to a variable to get more information on that variable at that location (we have *lots* of this kind of information: how do we display it without cluttering the UI?) Clearly these ideas are at the "early draft" stage. Does anyone else have ideas on improving the reports? I hope that any such code can be of use to other people's error-reporting scripts. I had a productive chat with Buck Golemon (another PyCon attendee, and a web frontend specialist), who pointed out that the current UI implementation within cpychecker is very tightly coupled to the analyzer, making it hard to hack on, and that a good way of decoupling things would be to capture the analysis results in JSON form. So I've hacked in JSON output, and checked in some sample JSON data into the source tree in libcpychecker/html/test http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commit;h=729be64… By doing this, an HTML/CSS/JS expert has a chance of implementing some of the above ideas without needing to know about the insides of the tool: you could write a standalone script that walks the JSON and generates HTML (last time I saw Buck he was having a go at writing such a thing; I don't know how far he got with it). Anyone else got ideas for how the readability of the error reports could be improved? Dave [1] https://us.pycon.org/2012/schedule/presentation/301/

2 8

Submitting a series of patches for inclusion?
by gcc-python-plugin.5.8eaf7cd8e5128d8191fe＠spamgourmet.com 19 Mar '12

19 Mar '12

A comment by David Malcolm in the thread "Trying to build gcc-python-plugin-0.9 on Debian unstable" <https://fedorahosted.org/pipermail/gcc-python-plugin/2012-March/000203.html> drew my attention to the build system. After looking at it, there are a few tweaks that could be made to simplify maintenance of the Makefile, match typical conventions for variable usage, and so forth. I have some patches pending for this and would like to submit them. My first inclination was to file a ticket. However, it looks like Trac is read-only to anonymous users and does not allow anonymous users to create accounts. Maintainers, how would you prefer to receive these patches? The shortlog for the changes so far is: Build: use $CC, not $GCC, for convention's sake Build: mark target 'all' as .PHONY Build: use $(CURDIR) instead of shell call. Build: use implicit rules for building generated files. Build: split CPPFLAGS/CFLAGS/LDFLAGS. Build: use $(RM), not explicit "rm -f". Build: reuse Python CFLAGS variable Build: rework tarball target to use .PHONY Build: tarball: disallow empty $(VERSION)

3 3

PyCon 2012 talk on gcc-python-plugin
by David Malcolm 15 Mar '12

15 Mar '12

I gave a talk at PyCon 2012 on Friday about my Python plugin for GCC, how this lowers the barrier for entry to potential GCC hackers, and how I've been using this to find reference-counting errors in 3rd-party Python C extensions. A video of the talk can be seen here: http://pyvideo.org/video/648/static-analysis-of-python-extension-modules-us… The slides are here: http://fedorapeople.org/~dmalcolm/presentations/pycon-2012/StaticAnalysisOf… We had a mini-sprint after the talk, and another yesterday in the main sprints, covering these topics: - getting the plugin to build on OS X (using MacPorts' build of gcc-4.6.1): this works, but needed some compat patching around some of the differences between glibc and OS X's libc (also case-sensitivity of filenames). I'll try to merge the changes once I've got back from PyCon - improving the static analysis engine: currently it takes the simplistic approach of trying to generate all traces of execution through the function in a big tree, which suffers from exponentially explosions. I'm hoping the insides can be reworked to implement an iterative solver that can handle loops more robustly (data flow equations) - improving the HTML error reports that the analyzer generates: I have lots of ideas on this, which I need to sketch out. I'm hoping to decouple the analysis engine from the presentation layer, to make it easier for a graphic designer who knows CSS, JS, etc to hack up a report generator that gives better readability and aesthetics compared to my crudely functional HTML+canvas abominations. Hope this is interesting Dave

2 1

Trying to build gcc-python-plugin-0.9 on Debian unstable
by Iustin Pop 10 Mar '12

10 Mar '12

Hi all, (Please keep me Cc-ed, I'm not subscribed) I'm trying to build the plugin, but I have lots of unittest failures, like: libcpychecker.compat.CouldNotFindVarDecl: could not find expected global variable 'PyExc_TypeError' It seems to me that this is https://fedorahosted.org/gcc-python-plugin/ticket/21, as Debian has gcc 4.6.3 which probably includes the fix. Any hint on how to workaround this issue? I've tried to build with Debian's gcc-4.7, but I get into other issues (missing warning_at…). thanks, iustin

2 4

Functions setting exceptions
by Daniele Varrazzo 02 Mar '12

02 Mar '12

Hello, attached there is a patch to implement PySequence_SetItem, which *doesn't* steal a reference as the PyList/PyTuple variants do, and returns 0 on success, -1 on error. After cleaning many genuine errors in psycopg, mostly about unchecked return values, I'm left with several false positives related exactly to this type of functions, returning 0 on success, -1 on failure and setting an exception. Each of these functions caller is reported as "returning (PyObject*)NULL without setting an exception". Is there any way to flag the behaviour of these functions? An __attribute__? I'm trying implementing it but time is out for tonight. I've find how to register the attribute callback but haven't found yet where and how to use it. -- Daniele

3 10

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

gcc-python-plugin March 2012