future of python3...

[David Malcolm]

On Wed, 2011-05-04 at 03:57 +0530, Rahul Sundaram wrote:
> On 05/03/2011 11:47 PM, David Malcolm wrote:
> > I'm also working on this Fedora 16 idea, which should help both PyPy and
> > CPython more robust:
> > http://fedoraproject.org/wiki/Features/StaticAnalysisOfCPythonExtensions
> > though it's early days yet, and I don't yet know how good the
> > signal:noise ratio for the tool will be, which will affect how tightly
> > we will want to integrate this into Fedora's build system.  (I'm about
> > to go on vacation, and I don't want to open up the code until I get
> > back, later this month).
> >
> > Hope this all sounds sane
> 
> Yes indeed and thanks for the info.  http://speed.pypy.org/ looks very
> impressive and with the tradeoff of less debugability,  I wasn't sure
> how much it impacts the speed of say yum.  Have you had a chance to look
> at that?

I haven't specifically looked at yum yet, but yum is heavily enmeshed
with the rpm python bindings, which are largely in C

Note that PyPy has to do extra work for C extension modules: the
PyObject* it provides are all just wrappers around PyPy's "real"
objects, so there is a CPU cost at the python/C boundary that isn't
present in CPython.

This, and the fact that the C code is invisible to the JIT compiler
means that a workload that's calling into C extension code a lot might
not gain much from the JIT compiler.  (Optimization advice for PyPy is
often to recode C back into Python, since pure Python code is amenable
to optimizations when being JIT-compiled).  So additional work may be
needed to support pypy running yum and to get speed and memory usage
improvements, and I don't know if it would be worth it.

So far, every time I've tried to use PyPy's C extension support, the
extension has either not built, or it's segfaulted on me very quickly.
This was with 1.4 and 1.4.1 though, and a lot of work was done in 1.5 on
this area.

If you want to play with this, you could try the pypy-devel rpm, or try
building pypy from source, but obviously you'll want to do it on a
testing rpm db, not on the rpm db of a machine you care about ;)

It's probably only worth doing if you're familiar with gdb, too, and are
willing to dive deep into the guts of PyPy, rpm and yum :)  I may have a
chance to do this when I get back from vacation.

Dave