Status of Fedora 13 Feature: EasierPythonDebugging
by David Malcolm
https://fedoraproject.org/wiki/Features/EasierPythonDebugging
is currently marked as 95%, but a better answer is "it's complicated".
I've done all I can for now to finish it, and it does provide some
useful improvements, but it's being heavily impacted by a bug outside of
my expertise (see (b) below).
I've been testing the feature, building out a test suite, and sending
upstream as http://bugs.python.org/issue8032
The torture test for this feature that I've been using is to run "yum
update", then hit CTRL-Z as it's running, then:
[david@brick ~]$ gdb attach $(pidof -x yum)
(gdb) bt
and see what you can see.
There are two outstanding bugs with the feature:
(a) the attach-to-yum testcase found a nasty issue: when there are
cycles in the graph of python object references, my pretty-print code
would take gdb into an infinite recursion. I've fixed this (and various
other bugs) in v4 of the hooks so that with v4 of the hooks it seems to
be robust. I've built python/python3 with v4; the relevant updates are:
https://admin.fedoraproject.org/updates/python-2.6.4-24.fc13
https://admin.fedoraproject.org/updates/python3-3.1.2-3.fc13
and ought to be installable on a f13 host with:
su -c 'yum --enablerepo=updates-testing update python python3'
so this is in "testing", though not in the tree yet.
(b) the critically important frame information within the heart of the
bytecode excution loop ought to tell us what code we're running, but is
showing up in gdb as "optimized out". (specifically, the "f" parameter
to function PyEval_EvalFrameEx).
Unfortunately, this greatly reduces the effectiveness of the feature
(it's still somewhat useful without this, but the feature page would
need a significant rewrite if this doesn't get fixed).
This looks like a regression of rhbz:556975:
https://bugzilla.redhat.com/show_bug.cgi?id=556975
I don't know if this is a gcc/gdb/python issue.
I've reopened this bug. I haven't yet added it to any of the tracker
bugs: http://fedoraproject.org/wiki/BugZappers/HouseKeeping/Trackers
Which tracker should this be in?
Dave
13 years, 8 months
Automatic Provides and Requires for Python modules
by Aurelien Bompard
Hi people,
At the top of our Python packaging page
(https://fedoraproject.org/wiki/Packaging:Python), there's a note which
reads 'In theory /usr/lib/rpm/pythondeps.sh would also automatically
generate "Provides" lines'
This is also true for python modules : distutils and setuptools have a way
to specify provides and requires, but those are not reflected in the RPM.
It means that they must be entered and maintained by hand in the spec file.
In my opinion, this is not optimal.
The perl modules have an interesting way of reflecting their dependencies in
the rpm: they add a dependency on "perl(Module::Name)". I believe the same
system can be applicable to python modules.
Currently, there are two main packaging systems in python : distutils and
setuptools. Distutils declares the dependencies in an egg-info file, which
is RFC-822-formatted. Setuptools turns this file into a PKG-INFO file in a
subdirectory, and adds a requires.txt files with additional dependencies, in
a different format.
The pythondeps.sh script should be able to extract requires from these
files. To match the dependencies, pythondeps.sh should create virtual
provides like "python(ModuleName) = version".
The good news is : I've written it already ;-)
It's based on the pythondeps.sh script from Git master (which changed a
little bit due to python3, see bug 532118). Also, the script does not try to
be too smart with versionned dependencies, because the format is a little
bit different in python and in rpm. For those complicated cases, handwritten
requirements can still be added to the spec file. The script only covers the
usual cases.
For reference, the dependency format in distutils is described here:
http://www.python.org/dev/peps/pep-0314/
The dependency format in setuptools is described here:
http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies
A patch would not make much sense due to the size of the addition, so here's
the full script: http://aurelien.bompard.org/projects/divers/pythondeps.sh
I've tested it with quite a few packages, but to make sure I've written a
few unit tests (very simple, bash-based) :
http://aurelien.bompard.org/projects/divers/test-pythondeps.sh
I do believe it would be a valuable addition to RPM (of course, even if
accepted shortly, I don't expect it to land in F-13 since it requires
recompiling all the python packages).
Do you think it's a good idea ? What about the implementation ?
If it looks good to you I'll propose it to rpm.org.
Cheers,
Aurélien
--
http://aurelien.bompard.org ~~~~ Jabber : abompard(a)jabber.fr
"Everyone thinks of changing the world, but no one thinks of changing
himself." -- Tolstoï
13 years, 8 months
Re: Python SIG
by Stanley A. Klein
I think the issue of Python distutils/setuptools is important. I posted a
note to the Python Distutils SIG asking if there was any way I could do an
RPN package (using bdist_rpm) with results going to my usual rpmbuild
directory tree instead of being placed in the same directory tree as the
sources and setup.py. I didn't get a response.
Meanwhile, everything seems to be eggs and easy_install that conflict with
the overall system management aspects (and Linux Standards Base aspects)
of RPM.
I think that any changes to fix the Python distutils/setuptools for RPMs
will probably need to originate in the Fedora community.
Stan Klein
On Thu, March 18, 2010 8:00 am, David Malcolm <dmalcolm(a)redhat.com> wrote:
> Message: 1
> Date: Wed, 17 Mar 2010 15:36:35 -0400
> From: David Malcolm <dmalcolm(a)redhat.com>
> Subject: Python SIG
> To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org>
> Message-ID: <1268854595.30012.2147.camel@brick>
> Content-Type: text/plain; charset="UTF-8"
>
> I noticed that there's an effort to better organize the various Special
> Interest Groups in Fedora:
> https://fedorahosted.org/fedora-engineering-services/ticket/2
>
> I noticed that although https://fedoraproject.org/wiki/SIGs/Python
> had "[[Category:Language-specific SIGs]]" we didn't have:
> "[[Category:SIGs]]" and so we weren't showing anywhere on that page.
>
> I've fixed this, and the page is now at least showing on the list at the
> bottom of:
> https://fedoraproject.org/wiki/Category:SIGs
>
> However, we're not in the hand-written list on that page, and I feel
> that we should be (hey, Ruby gets a mention, why can't we :) )
>
> Any ideas what the summary info for the SIG should be? At the risk of
> plagiarism, Ruby's says:
> "A SIG for people who are interested in improving the state of Ruby in
> Fedora. This includes packaging Ruby libraries and applications, setting
> and improving standards for packaging them as RPM's and maintaining Ruby
> packages for Fedora."
>
> and that might be a good first start, but I'd also include the runtime
> as well as libraries and applications - I don't want people to feel that
> the runtime is somehow sacrosanct (just more care required!).
>
> How about this:
> "A SIG for people who are interested in Python on Fedora. This includes
> packaging and optimizing the various Python 2 and Python 3 runtimes
> (CPython, Jython), packaging libraries and applications, setting and
> improving standards for packaging them as RPM's and maintaining Python
> packages for Fedora."
> (somewhat copied from Ruby)
>
> I think the various questions on the F-E-S ticket are good ones. I'm
> willing to do some work towards some of that ticket, but any help would
> be welcome.
>
> Thoughts?
> Dave
>
13 years, 8 months
Re: Python SIG
by Stanley A. Klein
Some of the Enthought packages are already in yum repositories. However,
they all are not named so you can easily see that they are indeed
Enthought packages. So if you want to install a collection of packages,
you have to go through the list of packages (such as using "yum info
python*") and notice the names.
Also, over the past few years I have built rpms for the Enthought toolset.
However, the last few times I ran into problems. I recently tried to
download the latest tagged version recently from the Enthought svn
repository and ran into the problem that the version I requested was not
recognized. The previous time I tried I ran into the "unpackaged files"
problem, because there was some kind of glitch in rpm.
I did take the tool for building eggs of all their packages and modified
it into a tool for fixing the setup.cfg files. Another problem I ran into
in the past was files and directories named appropriately for Windows but
not for Linux, i.e., with spaces in the file/directory names. I worked
most of them out with the people at Enthought, but I later found that it
is a problem that still needs watching. There are several other glitches
that you run into trying to build the rpm packages.
As I said in a previous posting, it would also be nice if bdist_rpm made
use of the same rpmbuild directory tree as other rpm builds and did not
build the rpms within the svn repository (or at least provided an option
to do so).
Stan Klein
On Fri, March 19, 2010 8:00 am, Toshio Kuratomi <a.badger(a)gmail.com> and
Gael Varoquaux <gael.varoquaux(a)normalesup.org> wrote:
>
> Message: 3
> Date: Thu, 18 Mar 2010 15:17:23 -0400
> From: Toshio Kuratomi <a.badger(a)gmail.com>
> Subject: Re: Python SIG
> To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org>
>
> On Thu, Mar 18, 2010 at 03:11:15PM +0000, Jos? Matos wrote:
>> On Wednesday 17 March 2010 19:36:35 David Malcolm wrote:
>> > How about this:
>> > "A SIG for people who are interested in Python on Fedora. This
>> includes
>> > packaging and optimizing the various Python 2 and Python 3 runtimes
>> > (CPython, Jython), packaging libraries and applications, setting and
>> > improving standards for packaging them as RPM's and maintaining Python
>> > packages for Fedora."
>> > (somewhat copied from Ruby)
>>
>> I find this a good start (just as you do). :-)
>>
>> Just to add to the discussion my main interest in this SIG is on
>> scientific
>> and/or educational python packages.
>>
>> What we see, in terms of packaging, in projects like sage
>> http://www.sagemath.org/ or in other projects (proprietary) like EPD
>> (Enthought Python distribution) or Python(x,y) is that put everything
>> under
>> the sun on those packages.
>>
>> It would be nice to have the free parts in Fedora...
>>
>> Such as the above statement is written I see these goals there.
>>
> WRT enthought, they're very interested in getting their software packaged
> for individual distributions but don't have any manpower or inhouse
> knowledge on how to do that. If someone were to start working on those
> packages, I could pass along contact information and we could potentially
> have help from them on changing their packaging over time to address our
> needs.
>
> -Toshio
> ------------------------------
>
> Message: 4
> Date: Thu, 18 Mar 2010 20:23:32 +0100
> From: Gael Varoquaux <gael.varoquaux(a)normalesup.org>
> Subject: Re: Python SIG
> To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org>
> Message-ID: <20100318192332.GB16916(a)phare.normalesup.org>
> Content-Type: text/plain; charset=iso-8859-1
>
> On Thu, Mar 18, 2010 at 03:17:23PM -0400, Toshio Kuratomi wrote:
>> > Such as the above statement is written I see these goals there.
>
>> WRT enthought, they're very interested in getting their software
>> packaged
>> for individual distributions but don't have any manpower or inhouse
>> knowledge on how to do that. If someone were to start working on those
>> packages, I could pass along contact information and we could
>> potentially
>> have help from them on changing their packaging over time to address our
>> needs.
>
> AFAIK, that's been partly done (mayavi2 is packaged under Fedora, I
> believe).
>
> I believe that Rakesh Pandit is the one to thank for that work.
>
> I second that assersion though: Enthought is interested in protecting and
> selling only the binaries of EPD. Enthought would like the corresponding
> software (all open source) to have as much success as possible, and thus
> to have high-quality packages on every platform.
>
> Disclaimer: I worked for Enthought (although only a very brief amount of
> time, as I am pursuing a career in academia), and I have much personal
> interests in the success of the various Enthought-related packages, but
> little time for packaging work.
>
> Ga?l
13 years, 8 months
Python SIG
by David Malcolm
I noticed that there's an effort to better organize the various Special
Interest Groups in Fedora:
https://fedorahosted.org/fedora-engineering-services/ticket/2
I noticed that although https://fedoraproject.org/wiki/SIGs/Python
had "[[Category:Language-specific SIGs]]" we didn't have:
"[[Category:SIGs]]" and so we weren't showing anywhere on that page.
I've fixed this, and the page is now at least showing on the list at the
bottom of:
https://fedoraproject.org/wiki/Category:SIGs
However, we're not in the hand-written list on that page, and I feel
that we should be (hey, Ruby gets a mention, why can't we :) )
Any ideas what the summary info for the SIG should be? At the risk of
plagiarism, Ruby's says:
"A SIG for people who are interested in improving the state of Ruby in
Fedora. This includes packaging Ruby libraries and applications, setting
and improving standards for packaging them as RPM's and maintaining Ruby
packages for Fedora."
and that might be a good first start, but I'd also include the runtime
as well as libraries and applications - I don't want people to feel that
the runtime is somehow sacrosanct (just more care required!).
How about this:
"A SIG for people who are interested in Python on Fedora. This includes
packaging and optimizing the various Python 2 and Python 3 runtimes
(CPython, Jython), packaging libraries and applications, setting and
improving standards for packaging them as RPM's and maintaining Python
packages for Fedora."
(somewhat copied from Ruby)
I think the various questions on the F-E-S ticket are good ones. I'm
willing to do some work towards some of that ticket, but any help would
be welcome.
Thoughts?
Dave
13 years, 8 months
Upstream revamp of byte-compilation
by David Malcolm
IIRC, we currently jump through some hoops in brp-python-bytecompile;
upstream landed some new ways of doing this in this patch for 2.7/3.2:
http://bugs.python.org/issue8140
(Not sure if it's helpful as I think Toshio solved this already in F13's
rpm-build, but I wanted to at least send it to the SIG in case it's of
interest)
Dave
13 years, 9 months
[Fwd: [unladen-swallow] Upgrading to llvm-2.7]
by David Malcolm
[CCing Fedora Python SIG: context is that unladen-swallow is the
optimized branch of python with JIT, and Jeffrey Yasskin is working
towards being able to dynamically link Python against LLVM, in the
python 3.3 timeframe]
Does Fedora's LLVM build have just a single configuration, or would it
be easy to support two? (Not sure if it should)
13 years, 9 months
Trying out a new way of packaging Python modules
by David Malcolm
= BACKGROUND : Multiple python stacks =
For Fedora 13 we've added a parallel-installable Python 3 stack,
doubling the number of Python runtimes from one to two.
I've ben mulling over things we could do for Python in Fedora 14. An
idea I'd like to borrow from Debian is to add debug builds of the
runtimes, which would give the developer the option of installing a
python stack built with full reference tracking information which is a
great boon during development of a C extension, and for analysing memory
usage; you can e.g. interate over all live objects. In my experience
this roughly halves the speed of the interpreter, and increases memory
usage. This would double the number of our runtimes from 2 to 4. (As
I understand it Debaian have shipped a standard vs debug pair of python
stacks for a couple of years now).
There's also the possibility of packaging pre-releases of python27 and
python32 within Fedora, and of shipping experimental stacks using the
other runtime implementations: Unladen Swallow, PyPy, Jython, Pynie,
etc.
In EPEL5 I've recently proposed a parallel-installable "python26"
alternate stack (see
https://www.redhat.com/archives/epel-devel-list/2010-March/msg00064.html ). Although I want a python26 stack, I suspect EPEL5 will also want python27 and python32 stacks in the future.
Given that Python add-ons aren't compatible between major or minor
releases [1], we need to somehow repackage add-on python modules for the
python stacks.
We updated Fedora's python packaging guidelines for Fedora 13 in order
to cover the "python3" runtime:
http://fedoraproject.org/wiki/Packaging:Python
I like that we have a single src.rpm per upstream tarball (for those
cases where both python 2 and python 3 can be built), and have a single
build per-arch within Koji that emits a subpackage per python runtime.
However I dislike the copy&paste approach we're using within .spec
files. For me, .spec files are source code, and I passionately hate
copy&paste of source code in programming: it's too easy for things to
get out of sync, tends to leads to bugs, and we end up having to work
harder than we should need to. Computers give us many ways of sharing
and reusing information, and we should use these to make our lives
easier (the computer should serve us, not the other way around!).
The way I see this: for every OS release, there would be a set of
supported Python runtimes, but I want us to be able to change it from
release to release, and I don't want us to be tied down in manual
specfile rewriting to achieve this; if we have to rewrite all our
specfiles to add a new runtime, we will never want to add a new runtime.
For Fedora 12, this was just "python". For Fedora 13 this is "python"
and "python3". Similarly for EPEL5: the system "python: is 2.4, but I
want to add a python26, and I suspect we will want more in the future.
For Fedora 14 I don't know what we should do yet, but I'd love to be
able to support more of the Python runtimes, and having a debug stack a
"yum install" away would be handy.
So I want to move from a "static" model where we hardcode information
about the available python runtimes into all of .spec files into a more
dynamic model, where the .spec file for a python module queries a tool
about what python runtimes it ought to support based on this OS, and
acts accordingly.
= THE EXPERIMENT =
I've implemented a tool, currently named "rpm-pyconfig" which can be
used to query information about Python runtimes. Currently I'm
hardcoding this data within the tool, but the idea is that every runtime
would drop a .conf file (perhaps in .ini format) into a config dir
"/etc/rpm-pyconfig.d/", and the tool would query this. (somewhat like
pkg-config).
One idea is that we would have a meta-package "python-meta-config" or
somesuch, which "Requires" all of the rpms holding the .conf files, thus
defining what the expected set of runtimes we're targetting for this OS
release. We'd change this early during the development cycle for a
Fedora release, and then keep it stable.
I've used this new tool to rewrite "python-coverage.spec" so that it
dynamically queries what python runtimes it should support. You can see
the specfile I wrote here:
http://fedorapeople.org/gitweb?p=dmalcolm/public_git/rpm-pyconfig.git;a=b...
This specfile _does_ build, and emits subpackages.
The tool has its own expression-evaluation language. For example
"@confbin" means "the python binary for the current configuration", and
is expanded to "python" for the default python2 build, and to "python3"
for the default python3 build.
The main commands are:
--for-each (and --for-each-2 and --for-each-3) which iterates over
every python configuration (or just those that are Python 2/Python3)
--eval which evaluates an expression then prints the result
--exe which evaluates an expression then executes the result
You can then use this within a specfile in a various ways.
For example, you can write:
rpm-pyconfig --foreach-2 --exe \
"cp -a ../pristine @confsrcdir"
rpm-pyconfig --foreach-3 --exe \
"cp -a %{py3dir} @confsrcdir"
and have each subbuild get its own copy of the source tree
("@confsrcdir" expands differently for each one), the python2 subbuilds
getting a pristine copy, the python3 subbuilds getting one that's had
2to3 run on it.
You can also use it with rpm's subshell capture syntax: %()
This is a little-known rpm construct in which the commands within the
parentheses are executed, and the stdout is captured, and then processed
further.
For example, the following uses it to iterate over all python runtimes
and express that there should be a -coverage subpackage for each one,
leading to "python-coverage" and "python3-coverage":
# Define the metadata for each built package:
%(rpm-pyconfig --foreach --eval "
%package -n @confpkg-coverage
Summary: Code coverage testing module for @confpkg
Group: System Environment/Libraries
Requires: @confpkg-setuptools
%description -n @confpkg-coverage
%SHARED_DESC
@confdescline
")
Similarly we can programatically generate multiple %files stanzas as
desired:
%(rpm-pyconfig --foreach --eval "
%files -n @confpkg-coverage
%defattr(-,root,root,-)
%doc README.txt
%{_bindir}/@confbin-coverage
@conf_sitearch/coverage/
@conf_sitearch/coverage*.egg-info/
")
Note how in this world the "%python_sitearch" rpm macro goes away, and
we instead use the "@conf_sitearch" rpm-pyconfig macro, which will vary
per configuration.
The current implementation of the tool is here:
http://fedorapeople.org/gitweb?p=dmalcolm/public_git/rpm-pyconfig.git;a=b...
(it's just an experiment at this stage, so is somewhat hackish)
This is what I've got so far. It's a very different way of doing python
packaging to what we've done in the past. I like it, in that it
expresses the stacks programmatically, rather than requiring copy&paste
of specfile fragments.
I'm sure there are issues with this approach, but I suspect they are
fixable, and I think this approach is more sustainable than manually
handcoding information about the supported Python runtimes into every
specfile: it gives us the freedom to add runtimes. Doing so is a major
event, but it at least reduces the effort required.
An example of an issue: currently it emits "python2-coverage" for the
core python2 package, rather than "python-coverage" (that's fixable).
Some more thoughts on the brainstorm that led to this can be seen at:
https://fedoraproject.org/wiki/DaveMalcolm/PythonIdeas
My biggest concern with this is that %() is little-known, and I don't
know how well-supported it is e.g. in RHEL5's rpm, and going forward.
Oh, and it's a total change in how we do Python packaging :)
The syntax is obviously up for grabs, and is somewhat clunky in places.
I used "@" to avoid having to deal with "%" and "$" already having
meaning in the rpm and shell worlds (since we have to pass around
strings that have functional content for all three langugages).
I suspect we'd need an exclusion syntax (e.g. "this module isn't
buildable on PyPy", "this module isn't buildable on 2.7 with ppc"); you
can perhaps express this using things like:
if [ $(rpm-pyconfig --eval @major_version.@minor_version) -eq 2.7 ]
%ifarch ppc
%endif
elif
or somesuch
Thoughts?
Dave
[1] both compiled C extensions and compiled .pyc/.pyo bytecode files are
incompatible (but in both cases this is detectable, the former via the
"NEEDED" field in the ELF metadata, the latter via the magic number in
their header); see PEP 384 and PEP 3147 for future ideas towards being
able to share more, but they're not done yet.
13 years, 9 months