Proposal: Python 3 in Fedora 13 - devel - Fedora mailing-lists

1 Oct 2009


      Proposal: Python 3 in Fedora 13
"Evolutionary, not revolutionary": build a python 3 stack
parallel-installable with the python 2 stack.
= High-level summary =
  - Python 3.0 was released almost 10 months ago, on 2008-12-03, and the
latest release of the 3.* branch is 3.1.1, released on 2009-08-17.
  - Other distros have python 3, though not necessarily with anything
"on top" resembling the full python 2 stack.
  - We have a working, valuable python 2 stack, which is used by
critical system components (yum and anaconda): we must not destabilize
the python 2 stack.
  - Python 3 is sufficiently different from python 2 that we need them
to be independent software stacks.
  - I plan to spend a large chunk of my $DAYJOB over the next few months
trying to build a useful Python 3 stack for Fedora 13, for some
definition of "useful" (help will be appreciated!)
  - I don't want to add extra work for package maintainers:  if you
maintain an SRPM of a python 2 module that's working for you, you
shouldn't feel obligated to own a separate SRPM for python 3.  If
someone has a need for the module on python 3, they can take on that
work.
= Background =
Python 3 is intended by upstream to be the future of Python, but we have
many critical components that use Python 2.  Python 2 and Python 3 are
sufficiently different that we need both (try writing "print" in each).
Python 2 will be around for a long time.
An interesting summary of Python 3 adoption can be seen here:
http://renesd.blogspot.com/2009/09/py3kpython3-more-than-one-year-on-096.htm...
An earlier proposal about python 3 in Fedora is here:
https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html
Going forward, I will have plenty of time to spend, as part of my
dayjob, on Python in Fedora [1]
= Proposal =
I want to get python 3 into Fedora, but I don't want to break yum or
anaconda (or anything else, for that matter!)
How to do this?  I propose that Fedora shall have separate,
parallel-installable Python 2 and Python 3 stacks.  I believe we can get
things to the point where on a Fedora box you'd be able to install both
stacks, and have some processes running python 2 code, and some running
python 3, simultaneously.
Where I would draw the line is on having both python 2 and python 3
running within the same _process_: the two libraries share most of their
symbol names, but with differing implementations, and the result of
trying to dynamically link the two into the same address space would be
highly unstable.
As an example, you'd be able to install both mod_python and mod_python3
rpms, but you wouldn't be able to (sanely) configure httpd to have both
running simultaneously (I guess we should add a run-time warning for
this case)
Scoping:
  - this work would target Fedora 13.  I'd avoid pushing it into F12
until it's proven safe to do so
  - the proposal is for python 2 vs python 3.  It could be extended to
support having multiple minor-versions of Python as well, but that's a
big extension of the work involved and not something I'm planning to
work on myself.
= Details =
We should split python 2 and python 3 at the source RPM level, where
possible.
The easy case is when upstream release separate tarballs for the python
2 and python 3 versions of code.  For example, given package
"python-foo" in packaging CVS, there would be a separate "python3-foo"
for the python 3 version.  There would be no expectation that the two
would need to upgrade in lock-step.  (The two SRPMS could have different
maintainers within Fedora: the packager of a python 2 module might not
yet have any interest in python 3)
The more difficult case is when the python module is emitted as part of
the build of a larger module.  Some examples:
  - the build of "rpm" itself emits an "rpm-python" subpackage.
  - Another example is the "postgres" srpm, which emits a
"postgresql-python" subpackage.
In a quick attempt at seeing other examples, on my laptop (F11), here
are the packages installed that provide python modules where the package
name differs from the srpm name:
$ rpm -qf /usr/lib/python2.*/site-packages/* | grep -v "is not owned" |
sort | uniq | xargs rpm -q --qf "%{NAME}-%{VERSION}-%{RELEASE}
%{SOURCERPM}\n" | sed -e"s/.src.rpm//" | squeal -f table col0, col1 from
- where col0 != col1
                                    col0|                              col1|
----------------------------------------+----------------------------------+
             at-spi-python-1.26.0-1.fc11|              at-spi-1.26.0-1.fc11|
         audit-libs-python-1.7.13-1.fc11|               audit-1.7.13-1.fc11|
                cracklib-python-2.8.13-4|                 cracklib-2.8.13-4|
              gamin-python-0.1.10-4.fc11|               gamin-0.1.10-4.fc11|
                hplip-libs-3.9.8-12.fc11|               hplip-3.9.8-12.fc11|
           libproxy-python-0.2.3-10.fc11|            libproxy-0.2.3-10.fc11|
         libselinux-python-2.0.80-1.fc11|          libselinux-2.0.80-1.fc11|
        libsemanage-python-2.0.31-4.fc11|         libsemanage-2.0.31-4.fc11|
                 libuser-python-0.56.9-3|                  libuser-0.56.9-3|
             libxml2-python-2.7.3-3.fc11|              libxml2-2.7.3-3.fc11|
              newt-python-0.52.10-4.fc11|               newt-0.52.10-4.fc11|
plague-common-0.4.5.7-5.20090612cvs.fc11| plague-0.4.5.7-5.20090612cvs.fc11|
policycoreutils-python-2.0.62-12.12.fc11| policycoreutils-2.0.62-12.12.fc11|
                python-magic-5.03-2.fc11|                  file-5.03-2.fc11|
          python-slip-dbus-0.1.15-3.fc11|         python-slip-0.1.15-3.fc11|
           python-slip-gtk-0.1.15-3.fc11|         python-slip-0.1.15-3.fc11|
                 rpm-python-4.7.1-1.fc11|                  rpm-4.7.1-1.fc11|
     setroubleshoot-server-2.1.14-3.fc11|      setroubleshoot-2.1.14-3.fc11|
 system-config-printer-libs-1.1.8-6.fc11|system-config-printer-1.1.8-6.fc11|
Such SRPMS have a:
  BuildRequires: python-devel
which is a subpackage from the python build (2.6)
My plan here is that we should create a python3-devel subpackage as a
subpackage of the python3 build, and have it parallel-installable with
the python-devel package.
We could then %prep the rpm build for each of the above so that the
python 3 support is built as a parallel component of the build,
independently of the python 2 support e.g. by copying the python2
support into a separate dir, then applying a patch as necessary (and
somehow wiring up the configuration/make so it builds both...)  The
caveat here is that I haven't tried actually doing this yet for any of
these packages.  Issues with this approach:
  - I don't yet know if autoconfiguration will work well with both
-devel packages installed
  - It will probably involve actually doing the porting work for each
package (yay, we get to be leaders!)
  - Whoever does this for a package needs to work closely with the
upstream for that package
I'll have a go at doing this for rpm when I get back from vacation.
Arguably the existing python 2 binding of rpm isn't great, but it's
probably best to go for close compatibility between python 2 and python
3 rather than try to overhaul the bindings as part of the port to python
3: one thing at a time!
"Naming convention" proposal:
How does this sound:
  - an rpm with a "python-" prefix means a python 2 rpm, of the
"default" python 2 minor version (for Fedora this will be the most
recent stable upstream minor release, for EPEL it will be the minor
release of 2 that came with the distro, so 2.4 for EPEL5)
- an rpm with a "python3-" prefix means a python 3 rpm, of the
"default" python 3 minor version (for Fedora this will be the most
recent stable upstream release)
(we could extend this to have specific minor-releases (e.g. use
"python24-" for a python 2.4 stack, in case some brave soul wants to get
zope/plone running.  But may be better to try to fix the zope/2.6
incompatibility at that point (caveat: haven't looked at the details of
the issue).  I don't intend to work on such versions))
What about packages without a "python-" prefix?  Proposal:  If upstream
has a naming convention for python2 vs python3, use it.  Otherwise, add
a "python3-" prefix to make things clear.  I'm not sure about the
details here.  Examples?
There have been various discussions upstream about what to call the
python 3 binary.  My favorite quote on the subject is from Guido,
http://mail.python.org/pipermail/python-3000/2008-March/012421.html :
...
During the next 3 years or so, installing Py3k as the default "python"
will be a deed of utter irresponsibility and is likely to break your
system in subtle ways (both OSX and Linux these days use Python for
certain system tasks). If you *really* want to shoot yourself in the
foot this way, go ahead and explicitly use "make altinstall
bininstall" or link it yourself.
I propose that for Fedora we have "/usr/bin/python3" for the
system/default version of python 3 and "/usr/bin/python" for the
system/default version of python 2.  Both would be symlinks to a binary
with the minor-release embedded in the name ("/usr/bin/python3.1" and
"/usr/bin/python/2.6").
As I understand things, this should make us broadly in agreement with
upstream; see e.g.:
http://mail.python.org/pipermail/python-dev/2009-April/088862.html
http://mail.python.org/pipermail/python-dev/2009-April/088884.html
A rough plan for Fedora 13 might be:
  - get python3 packaged in a manner compatible with the above
    - (persuade /usr/lib/rpm/brp-python-bytecompile to use the correct
python when building rpms containing .py files)
  - get rpm bindings working with python3
  - get some useful components working e.g. a web stack: Django,
TurboGears etc (though e.g. Django's py3k support is a long way off
IIRC); ideas?
  - solidify packaging guidelines for python 2 vs python 3 once we've
got some experience with the above and hopefully proven the techniques
  - look at porting major components over to python3 (but probably don't
actually do this for F13; leave python 2 as the critical component, I
suspect):
    - yum (rpm)
    - anaconda
However "no plan survives contact with the enemy", we'll see how things
turn out in reality when trying to get a full integrated stack working.
Future work (F14?) could involve cutting over the major components, so
that the base install would bring in "python3", and "python" would
become optional.  Obviously there's a _lot_ to be done before that can
be done sanely.
= Progress so far =
I've put together a somewhat-working python3 srpm, based on the python
srpm (with lots of FIXMEs added...)  It's not yet ready for a formal
package review (I'm working through the various patches, and it's not
yet fully installable in parallel with the python 2 stack), but you can
see the specfile here:
  http://people.redhat.com/dmalcolm/python3/python3.spec
and an SRPM here:
  http://people.redhat.com/dmalcolm/python3/python3-3.1.1-1.fc13.src.rpm
After I did this work, I saw that a couple of other people have written
Python 3 srpms for Fedora:
  http://ivazquez.fedorapeople.org/packages/python3000/
and
  https://bugzilla.redhat.com/show_bug.cgi?id=526126
and there was this proposal for doing Python 3 in F10:
https://www.redhat.com/archives/fedora-devel-list/2008-May/msg02417.html
similar to this one.  Obviously I want to work with those people to come
up with a working python 3 rpm in Fedora.
There's also the merge-review for python:
https://bugzilla.redhat.com/show_bug.cgi?id=226342 ; many of the
comments there would thus also apply to my srpm, given that I used the
former as my starting point.
On a tangent: we currently have 2.6.2 in F12/rawhide.  Upstream is
preparing a 2.6.3 release, with many bugfixes.  This seems to me like a
candidate for F13.  (the release notes for the 2.6.3 rc are long; I'd
want it to have a _lot_ of testing before pushing it to F12))
Thoughts?
Dave
[1] I'm being transferred at work, and will be able to spend a lot of
time on Python within Fedora.  Previously, my job involved keeping
various internal RH systems working (with any Fedora work done
afterhours or as a side project).  Having said that, I'm about to go on
vacation with no access to a computer for the period October 3rd-10th