python-2.6 and python-3.x

[Toshio Kuratomi]

I had the fortune to attend pycon this past week and went to a talk
about python-2.6 compatibility with python-3.

http://us.pycon.org/2009/conference/schedule/event/74/

The slides from that are available and largely self-explanatory.  They
list tricks for getting python code that runs on both py2.6 and py3.x.

However, one of the things that isn't obvious is what's missing from the
equation to make the idea of porting one library at a time work.  The
ideas in the slides hinge upon writing code that runs on python3.x and
then making that backwards compatible with python-2.6's py3
compatibility.  In Fedora, we're going to have the problem of taking
code and trying to port it to python-2.6's py3 compatibility layer and
then fixing it to run on py3k if necessary.  This is much harder as
python-2.6 will accept most python-2.5 and less code so we won't know
when we're using idioms that will not work in python-3.x.

Another thing that's missing is a 2to3 like tool that targets 2.6+compat
instead of python3.  However, this one might be possible if we ask.

A side note is that the Core devs apparently talked about naming the
interpreter binary for python /usr/bin/python3 at the language summit.
If they do this, /usr/bin/python will always mean python2 and
/usr/bin/python3 will be for python-3.x.  Not sure if that influences us
or not.

I don't have a solution for this yet.  Just throwing this out there as
something we need to consider as we start to port code to python-2.6
+py3 compat.

-Toshio

[Jochen Schmitt]

Toshio Kuratomi schrieb:
>  I don't have a solution for this yet.  Just throwing this out
> there as
>  something we need to consider as we start to port code to python-2.6
>  +py3 compat.
>     
I thnk there is an additional issue we have. On Fedora we have several 
packages
which use the C-API of python to extent the python environment. One example
it blender which contains python scripts which need a python extension 
provided
by blender.

I think porting this kind of application may nned a lot of work to 
porting to
Python 3000. So I thin wee need a python-2.6 compat package for a while 
until
all applications will been ported.

Best Regards:

Jochen Schmitt

[Toshio Kuratomi]

Jochen Schmitt wrote:
>  Toshio Kuratomi schrieb:
> > I don't have a solution for this yet.  Just throwing this out there as
> > something we need to consider as we start to port code to python-2.6
> > +py3 compat.
> >   
>  I thnk there is an additional issue we have. On Fedora we have several
>  packages
>  which use the C-API of python to extent the python environment. One example
>  it blender which contains python scripts which need a python extension
>  provided
>  by blender.
>  
>  I think porting this kind of application may nned a lot of work to
>  porting to
>  Python 3000. So I thin wee need a python-2.6 compat package for a while
>  until
>  all applications will been ported.
>   
I think that the C API is an additional barrier but if we can't do the
pure python modules in a gradual transition, whether we're able to port
the C API without a compat package period will be moot.

-Toshio

[Tom Callaway]

On 04/02/2009 01:46 PM, Jochen Schmitt wrote:

>  I think porting this kind of application may nned a lot of work to
>  porting to
>  Python 3000. So I thin wee need a python-2.6 compat package for a while
>  until
>  all applications will been ported. 
The problem with this approach is that it does not motivate upstreams to
move to Python 3, they will just depend on the compat package, and we'll
have a lot of trouble convincing anyone to move.

I think part of the solution will be for Fedora developers to help port
such applications to Python 3, and do so in a way that they detect the
version of python at runtime and set defines appropriately. That way, we
can have applications ready for Python3 before we actually make the switch.

~spot

[James Antill]

On Thu, 2009-04-02 at 13:55 -0400, Tom "spot" Callaway wrote:
>  On 04/02/2009 01:46 PM, Jochen Schmitt wrote:
>  
>  > I think porting this kind of application may nned a lot of work to
>  > porting to
>  > Python 3000. So I thin wee need a python-2.6 compat package for a while
>  > until
>  > all applications will been ported.
>  
>  The problem with this approach is that it does not motivate upstreams to
>  move to Python 3, they will just depend on the compat package, and we'll
>  have a lot of trouble convincing anyone to move.
>  
>  I think part of the solution will be for Fedora developers to help port
>  such applications to Python 3 
 Personally I'm in 100% agreement with the first part of this. We
(Fedora) shouldn't be forcing this move. Upstreams will be motivated or
not and that will give a timeline for the move. Obviously people doing
upstream work and Fedora work can motivate their upstream (or not) as
they wish.

 If we get to the point where say > 75% of upstreams have moved
(presumably in a way that their code will also work with 2.6, at least
in the short term) then at that point I think we as a distro. should
look at what to do with the other 25% ... either spending most of a
release helping them, or just dropping them. Whatever.

 Another, maybe less significant to Fedora point, is that until about
~F13 a non-trivial number of python developers will need to have code
working in 2.6 _and_ 2.4. So any forced migration so that they have to
be compatible with 2.4, 2.6 and 3.x would be a major PITA for them.

-- 
James Antill <james(a)fedoraproject.org&gt;
Fedora

[Toshio Kuratomi]

James Antill wrote:
>  On Thu, 2009-04-02 at 13:55 -0400, Tom "spot" Callaway
> wrote:
> > On 04/02/2009 01:46 PM, Jochen Schmitt wrote:
> >
> >> I think porting this kind of application may nned a lot of work to
> >> porting to
> >> Python 3000. So I thin wee need a python-2.6 compat package for a while
> >> until
> >> all applications will been ported.
> > The problem with this approach is that it does not motivate upstreams to
> > move to Python 3, they will just depend on the compat package, and we'll
> > have a lot of trouble convincing anyone to move.
> >
> > I think part of the solution will be for Fedora developers to help port
> > such applications to Python 3
>  
>   Personally I'm in 100% agreement with the first part of this. We
>  (Fedora) shouldn't be forcing this move. Upstreams will be motivated or
>  not and that will give a timeline for the move. Obviously people doing
>  upstream work and Fedora work can motivate their upstream (or not) as
>  they wish.
>  
>   If we get to the point where say > 75% of upstreams have moved
>  (presumably in a way that their code will also work with 2.6, at least
>  in the short term) then at that point I think we as a distro. should
>  look at what to do with the other 25% ... either spending most of a
>  release helping them, or just dropping them. Whatever.
>   
So if we take a wait for upstream path we need to make some changes in
our policy towards parallel python2/3 installs.  I think we need to
package the python3 interpreter so that people can do development of
python3 modules on Fedora.  Whether we allow python3 modules (so that
dependencies on more than the stdlib are easy) is more up in the air.
Certainly, it is going to be hard to develop very much of anything
without python3 modules but for things that are pure python, it is
possible for a developer to use easy_install to download those libraries
a lot of the time.  This does drive them away from distro packaging,
though, something I've been trying to get upstreams to see the benefit
of :-(

Note that including python3 modules opens the door for use of python3 in
applications while lack of modules pretty much means all but the
simplest command line applications will be stuck on python2.  For better
or for worse.

One thing I'm afraid of is how much trouble python3 is going to bring
for Unix.  Right now we have the opportunity to find issues and get
attention from python core devs for fixing them.  If we wait until
python-3.100 we'll be seen as johnny-come-latelys saying our platforms
are not compatible with what has already become standard practice in python.

>   Another, maybe less significant to Fedora point, is that until
> about
>  ~F13 a non-trivial number of python developers will need to have code
>  working in 2.6 _and_ 2.4. So any forced migration so that they have to
>  be compatible with 2.4, 2.6 and 3.x would be a major PITA for them.
>   
About a year, year and a half?  I think that's viable but it will be
beginning to get painful.  python-2.4 and python-2.5 just aren't
compatible with python-3.x and Fedora shouldn't be staying on old
packages solely to make compatibility with python-2.4 possible.

If we want people to be able to do that, we should admit that python3
and python2.x are different languages and install both.  (potentially
doubling the number of python modules during transition).  If we want to
hold to the python3 is an upgrade of python2 path, we need to be aware
that breaking compatibility with python-2.4 is in the cards and accept
that Fedora 12 is likely to contain versions of python modules that
won't run on python-2.4/2.5

-Toshio

[Toshio Kuratomi]

Jochen Schmitt wrote:
>  Toshio Kuratomi schrieb:
> > I don't have a solution for this yet.  Just throwing this out there as
> > something we need to consider as we start to port code to python-2.6
> > +py3 compat.
> >   
>  I thnk there is an additional issue we have. On Fedora we have several
>  packages
>  which use the C-API of python to extent the python environment. One example
>  it blender which contains python scripts which need a python extension
>  provided
>  by blender.
>  
>  I think porting this kind of application may nned a lot of work to
>  porting to
>  Python 3000. So I thin wee need a python-2.6 compat package for a while
>  until
>  all applications will been ported.
>   
As a note:
  http://docs.python.org/3.0/howto/cporting.html#cporting-howto

That doesn't look as though porting is going to be outrageously hard
(lots of grunt work but there's recipes for each change).  However, I
don't know if that guide is complete.  That's something we'll have to
explore when we start porting to 3.x.

-Toshio

[James Antill]

On Thu, 2009-04-02 at 10:20 -0700, Toshio Kuratomi wrote:
>  
>  A side note is that the Core devs apparently talked about naming the
>  interpreter binary for python /usr/bin/python3 at the language summit.
>  If they do this, /usr/bin/python will always mean python2 and
>  /usr/bin/python3 will be for python-3.x.  Not sure if that influences
>  us or not. 
 I don't see why it would. We've had the possibility to
have /usr/bin/python2.4 etc. for a long time ... but it doesn't really
help us due the binary name only being part of the problem.
 In theory we could take this as an official "yes, this is a different
language" hint, and make a python3deps.sh for rpmbuild, rename all the
python pkgs etc. ... but even then, can gedit etc. embend both
languages?

-- 
James Antill <james(a)fedoraproject.org&gt;
Fedora

[Toshio Kuratomi]

James Antill wrote:
>   In theory we could take this as an official "yes, this is a
> different
>  language" hint, and make a python3deps.sh for rpmbuild, rename all the
>  python pkgs etc. ... but even then, can gedit etc. embend both
>  languages?
>   
yep, this is what I was getting at (both the "official hint" and the
"we'd have to look for other issues").

-Toshio

[Ignacio Vazquez-Abrams]

On Thu, 2009-04-02 at 10:20 -0700, Toshio Kuratomi wrote:
>  I had the fortune to attend pycon this past week and went to a talk
>  about python-2.6 compatibility with python-3.
>  
>  http://us.pycon.org/2009/conference/schedule/event/74/
>  
>  The slides from that are available and largely self-explanatory.  They
>  list tricks for getting python code that runs on both py2.6 and py3.x. 
The video of the session (and in fact all PyCon 2009 sessions) are up:

http://pycon.blip.tv/file/1949281/
http://pycon.blip.tv/

The 3to2 tool he talks about briefly also has a place on the web:

http://wiki.python.org/moin/3to2

-- 
Ignacio Vazquez-Abrams <ivazqueznet(a)gmail.com&gt;

PLEASE don't CC me; I'm already subscribed

[Toshio Kuratomi]

Ignacio Vazquez-Abrams wrote:
>  On Thu, 2009-04-02 at 10:20 -0700, Toshio Kuratomi wrote:
> > I had the fortune to attend pycon this past week and went to a talk
> > about python-2.6 compatibility with python-3.
> >
> > http://us.pycon.org/2009/conference/schedule/event/74/
> >
> > The slides from that are available and largely self-explanatory.  They
> > list tricks for getting python code that runs on both py2.6 and py3.x.
>  
>  The video of the session (and in fact all PyCon 2009 sessions) are up:
>   
Not all the sessions yet, but a good chunk have been processed and
uploaded :-)

>  http://pycon.blip.tv/file/1949281/
>  http://pycon.blip.tv/
>  
>  The 3to2 tool he talks about briefly also has a place on the web:
>  
>  http://wiki.python.org/moin/3to2
>  
>   
-Toshio

[Kevin Kofler]

Toshio Kuratomi wrote:
>  I had the fortune to attend pycon this past week and went to a talk
>  about python-2.6 compatibility with python-3. 
I still think a compat package is the only viable approach, just like for
GTK+ 1 and 2 or for Qt/KDE 3 and 4.

The language is very different, so it's pretty much nontrivial to port all
the Python programs in Fedora, of which there are a lot.

        Kevin Kofler

[Jeremy Katz]

On Saturday, April 04 2009, Kevin Kofler said:
>  Toshio Kuratomi wrote:
>  > I had the fortune to attend pycon this past week and went to a talk
>  > about python-2.6 compatibility with python-3.
>  
>  I still think a compat package is the only viable approach, just like for
>  GTK+ 1 and 2 or for Qt/KDE 3 and 4.
>  
>  The language is very different, so it's pretty much nontrivial to port all
>  the Python programs in Fedora, of which there are a lot. 
It's also pretty non-trivial to include compat modules for every python
binding that is shipped.  It's not just like GTK+ 1 vs 2 or KDE3 vs
KDE4.

If only it were so simple

Jeremy