Multirelease effort: Moving to Python 3

Thu Jul 18 16:53:40 UTC 2013

On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
> Hi all,
> as a new Fedora Python maintainer, I have set myself a goal of moving
> Fedora to Python 3 as a default.

I'm not sure we want to make python3 default depending on what your
definition of default is.

/usr/bin/python should refer to python2 --
    http://www.python.org/dev/peps/pep-0394/  I'd be -1 to changing this

The python package itself should probably also remain python2 due to
    dependencies and expectations from other distros and documentation --
    I think I'd be -1 to changing this

The Fedora live images contain only python3, not python2 -- I'd be heavily
    in favour of this. +1

> This is going to be a multirelease effort
> that is going to affect lots of Fedora parts. Since we will need to switch
> default package manager from Yum to DNF (which is supposed to work with
> Python 3), we will need to wait for that. I've been told that DNF should
> be default in F22, so that's my target, too. That should also give
> everyone else plenty of time to work on other essential packages to make
> this happen.

Getting there at the same time as we get to DNF sounds like a good timeline.
(But see my note on anaconda below).  +1

> Here is my analysis/proposal:
> Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
> 1) Getting rid of Python 2 in mock minimal buildroot.

I'm not sure about this one as it will cause a lot of package churn.  It
might be a necessary pain pointi or it might be a pain point we want to
defer until later in our porting efforts.  Have to think about it more.

> 2) Porting Anaconda to Python 3.

+1 -- unfortunately, this probably depends on DNF.... So we may need to push
DNF in F22 and anaconda compatible with python3 in F23.

> 3) Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.

+1 -- this is what I see as the eventual goal (or perhaps, livecd python2
free followed by DVD python2 free followed by distro python2 free).

3.5) Switch tools that could target either python2 or python3 to target
python3.  Currently the packaging guidelines say to target python2 to
control dep proliferation and because that's the most supported by the
larger python ecosystem.  We should switch the recommendation when our
minimal environment must have python3 but does not need to have python2.

> ( 4) Making as much of the remaining packages Python 3 compatible )
> 

We could talk quite a bit on this point -- How active do we want to be with
the things that aren't in one of the essential buckets from further up.  We
could defer thinking about this until after we get the livecd python2-free,
though.

> In past few days, I've been going through packages that are part of the above steps. I have reported numerous bugs asking upstream and/or Fedora maintainers for help with porting to Python 3. We have some spare cycles in our small Python packaging team, that we will try to provide to whoever needs them most, but we're limited and we'll have to rely on the upstreams to do most of the work. I'm attaching a document with list of packages that need porting with some notes/links to opened bugs. Sometime soon, I'll open a tracking bug for this, so that everyone can see where we are quickly.
>
> (*) I call these "important" packages (in terms of being important for the Python 3 switch)
>
Cool.  A list of packages that are on the livecd is good.  One thing to
remember, though, is that the current Python Guidelines specify that we are
not to ship python3 versions of packages if upstream is not going to support
us in that effort:
https://fedoraproject.org/wiki/Packaging:Python#Subpackages

We could change that but I'm not 100% behind the idea of changing it.  As
stated in the Guidelines: 

"
[...]doing this on our own in Fedora is essentially creating a fork. That has
a large burden for maintaining the code, fixing bugs, porting when a new
version of upstream's code appears, managing a release schedule, and other
tasks normally handled by upstream. It's much better if we can cooperate
with upstream to share this work than doing it all on our own.
"

Luckily, in recent years I've only encountered a few upstreams that are
unwilling to look at python3 patches.  Most upstreams are amenable to
taking patches that establish python3 compatibility.  We just need to remain
clear that we have to work upstream to get these python3 versions into
fedora, not do it in our packages without upstream being on board.

> From packaging point of view, this will probably require:
> 1) Renaming python package to python2
> 2) Renaming python3 package to python

-1: What are the benefits of this as the cost of this is very high in
several ways:
* updating our dependencies
* divergence from other distros (I believe that arch is the only distro that
  has decided to ship python3 as "python".  everyone else ships python3 as
  python3)
* updating our documentation
* divergence from other upstream/googlable documentation

I could see us renaming the python package to python2, keeping a Virtual
Provide in the python2 package for python (and similar for all of the
subpackages and python-doc package), and leaving python3 as it is.  This
might be a stepping stone to when the internet's memory hasstarted
associating "python" with python3 instead of python2.

> 3) Switching the %{?with_python3} conditionals in specfiles to %{?with_python2} (we will probably create a script to automate this, at least partially)
> 
-1: This one doesn't make any sense to do.  The third-party python library
ecosystem is highly weighted for python2.  There are only a handful of
libraries that support python3 and not python2.  There are a boatload of
libraries that support python2 and not python3.  We're starting from a base
of existing python2 packages that may add support for python3.  The
conditionals are there to enable packaging of that situation.

> 
> FAQ:
> Q: Why do we need to switch to Python 3?
> A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
> 
Although I agree with the need to switch to python3, I don't think the first
three reasons are very compelling arguments (they're only half-truths) -- we
should concentrate on the last reason and also on features that python3
has that pyhton2 doesn't.  Chained exceptions are a pretty nice thing, for
instance.

> Q: How do I port to Python 3?
> A: There are tons of tutorials and howtos about porting and the differencies in general. E.g. [2] (general), [3] (c-extensions)
> 
The best two tutorials for python3 porting are likely:
https://wiki.ubuntu.com/Python/3 (Will be moving to the python.org wiki in
the near future)

http://python3porting.com/

> Q: What about Python 2?
> A: We will maintain that at least as long as upstream supports it. After that, I'd prefer dropping it, but since I know there will be people wanting to keep it around, I'll gladly give the maintenance to someone else.
> 
<nod>  2015 is right around the corner... I think someone else will get
stuck maintaining the package :-/
> 
> I'll be glad to answer all your questions and discuss the above points. Nothing is set into stone and I'd love to hear your ideas and comments.
>
I sent out a message earlier that we should have a python sig/python
guidelines discussion at flock.  I think that nick and I are the only two
that can definitely attend that in person.

Can anyone else make this timeslot on IRC?

http://flock2013.sched.org/event/281138262885f34d97408cfe65cdf21b?iframe=yes&w=800&sidebar=no&bg=no#?iframe=yes&w=800&sidebar=no&bg=no

Planning for python3 and any needed updates to the Guidelines surrounding
this are one of the things I wanted us to discuss.

[..]

One thing it might be nice to see in the below list is what things we have
some upstream control over already.  I believe the gdb work is being driven
by dmalcolm.  anaconda and yum/dnf are things we are upstream for. etc.
Knowing about this responsibility will help us to understand where we
control our own destiny and where we're dependent on other upstreams.

In some cases where upstream isn't going to port (for instance, dead
upstream), we may need to either port to a different upstream (potentially
large one-time cost) or fork upstream (ongoing maintainance burden).

One specific note:
>                     - python-pycurl - TODO - https://github.com/p/pycurl/pull/28 (is this the official upstream?)

You probably need to find someone to take over upstream maintainance of
python-pycurl.  Over the past two or three years, various people have
stepped up to take over upstream and never gotten more than a release out
the door.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20130718/9be1532c/attachment.sig>