Multirelease effort: Moving to Python 3

Fri Jul 19 06:41:23 UTC 2013

----- Original Message -----
> On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
> > Hi all,
> > as a new Fedora Python maintainer, I have set myself a goal of moving
> > Fedora to Python 3 as a default.
> 
> I'm not sure we want to make python3 default depending on what your
> definition of default is.
> 
> /usr/bin/python should refer to python2 --
>     http://www.python.org/dev/peps/pep-0394/  I'd be -1 to changing this
> 

So, my definition of default is "all system tools use Python 3, it is the only Python that gets to minimal buildroot/minimal Fedora installation" - that means:
- livecd can still ship Python 2
- /usr/bin/python points to Python 3
  - Please note, that the pep you're referring to also states that "python should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions", so this wouldn't really be going against the pep.

> The python package itself should probably also remain python2 due to
>     dependencies and expectations from other distros and documentation --
>     I think I'd be -1 to changing this
> 
> The Fedora live images contain only python3, not python2 -- I'd be heavily
>     in favour of this. +1
> 
> > This is going to be a multirelease effort
> > that is going to affect lots of Fedora parts. Since we will need to switch
> > default package manager from Yum to DNF (which is supposed to work with
> > Python 3), we will need to wait for that. I've been told that DNF should
> > be default in F22, so that's my target, too. That should also give
> > everyone else plenty of time to work on other essential packages to make
> > this happen.
> 
> Getting there at the same time as we get to DNF sounds like a good timeline.
> (But see my note on anaconda below).  +1
> 
> > Here is my analysis/proposal:
> > Before switching, we need to make sure that everything "important" (*) is
> > Python 3 compatible. There are three steps I see in this transition:
> > 1) Getting rid of Python 2 in mock minimal buildroot.
> 
> I'm not sure about this one as it will cause a lot of package churn.  It
> might be a necessary pain pointi or it might be a pain point we want to
> defer until later in our porting efforts.  Have to think about it more.
> 

If you look at the minimal mock buildroot for rawhide now, the only thing that is drawing in Python is gdb because of it's Python bindings (if I'm not mistaken). So compiling GDB against Python 3, which should work with newest gdb, will accomplish this AFAICS.

> > 2) Porting Anaconda to Python 3.
> 
> +1 -- unfortunately, this probably depends on DNF.... So we may need to push
> DNF in F22 and anaconda compatible with python3 in F23.
> 

DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.

> > 3) Making all livecd packages depend on Python 3 by default (and eventually
> > getting rid of Python 2 from livecd) - this will also require switching
> > from Yum to DNF as a default, that is supposed to support Python 3.
> 
> +1 -- this is what I see as the eventual goal (or perhaps, livecd python2
> free followed by DVD python2 free followed by distro python2 free).
> 
> 
> 3.5) Switch tools that could target either python2 or python3 to target
> python3.  Currently the packaging guidelines say to target python2 to
> control dep proliferation and because that's the most supported by the
> larger python ecosystem.  We should switch the recommendation when our
> minimal environment must have python3 but does not need to have python2.
> 

IMO we should switch this for F21, since livecd ships Python 3 anyway, so the switch doesn't have to happen in one point, but can be continuous.

> > ( 4) Making as much of the remaining packages Python 3 compatible )
> > 
> 
> We could talk quite a bit on this point -- How active do we want to be with
> the things that aren't in one of the essential buckets from further up.  We
> could defer thinking about this until after we get the livecd python2-free,
> though.
> 

This is really the last step, that is somehow tied what you mentioned as a reaction to 3) - going through the rest of packages on DVD and then whole distro. This will take few more releases I guess, but it is not that important as sorting out livecd.

> > In past few days, I've been going through packages that are part of the
> > above steps. I have reported numerous bugs asking upstream and/or Fedora
> > maintainers for help with porting to Python 3. We have some spare cycles
> > in our small Python packaging team, that we will try to provide to whoever
> > needs them most, but we're limited and we'll have to rely on the upstreams
> > to do most of the work. I'm attaching a document with list of packages
> > that need porting with some notes/links to opened bugs. Sometime soon,
> > I'll open a tracking bug for this, so that everyone can see where we are
> > quickly.
> >
> > (*) I call these "important" packages (in terms of being important for the
> > Python 3 switch)
> >
> Cool.  A list of packages that are on the livecd is good.  One thing to
> remember, though, is that the current Python Guidelines specify that we are
> not to ship python3 versions of packages if upstream is not going to support
> us in that effort:
> https://fedoraproject.org/wiki/Packaging:Python#Subpackages
> 
> We could change that but I'm not 100% behind the idea of changing it.  As
> stated in the Guidelines:
> 
> "
> [...]doing this on our own in Fedora is essentially creating a fork. That has
> a large burden for maintaining the code, fixing bugs, porting when a new
> version of upstream's code appears, managing a release schedule, and other
> tasks normally handled by upstream. It's much better if we can cooperate
> with upstream to share this work than doing it all on our own.
> "
> 
> Luckily, in recent years I've only encountered a few upstreams that are
> unwilling to look at python3 patches.  Most upstreams are amenable to
> taking patches that establish python3 compatibility.  We just need to remain
> clear that we have to work upstream to get these python3 versions into
> fedora, not do it in our packages without upstream being on board.
> 

Yep. When I was opening the bugs, I tried to open them in the sense "please work with upstream to port to Python 3". Only when I found out that upstream supports Python 3, I asked the maintainer to add python3- subpackage.

> > From packaging point of view, this will probably require:
> > 1) Renaming python package to python2
> > 2) Renaming python3 package to python
> 
> -1: What are the benefits of this as the cost of this is very high in
> several ways:
> * updating our dependencies
> * divergence from other distros (I believe that arch is the only distro that
>   has decided to ship python3 as "python".  everyone else ships python3 as
>   python3)
> * updating our documentation
> * divergence from other upstream/googlable documentation
> 
> I could see us renaming the python package to python2, keeping a Virtual
> Provide in the python2 package for python (and similar for all of the
> subpackages and python-doc package), and leaving python3 as it is.  This
> might be a stepping stone to when the internet's memory hasstarted
> associating "python" with python3 instead of python2.
> 
> > 3) Switching the %{?with_python3} conditionals in specfiles to
> > %{?with_python2} (we will probably create a script to automate this, at
> > least partially)
> > 
> -1: This one doesn't make any sense to do.  The third-party python library
> ecosystem is highly weighted for python2.  There are only a handful of
> libraries that support python3 and not python2.  There are a boatload of
> libraries that support python2 and not python3.  We're starting from a base
> of existing python2 packages that may add support for python3.  The
> conditionals are there to enable packaging of that situation.
> 

And this situation will be changing in the future. Right now, there are not so many Python packages in Fedora that only support Python 2 (I didn't count, but you don't see them too often these days). IMO Fedora should lead the way of making Python 3 "the Python" and Python 2 "the old compat version". This also makes sense in the traditional linux-distro "one version of package" that we should be trying to pursue.

> > 
> > FAQ:
> > Q: Why do we need to switch to Python 3?
> > A: Because Python 2 is old, slower, less pythonic, doesn't get any more
> > functionality and it won't be that long before the official upstream
> > support ends [1]
> > 
> Although I agree with the need to switch to python3, I don't think the first
> three reasons are very compelling arguments (they're only half-truths) -- we
> should concentrate on the last reason and also on features that python3
> has that pyhton2 doesn't.  Chained exceptions are a pretty nice thing, for
> instance.
> 

So first three reason:
- Python 2 is old - how is that a half-truth?
- Slower - yes, in the beginning, Python 3 was significantly slower because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
- Less Pythonic - where do I start with this? Python 3 got rid of tons of unnecessary syntactic constructs as well as builtin object methods. E.g. "print" vs. "print()"; exception raising syntax; dict.iteritems() removed and dict.items() only left, more consistent unicode handling etc. So in the sense of having only one way to do things, Python 3 is more pythonic than Python 2. If you read through zen of python, you can find more arguments for this (e.g. making int and long one type - "Special cases aren't special enough to break the rules."; simplification/rewrite of parts of stdlib - "Simple is better than complex.", etc.)

> > Q: How do I port to Python 3?
> > A: There are tons of tutorials and howtos about porting and the
> > differencies in general. E.g. [2] (general), [3] (c-extensions)
> > 
> The best two tutorials for python3 porting are likely:
> https://wiki.ubuntu.com/Python/3 (Will be moving to the python.org wiki in
> the near future)
> 
> http://python3porting.com/
> 

Thanks for the links.

> > Q: What about Python 2?
> > A: We will maintain that at least as long as upstream supports it. After
> > that, I'd prefer dropping it, but since I know there will be people
> > wanting to keep it around, I'll gladly give the maintenance to someone
> > else.
> > 
> <nod>  2015 is right around the corner... I think someone else will get
> stuck maintaining the package :-/
> > 
> > I'll be glad to answer all your questions and discuss the above points.
> > Nothing is set into stone and I'd love to hear your ideas and comments.
> >
> I sent out a message earlier that we should have a python sig/python
> guidelines discussion at flock.  I think that nick and I are the only two
> that can definitely attend that in person.
> 
> Can anyone else make this timeslot on IRC?
> 
> http://flock2013.sched.org/event/281138262885f34d97408cfe65cdf21b?iframe=yes&w=800&sidebar=no&bg=no#?iframe=yes&w=800&sidebar=no&bg=no
> 

I'll try.

> Planning for python3 and any needed updates to the Guidelines surrounding
> this are one of the things I wanted us to discuss.
> 
> [..]
> 
> One thing it might be nice to see in the below list is what things we have
> some upstream control over already.  I believe the gdb work is being driven
> by dmalcolm.  anaconda and yum/dnf are things we are upstream for. etc.
> Knowing about this responsibility will help us to understand where we
> control our own destiny and where we're dependent on other upstreams.
> 
> In some cases where upstream isn't going to port (for instance, dead
> upstream), we may need to either port to a different upstream (potentially
> large one-time cost) or fork upstream (ongoing maintainance burden).
> 
> One specific note:
> >                     - python-pycurl - TODO -
> >                     https://github.com/p/pycurl/pull/28 (is this the
> >                     official upstream?)
> 
> You probably need to find someone to take over upstream maintainance of
> python-pycurl.  Over the past two or three years, various people have
> stepped up to take over upstream and never gotten more than a release out
> the door.
> 

Yeah, I'll try to speak to Fedora's maintainer of pycurl to see if he has any updates on this and we'll see.

> -Toshio
> 

Thanks for your thoughts.
Slavek.

-- 
Regards,
Bohuslav "Slavek" Kabrda.

[1] https://speakerdeck.com/pyconslides/python-3-dot-3-trust-me-its-better-than-python-2-dot-7-by-dr-brett-cannon