Multirelease effort: Moving to Python 3

[Toshio Kuratomi]

On Fri, Jul 19, 2013 at 02:41:23AM -0400, Bohuslav Kabrda wrote:
> ----- Original Message -----
> > On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
> > > Hi all,
> > > as a new Fedora Python maintainer, I have set myself a goal of moving
> > > Fedora to Python 3 as a default.
> > 
> > I'm not sure we want to make python3 default depending on what your
> > definition of default is.
> > 
> > /usr/bin/python should refer to python2 --
> >     http://www.python.org/dev/peps/pep-0394/  I'd be -1 to changing this
> > 
> 
> So, my definition of default is "all system tools use Python 3, it is the only Python that gets to minimal buildroot/minimal Fedora installation" - that means:

I'm okay with this portion of the definition.  One note is I would be hesitant
about the timing of python3 being the only python that is installed into the
minimal buildroot. This should probably happen in rawhide right after
a branching.  

> - livecd can still ship Python 2

I would consider this to be the goal that we should shoot for, though.  We
are constantly fighting for space on the install images and we know that
people who install Fedora would like to have the ability to slim down what
is installed.  Shooting for no-python2 on the livecds and after that, no
python2 on the install dvds, and still later, no need for python2 in the
packages in the repository seem like milestones that have actual real value
for end users.

> - /usr/bin/python points to Python 3

I am firmly against this.  more depth was in my reply to mmaslano although
I'll reply to one thing here:

>   - Please note, that the pep you're referring to also states that "python
>   should refer to the same target as python2 but may refer to python3 on
>   some bleeding edge distributions", so this wouldn't really be going
>   against the pep.
>
This is a misinterpretion of the PEP.  (This section is confusing, though:
"python should refer to the same target as python2" is a recommendation to
distributions.  "but may refer to python3 on some bleeding edge
distributions" is a statement of fact for end users to watch out for) See
the recommendation section:

"For the time being, it is recommended that python should refer to python2
(however, some distributions have already chosen otherwise; see the
Rationale and Migration Notes below)."

and Future Changes Section:
http://www.python.org/dev/peps/pep-0394/#future-changes-to-this-recommendation

"
It is anticipated that there will eventually come a time where the third
party ecosystem surrounding Python 3 is sufficiently mature for this
recommendation to be updated to suggest that the python symlink refer to
python3 rather than python2.

This recommendation will be periodically reviewed over the next few years,
and updated when the core development team judges it appropriate.
"

The "may refer to python3" phrase is an acknowledgment that arch has moved
to /usr/bin/python == python3 and isn't going to revert even though upstream
thinks it's a... premature time to make that switch.  (To be fair to arch,
the discussion and PEP happened as a result of arch making that switch so
they'd already committed to that before the consensus was formed that this
would be a bad thing to do atthis time.  We don't have that excuse ;-)

If you'd like to read the discussions for yourself, there are three threads
linked from the PEP.  An even earlier one is at:
http://mail.python.org/pipermail/python-dev/2010-November/105252.html


> 
> > The python package itself should probably also remain python2 due to
> >     dependencies and expectations from other distros and documentation --
> >     I think I'd be -1 to changing this
> > 
> > The Fedora live images contain only python3, not python2 -- I'd be heavily
> >     in favour of this. +1
> > 
> > > This is going to be a multirelease effort
> > > that is going to affect lots of Fedora parts. Since we will need to switch
> > > default package manager from Yum to DNF (which is supposed to work with
> > > Python 3), we will need to wait for that. I've been told that DNF should
> > > be default in F22, so that's my target, too. That should also give
> > > everyone else plenty of time to work on other essential packages to make
> > > this happen.
> > 
> > Getting there at the same time as we get to DNF sounds like a good timeline.
> > (But see my note on anaconda below).  +1
> > 
> > > Here is my analysis/proposal:
> > > Before switching, we need to make sure that everything "important" (*) is
> > > Python 3 compatible. There are three steps I see in this transition:
> > > 1) Getting rid of Python 2 in mock minimal buildroot.
> > 
> > I'm not sure about this one as it will cause a lot of package churn.  It
> > might be a necessary pain pointi or it might be a pain point we want to
> > defer until later in our porting efforts.  Have to think about it more.
> > 
> 
> If you look at the minimal mock buildroot for rawhide now, the only thing that is drawing in Python is gdb because of it's Python bindings (if I'm not mistaken). So compiling GDB against Python 3, which should work with newest gdb, will accomplish this AFAICS.
> 
<nod>  I thought that python might be one of the packages that showed up
in the dep chain here: http://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions_2

which would mean that packages might be leaving it out of BuildRequires
right now.  I wasn't able to find a dep chain leading back to python from
one of those so I think that any package which isn't explicitly
BuildRequiring python just has a bug.  No objection here although as noted
earlier, we should probably do this right after a rawhide branch so that any
of these bugs are found and fixed in plenty of time.

> > > 2) Porting Anaconda to Python 3.
> > 
> > +1 -- unfortunately, this probably depends on DNF.... So we may need to push
> > DNF in F22 and anaconda compatible with python3 in F23.
> > 
> 
> DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.
> 
Well.... 1) If DNF lands their python3 bindings sooner, that's fine for the
timeline.  But if they don't, anaconda can't be finished until after.  So
this is something to note as a key piece of the switch.  2) Fedora
anaconda experience (and general open source development experience as
well... you've read the Joel on Software article about netscape, right?)
would tend to show that big rewrites are worse than several smaller ones.

Sure, smaller ones mean that you touch the same code a few times before
you're satisfied with it.  But smaller ones mean that you can stop partway
through (say, for instance, because we needed a few extra weeks to port to
DNF and that doesn't leave us enough time to complete the python3 port
[even assuming that that doesn't take longer than anticipated] in time for
Fedora 22's release date) which is just anticipating that Murphy's Law is
inevitably going to throw a wrench into your timeline for completion.

> > > 3) Making all livecd packages depend on Python 3 by default (and eventually
> > > getting rid of Python 2 from livecd) - this will also require switching
> > > from Yum to DNF as a default, that is supposed to support Python 3.
> > 
> > +1 -- this is what I see as the eventual goal (or perhaps, livecd python2
> > free followed by DVD python2 free followed by distro python2 free).
> > 
> > 
> > 3.5) Switch tools that could target either python2 or python3 to target
> > python3.  Currently the packaging guidelines say to target python2 to
> > control dep proliferation and because that's the most supported by the
> > larger python ecosystem.  We should switch the recommendation when our
> > minimal environment must have python3 but does not need to have python2.
> > 
> 
> IMO we should switch this for F21, since livecd ships Python 3 anyway, so the switch doesn't have to happen in one point, but can be continuous.
> 
Ehhh...  I don't think the livecd having to ship python3 is a good measure
for this.  I think something considerably more minimal than the livecd would
be better.  talked to mattdm (since he's been working on minimal
environments) and he suggested @core or @standard groups might be
appropriate.)  The idea is to avoid doubling the needed python stacks on
minimal environments until necessary.  Switching tools that have the option
of running on either one or the other of the stacks to python3 prematurely
means that we start doubling the the python stacks needed before it's
necessary.

> > > ( 4) Making as much of the remaining packages Python 3 compatible )
> > > 
> > 
> > We could talk quite a bit on this point -- How active do we want to be with
> > the things that aren't in one of the essential buckets from further up.  We
> > could defer thinking about this until after we get the livecd python2-free,
> > though.
> > 
> 
> This is really the last step, that is somehow tied what you mentioned as
> a reaction to 3) - going through the rest of packages on DVD and then
> whole distro. This will take few more releases I guess, but it is not that
> important as sorting out livecd.
>
yeah, this strikes me as extending far into the future.  I will note,
though, that ideas about changing /usr/bin/python to point to python3
probably come in the latter stages of this step rather than before.

> > 
> > > 3) Switching the %{?with_python3} conditionals in specfiles to
> > > %{?with_python2} (we will probably create a script to automate this, at
> > > least partially)
> > > 
> > -1: This one doesn't make any sense to do.  The third-party python library
> > ecosystem is highly weighted for python2.  There are only a handful of
> > libraries that support python3 and not python2.  There are a boatload of
> > libraries that support python2 and not python3.  We're starting from a base
> > of existing python2 packages that may add support for python3.  The
> > conditionals are there to enable packaging of that situation.
> > 
> 
> And this situation will be changing in the future. Right now, there are not so many Python packages in Fedora that only support Python 2 (I didn't count, but you don't see them too often these days).
>
Uh.... What's your methodology?  This is a very, very, very bad estimate but
I think it'll show that we need more than an anecdote to prove that
statement:

$ repoquery -q 'python3-*' |wc -l
259
$ repoquery -q 'python-*' |wc -l
1099

Also, just to be clear, you do understand why switching the conditionals
doesn't work for our existing packages, correct?

> IMO Fedora should lead the way of making Python 3 "the Python" and Python 2 "the old compat version". This also makes sense in the traditional linux-distro "one version of package" that we should be trying to pursue.
>

-1.  I'm serious, you've got the wrong conceptual model of the relationship
between python2 and python3 stuck in your head and it's coloring what you're
trying to achieve in bad ways.  Python3 is not an upgrade to Python2.
Python3 is a new language.  It is compatible in many ways.  If you can
target recent enough versions (at least python-2.6 but python2.7 is better
and python-3.3) you can set out to purposefully code things that work on
both languages.  But if you're writing general, working python2 code using
idioms and thought processes that you've mastered over the last 10 years,
chances are extremely high that not even your simple scripts are going to
run without modification.


> 
> > > 
> > > FAQ:
> > > Q: Why do we need to switch to Python 3?
> > > A: Because Python 2 is old, slower, less pythonic, doesn't get any more
> > > functionality and it won't be that long before the official upstream
> > > support ends [1]
> > > 
> > Although I agree with the need to switch to python3, I don't think the first
> > three reasons are very compelling arguments (they're only half-truths) -- we
> > should concentrate on the last reason and also on features that python3
> > has that pyhton2 doesn't.  Chained exceptions are a pretty nice thing, for
> > instance.
> > 
> 
> So first three reason:
> - Python 2 is old - how is that a half-truth?
>
C is older.  Let's get rid of that first.  Old is not a reason to switch
or get rid of it.


> - Slower - yes, in the beginning, Python 3 was significantly slower because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.

* pypy is faster and mostly python2 compatible.  Many shops that want speed
  are switching to that rather than python3.  (Not saying this is a good
  idea for a distro to do.  I'm just saying that making the speed argument
  is not compelling).
* people don't use python for raw speed of processing.  They really just
  care if it's fast enough.  People who write python code would be happy to
  take speed increases if they were free.  But python2 to python3 requires
  porting code so it comes with a significant cost.  Speed is a side effect
  of switching, not a reason to switch.

> - Less Pythonic - where do I start with this? Python 3 got rid of tons of unnecessary syntactic constructs as well as builtin object methods. E.g. "print" vs. "print()"; exception raising syntax; dict.iteritems() removed and dict.items() only left, more consistent unicode handling etc. So in the sense of having only one way to do things, Python 3 is more pythonic than Python 2. If you read through zen of python, you can find more arguments for this (e.g. making int and long one type - "Special cases aren't special enough to break the rules."; simplification/rewrite of parts of stdlib - "Simple is better than complex.", etc.)
> 
pythonic is a very vague statement and I wouldn't consider most of your list
to be examples of those.  Yes, python3 may be a *better* language (and
I would include most of your list as "features of python3 that python2 does
not have) but a more pythonic language... that's not something that you can
readily measure.  For instance, I can make the case that python3's unicode
handling is less pythonic than python2 as it violates three rules of the zen
of python:

Explicit is better than implicit.
Errors should never pass silently.
In the face of ambiguity, refuse the temptation to guess.

(To be fair, python2 violated some of these rules in its unicode handling as
well, although errors should never pass silently would probably take some
work to convince most people :-)

Anyhow, I stick to my assertion that we should be talking mostly about
upstream support ending as the reason to switch and also the features that
python3 provides that python2 does not [and as noted, I'd lump your
"pythonic" list into this category.]  Stating a compelling argument of why
people should change isn't just about identifying all the things that
the change will do.  It's also about identifying the things that the change
will do that are important to the person and resonate with their needs.

-Toshio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/devel/attachments/20130719/c945aaf1/attachment-0001.sig>