Multirelease effort: Moving to Python 3

[Bohuslav Kabrda]

----- Original Message -----
> I'm okay with this portion of the definition.  One note is I would be
> hesitant
> about the timing of python3 being the only python that is installed into the
> minimal buildroot. This should probably happen in rawhide right after
> a branching.
> 

To be more specific, I meant "installed into the minimal buildroot by default", which only means building GDB with Python 3, as I've written elsewhere. IMO it shouldn't cause any problems, because all the packages that use python 2 during build should BR: python2-devel anyway, right? (maybe some packages with build scripts written in python 2 may have minor issues, but these should be solvable by simply adding BR: python or rewriting to python 3)

> > - livecd can still ship Python 2
> 
> I would consider this to be the goal that we should shoot for, though.  We
> are constantly fighting for space on the install images and we know that
> people who install Fedora would like to have the ability to slim down what
> is installed.  Shooting for no-python2 on the livecds and after that, no
> python2 on the install dvds, and still later, no need for python2 in the
> packages in the repository seem like milestones that have actual real value
> for end users.
> 

Ok, +1

> > - /usr/bin/python points to Python 3
> 
> I am firmly against this.  more depth was in my reply to mmaslano although
> I'll reply to one thing here:
> 
> >   - Please note, that the pep you're referring to also states that "python
> >   should refer to the same target as python2 but may refer to python3 on
> >   some bleeding edge distributions", so this wouldn't really be going
> >   against the pep.
> >
> This is a misinterpretion of the PEP.  (This section is confusing, though:
> "python should refer to the same target as python2" is a recommendation to
> distributions.  "but may refer to python3 on some bleeding edge
> distributions" is a statement of fact for end users to watch out for) See
> the recommendation section:
> 
> "For the time being, it is recommended that python should refer to python2
> (however, some distributions have already chosen otherwise; see the
> Rationale and Migration Notes below)."
> 
> and Future Changes Section:
> http://www.python.org/dev/peps/pep-0394/#future-changes-to-this-recommendation
> 
> "
> It is anticipated that there will eventually come a time where the third
> party ecosystem surrounding Python 3 is sufficiently mature for this
> recommendation to be updated to suggest that the python symlink refer to
> python3 rather than python2.
> 
> This recommendation will be periodically reviewed over the next few years,
> and updated when the core development team judges it appropriate.
> "
> 
> The "may refer to python3" phrase is an acknowledgment that arch has moved
> to /usr/bin/python == python3 and isn't going to revert even though upstream
> thinks it's a... premature time to make that switch.  (To be fair to arch,
> the discussion and PEP happened as a result of arch making that switch so
> they'd already committed to that before the consensus was formed that this
> would be a bad thing to do atthis time.  We don't have that excuse ;-)
> 
> If you'd like to read the discussions for yourself, there are three threads
> linked from the PEP.  An even earlier one is at:
> http://mail.python.org/pipermail/python-dev/2010-November/105252.html
> 

One way or the other, the PEP states that people should be careful about what /usr/bin/python points to, since it may be python 2 as well as python 3. This gives us the freedom to do the switch when we want and not break the upstream expectations (yes, I know that huge number of people expect /usr/bin/python to point to python 3, but this pep basically says that it's a bad idea to assume that).

> 
> > 
> > > The python package itself should probably also remain python2 due to
> > >     dependencies and expectations from other distros and documentation --
> > >     I think I'd be -1 to changing this
> > > 
> > > The Fedora live images contain only python3, not python2 -- I'd be
> > > heavily
> > >     in favour of this. +1
> > > 
> > > > This is going to be a multirelease effort
> > > > that is going to affect lots of Fedora parts. Since we will need to
> > > > switch
> > > > default package manager from Yum to DNF (which is supposed to work with
> > > > Python 3), we will need to wait for that. I've been told that DNF
> > > > should
> > > > be default in F22, so that's my target, too. That should also give
> > > > everyone else plenty of time to work on other essential packages to
> > > > make
> > > > this happen.
> > > 
> > > Getting there at the same time as we get to DNF sounds like a good
> > > timeline.
> > > (But see my note on anaconda below).  +1
> > > 
> > > > Here is my analysis/proposal:
> > > > Before switching, we need to make sure that everything "important" (*)
> > > > is
> > > > Python 3 compatible. There are three steps I see in this transition:
> > > > 1) Getting rid of Python 2 in mock minimal buildroot.
> > > 
> > > I'm not sure about this one as it will cause a lot of package churn.  It
> > > might be a necessary pain pointi or it might be a pain point we want to
> > > defer until later in our porting efforts.  Have to think about it more.
> > > 
> > 
> > If you look at the minimal mock buildroot for rawhide now, the only thing
> > that is drawing in Python is gdb because of it's Python bindings (if I'm
> > not mistaken). So compiling GDB against Python 3, which should work with
> > newest gdb, will accomplish this AFAICS.
> > 
> <nod>  I thought that python might be one of the packages that showed up
> in the dep chain here:
> http://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions_2
> 
> which would mean that packages might be leaving it out of BuildRequires
> right now.  I wasn't able to find a dep chain leading back to python from
> one of those so I think that any package which isn't explicitly
> BuildRequiring python just has a bug.  No objection here although as noted
> earlier, we should probably do this right after a rawhide branch so that any
> of these bugs are found and fixed in plenty of time.
> 
> > > > 2) Porting Anaconda to Python 3.
> > > 
> > > +1 -- unfortunately, this probably depends on DNF.... So we may need to
> > > push
> > > DNF in F22 and anaconda compatible with python3 in F23.
> > > 
> > 
> > DNF is a continuous effort. I believe that DNF will provide it's Python 3
> > bindings sooner than in F22, so Anaconda devels can simultaneously do
> > porting to Python 3 as well as to DNF. IMO this is good thing, since they
> > will just do one big rewrite instead of two smaller.
> > 
> Well.... 1) If DNF lands their python3 bindings sooner, that's fine for the
> timeline.  But if they don't, anaconda can't be finished until after.  So
> this is something to note as a key piece of the switch.  2) Fedora
> anaconda experience (and general open source development experience as
> well... you've read the Joel on Software article about netscape, right?)
> would tend to show that big rewrites are worse than several smaller ones.
> 
> Sure, smaller ones mean that you touch the same code a few times before
> you're satisfied with it.  But smaller ones mean that you can stop partway
> through (say, for instance, because we needed a few extra weeks to port to
> DNF and that doesn't leave us enough time to complete the python3 port
> [even assuming that that doesn't take longer than anticipated] in time for
> Fedora 22's release date) which is just anticipating that Murphy's Law is
> inevitably going to throw a wrench into your timeline for completion.
> 

And still, they can write Python 3 code that is compatible with both Python 2 and Python 3, so if DNF fails to provide Python 3 bindings in time, they can run on Python 2.

> > > > 3) Making all livecd packages depend on Python 3 by default (and
> > > > eventually
> > > > getting rid of Python 2 from livecd) - this will also require switching
> > > > from Yum to DNF as a default, that is supposed to support Python 3.
> > > 
> > > +1 -- this is what I see as the eventual goal (or perhaps, livecd python2
> > > free followed by DVD python2 free followed by distro python2 free).
> > > 
> > > 
> > > 3.5) Switch tools that could target either python2 or python3 to target
> > > python3.  Currently the packaging guidelines say to target python2 to
> > > control dep proliferation and because that's the most supported by the
> > > larger python ecosystem.  We should switch the recommendation when our
> > > minimal environment must have python3 but does not need to have python2.
> > > 
> > 
> > IMO we should switch this for F21, since livecd ships Python 3 anyway, so
> > the switch doesn't have to happen in one point, but can be continuous.
> > 
> Ehhh...  I don't think the livecd having to ship python3 is a good measure
> for this.  I think something considerably more minimal than the livecd would
> be better.  talked to mattdm (since he's been working on minimal
> environments) and he suggested @core or @standard groups might be
> appropriate.)  The idea is to avoid doubling the needed python stacks on
> minimal environments until necessary.  Switching tools that have the option
> of running on either one or the other of the stacks to python3 prematurely
> means that we start doubling the the python stacks needed before it's
> necessary.
> 

As I replied to mattdm, I'm not against switching this at a single time (although doing more little steps is better than doing a single huge step ;)).

> > > > ( 4) Making as much of the remaining packages Python 3 compatible )
> > > > 
> > > 
> > > We could talk quite a bit on this point -- How active do we want to be
> > > with
> > > the things that aren't in one of the essential buckets from further up.
> > > We
> > > could defer thinking about this until after we get the livecd
> > > python2-free,
> > > though.
> > > 
> > 
> > This is really the last step, that is somehow tied what you mentioned as
> > a reaction to 3) - going through the rest of packages on DVD and then
> > whole distro. This will take few more releases I guess, but it is not that
> > important as sorting out livecd.
> >
> yeah, this strikes me as extending far into the future.  I will note,
> though, that ideas about changing /usr/bin/python to point to python3
> probably come in the latter stages of this step rather than before.
> 
> > > 
> > > > 3) Switching the %{?with_python3} conditionals in specfiles to
> > > > %{?with_python2} (we will probably create a script to automate this, at
> > > > least partially)
> > > > 
> > > -1: This one doesn't make any sense to do.  The third-party python
> > > library
> > > ecosystem is highly weighted for python2.  There are only a handful of
> > > libraries that support python3 and not python2.  There are a boatload of
> > > libraries that support python2 and not python3.  We're starting from a
> > > base
> > > of existing python2 packages that may add support for python3.  The
> > > conditionals are there to enable packaging of that situation.
> > > 
> > 
> > And this situation will be changing in the future. Right now, there are not
> > so many Python packages in Fedora that only support Python 2 (I didn't
> > count, but you don't see them too often these days).
> >
> Uh.... What's your methodology?  This is a very, very, very bad estimate but
> I think it'll show that we need more than an anecdote to prove that
> statement:
> 
> $ repoquery -q 'python3-*' |wc -l
> 259
> $ repoquery -q 'python-*' |wc -l
> 1099
> 

Heh, true :) I based this on packages that I review (out of which likely 90 % or so have python3- subpackage) and I didn't really count. Well, it is high time we start working on these 700 :)

> Also, just to be clear, you do understand why switching the conditionals
> doesn't work for our existing packages, correct?
> 

Please be more specific. What do you mean "doesn't work"?

> > IMO Fedora should lead the way of making Python 3 "the Python" and Python 2
> > "the old compat version". This also makes sense in the traditional
> > linux-distro "one version of package" that we should be trying to pursue.
> >
> 
> -1.  I'm serious, you've got the wrong conceptual model of the relationship
> between python2 and python3 stuck in your head and it's coloring what you're
> trying to achieve in bad ways.  Python3 is not an upgrade to Python2.
> Python3 is a new language.  It is compatible in many ways.  If you can
> target recent enough versions (at least python-2.6 but python2.7 is better
> and python-3.3) you can set out to purposefully code things that work on
> both languages.  But if you're writing general, working python2 code using
> idioms and thought processes that you've mastered over the last 10 years,
> chances are extremely high that not even your simple scripts are going to
> run without modification.
> 

And again, I'm saying that your conceptual model is not necessarily the correct one.

> 
> > 
> > > > 
> > > > FAQ:
> > > > Q: Why do we need to switch to Python 3?
> > > > A: Because Python 2 is old, slower, less pythonic, doesn't get any more
> > > > functionality and it won't be that long before the official upstream
> > > > support ends [1]
> > > > 
> > > Although I agree with the need to switch to python3, I don't think the
> > > first
> > > three reasons are very compelling arguments (they're only half-truths) --
> > > we
> > > should concentrate on the last reason and also on features that python3
> > > has that pyhton2 doesn't.  Chained exceptions are a pretty nice thing,
> > > for
> > > instance.
> > > 
> > 
> > So first three reason:
> > - Python 2 is old - how is that a half-truth?
> >
> C is older.  Let's get rid of that first.  Old is not a reason to switch
> or get rid of it.
> 

Python 2 has Python 3 as a successor, C does not (yes, I know you'll say C++, but this is really not the case).

> 
> > - Slower - yes, in the beginning, Python 3 was significantly slower because
> > of nonoptimal code after the rewrite from Python 3. But with Python 3.3
> > for instance, you get tons of speed improvements - decimal module for
> > instance got a significant boost. Brett Cannon had a nice presentation
> > about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but
> > mostly it's faster.
> 
> * pypy is faster and mostly python2 compatible.  Many shops that want speed
>   are switching to that rather than python3.  (Not saying this is a good
>   idea for a distro to do.  I'm just saying that making the speed argument
>   is not compelling).
> * people don't use python for raw speed of processing.  They really just
>   care if it's fast enough.  People who write python code would be happy to
>   take speed increases if they were free.  But python2 to python3 requires
>   porting code so it comes with a significant cost.  Speed is a side effect
>   of switching, not a reason to switch.
> 

It is one of the reasons to switch for me.

> > - Less Pythonic - where do I start with this? Python 3 got rid of tons of
> > unnecessary syntactic constructs as well as builtin object methods. E.g.
> > "print" vs. "print()"; exception raising syntax; dict.iteritems() removed
> > and dict.items() only left, more consistent unicode handling etc. So in
> > the sense of having only one way to do things, Python 3 is more pythonic
> > than Python 2. If you read through zen of python, you can find more
> > arguments for this (e.g. making int and long one type - "Special cases
> > aren't special enough to break the rules."; simplification/rewrite of
> > parts of stdlib - "Simple is better than complex.", etc.)
> > 
> pythonic is a very vague statement and I wouldn't consider most of your list
> to be examples of those.  Yes, python3 may be a *better* language (and
> I would include most of your list as "features of python3 that python2 does
> not have) but a more pythonic language... that's not something that you can
> readily measure.  For instance, I can make the case that python3's unicode
> handling is less pythonic than python2 as it violates three rules of the zen
> of python:
> 
> Explicit is better than implicit.
> Errors should never pass silently.
> In the face of ambiguity, refuse the temptation to guess.
> 

Eh, I really don't see where Python 3 unicode handling violates these. Could you be more specific?

> (To be fair, python2 violated some of these rules in its unicode handling as
> well, although errors should never pass silently would probably take some
> work to convince most people :-)
> 
> Anyhow, I stick to my assertion that we should be talking mostly about
> upstream support ending as the reason to switch and also the features that
> python3 provides that python2 does not [and as noted, I'd lump your
> "pythonic" list into this category.]  Stating a compelling argument of why
> people should change isn't just about identifying all the things that
> the change will do.  It's also about identifying the things that the change
> will do that are important to the person and resonate with their needs.
> 
> -Toshio

-- 
Regards,
Bohuslav "Slavek" Kabrda.