Multirelease effort: Moving to Python 3

Tue Jul 23 08:18:01 UTC 2013

On 07/23/2013 05:13 PM, Bohuslav Kabrda wrote:
> One way or the other, the PEP states that people should be careful about what /usr/bin/python points to, since it may be python 2 as well as python 3. This gives us the freedom to do the switch when we want and not break the upstream expectations (yes, I know that huge number of people expect /usr/bin/python to point to python 3, but this pep basically says that it's a bad idea to assume that).

As the migration notes in PEP 394 say: "More conservative distributions
that are less willing to tolerate breakage of third party scripts
continue to alias it to python2. Until the conventions described in this
PEP are more widely adopted, having python invoke python2 will remain
the recommended option."

So I guess it depends if Fedora sees itself as being in the "More
conservative" camp or not :)

>> Also, just to be clear, you do understand why switching the conditionals
>> doesn't work for our existing packages, correct?
>>
> 
> Please be more specific. What do you mean "doesn't work"?
> 
>>> IMO Fedora should lead the way of making Python 3 "the Python" and Python 2
>>> "the old compat version". This also makes sense in the traditional
>>> linux-distro "one version of package" that we should be trying to pursue.
>>>
>>
>> -1.  I'm serious, you've got the wrong conceptual model of the relationship
>> between python2 and python3 stuck in your head and it's coloring what you're
>> trying to achieve in bad ways.  Python3 is not an upgrade to Python2.
>> Python3 is a new language.  It is compatible in many ways.  If you can
>> target recent enough versions (at least python-2.6 but python2.7 is better
>> and python-3.3) you can set out to purposefully code things that work on
>> both languages.  But if you're writing general, working python2 code using
>> idioms and thought processes that you've mastered over the last 10 years,
>> chances are extremely high that not even your simple scripts are going to
>> run without modification.
>>
> 
> And again, I'm saying that your conceptual model is not necessarily the correct one.

For simple scripts, I think Toshio is correct. The basic syntax changes,
like print becoming a function, or the way you bind a caught exception
to a name, are easy to handle for applications and libraries with
something like python-modernize, but that's not going to happen for the
vast majority of sysadmin and general user scripts out there.

Consider the flak Canonical got when they switched /bin/sh from bash to
dash, and exposed all the scripts people had written that used bash
extensions, but still had /bin/sh in the shebang line. The risk in
switching the python symlink to python3 isn't really in breaking
*applications*, it's in breaking *scripts*.

There are *lots* of ways for things to break if Python 2 code that isn't
expecting it is run under Python 3. The "Common Stumbling Blocks"
(http://docs.python.org/dev/whatsnew/3.0.html#common-stumbling-blocks)
in the release notes for 3.0 cover some highlights. Most migration
guides gloss over these, since they're looking at things from the
perspective of development project that are taking steps to get ready
for the change, rather than an end user who has just run "fedup" and
finds that all of their homegrown Python scripts are now throwing syntax
errors and other strange things.

That's why I actually advocate for distros (including Fedora) to design
all their packages to not care, and always using a name qualified with
the major language version. If a sysadmin changes /bin/python to point
to something else, Fedora itself shouldn't be affected at all, only user
scripts.

As the migration notes in PEP 394 say:

  * It is suggested that even distribution-specific packages follow the
python2/python3 convention, even in code that is not intended to operate
on other distributions. This will reduce problems if the distribution
later decides to change the version of the Python interpreter that the
python command invokes, or if a sysadmin installs a custom python
command with a different major version than the distribution default.
Distributions can test whether they are fully following this convention
by changing the python interpreter on a test box and checking to see if
anything breaks.
  * If the above point is adhered to and sysadmins are permitted to
change the python command, then the python command should always be
implemented as a link to the interpreter binary (or a link to a link)
and not vice versa. That way, if a sysadmin does decide to replace the
installed python file, they can do so without inadvertently deleting the
previously installed binary.
  * If the Python 2 interpreter becomes uncommon, scripts should
nevertheless continue to use the python3 convention rather that just
python. This will ease transition in the event that yet another major
version of Python is released.
  * If these conventions are adhered to, it will become the case that
the python command is only executed in an interactive manner as a user
convenience, or to run scripts that are source compatible with both
Python 2 and Python 3.

It's important to read *all* of the migration notes in PEP 394, not just
the parts that agree with your preferred plan - back when we wrote it,
we did put a fair bit of thought into how a distribution could migrate
their own software to Python 3 without breaking end user scripts that
are still expecting Python 2.

It may be that in trying to be diplomatic about what Arch did, we
softened the wording too much so it can be interpreted as "what the Arch
folks did wasn't clearly denounced, so we can copy them rather than
doing what the PEP suggests!" :(

>>> So first three reason:
>>> - Python 2 is old - how is that a half-truth?
>>>
>> C is older.  Let's get rid of that first.  Old is not a reason to switch
>> or get rid of it.
>>
> 
> Python 2 has Python 3 as a successor, C does not (yes, I know you'll say C++, but this is really not the case).

A more accurate comparison would be the switch from K&R C to ANSI C (and
that took a long time).

Python 2.7 is a stable, well liked language. While the core development
team plans to stop making new binary releases of 2.7 in 2015, it will
still be receiving upstream source only security releases for some time
after that.

>>> - Slower - yes, in the beginning, Python 3 was significantly slower because
>>> of nonoptimal code after the rewrite from Python 3. But with Python 3.3
>>> for instance, you get tons of speed improvements - decimal module for
>>> instance got a significant boost. Brett Cannon had a nice presentation
>>> about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but
>>> mostly it's faster.
>>
>> * pypy is faster and mostly python2 compatible.  Many shops that want speed
>>   are switching to that rather than python3.  (Not saying this is a good
>>   idea for a distro to do.  I'm just saying that making the speed argument
>>   is not compelling).
>> * people don't use python for raw speed of processing.  They really just
>>   care if it's fast enough.  People who write python code would be happy to
>>   take speed increases if they were free.  But python2 to python3 requires
>>   porting code so it comes with a significant cost.  Speed is a side effect
>>   of switching, not a reason to switch.
>>
> 
> It is one of the reasons to switch for me.

One key advantage of Python 3.3+ is drastically reduced memory usage for
applications that deal almost entirely in unicode strings (courtesy of
PEP 393).

Cheers,
Nick.

-- 
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane

Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)