Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default. This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition: 1) Getting rid of Python 2 in mock minimal buildroot. 2) Porting Anaconda to Python 3. 3) Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3. ( 4) Making as much of the remaining packages Python 3 compatible )
In past few days, I've been going through packages that are part of the above steps. I have reported numerous bugs asking upstream and/or Fedora maintainers for help with porting to Python 3. We have some spare cycles in our small Python packaging team, that we will try to provide to whoever needs them most, but we're limited and we'll have to rely on the upstreams to do most of the work. I'm attaching a document with list of packages that need porting with some notes/links to opened bugs. Sometime soon, I'll open a tracking bug for this, so that everyone can see where we are quickly. (*) I call these "important" packages (in terms of being important for the Python 3 switch)
From packaging point of view, this will probably require:
1) Renaming python package to python2 2) Renaming python3 package to python 3) Switching the %{?with_python3} conditionals in specfiles to %{?with_python2} (we will probably create a script to automate this, at least partially)
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Q: How do I port to Python 3? A: There are tons of tutorials and howtos about porting and the differencies in general. E.g. [2] (general), [3] (c-extensions)
Q: What about Python 2? A: We will maintain that at least as long as upstream supports it. After that, I'd prefer dropping it, but since I know there will be people wanting to keep it around, I'll gladly give the maintenance to someone else.
I'll be glad to answer all your questions and discuss the above points. Nothing is set into stone and I'd love to hear your ideas and comments. Thanks for reading this through! Slavek.
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default.
That's a worthy goal!
From packaging point of view, this will probably require:
- Renaming python package to python2
- Renaming python3 package to python
- Switching the %{?with_python3} conditionals in specfiles to %{?with_python2} (we will probably create a script to automate this, at least partially)
But that seems like pointless busywork and churn. Sure, when python3 is ubiquitous, those checks can be dropped or reversed... But right now it would be better to focus on actual python3 versions of all packages — we're not there yet.
Zbyszek
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default.
That's a worthy goal!
From packaging point of view, this will probably require:
- Renaming python package to python2
- Renaming python3 package to python
- Switching the %{?with_python3} conditionals in specfiles to
%{?with_python2} (we will probably create a script to automate this, at least partially)
But that seems like pointless busywork and churn. Sure, when python3 is ubiquitous, those checks can be dropped or reversed... But right now it would be better to focus on actual python3 versions of all packages — we're not there yet.
Yes, current efforts should be directed to porting for Python 3, the packaging points I mentioned will really be the very last step. Thanks, Slavek.
Zbyszek
they are not broken. they are refucktored -- alxchk
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default.
I'm not sure we want to make python3 default depending on what your definition of default is.
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
The python package itself should probably also remain python2 due to dependencies and expectations from other distros and documentation -- I think I'd be -1 to changing this
The Fedora live images contain only python3, not python2 -- I'd be heavily in favour of this. +1
This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Getting there at the same time as we get to DNF sounds like a good timeline. (But see my note on anaconda below). +1
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
I'm not sure about this one as it will cause a lot of package churn. It might be a necessary pain pointi or it might be a pain point we want to defer until later in our porting efforts. Have to think about it more.
- Porting Anaconda to Python 3.
+1 -- unfortunately, this probably depends on DNF.... So we may need to push DNF in F22 and anaconda compatible with python3 in F23.
- Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
+1 -- this is what I see as the eventual goal (or perhaps, livecd python2 free followed by DVD python2 free followed by distro python2 free).
3.5) Switch tools that could target either python2 or python3 to target python3. Currently the packaging guidelines say to target python2 to control dep proliferation and because that's the most supported by the larger python ecosystem. We should switch the recommendation when our minimal environment must have python3 but does not need to have python2.
( 4) Making as much of the remaining packages Python 3 compatible )
We could talk quite a bit on this point -- How active do we want to be with the things that aren't in one of the essential buckets from further up. We could defer thinking about this until after we get the livecd python2-free, though.
In past few days, I've been going through packages that are part of the above steps. I have reported numerous bugs asking upstream and/or Fedora maintainers for help with porting to Python 3. We have some spare cycles in our small Python packaging team, that we will try to provide to whoever needs them most, but we're limited and we'll have to rely on the upstreams to do most of the work. I'm attaching a document with list of packages that need porting with some notes/links to opened bugs. Sometime soon, I'll open a tracking bug for this, so that everyone can see where we are quickly.
(*) I call these "important" packages (in terms of being important for the Python 3 switch)
Cool. A list of packages that are on the livecd is good. One thing to remember, though, is that the current Python Guidelines specify that we are not to ship python3 versions of packages if upstream is not going to support us in that effort: https://fedoraproject.org/wiki/Packaging:Python#Subpackages
We could change that but I'm not 100% behind the idea of changing it. As stated in the Guidelines:
" [...]doing this on our own in Fedora is essentially creating a fork. That has a large burden for maintaining the code, fixing bugs, porting when a new version of upstream's code appears, managing a release schedule, and other tasks normally handled by upstream. It's much better if we can cooperate with upstream to share this work than doing it all on our own. "
Luckily, in recent years I've only encountered a few upstreams that are unwilling to look at python3 patches. Most upstreams are amenable to taking patches that establish python3 compatibility. We just need to remain clear that we have to work upstream to get these python3 versions into fedora, not do it in our packages without upstream being on board.
From packaging point of view, this will probably require:
- Renaming python package to python2
- Renaming python3 package to python
-1: What are the benefits of this as the cost of this is very high in several ways: * updating our dependencies * divergence from other distros (I believe that arch is the only distro that has decided to ship python3 as "python". everyone else ships python3 as python3) * updating our documentation * divergence from other upstream/googlable documentation
I could see us renaming the python package to python2, keeping a Virtual Provide in the python2 package for python (and similar for all of the subpackages and python-doc package), and leaving python3 as it is. This might be a stepping stone to when the internet's memory hasstarted associating "python" with python3 instead of python2.
- Switching the %{?with_python3} conditionals in specfiles to %{?with_python2} (we will probably create a script to automate this, at least partially)
-1: This one doesn't make any sense to do. The third-party python library ecosystem is highly weighted for python2. There are only a handful of libraries that support python3 and not python2. There are a boatload of libraries that support python2 and not python3. We're starting from a base of existing python2 packages that may add support for python3. The conditionals are there to enable packaging of that situation.
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
Q: How do I port to Python 3? A: There are tons of tutorials and howtos about porting and the differencies in general. E.g. [2] (general), [3] (c-extensions)
The best two tutorials for python3 porting are likely: https://wiki.ubuntu.com/Python/3 (Will be moving to the python.org wiki in the near future)
Q: What about Python 2? A: We will maintain that at least as long as upstream supports it. After that, I'd prefer dropping it, but since I know there will be people wanting to keep it around, I'll gladly give the maintenance to someone else.
<nod> 2015 is right around the corner... I think someone else will get stuck maintaining the package :-/
I'll be glad to answer all your questions and discuss the above points. Nothing is set into stone and I'd love to hear your ideas and comments.
I sent out a message earlier that we should have a python sig/python guidelines discussion at flock. I think that nick and I are the only two that can definitely attend that in person.
Can anyone else make this timeslot on IRC?
http://flock2013.sched.org/event/281138262885f34d97408cfe65cdf21b?iframe=yes...
Planning for python3 and any needed updates to the Guidelines surrounding this are one of the things I wanted us to discuss.
[..]
One thing it might be nice to see in the below list is what things we have some upstream control over already. I believe the gdb work is being driven by dmalcolm. anaconda and yum/dnf are things we are upstream for. etc. Knowing about this responsibility will help us to understand where we control our own destiny and where we're dependent on other upstreams.
In some cases where upstream isn't going to port (for instance, dead upstream), we may need to either port to a different upstream (potentially large one-time cost) or fork upstream (ongoing maintainance burden).
One specific note:
- python-pycurl - TODO - https://github.com/p/pycurl/pull/28 (is this the official upstream?)
You probably need to find someone to take over upstream maintainance of python-pycurl. Over the past two or three years, various people have stepped up to take over upstream and never gotten more than a release out the door.
-Toshio
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" mike.catanzaro@gmail.com wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
On 07/19/2013 05:44 AM, Toshio Kuratomi wrote:
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" <mike.catanzaro@gmail.com mailto:mike.catanzaro@gmail.com> wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
Why do you want to use old version as default? That's very conservative approach for Fedora ;-) Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
Marcela
On Fri, Jul 19, 2013 at 08:20:02AM +0200, Marcela Mašláňová wrote:
On 07/19/2013 05:44 AM, Toshio Kuratomi wrote:
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" <mike.catanzaro@gmail.com mailto:mike.catanzaro@gmail.com> wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
Why do you want to use old version as default? That's very conservative approach for Fedora ;-)
This is not just about porting packages that Fedora is shipping. There are a quadrillion and one random scripts out in the wild with #!/usr/bin/python in them, that only work with python2. If Fedora makes /usr/bin/python be a python3 interpretor, we'll break pretty much all of them. I don't think we want to be doing that for a very long time yet, if ever.
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python, but definitely not change the semantics of the latter.
Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
Given the amount of python2 code out there, I wouldn't be at all surprised if someone steps up to maintain python2 beyond the date at which its current maintainer ceases work. Of course such maintainence work would likely be important bug fixes & security updates only, not feature work.
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
Daniel
From: berrange@redhat.com
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain
/usr/bin/python,
but definitely not change the semantics of the latter.
I think this is by far the best thing to do. After all "import this" says:
Explicit is better than implicit.
-- John Florian
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
On Fri, Jul 19, 2013 at 08:20:02AM +0200, Marcela Mašláňová wrote:
On 07/19/2013 05:44 AM, Toshio Kuratomi wrote:
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" <mike.catanzaro@gmail.com mailto:mike.catanzaro@gmail.com> wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
Why do you want to use old version as default? That's very conservative approach for Fedora ;-)
This is not just about porting packages that Fedora is shipping. There are a quadrillion and one random scripts out in the wild with #!/usr/bin/python in them, that only work with python2. If Fedora makes /usr/bin/python be a python3 interpretor, we'll break pretty much all of them. I don't think we want to be doing that for a very long time yet, if ever.
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python, but definitely not change the semantics of the latter.
Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
Given the amount of python2 code out there, I wouldn't be at all surprised if someone steps up to maintain python2 beyond the date at which its current maintainer ceases work. Of course such maintainence work would likely be important bug fixes & security updates only, not feature work.
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to change shebangs as short-term workaround) than drop users cold at python2 removal time (which *will* eventually come). Some people really need some breakage to notice they're missing a migration.
From: nicolas.mailhot@laposte.net
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to change shebangs as short-term workaround) than drop users cold at python2
removal
time (which *will* eventually come). Some people really need some
breakage
to notice they're missing a migration.
Excellent point! If I were caught unaware, I'd much rather have to fix up a bunch shebang lines and learn that I need to get going on my migration pronto than the alternative of finding one day that python2 is just gone where one is left with either a hurried port or downloading python2 from external sources.
-- John Florian
On Fri, 2013-07-19 at 11:24 -0400, John.Florian@dart.biz wrote:
From: nicolas.mailhot@laposte.net
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to change shebangs as short-term workaround) than drop users cold at python2
removal
time (which *will* eventually come). Some people really need some
breakage
to notice they're missing a migration.
Excellent point! If I were caught unaware, I'd much rather have to fix up a bunch shebang lines and learn that I need to get going on my migration pronto than the alternative of finding one day that python2 is just gone where one is left with either a hurried port or downloading python2 from external sources.
The heads-up can be done before the python2 dropping by removing /usr/bin/python from python2 package.
From: tmraz@redhat.com
On Fri, 2013-07-19 at 11:24 -0400, John.Florian@dart.biz wrote:
From: nicolas.mailhot@laposte.net
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by
default,
well before 2015, I don't think we should assume that Python2
support
is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way
is
really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to
change
shebangs as short-term workaround) than drop users cold at python2
removal
time (which *will* eventually come). Some people really need some
breakage
to notice they're missing a migration.
Excellent point! If I were caught unaware, I'd much rather have to
fix up
a bunch shebang lines and learn that I need to get going on my
migration
pronto than the alternative of finding one day that python2 is just
gone
where one is left with either a hurried port or downloading python2
from
external sources.
The heads-up can be done before the python2 dropping by removing /usr/bin/python from python2 package.
Ah, true indeed. However, I still like the idea of being explicit. If you want py2, say so in the shebang line. Likewise with py3. How can we get the very blunt "head's up" and be explicit too? I have no idea when py2 will truly go away, but I'm convinced it will eventually and I'd like to create my works in the best possible fashion. Thus far I've relied on the implicit py->py2 and explicit py3 invocations as that's the way I've seen Fedora set examples, but I think this should really be more detailed in the packaging guidelines.
-- John Florian
On Fri, Jul 19, 2013 at 11:58:35AM -0400, John.Florian@dart.biz wrote:
From: tmraz@redhat.com
On Fri, 2013-07-19 at 11:24 -0400, John.Florian@dart.biz wrote:
Excellent point! If I were caught unaware, I'd much rather have to fix up a bunch shebang lines and learn that I need to get going on my migration pronto than the alternative of finding one day that python2 is just gone where one is left with either a hurried port or downloading python2 from external sources.
The heads-up can be done before the python2 dropping by removing /usr/bin/python from python2 package.
Ah, true indeed. However, I still like the idea of being explicit. If you want py2, say so in the shebang line. Likewise with py3. How can we get the very blunt "head's up" and be explicit too? I have no idea when py2 will truly go away, but I'm convinced it will eventually and I'd like to create my works in the best possible fashion. Thus far I've relied on the implicit py->py2 and explicit py3 invocations as that's the way I've seen Fedora set examples, but I think this should really be more detailed in the packaging guidelines.
I'm not sure if this is a good idea (or in what time frame it may be a good idea. It's definitely too early now and likely too early for F22) but -- if you want something explicit, we could replace /usr/bin/python with something that prints out an error message to switch shebang lines to use /usr/bin/python2 and exits.
(Perhaps a warning message to switch shebang lines could be done now, though).
-Toshio
On Fri, Jul 19, 2013 at 10:11 PM, Toshio Kuratomi a.badger@gmail.com wrote:
I'm not sure if this is a good idea (or in what time frame it may be a good idea. It's definitely too early now and likely too early for F22) but -- if you want something explicit, we could replace /usr/bin/python with something that prints out an error message to switch shebang lines to use /usr/bin/python2 and exits.
(Perhaps a warning message to switch shebang lines could be done now, though).
I personally think it is too early for even that. If one asks me to make necessary changes only to the projects I am upstream, I will have to clone myself many times to do that in any near future.
Kushal
On Fri, Jul 19, 2013 at 05:16:03PM +0200, Nicolas Mailhot wrote:
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to change shebangs as short-term workaround) than drop users cold at python2 removal time (which *will* eventually come). Some people really need some breakage to notice they're missing a migration.
I don't think we're anywhere near the point in time where such an approach is worth the pain it will inflict on people. As above, I'm pretty sceptical that maintainence of Python2 is actually going to stop in 2015, as I think there are too many people and/or organizations who will find they have reasons for wanting to stick on python2 for a long time. Maybe I'll be proved wrong on this in time, but I doubt it, as there's far too many prior examples of people/companies sticking with ancient software versions because the cost of maintaining them is less than the cost of porting them.
Daniel
Le Ven 19 juillet 2013 17:30, Daniel P. Berrange a écrit :
On Fri, Jul 19, 2013 at 05:16:03PM +0200, Nicolas Mailhot wrote:
Le Ven 19 juillet 2013 17:04, Tomas Mraz a écrit :
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
+1 Breaking completely needlessly backwards compatibility this way is really not a good idea.
OTOH, it's better to give a heads-up this way (with the option to change shebangs as short-term workaround) than drop users cold at python2 removal time (which *will* eventually come). Some people really need some breakage to notice they're missing a migration.
I don't think we're anywhere near the point in time where such an approach is worth the pain it will inflict on people. As above, I'm pretty sceptical that maintainence of Python2 is actually going to stop in 2015, as I think there are too many people and/or organizations who will find they have reasons for wanting to stick on python2 for a long time. Maybe I'll be proved wrong on this in time, but I doubt it, as there's far too many prior examples of people/companies sticking with ancient software versions because the cost of maintaining them is less than the cost of porting them.
Well, in practical terms if you want to do it graciously you need
time of python renaming in fedora = fedora python2 removal time - one rhel cycle
because some people won't migrate before their stuff is broken in an enterprise distro, so for the heads up to be effective it needs to be advertised a full rhel cycle
Daniel P. Berrange (berrange@redhat.com) said:
I don't think we're anywhere near the point in time where such an approach is worth the pain it will inflict on people. As above, I'm pretty sceptical that maintainence of Python2 is actually going to stop in 2015, as I think there are too many people and/or organizations who will find they have reasons for wanting to stick on python2 for a long time. Maybe I'll be proved wrong on this in time, but I doubt it, as there's far too many prior examples of people/companies sticking with ancient software versions because the cost of maintaining them is less than the cost of porting them.
Especially since I do believe there are large software infrastructures even more tied to python2 than Fedora is with yum/anaconda/etc. (Hellooooooo, OpenStack!)
Bill
On Fri, Jul 19, 2013 at 03:21:12PM -0400, Bill Nottingham wrote:
Daniel P. Berrange (berrange@redhat.com) said:
I don't think we're anywhere near the point in time where such an approach is worth the pain it will inflict on people. As above, I'm pretty sceptical that maintainence of Python2 is actually going to stop in 2015, as I think there are too many people and/or organizations who will find they have reasons for wanting to stick on python2 for a long time. Maybe I'll be proved wrong on this in time, but I doubt it, as there's far too many prior examples of people/companies sticking with ancient software versions because the cost of maintaining them is less than the cost of porting them.
Especially since I do believe there are large software infrastructures even more tied to python2 than Fedora is with yum/anaconda/etc. (Hellooooooo, OpenStack!)
There is an effort in openstack to try to make the code python 3 friendly, but they wish to retain python 2.x compatibility for a while yet since that's what most of the OS platforms they're targetting have most widely available. So they're seeing if it is possible to get code that works on say python 2.7 and 3.x concurrently. AFAIK, there's not official timeline yet for declaring that openstack formally supports python 3.
Daniel
On Fri, Jul 19, 2013 at 10:17:51AM +0100, Daniel P. Berrange wrote:
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python, but definitely not change the semantics of the latter.
+1*10^6
Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
Given the amount of python2 code out there, I wouldn't be at all surprised if someone steps up to maintain python2 beyond the date at which its current maintainer ceases work. Of course such maintainence work would likely be important bug fixes & security updates only, not feature work.
So while I encourage a Fedora effort to get onto Python3 by default, well before 2015, I don't think we should assume that Python2 support is definitely going to stop in 2015.
<nod> At this time, I'm pretty sure that someone will step up to maintain python2 packages in all major distros after the 2015 date that upstream stops supporting a CPython2. There's several different ways that could shake out --
* Distros independently maintain python2 packages * Distros get together and maintain python2 as a unified project. Possibly even using the CPython upstream revision control system and issue tracker. * Distros switch to alternate implementations of python2 (such as pypy) for their python2 compatible code.
It's too early for me to tell which of these is likely to happen. It depends mostly on how important python2 still is when upstream CPython finally decides to stop supporting python2.
-Toshio
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python
Note the GNOME discussion here:
https://mail.gnome.org/archives/desktop-devel-list/2012-November/msg00005.ht...
One ugly part is that Debian Wheezy does not have "python2". That combined with the Arch Linux people who made /usr/bin/python = python3 is a terrible mess for people who just want random portable scripts.
For GNOME we decided to just try to ensure python2 always exists: https://mail.gnome.org/archives/desktop-devel-list/2012-November/msg00017.ht...
On Fri, Jul 19, 2013 at 12:53:20PM -0400, Colin Walters wrote:
On Fri, 2013-07-19 at 10:17 +0100, Daniel P. Berrange wrote:
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python
Note the GNOME discussion here:
https://mail.gnome.org/archives/desktop-devel-list/2012-November/msg00005.ht...
One ugly part is that Debian Wheezy does not have "python2". That combined with the Arch Linux people who made /usr/bin/python = python3 is a terrible mess for people who just want random portable scripts.
For GNOME we decided to just try to ensure python2 always exists: https://mail.gnome.org/archives/desktop-devel-list/2012-November/msg00017.ht...
<nod> These two concerns were reasons that PEP394 was written.
http://www.python.org/dev/peps/pep-0394/#rationale
and CPython2's install was changed to create the python2 symlink: http://www.python.org/dev/peps/pep-0394/#application-to-the-cpython-referenc...
which should make it into Debian at some point.
-Toshio
On Fri, Jul 19, 2013 at 11:17 AM, Daniel P. Berrange berrange@redhat.com wrote:
Far better to encourage people to explicitly use /usr/bin/python2 and /usr/bin/python3 explicitly and discourage any use of plain /usr/bin/python, but definitely not change the semantics of the latter.
Right, treating the existence of /usr/bin/python as a mistake/historical accident is reasonable.
(As a philosophical aside, isn't it rather sad that we don't have any established conventions for handling such transition/compatibility issues? We are treating project and every every case as unique, when they really aren't; just about any API provider faces the same versioning/compatibility issues sooner or later. If only there were some kind of consensus for "how things are done" that any project could adopt - and were expected to adopt - already in the early stages of the project when nobody thinks a migration will ever be needed.) Mirek
On Fri, Jul 19, 2013 at 08:20:02AM +0200, Marcela Mašláňová wrote:
On 07/19/2013 05:44 AM, Toshio Kuratomi wrote:
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" <mike.catanzaro@gmail.com mailto:mike.catanzaro@gmail.com> wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
Why do you want to use old version as default? That's very conservative approach for Fedora ;-)
This is the wrong mental model of the relationship between python2 and python3. As a comparison this argument is akin to saying, "C is the old version of C++. We should be porting all our code to use the C++ compiler in Fedora" C is much more compatible to C++ than python2 is to python3 so this would be a less drastic change.
python2 and python3 are separate languages. There is a lot of similarity between the two and with recent enough versions of python2 (2.7) and python3 (python3.4) and some external libraries (python-six) and by sticking to some specific coding styles ( http://python3porting.com/noconv.html ) and by sometimes resorting to having separate files for some python2-specific routines vs python3-specific routines you can write code that is valid and runs under either language. That does not mean that they are the same language.
Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
I am wholeheartedly in favor of "working on it". But things like switching the default /usr/bin/python are not "working on it". Those are backwards incompatible, end user visible, changes of expectations that we do not need to implement now in order to be a leader. When people invoke python, they expect python2. When they want python3, they expect to type python3 to get that. At the moment, causing /usr/bin/python to invoke python3 is just going to cause confusion and end user confusion (arch is the only linux distro that sets /usr/bin/python to point to python3).
As an example here, let's suppose that we were to stop shipping firefox by default and started shipping chromium instead. Would we make /usr/bin/firefox invoke /usr/bin/chromium? Would we have a firefox icon that started chromium when clicked? When applied here, it does seem to make more sense for end user's to notice that there isn't a firefox icon and either consciously open up a different web browser or else install firefox to get their preferred browsing experience, right? /usr/bin/python is the same way. People have tons of scripts on their systems with #!/usr/bin/python shebang lines. If we change /usr/bin/python to point to /usr/bin/python3 then we'll force them to deal with either porting, going through all their scripts and replacing them with /usr/bin/python2, or switching to another distribution which doesn't change the meanings of well known interpreter paths for no gain.
All this is not to say that there might not be a time in the future when it makes sense to have python3 assume the /usr/bin/python name. However, that time is *after* the collective user consciousness has stopped using /usr/bin/python, not before. When we can go to pycon and talks are assumed to be targeting python3 rather than python2. After people wanting to do a simple calculation on the CLI stop doing /usr/bin/python -c 'print 5 + 6'. This is probably after alternate python implementations like pypy and jython either die off or finally make the switch to implementing python3. My guess is this will be after there's a RHEL release that includes /usr/bin/python3 in the default install since RHEL and CentOS are major parts of our ecosystem and will have to be able to use the python3 syntax on both their RHEL servers and Fedora machines. To some extent, this also depends on what other large distros set as their expectations: Debian and Ubuntu currently point /usr/bin/python to python2. AFAIK, only Arch is pointing to python3. My intuition is this will be at least two years after we stop shipping /usr/bin/python (pointing to python2) by default but this could be wrong -- it's about the expectations of the Fedora community and the Linux and python development communities as a whole.
-Toshio
----- Original Message -----
On Fri, Jul 19, 2013 at 08:20:02AM +0200, Marcela Mašláňová wrote:
On 07/19/2013 05:44 AM, Toshio Kuratomi wrote:
On Jul 18, 2013 5:42 PM, "Michael Catanzaro" <mike.catanzaro@gmail.com mailto:mike.catanzaro@gmail.com> wrote:
On Thu, 2013-07-18 at 09:53 -0700, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
But when python2 is no longer installed by default, surely you want to get a python prompt when you type 'python'?
Yes, and for a long time, I'm going to want to get a python2 prompt. Which means installing the package for python2, not having python3 start on that case.
Why do you want to use old version as default? That's very conservative approach for Fedora ;-)
This is the wrong mental model of the relationship between python2 and python3. As a comparison this argument is akin to saying, "C is the old version of C++. We should be porting all our code to use the C++ compiler in Fedora" C is much more compatible to C++ than python2 is to python3 so this would be a less drastic change.
python2 and python3 are separate languages. There is a lot of similarity between the two and with recent enough versions of python2 (2.7) and python3 (python3.4) and some external libraries (python-six) and by sticking to some specific coding styles ( http://python3porting.com/noconv.html ) and by sometimes resorting to having separate files for some python2-specific routines vs python3-specific routines you can write code that is valid and runs under either language. That does not mean that they are the same language.
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Upstream plans to support it until 2015 (maybe little longer). Fedora needs to be prepared for such step, so it's the right time to start working on it.
I am wholeheartedly in favor of "working on it". But things like switching the default /usr/bin/python are not "working on it". Those are backwards incompatible, end user visible, changes of expectations that we do not need to implement now in order to be a leader. When people invoke python, they expect python2. When they want python3, they expect to type python3 to get that. At the moment, causing /usr/bin/python to invoke python3 is just going to cause confusion and end user confusion (arch is the only linux distro that sets /usr/bin/python to point to python3).
As an example here, let's suppose that we were to stop shipping firefox by default and started shipping chromium instead. Would we make /usr/bin/firefox invoke /usr/bin/chromium? Would we have a firefox icon that started chromium when clicked? When applied here, it does seem to make more sense for end user's to notice that there isn't a firefox icon and either consciously open up a different web browser or else install firefox to get their preferred browsing experience, right? /usr/bin/python is the same way. People have tons of scripts on their systems with #!/usr/bin/python shebang lines. If we change /usr/bin/python to point to /usr/bin/python3 then we'll force them to deal with either porting, going through all their scripts and replacing them with /usr/bin/python2, or switching to another distribution which doesn't change the meanings of well known interpreter paths for no gain.
Again, this is all based on your "mental model" and on your assumption that it is the correct one. I just don't agree.
All this is not to say that there might not be a time in the future when it makes sense to have python3 assume the /usr/bin/python name. However, that time is *after* the collective user consciousness has stopped using /usr/bin/python, not before. When we can go to pycon and talks are assumed to be targeting python3 rather than python2. After people wanting to do a simple calculation on the CLI stop doing /usr/bin/python -c 'print 5 + 6'. This is probably after alternate python implementations like pypy and jython either die off or finally make the switch to implementing python3. My guess is this will be after there's a RHEL release that includes /usr/bin/python3 in the default install since RHEL and CentOS are major parts of our ecosystem and will have to be able to use the python3 syntax on both their RHEL servers and Fedora machines. To some extent, this also depends on what other large distros set as their expectations: Debian and Ubuntu currently point /usr/bin/python to python2. AFAIK, only Arch is pointing to python3. My intuition is this will be at least two years after we stop shipping /usr/bin/python (pointing to python2) by default but this could be wrong -- it's about the expectations of the Fedora community and the Linux and python development communities as a whole.
-Toshio
On Tue, 2013-07-23 at 02:40 -0400, Bohuslav Kabrda wrote:
python2 and python3 are separate languages. There is a lot of
similarity
between the two and with recent enough versions of python2 (2.7) and
python3
(python3.4) and some external libraries (python-six) and by sticking
to some
specific coding styles ( http://python3porting.com/noconv.html ) and
by
sometimes resorting to having separate files for some
python2-specific
routines vs python3-specific routines you can write code that is
valid and
runs under either language. That does not mean that they are the
same
language.
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Python 2.7 might not be backward compatible with 2.6 or 2.5 but 2.5 and 2.6 is forward compatible with 2.7, which is not true for python 2.* vs 3.*.
That makes them more of different languages than 2.6 vs 2.7.
Pierre
----- Original Message -----
On Tue, 2013-07-23 at 02:40 -0400, Bohuslav Kabrda wrote:
python2 and python3 are separate languages. There is a lot of
similarity
between the two and with recent enough versions of python2 (2.7) and
python3
(python3.4) and some external libraries (python-six) and by sticking
to some
specific coding styles ( http://python3porting.com/noconv.html ) and
by
sometimes resorting to having separate files for some
python2-specific
routines vs python3-specific routines you can write code that is
valid and
runs under either language. That does not mean that they are the
same
language.
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Python 2.7 might not be backward compatible with 2.6 or 2.5 but 2.5 and 2.6 is forward compatible with 2.7, which is not true for python 2.* vs 3.*.
Good point, but not entirely true, IMHO. If you look at Python 2.7 release notes [1], you'll find out that e.g. float-int conversions round differently. So if you targeted your code for 2.6 working under the assumption that the rounding will have a certain result, your script will likely break (e.g. what you wrote for 2.6 may fail in 2.7).
That makes them more of different languages than 2.6 vs 2.7.
Pierre
[1] http://docs.python.org/dev/whatsnew/2.7.html
On Tue, Jul 23, 2013 at 02:40:56AM -0400, Bohuslav Kabrda wrote:
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Even more directly, I am saying that your mental model is wrong. Now then, I assumed that you would not find that palatable therefore I backed my mental model up with links to the python-dev mailing list. (The three threads from the PEP and the earlier one that I linked to explicitly in my email). What are the links to upstream that you are basing your mental model on?
-Toshio
Dne 24.7.2013 06:32, Toshio Kuratomi napsal(a):
On Tue, Jul 23, 2013 at 02:40:56AM -0400, Bohuslav Kabrda wrote:
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Even more directly, I am saying that your mental model is wrong. Now then, I assumed that you would not find that palatable therefore I backed my mental model up with links to the python-dev mailing list. (The three threads from the PEP and the earlier one that I linked to explicitly in my email). What are the links to upstream that you are basing your mental model on?
-Toshio
So what is the story with Arch and Python 3? Is that considered total failure after almost 3 years they did that? Just asking ...
Vít
On Wed, Jul 24, 2013 at 6:32 AM, Toshio Kuratomi a.badger@gmail.com wrote:
On Tue, Jul 23, 2013 at 02:40:56AM -0400, Bohuslav Kabrda wrote:
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Even more directly, I am saying that your mental model is wrong. Now then, I assumed that you would not find that palatable therefore I backed my mental model up with links to the python-dev mailing list. (The three threads from the PEP and the earlier one that I linked to explicitly in my email). What are the links to upstream that you are basing your mental model on?
Ignoring the naming/versioning/PR aspects, Python 3 is developed by the approximately the same group of developers, who have been quite public about planning to stop maintenance of the Python 2 branch.
That means Fedora will need to migrate eventually[1], so thinking about the two versions as just continuing the same language makes more _practical sense for Fedora_. Treating them as two separate languages that can just peacefully coexist obfuscates the need to migrate, and the need for every Python package maintainer to plan for migration. Mirek
[1] Ignoring the possibilities that upstream will change their mind about Python 2, or that somebody will pick it up and continue maintaining it. For all I know that might well happen - but from what I've seen I don't feel comfortable betting the ability to continue to run anaconda, yum, koji and other critical infrastructure on such an outcome.
----- Original Message -----
On Tue, Jul 23, 2013 at 02:40:56AM -0400, Bohuslav Kabrda wrote:
The problem is that you're basically saying "my mental model is the right one", which is not necessarily true for everyone (and not necessarily true generally). Taking your arguments a bit further, Python 2.6 and 2.7 are different languages too, since there are some backward incompatible additions to Python 2.7.
Even more directly, I am saying that your mental model is wrong. Now then, I assumed that you would not find that palatable therefore I backed my mental model up with links to the python-dev mailing list. (The three threads from the PEP and the earlier one that I linked to explicitly in my email). What are the links to upstream that you are basing your mental model on?
Since Python upstream really cares about these things, I started a discussion about this on their mailing list [1]. So far, it seems that people prefer my mental model, but this is judging from just 4 answers, 2 of which mentioned this. So let's wait and see. BTW, if Python 2 and Python 3 were different languages, then IMO it wouldn't make sense to point /usr/bin/python to Python 3, while upstream plans to actually give the recommendation to do so sometime in the future.
-Toshio
----- Original Message -----
Since Python upstream really cares about these things, I started a discussion about this on their mailing list [1]. So far, it seems that people prefer my mental model, but this is judging from just 4 answers, 2 of which mentioned this. So let's wait and see. BTW, if Python 2 and Python 3 were different languages, then IMO it wouldn't make sense to point /usr/bin/python to Python 3, while upstream plans to actually give the recommendation to do so sometime in the future.
So, judging from what upstream has recommended (and how they will modify pep 394), we should: - Mandate usage of /usr/bin/python2 and prohibit usage of /usr/bin/python (I think this can be done automatically in %__os_install_post, or somewhere similar) - we should probably do that ASAP - Rename python to python2 and add "Provides: python" for the time being (the similar should probably be done for all python packages to keep things consistent) - we can do this for F21 - Somewhere in the future switch "Provides: python" to python3 stuff and possibly /usr/bin/python to point to python3, according to upstream recommendation (2015?) to keep consistent across multiple platforms as much as possible.
Just BTW: $ repoquery --whatrequires /usr/bin/python | wc -l 1262
If this seems ok, then I'll go on and propose the first step to FPC. Thanks, Slavek.
On Jul 25, 2013 12:13 AM, "Bohuslav Kabrda" bkabrda@redhat.com wrote:
----- Original Message -----
Since Python upstream really cares about these things, I started a
discussion
about this on their mailing list [1]. So far, it seems that people
prefer my
mental model, but this is judging from just 4 answers, 2 of which
mentioned
this. So let's wait and see. BTW, if Python 2 and Python 3 were different languages, then IMO it
wouldn't
make sense to point /usr/bin/python to Python 3, while upstream plans to actually give the recommendation to do so sometime in the future.
So, judging from what upstream has recommended (and how they will modify
pep 394), we should:
- Mandate usage of /usr/bin/python2 and prohibit usage of /usr/bin/python
(I think this can be done automatically in %__os_install_post, or somewhere similar) - we should probably do that ASAP
<nod>. Which means patching in a bunch of packages. For os_install_post modification you're talking about a check that fails the build if /usr/bin/python is used? I note that upstream envisions some cases where /usr/bin/python is correct but I don't recall any package where we'd have invoked that clause (the case is when the code will run on either python2 or on python3)
/me goes to the upstream thread to ask Nick what distutils/setuptools/etc do when they rewrite shebang lines.
- Rename python to python2 and add "Provides: python" for the time being
(the similar should probably be done for all python packages to keep things consistent) - we can do this for F21
Renaming other packages gets problematic. See the previous discussion on python-devel@lists.fp.o that tomspur championed. (This was one of the items I was hoping could be discussed at flock) You wouldn't expect to report bugs for the python3-foo library against the python2-foo package for instance. One possibility that we talked about was to stop shipping python3 packages as subpackages of the python2 modules.
- Somewhere in the future switch "Provides: python" to python3 stuff and
possibly /usr/bin/python to point to python3, according to upstream recommendation (2015?) to keep consistent across multiple platforms as much as possible.
Rather than 2015 I'd tie it to when upstream switches the recommendation in pep 394.
-Toshio
----- Original Message -----
On Jul 25, 2013 12:13 AM, "Bohuslav Kabrda" < bkabrda@redhat.com > wrote:
----- Original Message -----
Since Python upstream really cares about these things, I started a discussion about this on their mailing list [1]. So far, it seems that people prefer my mental model, but this is judging from just 4 answers, 2 of which mentioned this. So let's wait and see. BTW, if Python 2 and Python 3 were different languages, then IMO it wouldn't make sense to point /usr/bin/python to Python 3, while upstream plans to actually give the recommendation to do so sometime in the future.
So, judging from what upstream has recommended (and how they will modify pep 394), we should:
- Mandate usage of /usr/bin/python2 and prohibit usage of /usr/bin/python
(I think this can be done automatically in %__os_install_post, or somewhere similar) - we should probably do that ASAP
<nod>. Which means patching in a bunch of packages. For os_install_post modification you're talking about a check that fails the build if /usr/bin/python is used? I note that upstream envisions some cases where /usr/bin/python is correct but I don't recall any package where we'd have invoked that clause (the case is when the code will run on either python2 or on python3)
I meant something that would actually rewrite the lines, even though that may be very expensive to run for every build... Failing the build is another option, yes. I'll try to have a look at how these would be implemented.
/me goes to the upstream thread to ask Nick what distutils/setuptools/etc do when they rewrite shebang lines.
Hmm, patching setuptools to do this for us sounds interesting, although not everything that has /usr/bin/python uses setuptools for build, unfortunately.
- Rename python to python2 and add "Provides: python" for the time being
(the similar should probably be done for all python packages to keep things consistent) - we can do this for F21
Renaming other packages gets problematic. See the previous discussion on python-devel@lists.fp.o that tomspur championed. (This was one of the items I was hoping could be discussed at flock) You wouldn't expect to report bugs for the python3-foo library against the python2-foo package for instance. One possibility that we talked about was to stop shipping python3 packages as subpackages of the python2 modules.
Yep, I remember. I'm thinking about this (hope I'm not repeating anyone's idea from before): What if we use one python-foo.srpm to produce two python2-foo.rpm and python3-foo.rpm, but not produce python-foo.rpm? Assuming that python2-foo has Provides: python-foo, I think that pretty much achieves what we want - unless I'm missing something. The only glitch I can see is that we would need to tell users that they need to report all bugs for python-foo component. But they can do the mapping from python3-foo to python-foo now, so it shouldn't hurt so much, IMO.
- Somewhere in the future switch "Provides: python" to python3 stuff and
possibly /usr/bin/python to point to python3, according to upstream recommendation (2015?) to keep consistent across multiple platforms as much as possible.
Rather than 2015 I'd tie it to when upstream switches the recommendation in pep 394.
That is what I meant :) 2015 was just a wild guess of when that might happen.
-Toshio
On Thu, Jul 25, 2013 at 06:49:41AM -0400, Bohuslav Kabrda wrote:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
<nod>. Which means patching in a bunch of packages. For os_install_post modification you're talking about a check that fails the build if /usr/bin/ python is used? I note that upstream envisions some cases where /usr/bin/ python is correct but I don't recall any package where we'd have invoked that clause (the case is when the code will run on either python2 or on python3)
I meant something that would actually rewrite the lines, even though that may be very expensive to run for every build... Failing the build is another option, yes. I'll try to have a look at how these would be implemented.
I would be hesitant to do rewriting of the shebang lines. I suppose that we do re-byte-compile the .pyc and .pyo now but it seems different to be changing something that is also source code and human readable automatically.
Some other places to make this same change: we also have to have people change anywhere that they use "python" in scripts (for instance, some python applications ship a shell script that invokes python /path/to/python_file.py ) and our rpm macros should probably also change to use /usr/bin/python2 instead of /usr/bin/python
/me goes to the upstream thread to ask Nick what distutils/setuptools/etc do when they rewrite shebang lines.
Hmm, patching setuptools to do this for us sounds interesting, although not everything that has /usr/bin/python uses setuptools for build, unfortunately.
<nod> And not so unfortunately... setuptools has some pretty iffy code :-/
> - Rename python to python2 and add "Provides: python" for the time being (the similar should probably be done for all python packages to keep things consistent) - we can do this for F21 Renaming other packages gets problematic. See the previous discussion on python-devel@lists.fp.o that tomspur championed. (This was one of the items I was hoping could be discussed at flock) You wouldn't expect to report bugs for the python3-foo library against the python2-foo package for instance. One possibility that we talked about was to stop shipping python3 packages as subpackages of the python2 modules.
Yep, I remember. I'm thinking about this (hope I'm not repeating anyone's idea from before): What if we use one python-foo.srpm to produce two python2-foo.rpm and python3-foo.rpm, but not produce python-foo.rpm? Assuming that python2-foo has Provides: python-foo, I think that pretty much achieves what we want - unless I'm missing something. The only glitch I can see is that we would need to tell users that they need to report all bugs for python-foo component. But they can do the mapping from python3-foo to python-foo now, so it shouldn't hurt so much, IMO.
<nod> a main package that doesn't get built triggers an intense dislike for me but that may be the best thing to do. I do still think that having two separate SRPMs is advantageous to packages (I've worked on several packages: docutils, nose) where the package doesn't build or run correctly on one version of the interpreter and so it holds back package updates. dgilmore pointed out to me yesterday that because python3 is not currently building for arm, a lot of python packages won't build for arm despite their python2 versions (which is what actual software in the distribution depends on) building fine.
> - Somewhere in the future switch "Provides: python" to python3 stuff and possibly /usr/bin/python to point to python3, according to upstream recommendation (2015?) to keep consistent across multiple platforms as much as possible. > Rather than 2015 I'd tie it to when upstream switches the recommendation in pep 394.
That is what I meant :) 2015 was just a wild guess of when that might happen.
<nod> True enough :-)
-Toshio
On Thu, Jul 25, 2013 at 06:49:41AM -0400, Bohuslav Kabrda wrote:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
/me goes to the upstream thread to ask Nick what distutils/setuptools/etc do when they rewrite shebang lines.
Hmm, patching setuptools to do this for us sounds interesting, although not everything that has /usr/bin/python uses setuptools for build, unfortunately.
Haven't gotten much response yet but: http://mail.python.org/pipermail/distutils-sig/2013-July/022002.html
I think the use of sys.executable means if we use /usr/bin/python2 setup.py instead of /usr/bin/python setup.py then we'll start to get shebang lines with /usr/bin/python2 instead of embedded /usr/bin/python.
That will get us through many of the applications that are installed by packages.
-Toshio
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default.
I'm not sure we want to make python3 default depending on what your definition of default is.
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
So, my definition of default is "all system tools use Python 3, it is the only Python that gets to minimal buildroot/minimal Fedora installation" - that means: - livecd can still ship Python 2 - /usr/bin/python points to Python 3 - Please note, that the pep you're referring to also states that "python should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions", so this wouldn't really be going against the pep.
The python package itself should probably also remain python2 due to dependencies and expectations from other distros and documentation -- I think I'd be -1 to changing this
The Fedora live images contain only python3, not python2 -- I'd be heavily in favour of this. +1
This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Getting there at the same time as we get to DNF sounds like a good timeline. (But see my note on anaconda below). +1
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
I'm not sure about this one as it will cause a lot of package churn. It might be a necessary pain pointi or it might be a pain point we want to defer until later in our porting efforts. Have to think about it more.
If you look at the minimal mock buildroot for rawhide now, the only thing that is drawing in Python is gdb because of it's Python bindings (if I'm not mistaken). So compiling GDB against Python 3, which should work with newest gdb, will accomplish this AFAICS.
- Porting Anaconda to Python 3.
+1 -- unfortunately, this probably depends on DNF.... So we may need to push DNF in F22 and anaconda compatible with python3 in F23.
DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.
- Making all livecd packages depend on Python 3 by default (and eventually
getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
+1 -- this is what I see as the eventual goal (or perhaps, livecd python2 free followed by DVD python2 free followed by distro python2 free).
3.5) Switch tools that could target either python2 or python3 to target python3. Currently the packaging guidelines say to target python2 to control dep proliferation and because that's the most supported by the larger python ecosystem. We should switch the recommendation when our minimal environment must have python3 but does not need to have python2.
IMO we should switch this for F21, since livecd ships Python 3 anyway, so the switch doesn't have to happen in one point, but can be continuous.
( 4) Making as much of the remaining packages Python 3 compatible )
We could talk quite a bit on this point -- How active do we want to be with the things that aren't in one of the essential buckets from further up. We could defer thinking about this until after we get the livecd python2-free, though.
This is really the last step, that is somehow tied what you mentioned as a reaction to 3) - going through the rest of packages on DVD and then whole distro. This will take few more releases I guess, but it is not that important as sorting out livecd.
In past few days, I've been going through packages that are part of the above steps. I have reported numerous bugs asking upstream and/or Fedora maintainers for help with porting to Python 3. We have some spare cycles in our small Python packaging team, that we will try to provide to whoever needs them most, but we're limited and we'll have to rely on the upstreams to do most of the work. I'm attaching a document with list of packages that need porting with some notes/links to opened bugs. Sometime soon, I'll open a tracking bug for this, so that everyone can see where we are quickly.
(*) I call these "important" packages (in terms of being important for the Python 3 switch)
Cool. A list of packages that are on the livecd is good. One thing to remember, though, is that the current Python Guidelines specify that we are not to ship python3 versions of packages if upstream is not going to support us in that effort: https://fedoraproject.org/wiki/Packaging:Python#Subpackages
We could change that but I'm not 100% behind the idea of changing it. As stated in the Guidelines:
" [...]doing this on our own in Fedora is essentially creating a fork. That has a large burden for maintaining the code, fixing bugs, porting when a new version of upstream's code appears, managing a release schedule, and other tasks normally handled by upstream. It's much better if we can cooperate with upstream to share this work than doing it all on our own. "
Luckily, in recent years I've only encountered a few upstreams that are unwilling to look at python3 patches. Most upstreams are amenable to taking patches that establish python3 compatibility. We just need to remain clear that we have to work upstream to get these python3 versions into fedora, not do it in our packages without upstream being on board.
Yep. When I was opening the bugs, I tried to open them in the sense "please work with upstream to port to Python 3". Only when I found out that upstream supports Python 3, I asked the maintainer to add python3- subpackage.
From packaging point of view, this will probably require:
- Renaming python package to python2
- Renaming python3 package to python
-1: What are the benefits of this as the cost of this is very high in several ways:
- updating our dependencies
- divergence from other distros (I believe that arch is the only distro that has decided to ship python3 as "python". everyone else ships python3 as python3)
- updating our documentation
- divergence from other upstream/googlable documentation
I could see us renaming the python package to python2, keeping a Virtual Provide in the python2 package for python (and similar for all of the subpackages and python-doc package), and leaving python3 as it is. This might be a stepping stone to when the internet's memory hasstarted associating "python" with python3 instead of python2.
- Switching the %{?with_python3} conditionals in specfiles to
%{?with_python2} (we will probably create a script to automate this, at least partially)
-1: This one doesn't make any sense to do. The third-party python library ecosystem is highly weighted for python2. There are only a handful of libraries that support python3 and not python2. There are a boatload of libraries that support python2 and not python3. We're starting from a base of existing python2 packages that may add support for python3. The conditionals are there to enable packaging of that situation.
And this situation will be changing in the future. Right now, there are not so many Python packages in Fedora that only support Python 2 (I didn't count, but you don't see them too often these days). IMO Fedora should lead the way of making Python 3 "the Python" and Python 2 "the old compat version". This also makes sense in the traditional linux-distro "one version of package" that we should be trying to pursue.
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason: - Python 2 is old - how is that a half-truth? - Slower - yes, in the beginning, Python 3 was significantly slower because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster. - Less Pythonic - where do I start with this? Python 3 got rid of tons of unnecessary syntactic constructs as well as builtin object methods. E.g. "print" vs. "print()"; exception raising syntax; dict.iteritems() removed and dict.items() only left, more consistent unicode handling etc. So in the sense of having only one way to do things, Python 3 is more pythonic than Python 2. If you read through zen of python, you can find more arguments for this (e.g. making int and long one type - "Special cases aren't special enough to break the rules."; simplification/rewrite of parts of stdlib - "Simple is better than complex.", etc.)
Q: How do I port to Python 3? A: There are tons of tutorials and howtos about porting and the differencies in general. E.g. [2] (general), [3] (c-extensions)
The best two tutorials for python3 porting are likely: https://wiki.ubuntu.com/Python/3 (Will be moving to the python.org wiki in the near future)
Thanks for the links.
Q: What about Python 2? A: We will maintain that at least as long as upstream supports it. After that, I'd prefer dropping it, but since I know there will be people wanting to keep it around, I'll gladly give the maintenance to someone else.
<nod> 2015 is right around the corner... I think someone else will get stuck maintaining the package :-/
I'll be glad to answer all your questions and discuss the above points. Nothing is set into stone and I'd love to hear your ideas and comments.
I sent out a message earlier that we should have a python sig/python guidelines discussion at flock. I think that nick and I are the only two that can definitely attend that in person.
Can anyone else make this timeslot on IRC?
http://flock2013.sched.org/event/281138262885f34d97408cfe65cdf21b?iframe=yes...
I'll try.
Planning for python3 and any needed updates to the Guidelines surrounding this are one of the things I wanted us to discuss.
[..]
One thing it might be nice to see in the below list is what things we have some upstream control over already. I believe the gdb work is being driven by dmalcolm. anaconda and yum/dnf are things we are upstream for. etc. Knowing about this responsibility will help us to understand where we control our own destiny and where we're dependent on other upstreams.
In some cases where upstream isn't going to port (for instance, dead upstream), we may need to either port to a different upstream (potentially large one-time cost) or fork upstream (ongoing maintainance burden).
One specific note:
- python-pycurl - TODO - https://github.com/p/pycurl/pull/28 (is this the official upstream?)
You probably need to find someone to take over upstream maintainance of python-pycurl. Over the past two or three years, various people have stepped up to take over upstream and never gotten more than a release out the door.
Yeah, I'll try to speak to Fedora's maintainer of pycurl to see if he has any updates on this and we'll see.
-Toshio
Thanks for your thoughts. Slavek.
On 19. 7. 2013 at 02:41:23, Bohuslav Kabrda wrote: ... snip ...
+1 -- unfortunately, this probably depends on DNF.... So we may need to push DNF in F22 and anaconda compatible with python3 in F23.
DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.
That's right, we are already working on the API stabilization and testing dnf with Anaconda is one part of the entire effort. However there is only so much we can do ourselves. Stronger participation of the Anaconda team is required, as we need to know what parts are they missing and what parts need improving.
... snip ...
Thanks Jan
On Fri, Jul 19, 2013 at 02:41:23AM -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default.
I'm not sure we want to make python3 default depending on what your definition of default is.
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
So, my definition of default is "all system tools use Python 3, it is the only Python that gets to minimal buildroot/minimal Fedora installation" - that means:
I'm okay with this portion of the definition. One note is I would be hesitant about the timing of python3 being the only python that is installed into the minimal buildroot. This should probably happen in rawhide right after a branching.
- livecd can still ship Python 2
I would consider this to be the goal that we should shoot for, though. We are constantly fighting for space on the install images and we know that people who install Fedora would like to have the ability to slim down what is installed. Shooting for no-python2 on the livecds and after that, no python2 on the install dvds, and still later, no need for python2 in the packages in the repository seem like milestones that have actual real value for end users.
- /usr/bin/python points to Python 3
I am firmly against this. more depth was in my reply to mmaslano although I'll reply to one thing here:
- Please note, that the pep you're referring to also states that "python
should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions", so this wouldn't really be going against the pep.
This is a misinterpretion of the PEP. (This section is confusing, though: "python should refer to the same target as python2" is a recommendation to distributions. "but may refer to python3 on some bleeding edge distributions" is a statement of fact for end users to watch out for) See the recommendation section:
"For the time being, it is recommended that python should refer to python2 (however, some distributions have already chosen otherwise; see the Rationale and Migration Notes below)."
and Future Changes Section: http://www.python.org/dev/peps/pep-0394/#future-changes-to-this-recommendati...
" It is anticipated that there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2.
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. "
The "may refer to python3" phrase is an acknowledgment that arch has moved to /usr/bin/python == python3 and isn't going to revert even though upstream thinks it's a... premature time to make that switch. (To be fair to arch, the discussion and PEP happened as a result of arch making that switch so they'd already committed to that before the consensus was formed that this would be a bad thing to do atthis time. We don't have that excuse ;-)
If you'd like to read the discussions for yourself, there are three threads linked from the PEP. An even earlier one is at: http://mail.python.org/pipermail/python-dev/2010-November/105252.html
The python package itself should probably also remain python2 due to dependencies and expectations from other distros and documentation -- I think I'd be -1 to changing this
The Fedora live images contain only python3, not python2 -- I'd be heavily in favour of this. +1
This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Getting there at the same time as we get to DNF sounds like a good timeline. (But see my note on anaconda below). +1
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
I'm not sure about this one as it will cause a lot of package churn. It might be a necessary pain pointi or it might be a pain point we want to defer until later in our porting efforts. Have to think about it more.
If you look at the minimal mock buildroot for rawhide now, the only thing that is drawing in Python is gdb because of it's Python bindings (if I'm not mistaken). So compiling GDB against Python 3, which should work with newest gdb, will accomplish this AFAICS.
<nod> I thought that python might be one of the packages that showed up in the dep chain here: http://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions_2
which would mean that packages might be leaving it out of BuildRequires right now. I wasn't able to find a dep chain leading back to python from one of those so I think that any package which isn't explicitly BuildRequiring python just has a bug. No objection here although as noted earlier, we should probably do this right after a rawhide branch so that any of these bugs are found and fixed in plenty of time.
- Porting Anaconda to Python 3.
+1 -- unfortunately, this probably depends on DNF.... So we may need to push DNF in F22 and anaconda compatible with python3 in F23.
DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.
Well.... 1) If DNF lands their python3 bindings sooner, that's fine for the timeline. But if they don't, anaconda can't be finished until after. So this is something to note as a key piece of the switch. 2) Fedora anaconda experience (and general open source development experience as well... you've read the Joel on Software article about netscape, right?) would tend to show that big rewrites are worse than several smaller ones.
Sure, smaller ones mean that you touch the same code a few times before you're satisfied with it. But smaller ones mean that you can stop partway through (say, for instance, because we needed a few extra weeks to port to DNF and that doesn't leave us enough time to complete the python3 port [even assuming that that doesn't take longer than anticipated] in time for Fedora 22's release date) which is just anticipating that Murphy's Law is inevitably going to throw a wrench into your timeline for completion.
- Making all livecd packages depend on Python 3 by default (and eventually
getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
+1 -- this is what I see as the eventual goal (or perhaps, livecd python2 free followed by DVD python2 free followed by distro python2 free).
3.5) Switch tools that could target either python2 or python3 to target python3. Currently the packaging guidelines say to target python2 to control dep proliferation and because that's the most supported by the larger python ecosystem. We should switch the recommendation when our minimal environment must have python3 but does not need to have python2.
IMO we should switch this for F21, since livecd ships Python 3 anyway, so the switch doesn't have to happen in one point, but can be continuous.
Ehhh... I don't think the livecd having to ship python3 is a good measure for this. I think something considerably more minimal than the livecd would be better. talked to mattdm (since he's been working on minimal environments) and he suggested @core or @standard groups might be appropriate.) The idea is to avoid doubling the needed python stacks on minimal environments until necessary. Switching tools that have the option of running on either one or the other of the stacks to python3 prematurely means that we start doubling the the python stacks needed before it's necessary.
( 4) Making as much of the remaining packages Python 3 compatible )
We could talk quite a bit on this point -- How active do we want to be with the things that aren't in one of the essential buckets from further up. We could defer thinking about this until after we get the livecd python2-free, though.
This is really the last step, that is somehow tied what you mentioned as a reaction to 3) - going through the rest of packages on DVD and then whole distro. This will take few more releases I guess, but it is not that important as sorting out livecd.
yeah, this strikes me as extending far into the future. I will note, though, that ideas about changing /usr/bin/python to point to python3 probably come in the latter stages of this step rather than before.
- Switching the %{?with_python3} conditionals in specfiles to
%{?with_python2} (we will probably create a script to automate this, at least partially)
-1: This one doesn't make any sense to do. The third-party python library ecosystem is highly weighted for python2. There are only a handful of libraries that support python3 and not python2. There are a boatload of libraries that support python2 and not python3. We're starting from a base of existing python2 packages that may add support for python3. The conditionals are there to enable packaging of that situation.
And this situation will be changing in the future. Right now, there are not so many Python packages in Fedora that only support Python 2 (I didn't count, but you don't see them too often these days).
Uh.... What's your methodology? This is a very, very, very bad estimate but I think it'll show that we need more than an anecdote to prove that statement:
$ repoquery -q 'python3-*' |wc -l 259 $ repoquery -q 'python-*' |wc -l 1099
Also, just to be clear, you do understand why switching the conditionals doesn't work for our existing packages, correct?
IMO Fedora should lead the way of making Python 3 "the Python" and Python 2 "the old compat version". This also makes sense in the traditional linux-distro "one version of package" that we should be trying to pursue.
-1. I'm serious, you've got the wrong conceptual model of the relationship between python2 and python3 stuck in your head and it's coloring what you're trying to achieve in bad ways. Python3 is not an upgrade to Python2. Python3 is a new language. It is compatible in many ways. If you can target recent enough versions (at least python-2.6 but python2.7 is better and python-3.3) you can set out to purposefully code things that work on both languages. But if you're writing general, working python2 code using idioms and thought processes that you've mastered over the last 10 years, chances are extremely high that not even your simple scripts are going to run without modification.
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
C is older. Let's get rid of that first. Old is not a reason to switch or get rid of it.
- Slower - yes, in the beginning, Python 3 was significantly slower because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
* pypy is faster and mostly python2 compatible. Many shops that want speed are switching to that rather than python3. (Not saying this is a good idea for a distro to do. I'm just saying that making the speed argument is not compelling). * people don't use python for raw speed of processing. They really just care if it's fast enough. People who write python code would be happy to take speed increases if they were free. But python2 to python3 requires porting code so it comes with a significant cost. Speed is a side effect of switching, not a reason to switch.
- Less Pythonic - where do I start with this? Python 3 got rid of tons of unnecessary syntactic constructs as well as builtin object methods. E.g. "print" vs. "print()"; exception raising syntax; dict.iteritems() removed and dict.items() only left, more consistent unicode handling etc. So in the sense of having only one way to do things, Python 3 is more pythonic than Python 2. If you read through zen of python, you can find more arguments for this (e.g. making int and long one type - "Special cases aren't special enough to break the rules."; simplification/rewrite of parts of stdlib - "Simple is better than complex.", etc.)
pythonic is a very vague statement and I wouldn't consider most of your list to be examples of those. Yes, python3 may be a *better* language (and I would include most of your list as "features of python3 that python2 does not have) but a more pythonic language... that's not something that you can readily measure. For instance, I can make the case that python3's unicode handling is less pythonic than python2 as it violates three rules of the zen of python:
Explicit is better than implicit. Errors should never pass silently. In the face of ambiguity, refuse the temptation to guess.
(To be fair, python2 violated some of these rules in its unicode handling as well, although errors should never pass silently would probably take some work to convince most people :-)
Anyhow, I stick to my assertion that we should be talking mostly about upstream support ending as the reason to switch and also the features that python3 provides that python2 does not [and as noted, I'd lump your "pythonic" list into this category.] Stating a compelling argument of why people should change isn't just about identifying all the things that the change will do. It's also about identifying the things that the change will do that are important to the person and resonate with their needs.
-Toshio
Le Ven 19 juillet 2013 22:11, Toshio Kuratomi a écrit :
Python3 is not an upgrade to Python2. Python3 is a new language. It is compatible in many ways. If you can target recent enough versions (at least python-2.6 but python2.7 is better and python-3.3) you can set out to purposefully code things that work on both languages. But if you're writing general, working python2 code using idioms and thought processes that you've mastered over the last 10 years, chances are extremely high that not even your simple scripts are going to run without modification.
How exactly is it different from when gcc grew standard C++ behaviour and most C++ apps broke right and left?
http://gcc.gnu.org/gcc-2.96.html
On Saturday 20 July 2013 17:04:47 Nicolas Mailhot wrote:
How exactly is it different from when gcc grew standard C++ behaviour and most C++ apps broke right and left?
The same can be said about C++/11 that clearly looks (at least to me) as a new and better language than C++/03 (or older).
I am aware that this goes against Toshio's statement, a statement that I can understand and agree but really python3 is a better python than python2 was (is). Surely that sometimes age make us wiser.
On the other hand according to python's zen:
$ python -m 'this' | grep practical Although practicality beats purity.
That is why I agree with Toshio POV. FWIW, and before you ask, most of the time before expressing my thoughts I don't need to fire a python session. ;-)
Regards,
----- Original Message -----
I'm okay with this portion of the definition. One note is I would be hesitant about the timing of python3 being the only python that is installed into the minimal buildroot. This should probably happen in rawhide right after a branching.
To be more specific, I meant "installed into the minimal buildroot by default", which only means building GDB with Python 3, as I've written elsewhere. IMO it shouldn't cause any problems, because all the packages that use python 2 during build should BR: python2-devel anyway, right? (maybe some packages with build scripts written in python 2 may have minor issues, but these should be solvable by simply adding BR: python or rewriting to python 3)
- livecd can still ship Python 2
I would consider this to be the goal that we should shoot for, though. We are constantly fighting for space on the install images and we know that people who install Fedora would like to have the ability to slim down what is installed. Shooting for no-python2 on the livecds and after that, no python2 on the install dvds, and still later, no need for python2 in the packages in the repository seem like milestones that have actual real value for end users.
Ok, +1
- /usr/bin/python points to Python 3
I am firmly against this. more depth was in my reply to mmaslano although I'll reply to one thing here:
- Please note, that the pep you're referring to also states that "python
should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions", so this wouldn't really be going against the pep.
This is a misinterpretion of the PEP. (This section is confusing, though: "python should refer to the same target as python2" is a recommendation to distributions. "but may refer to python3 on some bleeding edge distributions" is a statement of fact for end users to watch out for) See the recommendation section:
"For the time being, it is recommended that python should refer to python2 (however, some distributions have already chosen otherwise; see the Rationale and Migration Notes below)."
and Future Changes Section: http://www.python.org/dev/peps/pep-0394/#future-changes-to-this-recommendati...
" It is anticipated that there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2.
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. "
The "may refer to python3" phrase is an acknowledgment that arch has moved to /usr/bin/python == python3 and isn't going to revert even though upstream thinks it's a... premature time to make that switch. (To be fair to arch, the discussion and PEP happened as a result of arch making that switch so they'd already committed to that before the consensus was formed that this would be a bad thing to do atthis time. We don't have that excuse ;-)
If you'd like to read the discussions for yourself, there are three threads linked from the PEP. An even earlier one is at: http://mail.python.org/pipermail/python-dev/2010-November/105252.html
One way or the other, the PEP states that people should be careful about what /usr/bin/python points to, since it may be python 2 as well as python 3. This gives us the freedom to do the switch when we want and not break the upstream expectations (yes, I know that huge number of people expect /usr/bin/python to point to python 3, but this pep basically says that it's a bad idea to assume that).
The python package itself should probably also remain python2 due to dependencies and expectations from other distros and documentation -- I think I'd be -1 to changing this
The Fedora live images contain only python3, not python2 -- I'd be heavily in favour of this. +1
This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Getting there at the same time as we get to DNF sounds like a good timeline. (But see my note on anaconda below). +1
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
I'm not sure about this one as it will cause a lot of package churn. It might be a necessary pain pointi or it might be a pain point we want to defer until later in our porting efforts. Have to think about it more.
If you look at the minimal mock buildroot for rawhide now, the only thing that is drawing in Python is gdb because of it's Python bindings (if I'm not mistaken). So compiling GDB against Python 3, which should work with newest gdb, will accomplish this AFAICS.
<nod> I thought that python might be one of the packages that showed up in the dep chain here: http://fedoraproject.org/wiki/Packaging:Guidelines#Exceptions_2
which would mean that packages might be leaving it out of BuildRequires right now. I wasn't able to find a dep chain leading back to python from one of those so I think that any package which isn't explicitly BuildRequiring python just has a bug. No objection here although as noted earlier, we should probably do this right after a rawhide branch so that any of these bugs are found and fixed in plenty of time.
- Porting Anaconda to Python 3.
+1 -- unfortunately, this probably depends on DNF.... So we may need to push DNF in F22 and anaconda compatible with python3 in F23.
DNF is a continuous effort. I believe that DNF will provide it's Python 3 bindings sooner than in F22, so Anaconda devels can simultaneously do porting to Python 3 as well as to DNF. IMO this is good thing, since they will just do one big rewrite instead of two smaller.
Well.... 1) If DNF lands their python3 bindings sooner, that's fine for the timeline. But if they don't, anaconda can't be finished until after. So this is something to note as a key piece of the switch. 2) Fedora anaconda experience (and general open source development experience as well... you've read the Joel on Software article about netscape, right?) would tend to show that big rewrites are worse than several smaller ones.
Sure, smaller ones mean that you touch the same code a few times before you're satisfied with it. But smaller ones mean that you can stop partway through (say, for instance, because we needed a few extra weeks to port to DNF and that doesn't leave us enough time to complete the python3 port [even assuming that that doesn't take longer than anticipated] in time for Fedora 22's release date) which is just anticipating that Murphy's Law is inevitably going to throw a wrench into your timeline for completion.
And still, they can write Python 3 code that is compatible with both Python 2 and Python 3, so if DNF fails to provide Python 3 bindings in time, they can run on Python 2.
- Making all livecd packages depend on Python 3 by default (and
eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
+1 -- this is what I see as the eventual goal (or perhaps, livecd python2 free followed by DVD python2 free followed by distro python2 free).
3.5) Switch tools that could target either python2 or python3 to target python3. Currently the packaging guidelines say to target python2 to control dep proliferation and because that's the most supported by the larger python ecosystem. We should switch the recommendation when our minimal environment must have python3 but does not need to have python2.
IMO we should switch this for F21, since livecd ships Python 3 anyway, so the switch doesn't have to happen in one point, but can be continuous.
Ehhh... I don't think the livecd having to ship python3 is a good measure for this. I think something considerably more minimal than the livecd would be better. talked to mattdm (since he's been working on minimal environments) and he suggested @core or @standard groups might be appropriate.) The idea is to avoid doubling the needed python stacks on minimal environments until necessary. Switching tools that have the option of running on either one or the other of the stacks to python3 prematurely means that we start doubling the the python stacks needed before it's necessary.
As I replied to mattdm, I'm not against switching this at a single time (although doing more little steps is better than doing a single huge step ;)).
( 4) Making as much of the remaining packages Python 3 compatible )
We could talk quite a bit on this point -- How active do we want to be with the things that aren't in one of the essential buckets from further up. We could defer thinking about this until after we get the livecd python2-free, though.
This is really the last step, that is somehow tied what you mentioned as a reaction to 3) - going through the rest of packages on DVD and then whole distro. This will take few more releases I guess, but it is not that important as sorting out livecd.
yeah, this strikes me as extending far into the future. I will note, though, that ideas about changing /usr/bin/python to point to python3 probably come in the latter stages of this step rather than before.
- Switching the %{?with_python3} conditionals in specfiles to
%{?with_python2} (we will probably create a script to automate this, at least partially)
-1: This one doesn't make any sense to do. The third-party python library ecosystem is highly weighted for python2. There are only a handful of libraries that support python3 and not python2. There are a boatload of libraries that support python2 and not python3. We're starting from a base of existing python2 packages that may add support for python3. The conditionals are there to enable packaging of that situation.
And this situation will be changing in the future. Right now, there are not so many Python packages in Fedora that only support Python 2 (I didn't count, but you don't see them too often these days).
Uh.... What's your methodology? This is a very, very, very bad estimate but I think it'll show that we need more than an anecdote to prove that statement:
$ repoquery -q 'python3-*' |wc -l 259 $ repoquery -q 'python-*' |wc -l 1099
Heh, true :) I based this on packages that I review (out of which likely 90 % or so have python3- subpackage) and I didn't really count. Well, it is high time we start working on these 700 :)
Also, just to be clear, you do understand why switching the conditionals doesn't work for our existing packages, correct?
Please be more specific. What do you mean "doesn't work"?
IMO Fedora should lead the way of making Python 3 "the Python" and Python 2 "the old compat version". This also makes sense in the traditional linux-distro "one version of package" that we should be trying to pursue.
-1. I'm serious, you've got the wrong conceptual model of the relationship between python2 and python3 stuck in your head and it's coloring what you're trying to achieve in bad ways. Python3 is not an upgrade to Python2. Python3 is a new language. It is compatible in many ways. If you can target recent enough versions (at least python-2.6 but python2.7 is better and python-3.3) you can set out to purposefully code things that work on both languages. But if you're writing general, working python2 code using idioms and thought processes that you've mastered over the last 10 years, chances are extremely high that not even your simple scripts are going to run without modification.
And again, I'm saying that your conceptual model is not necessarily the correct one.
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
C is older. Let's get rid of that first. Old is not a reason to switch or get rid of it.
Python 2 has Python 3 as a successor, C does not (yes, I know you'll say C++, but this is really not the case).
- Slower - yes, in the beginning, Python 3 was significantly slower because
of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
- pypy is faster and mostly python2 compatible. Many shops that want speed are switching to that rather than python3. (Not saying this is a good idea for a distro to do. I'm just saying that making the speed argument is not compelling).
- people don't use python for raw speed of processing. They really just care if it's fast enough. People who write python code would be happy to take speed increases if they were free. But python2 to python3 requires porting code so it comes with a significant cost. Speed is a side effect of switching, not a reason to switch.
It is one of the reasons to switch for me.
- Less Pythonic - where do I start with this? Python 3 got rid of tons of
unnecessary syntactic constructs as well as builtin object methods. E.g. "print" vs. "print()"; exception raising syntax; dict.iteritems() removed and dict.items() only left, more consistent unicode handling etc. So in the sense of having only one way to do things, Python 3 is more pythonic than Python 2. If you read through zen of python, you can find more arguments for this (e.g. making int and long one type - "Special cases aren't special enough to break the rules."; simplification/rewrite of parts of stdlib - "Simple is better than complex.", etc.)
pythonic is a very vague statement and I wouldn't consider most of your list to be examples of those. Yes, python3 may be a *better* language (and I would include most of your list as "features of python3 that python2 does not have) but a more pythonic language... that's not something that you can readily measure. For instance, I can make the case that python3's unicode handling is less pythonic than python2 as it violates three rules of the zen of python:
Explicit is better than implicit. Errors should never pass silently. In the face of ambiguity, refuse the temptation to guess.
Eh, I really don't see where Python 3 unicode handling violates these. Could you be more specific?
(To be fair, python2 violated some of these rules in its unicode handling as well, although errors should never pass silently would probably take some work to convince most people :-)
Anyhow, I stick to my assertion that we should be talking mostly about upstream support ending as the reason to switch and also the features that python3 provides that python2 does not [and as noted, I'd lump your "pythonic" list into this category.] Stating a compelling argument of why people should change isn't just about identifying all the things that the change will do. It's also about identifying the things that the change will do that are important to the person and resonate with their needs.
-Toshio
On 19/07/13 02:41 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
- Slower - yes, in the beginning, Python 3 was significantly slower
because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
Mind to share some grounds for this claim? My negligible experience told me the contrary, but perhaps timeit module is a bad indicator.
Otherwise yes, let's get gently beyond 3000.
----- Original Message -----
On 19/07/13 02:41 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
- Slower - yes, in the beginning, Python 3 was significantly slower
because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
Mind to share some grounds for this claim? My negligible experience told me the contrary, but perhaps timeit module is a bad indicator.
There is the presentation that I referenced in the email that you reacted to (referenced by [1]). You can also have a look at pystone benchmark results (a quick googling around gave me e.g. http://www.levigross.com/post/2340736877/pystone-benchmark-on-2-6-2-7-3-2)
Otherwise yes, let's get gently beyond 3000.
-- Jan -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On 23/07/13 03:16 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On 19/07/13 02:41 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
- Slower - yes, in the beginning, Python 3 was significantly slower
because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
Mind to share some grounds for this claim? My negligible experience told me the contrary, but perhaps timeit module is a bad indicator.
There is the presentation that I referenced in the email that you reacted to (referenced by [1]).
Saw the presentation before actually replying -- it doesn't show any evidence Python 3.3 would be noticably faster. And my micro tests unfortunately fell into the "slower" category as nicely demonstrated in the graphs there.
You can also have a look at pystone benchmark results (a quick googling around gave me e.g. http://www.levigross.com/post/2340736877/pystone-benchmark-on-2-6-2-7-3-2)
This is more than two years old. The score reliability of this benchmark is also questionable (not speaking about order of magnitude difference as with PyPy).
Otherwise yes, let's get gently beyond 3000.
----- Original Message -----
On 23/07/13 03:16 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On 19/07/13 02:41 -0400, Bohuslav Kabrda wrote:
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
FAQ: Q: Why do we need to switch to Python 3? A: Because Python 2 is old, slower, less pythonic, doesn't get any more functionality and it won't be that long before the official upstream support ends [1]
Although I agree with the need to switch to python3, I don't think the first three reasons are very compelling arguments (they're only half-truths) -- we should concentrate on the last reason and also on features that python3 has that pyhton2 doesn't. Chained exceptions are a pretty nice thing, for instance.
So first three reason:
- Python 2 is old - how is that a half-truth?
- Slower - yes, in the beginning, Python 3 was significantly slower
because of nonoptimal code after the rewrite from Python 3. But with Python 3.3 for instance, you get tons of speed improvements - decimal module for instance got a significant boost. Brett Cannon had a nice presentation about speed benchmarking [1]. Yes, Python 3 is slower in some areas, but mostly it's faster.
Mind to share some grounds for this claim? My negligible experience told me the contrary, but perhaps timeit module is a bad indicator.
There is the presentation that I referenced in the email that you reacted to (referenced by [1]).
Saw the presentation before actually replying -- it doesn't show any evidence Python 3.3 would be noticably faster. And my micro tests unfortunately fell into the "slower" category as nicely demonstrated in the graphs there.
Yes, it is not _noticably_ faster, I agree. The thing is that micro tests do tend to be slower on Python 3.3 because of different integer and string representations. Things start to change when you start to use stdlib.
You can also have a look at pystone benchmark results (a quick googling around gave me e.g. http://www.levigross.com/post/2340736877/pystone-benchmark-on-2-6-2-7-3-2)
This is more than two years old. The score reliability of this benchmark is also questionable (not speaking about order of magnitude difference as with PyPy).
Yeah, PyPy is known to be a lot faster than any cPython. The problem with PyPy is that C extension support is still in beta [1] and requires some patching. So unfortunately PyPy is not suitable for default Python implementation for distribution such as Fedora, IMO.
Otherwise yes, let's get gently beyond 3000.
-- Jan
On 07/18/2013 06:53 PM, Toshio Kuratomi wrote:
/usr/bin/python should refer to python2 -- http://www.python.org/dev/peps/pep-0394/ I'd be -1 to changing this
Quoting from this PEP: python should refer to the same target as python2 but may refer to python3 on some bleeding edge distributions
I always thought that Fedora (and especially rawhide) is bleading edge distribution :)
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
From packaging point of view, this will probably require:
- Renaming python package to python2
- Renaming python3 package to python
- Switching the %{?with_python3} conditionals in specfiles to %{?with_python2} (we will probably create a script to automate this, at least partially)
Renaming the python package to python2 kind of makes sense, but renaming the python3 package to python seems needlessly confusing. Wouldn't it make sense to just keep python2 and python3 side by side without ambiguity until some long future date when python2 disappears?
-- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default. This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
- Porting Anaconda to Python 3.
- Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
( 4) Making as much of the remaining packages Python 3 compatible )
If we do any work on python3 conversions, it must be done in the context of respective upstream projects, and not a Fedora custom addon. It will also quite likely require non-negligable dev resources to make a big dent in it. It is unlikely to be as simple as just telling upstream to run 2to3, because a great many upstreams are going to want to provide ongoing support for both Python2 and Python3 to satisfy non cutting edge distros.
From packaging point of view, this will probably require:
- Renaming python package to python2
Renaming this is fine, particularly if we also add a Provides: python
- Renaming python3 package to python
This is a bad idea IMHO. Python3 is not upgrade compatible with functionality previously provided in the current python package. So such a rename is going to needlessly cause pain & suffering.
- Switching the %{?with_python3} conditionals in specfiles to %{?with_python2}
(we will probably create a script to automate this, at least partially)
I don't think we need do any automated conversion of that sort. It is perfectly reasonable for packages to in fact have both %{with_python3} and %{with_python2} conditionals present. By all means add %{with_python2} to allow builds without the legacy python2 stuff, but no need to blanket remove %{with_python3} - leave it upto the maintainer to decide if they wish to have conditionals for this, or unconditionally build both.
Regards, Daniel
On Fri, Jul 19, 2013 at 11:27 AM, Daniel P. Berrange berrange@redhat.com wrote:
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default. This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
- Porting Anaconda to Python 3.
- Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
( 4) Making as much of the remaining packages Python 3 compatible )
If we do any work on python3 conversions, it must be done in the context of respective upstream projects, and not a Fedora custom addon.
IMHO this is precisely the kind of integration where a distribution is perfectly justified to carry local patches, even against explicit wishes of upstream if necessary (though cooperating with upstream and finding a way to integrate the patch upstream is much better). We shouldn't block the transition just because a one or two upstreams refuse to port (or, more likely, a dozen or two upstreams do not exist any more). Mirek
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
- Making all livecd packages depend on Python 3 by default (and
eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python
I have a concern about bloating @core and by extension the cloud image. Right now, python is about 5% of the total on-disk usage. I'd hate to see that go to 10%. Therefore, I'd like to see a goal of making the transition for usage in the base cloud image go entirely from python2 to python3 in one release cycle.
(Roughly, that's @core + cloud-init, which isn't currently on your list.)
On Fri, Jul 19, 2013 at 03:29:57PM -0400, Matthew Miller wrote:
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
- Making all livecd packages depend on Python 3 by default (and
eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python
I have a concern about bloating @core and by extension the cloud image. Right now, python is about 5% of the total on-disk usage. I'd hate to see that go to 10%. Therefore, I'd like to see a goal of making the transition for usage in the base cloud image go entirely from python2 to python3 in one release cycle.
(Roughly, that's @core + cloud-init, which isn't currently on your list.)
This seems like a very very worthy goal. I took a brief look at @core (I didn't recurse down the whole dep chain so hopefully I didn't miss an iceberg) and that looks doable. The switch from yum to DNF is the big thing.
cloud-init is python so it might need to be ported in and of itself. Some of its dependencies also don't have python3 ports:
* configobj -- no upstream python3 port and the third-party port never made it to a release (the code may work but it's unmaintained). If someone is ready to become the upstream for this, indications are that it would be welcomed by both the port's original author and the original ConfigObj maintainer. * policycoreutils-python -- not sure if this interfaces with the C libraries or is about implementing higher level functionality. Nearly anytime you interface with C you have to make choices about bytes vs text and problems with encoding. That could slow up this work. * python-boto -- Looks like there's a badly out-of-date branch in the upstream SCM repository to add python3 support. That may be based on an API incompatible update, though (I see that there's work on boto3...) boto is going to need to talk over the wire to AWS so there's going to be a lot of places where bytes vs text decisions have to be made. All in all, this package feels like it might be the hardest piece of the cloud-init porting. * python-cheetah -- Development slowed way down after they made their last release in 2010 and announced that the next release cheetah-3.0 would include python3 support. Probably need to contact upstream about this and may need to prepare the patch to do the port.
-Toshio
On Fri, Jul 19, 2013 at 01:45:41PM -0700, Toshio Kuratomi wrote:
- python-cheetah -- Development slowed way down after they made their last release in 2010 and announced that the next release cheetah-3.0 would include python3 support. Probably need to contact upstream about this and may need to prepare the patch to do the port.
What I'd _really_ like to do is get cheetah factored out of cloud-init. (https://bugzilla.redhat.com/show_bug.cgi?id=974327). It brings in a whole dependency chain of which python2 vs. python3 is the least of the troubles.
On Fri, Jul 19, 2013 at 05:30:36PM -0400, Matthew Miller wrote:
On Fri, Jul 19, 2013 at 01:45:41PM -0700, Toshio Kuratomi wrote:
- python-cheetah -- Development slowed way down after they made their last release in 2010 and announced that the next release cheetah-3.0 would include python3 support. Probably need to contact upstream about this and may need to prepare the patch to do the port.
What I'd _really_ like to do is get cheetah factored out of cloud-init. (https://bugzilla.redhat.com/show_bug.cgi?id=974327). It brings in a whole dependency chain of which python2 vs. python3 is the least of the troubles.
If your needs are very minimal, python3-tempita might be a good choice. If you actually do need more features than that, python-mako and python-jinja2 are popular. Note that both of those have a few deps (but hopefully not as bad as cheetah). (Also -- the python3 version of mako has less deps than the python2 version... I think that it just because those deps haven't been ported to python3 yet and the package can operate with reduced functionality without them. the deps fo the python3 version might expand i nthe future).
-Toshio
----- Original Message -----
On Thu, Jul 18, 2013 at 11:24:22AM -0400, Bohuslav Kabrda wrote:
- Making all livecd packages depend on Python 3 by default (and
eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python
I have a concern about bloating @core and by extension the cloud image. Right now, python is about 5% of the total on-disk usage. I'd hate to see that go to 10%. Therefore, I'd like to see a goal of making the transition for usage in the base cloud image go entirely from python2 to python3 in one release cycle.
(Roughly, that's @core + cloud-init, which isn't currently on your list.)
Doing everything in one shot sounds reasonable from this POV, I'll try to put this into my plan (I'll go through cloud-init the same way I went through livecd and post my findings here).
-- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mattdm@fedoraproject.org
On 18/07/13 08:24 AM, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default. This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
- Porting Anaconda to Python 3.
- Making all livecd packages depend on Python 3 by default (and eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3.
( 4) Making as much of the remaining packages Python 3 compatible )
I am very surprised infrastructure has not planned to gradually move to python 3 since its release. I understand that due to RHEL, python2 will remain used, in Fedora it is disappointing very little initiative were proposed in the past for a better transition. i applaud the effort which is really needed. Fedora is supposed to be a distribution with latest stable technologies. Python 2.x support is coming to the end.
On 23 July 2013 13:02, Luya Tshimbalanga luya@fedoraproject.org wrote:
On 18/07/13 08:24 AM, Bohuslav Kabrda wrote:
Hi all, as a new Fedora Python maintainer, I have set myself a goal of moving Fedora to Python 3 as a default. This is going to be a multirelease effort that is going to affect lots of Fedora parts. Since we will need to switch default package manager from Yum to DNF (which is supposed to work with Python 3), we will need to wait for that. I've been told that DNF should be default in F22, so that's my target, too. That should also give everyone else plenty of time to work on other essential packages to make this happen.
Here is my analysis/proposal: Before switching, we need to make sure that everything "important" (*) is Python 3 compatible. There are three steps I see in this transition:
- Getting rid of Python 2 in mock minimal buildroot.
- Porting Anaconda to Python 3.
- Making all livecd packages depend on Python 3 by default (and
eventually getting rid of Python 2 from livecd) - this will also require switching from Yum to DNF as a default, that is supposed to support Python 3. ( 4) Making as much of the remaining packages Python 3 compatible )
I am very surprised infrastructure has not planned to gradually move to
python 3 since its release. I understand that due to RHEL, python2 will remain used, in Fedora it is disappointing very little initiative were proposed in the past for a better transition.
Don't be disappointed. Go help. The above has been the plan since Python 3 came out. We can plan all we want, but if no one has time to work on it then it doesn't mean much. We can't shut down the old infrastructure and we can't stop implementing things that people require now versus 6-9 months from now when a python3 rewrite would be done with the people currently here.
On 23/07/13 12:23 PM, Stephen John Smoogen wrote:
Don't be disappointed. Go help. The above has been the plan since Python 3 came out. We can plan all we want, but if no one has time to work on it then it doesn't mean much. We can't shut down the old infrastructure and we can't stop implementing things that people require now versus 6-9 months from now when a python3 rewrite would be done with the people currently here.
-- Stephen J Smoogen.
I'll do as I can to help executing the plan. Thanks for the encouragement.
Luya
On Tue, Jul 23, 2013 at 12:02:04PM -0700, Luya Tshimbalanga wrote:
I am very surprised infrastructure has not planned to gradually move to python 3 since its release. I understand that due to RHEL, python2 will remain used, in Fedora it is disappointing very little initiative were proposed in the past for a better transition.
Note -- if you want infrastructure to move to python3, the first step would be to get python3 into EPEL. There are quite a few things that could go into an EPEL python3 stack once we have python-3.3 there. (Although EPEL will probably have to decide whether they want to create a bunch of forward compat packages [since they have older non-python3-capable versions of some packages there] or instead create separate python3 SRPM-level packages for the new code. And if we do move to separate packages in EPEL, whether we want to revise our packaging in Fedora to have separate packages for python2 and python3 in Fedora.
Infrastructure really doesn't have any chance to migrate to python3 at all until there's a python3 runtime available there. (Note -- there's other things holding it back as well -- for instance, the lack of a python3 port for the web framework that underlies a few critical services, and the fact that python2 will have to be run on the servers for basic system things like yum and ansible until we get a RHEL version with DNF and a python3 version of ansible. But without a python3 runtime, not even those things that can move independent of those constraints can migrate.)
-Toshio
On 23/07/13 09:28 PM, Toshio Kuratomi wrote:
Note -- if you want infrastructure to move to python3, the first step would be to get python3 into EPEL. There are quite a few things that could go into an EPEL python3 stack once we have python-3.3 there. (Although EPEL will probably have to decide whether they want to create a bunch of forward compat packages [since they have older non-python3-capable versions of some packages there] or instead create separate python3 SRPM-level packages for the new code. And if we do move to separate packages in EPEL, whether we want to revise our packaging in Fedora to have separate packages for python2 and python3 in Fedora.
Infrastructure really doesn't have any chance to migrate to python3 at all until there's a python3 runtime available there. (Note -- there's other things holding it back as well -- for instance, the lack of a python3 port for the web framework that underlies a few critical services, and the fact that python2 will have to be run on the servers for basic system things like yum and ansible until we get a RHEL version with DNF and a python3 version of ansible. But without a python3 runtime, not even those things that can move independent of those constraints can migrate.)
-Toshio
EPEL did not have python3 ? By looking at EL6 repository, I realized Blender is still at 2.49b instead of the current version 2.68. Will it be possible to bring on EL6 or future EL7? Now that the topic is raised, it is a good opportunity to gradually start the migration. Although my focus is currently on Design team, I may provide some assistance as I can.
Luya