Hey all, I just wanted to ask for some thoughts on a "problem" (or rather a "hardship") that is starting to show with the Python 3 transition in Fedora. I've been contacted by two maintainers of "applications" in Fedora for advice and have been thinking for some time how to solve this:
Note: by "applications" I mean packages that provide end-user benefit and don't need to be packaged for both Python interpreters. They just use some Python (preferably the default one) to deliver functionality to user. Let's take copr-cli as an example - this is a thin CLI wrapper around python-copr.
Current state: - Up until F21, maintainers were encouraged to build applications with Python 2, but weren't discouraged from building with Python 3. - From F22 on, packagers will be encouraged to build with Python 3 rather than Python 2. - Lots of packagers want to keep the same specfile for EPEL and Fedora. - Fedora guidelines mandate explicit usage of %__python2 and %__python3 (and all the sitelib/sitearch macros).
The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that.
Proposed Solution: After thinking a few days about this, here's what I propose: - Every specfile will have a minimal header with macro definitions that will look like this:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
%make_default_python %default_python
- The %make_default_python macro function will point all the unversioned macros to proper values for given %default_python:
### In Fedora %{__python} -> /usr/bin/python3 %{python_sitelib} -> /usr/lib/python3.X/site-packages # and other macros...
### In EPEL %{__python} -> /usr/bin/python2 %{python_sitelib} -> /usr/lib/python2.X/site-packages # and other macros...
- This means that packagers will be able to use the unversioned macros throughout their specfile. Requires and BuildRequires will look like this:
Requires: %{default_python}-flask BuildRequires %{default_python}-setuptools
Note: since BuildRequires need to be expanded in the minimal buildroot, it's necessary to define the %default_python macro in the specfile. Other way would be to define it in a macro file that would be added to the minimal buildroot, but that's neither very likely nor very nice :)
I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages.
Thoughts? Thanks, Slavek
[1] https://fedoraproject.org/wiki/User:Bkabrda/EPEL7_Python3
On 27 January 2015 at 21:26, Bohuslav Kabrda bkabrda@redhat.com wrote:
Hey all, I just wanted to ask for some thoughts on a "problem" (or rather a "hardship") that is starting to show with the Python 3 transition in Fedora. I've been contacted by two maintainers of "applications" in Fedora for advice and have been thinking for some time how to solve this:
Note: by "applications" I mean packages that provide end-user benefit and don't need to be packaged for both Python interpreters. They just use some Python (preferably the default one) to deliver functionality to user. Let's take copr-cli as an example - this is a thin CLI wrapper around python-copr.
Current state:
- Up until F21, maintainers were encouraged to build applications with Python 2, but weren't discouraged from building with Python 3.
- From F22 on, packagers will be encouraged to build with Python 3 rather than Python 2.
- Lots of packagers want to keep the same specfile for EPEL and Fedora.
- Fedora guidelines mandate explicit usage of %__python2 and %__python3 (and all the sitelib/sitearch macros).
The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that.
Proposed Solution: After thinking a few days about this, here's what I propose:
- Every specfile will have a minimal header with macro definitions that will look like this:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
I'm wary of this proposed solution mostly due to the fact that in the middle of last year, the Beaker team had to go through and completely redesign the way the automatic kickstart generation worked, because the templates were full of checks that assumed "RHEL 6" as the default basis for derived distros. Once RHEL 7 and CentOS 7 were generally available, that assumption became problematically wrong (e.g. systemd wasn't a Fedora only feature any more), so the templates were all rewritten to be based on operating system feature flags instead (which had the added bonus of also making them more tolerant of Fedora derivatives).
I see the seeds of a similar problem being planted with the above proposal: what happens when, at some point in the future, "Python 3 as default" is no longer a Fedora-only feature? Do we have to go edit all the spec files again?
What if, instead, we were able to add a new macro that let folks *explicitly* opt in to running in the "system Python", but then define the recommended spec file usage such that it falls back to Python 2 if the "system Python" macro isn't defined?
That would give people 3 explicit options to choose from:
* always run in Python 2 * always run in Python 3 * run in the same Python as Anaconda and yum
Single source Python 2/3 compatibility could then be made a policy requirement for packages opting in to running in the system Python rather than explicitly running in Python 3.
Regards, Nick.
----- Original Message -----
On 27 January 2015 at 21:26, Bohuslav Kabrda bkabrda@redhat.com wrote:
Hey all, I just wanted to ask for some thoughts on a "problem" (or rather a "hardship") that is starting to show with the Python 3 transition in Fedora. I've been contacted by two maintainers of "applications" in Fedora for advice and have been thinking for some time how to solve this:
Note: by "applications" I mean packages that provide end-user benefit and don't need to be packaged for both Python interpreters. They just use some Python (preferably the default one) to deliver functionality to user. Let's take copr-cli as an example - this is a thin CLI wrapper around python-copr.
Current state:
- Up until F21, maintainers were encouraged to build applications with
Python 2, but weren't discouraged from building with Python 3.
- From F22 on, packagers will be encouraged to build with Python 3 rather
than Python 2.
- Lots of packagers want to keep the same specfile for EPEL and Fedora.
- Fedora guidelines mandate explicit usage of %__python2 and %__python3
(and all the sitelib/sitearch macros).
The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that.
Proposed Solution: After thinking a few days about this, here's what I propose:
- Every specfile will have a minimal header with macro definitions that
will look like this:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
I'm wary of this proposed solution mostly due to the fact that in the middle of last year, the Beaker team had to go through and completely redesign the way the automatic kickstart generation worked, because the templates were full of checks that assumed "RHEL 6" as the default basis for derived distros. Once RHEL 7 and CentOS 7 were generally available, that assumption became problematically wrong (e.g. systemd wasn't a Fedora only feature any more), so the templates were all rewritten to be based on operating system feature flags instead (which had the added bonus of also making them more tolerant of Fedora derivatives).
I see the seeds of a similar problem being planted with the above proposal: what happens when, at some point in the future, "Python 3 as default" is no longer a Fedora-only feature? Do we have to go edit all the spec files again?
Yikes! Good point, I have put much thought into the macro solution and I wasn't giving enough thought to writing the actual condition...
What if, instead, we were able to add a new macro that let folks *explicitly* opt in to running in the "system Python", but then define the recommended spec file usage such that it falls back to Python 2 if the "system Python" macro isn't defined?
That would give people 3 explicit options to choose from:
- always run in Python 2
- always run in Python 3
- run in the same Python as Anaconda and yum
Single source Python 2/3 compatibility could then be made a policy requirement for packages opting in to running in the system Python rather than explicitly running in Python 3.
I've thought of this, but the problem is that we'd need to add this new macro to either RHEL's python-devel or minimal EPEL buildroot. We need this because the macros for BuildRequires need to be resolved in the minimal buildroot - so if the macro is not there, RPM can't expand BuildRequires and the build fails. The fact that adding such macros to RHEL's python-devel or EPEL buildroots is not likely made me come up with the proposed approach.
Unless... Unless we use the original unversioned %__python/%python_sitelib/%python_sitearch macros for this. The downside is that we'd need to go through tons of Fedora python-* specfiles and replace usage of unversioned macros by the versioned ones - but that's one-time effort. The upside is that these work in RHEL/EPEL without any additional effort *and* they kind of express precisely what we want them to express - *the* Python.
I do agree with Nick's 3 explicit options listed above - the first two being mostly for python-* extension packages and the third for applications that "don't care". I think that this is possibly a huge change that would be possible for F23 soonest, but it makes sense (it'd require changes in hundreds, maybe thousands of Fedora specfiles and also in RPM). I still think this only makes sense to use for "applications", not for "libraries", but I guess we're on the same page with this :)
Slavek
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 27, 2015 at 10:50:07PM +1000, Nick Coghlan wrote:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
I'm wary of this proposed solution mostly due to the fact that in the middle of last year, the Beaker team had to go through and completely redesign the way the automatic kickstart generation worked, because the templates were full of checks that assumed "RHEL 6" as the default basis for derived distros. Once RHEL 7 and CentOS 7 were generally available, that assumption became problematically wrong (e.g. systemd wasn't a Fedora only feature any more), so the templates were all rewritten to be based on operating system feature flags instead (which had the added bonus of also making them more tolerant of Fedora derivatives).
I see the seeds of a similar problem being planted with the above proposal: what happens when, at some point in the future, "Python 3 as default" is no longer a Fedora-only feature? Do we have to go edit all the spec files again?
Note that this ship has sailed long ago. Fedora packaging spec style is that someone (usually FPC) has to collect information about which Fedora/RHEL version a feature became relevant and then construct a conditional that accurately portrays that knowledge. So in the example above, at least a version check for fedora would be added. When we are able to build default python3 on rhel people would need to update spec files to reflect that (but that's an EPEL branching event anyway so people are going to have to do some manual work on to make the packages build on for the new EPEL anyway.)
It would be good if we could do better, though....
What if, instead, we were able to add a new macro that let folks *explicitly* opt in to running in the "system Python", but then define the recommended spec file usage such that it falls back to Python 2 if the "system Python" macro isn't defined?
Slavek raises the issue of how we get this into the buildroot. An idea could be to talk to rel-eng and the other packagers about adding these sorts of system-feature macros to a package in the buildroot. We could create a new package or add onto an existing one (is epel-release and fedora-release in the buildroot?) the package would contain a small set of macros that specified certain features about the OS that packagers need to know Then we really could write the conditionals as a feature test instead of a distro version test.
One drawback is that we would have to push the macros out to every release that we build for (epel and fedora) otherwise we'd still have to use distro+version conditionals.
-Toshio
On 28 January 2015 at 03:32, Toshio Kuratomi a.badger@gmail.com wrote:
On Tue, Jan 27, 2015 at 10:50:07PM +1000, Nick Coghlan wrote:
What if, instead, we were able to add a new macro that let folks *explicitly* opt in to running in the "system Python", but then define the recommended spec file usage such that it falls back to Python 2 if the "system Python" macro isn't defined?
Slavek raises the issue of how we get this into the buildroot. An idea could be to talk to rel-eng and the other packagers about adding these sorts of system-feature macros to a package in the buildroot. We could create a new package or add onto an existing one (is epel-release and fedora-release in the buildroot?) the package would contain a small set of macros that specified certain features about the OS that packagers need to know Then we really could write the conditionals as a feature test instead of a distro version test.
One drawback is that we would have to push the macros out to every release that we build for (epel and fedora) otherwise we'd still have to use distro+version conditionals.
That sounds vaguely analogous to the situation we have in Beaker: adding completely new "system features" may require a Beaker server update, while enabling and disabling already known features for a custom distro is just a configuration setting for that distro in the database. Actually switching to that model required updating the base templates for every distro we natively support.
The reason we decided that approach was worth the extra up front effort was because it meant we just had one place to update in the future (the code that handles the calculation of the distro -> feature mappings) rather than having to search the templates for all the cases where we were switching based on the distro.
I personally think "do it right" (i.e. figuring out how to enable feature based rather than version based checks) is the direction we should go for Fedora & EPEL, and then Slavek & I can separately tackle the challenge of getting key downstreams (i.e. RHEL & CentOS) to go along with that change. I'm more optimistic than Slavek is about that, as many of the reasons we made the change for the Beaker kickstart templates also apply to building for different environments.
Regards, Nick.
----- Original Message -----
On 28 January 2015 at 03:32, Toshio Kuratomi a.badger@gmail.com wrote:
On Tue, Jan 27, 2015 at 10:50:07PM +1000, Nick Coghlan wrote:
What if, instead, we were able to add a new macro that let folks *explicitly* opt in to running in the "system Python", but then define the recommended spec file usage such that it falls back to Python 2 if the "system Python" macro isn't defined?
Slavek raises the issue of how we get this into the buildroot. An idea could be to talk to rel-eng and the other packagers about adding these sorts of system-feature macros to a package in the buildroot. We could create a new package or add onto an existing one (is epel-release and fedora-release in the buildroot?) the package would contain a small set of macros that specified certain features about the OS that packagers need to know Then we really could write the conditionals as a feature test instead of a distro version test.
One drawback is that we would have to push the macros out to every release that we build for (epel and fedora) otherwise we'd still have to use distro+version conditionals.
That sounds vaguely analogous to the situation we have in Beaker: adding completely new "system features" may require a Beaker server update, while enabling and disabling already known features for a custom distro is just a configuration setting for that distro in the database. Actually switching to that model required updating the base templates for every distro we natively support.
The reason we decided that approach was worth the extra up front effort was because it meant we just had one place to update in the future (the code that handles the calculation of the distro -> feature mappings) rather than having to search the templates for all the cases where we were switching based on the distro.
I personally think "do it right" (i.e. figuring out how to enable feature based rather than version based checks) is the direction we should go for Fedora & EPEL, and then Slavek & I can separately tackle the challenge of getting key downstreams (i.e. RHEL & CentOS) to go along with that change. I'm more optimistic than Slavek is about that, as many of the reasons we made the change for the Beaker kickstart templates also apply to building for different environments.
I'll try to find out what it'd take to add such a package to epel minimal buildroot and fedora minimal buildroot and we'll see. I actually have one more use case for this and that is the Py3 in EPEL, where having some macros in minimal buildroot would help a lot, too.
I'll try to find out ASAP and post the result here.
Slavek
Regards, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 27, 2015 at 06:26:26AM -0500, Bohuslav Kabrda wrote:
Current state:
- Up until F21, maintainers were encouraged to build applications with Python 2, but weren't discouraged from building with Python 3.
Note -- this isn't quite right. If an application could run with either python2 or python3 maintainers in F21 and below were supposed to have the app run with python2. F22 is supposed to flip this so that maintainers are supposed make these packages run with python3 instead of python2.
- From F22 on, packagers will be encouraged to build with Python 3 rather than Python 2.
- Lots of packagers want to keep the same specfile for EPEL and Fedora.
- Fedora guidelines mandate explicit usage of %__python2 and %__python3 (and all the sitelib/sitearch macros).
The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that.
Proposed Solution: After thinking a few days about this, here's what I propose:
- Every specfile will have a minimal header with macro definitions that will look like this:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
%make_default_python %default_python
- The %make_default_python macro function will point all the unversioned macros to proper values for given %default_python:
### In Fedora %{__python} -> /usr/bin/python3 %{python_sitelib} -> /usr/lib/python3.X/site-packages # and other macros...
### In EPEL %{__python} -> /usr/bin/python2 %{python_sitelib} -> /usr/lib/python2.X/site-packages # and other macros...
I'm not really a fan of redefining existing macros like this. The problem is the same as the python2 "from __future__ import unicode_literals" statement. It causes your existing knowledge of how things work to betray you. And recognizing that the change is present requires you to look somewhere far away from the code that you are actually interested in.
- This means that packagers will be able to use the unversioned macros throughout their specfile. Requires and BuildRequires will look like this:
Requires: %{default_python}-flask BuildRequires %{default_python}-setuptools
Note: since BuildRequires need to be expanded in the minimal buildroot, it's necessary to define the %default_python macro in the specfile. Other way would be to define it in a macro file that would be added to the minimal buildroot, but that's neither very likely nor very nice :)
Have you talked to dennis? We've added packages to the minimal BuildRoot many times before in order to enable macro files. Not so problematic if it's only one macro, though.
I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages.
Thoughts?
If we were to use different macro names instead of overwriting currently well known ones I think this has merit.
-Toshio
----- Original Message -----
On Tue, Jan 27, 2015 at 06:26:26AM -0500, Bohuslav Kabrda wrote:
Current state:
- Up until F21, maintainers were encouraged to build applications with
Python 2, but weren't discouraged from building with Python 3.
Note -- this isn't quite right. If an application could run with either python2 or python3 maintainers in F21 and below were supposed to have the app run with python2. F22 is supposed to flip this so that maintainers are supposed make these packages run with python3 instead of python2.
I guess that I interpreted it my way since I didn't see any "must" in there... But ok, thanks for the clarification :)
- From F22 on, packagers will be encouraged to build with Python 3 rather
than Python 2.
- Lots of packagers want to keep the same specfile for EPEL and Fedora.
- Fedora guidelines mandate explicit usage of %__python2 and %__python3
(and all the sitelib/sitearch macros).
The Problem: If packagers want to build against Python 3 in Fedora and Python in EPEL *and* keep the same specfile, they have to invent some ugly hacks, since Fedora's guidelines require explicit usage of versioned Python macros. This affects Requires and BuildRequires and %prep, %build, %install, %check sections. People who want to do this either redefine %__python in Fedora to point to Python 3 or something like that - I'm afraid that we could end up with a huge pile of crazy macro redefinitions in tons of packages and I want to avoid that.
Proposed Solution: After thinking a few days about this, here's what I propose:
- Every specfile will have a minimal header with macro definitions that
will look like this:
%if 0%{?fedora} %global default_python python3 %else %global default_python python %endif
%make_default_python %default_python
- The %make_default_python macro function will point all the unversioned
macros to proper values for given %default_python:
### In Fedora %{__python} -> /usr/bin/python3 %{python_sitelib} -> /usr/lib/python3.X/site-packages # and other macros...
### In EPEL %{__python} -> /usr/bin/python2 %{python_sitelib} -> /usr/lib/python2.X/site-packages # and other macros...
I'm not really a fan of redefining existing macros like this. The problem is the same as the python2 "from __future__ import unicode_literals" statement. It causes your existing knowledge of how things work to betray you. And recognizing that the change is present requires you to look somewhere far away from the code that you are actually interested in.
Right, I guess I'll just agree on disagreeing with you on this :) I don't think it's a problem the same way I don't think "from __future__ import unicode_literals" is a problem.
- This means that packagers will be able to use the unversioned macros
throughout their specfile. Requires and BuildRequires will look like this:
Requires: %{default_python}-flask BuildRequires %{default_python}-setuptools
Note: since BuildRequires need to be expanded in the minimal buildroot, it's necessary to define the %default_python macro in the specfile. Other way would be to define it in a macro file that would be added to the minimal buildroot, but that's neither very likely nor very nice :)
Have you talked to dennis? We've added packages to the minimal BuildRoot many times before in order to enable macro files. Not so problematic if it's only one macro, though.
I will. Do you have any idea who I should talk to regarding EPEL? Is it also Dennis?
I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages.
Thoughts?
If we were to use different macro names instead of overwriting currently well known ones I think this has merit.
For me, introducing yet another set of macros is unnecessary. I think that it'd introduce a long(-ish) new part of guidelines that'll make them even more complicated than they are right now (compared to one new macro function and rules on how/when to use it).
-Toshio
Slavek
On Wed, Jan 28, 2015 at 11:59:01AM -0500, Bohuslav Kabrda wrote:
----- Original Message -----
On Tue, Jan 27, 2015 at 06:26:26AM -0500, Bohuslav Kabrda wrote:
Current state:
- Up until F21, maintainers were encouraged to build applications with
Python 2, but weren't discouraged from building with Python 3.
Note -- this isn't quite right. If an application could run with either python2 or python3 maintainers in F21 and below were supposed to have the app run with python2. F22 is supposed to flip this so that maintainers are supposed make these packages run with python3 instead of python2.
I guess that I interpreted it my way since I didn't see any "must" in there... But ok, thanks for the clarification :)
Yeah -- a product of Fedora Guidelines being written by different people at different times. We don't have strict RFC-like usage of should and may in most places. (must is usually unambiguous as to being a directive and a bolded "should", "must", or "may" has precise meaning. Other uses (or simple lack of any of those terms) leaves the question of how strict a question that would have to go back to the FPC to figure out what they meant in the past or what they want it to mean in the present day.)
- This means that packagers will be able to use the unversioned macros
throughout their specfile. Requires and BuildRequires will look like this:
Requires: %{default_python}-flask BuildRequires %{default_python}-setuptools
Note: since BuildRequires need to be expanded in the minimal buildroot, it's necessary to define the %default_python macro in the specfile. Other way would be to define it in a macro file that would be added to the minimal buildroot, but that's neither very likely nor very nice :)
Have you talked to dennis? We've added packages to the minimal BuildRoot many times before in order to enable macro files. Not so problematic if it's only one macro, though.
I will. Do you have any idea who I should talk to regarding EPEL? Is it also Dennis?
Also dennis. Rel-eng is growing as a group so you could also put in a ticket/go to one of their weekly meetings but Dennis is probably still the one that knows the most about koji and buildroots.
I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages.
Thoughts?
If we were to use different macro names instead of overwriting currently well known ones I think this has merit.
For me, introducing yet another set of macros is unnecessary. I think that it'd introduce a long(-ish) new part of guidelines that'll make them even more complicated than they are right now (compared to one new macro function and rules on how/when to use it).
Nick's breakdown of the three desired states seems like a non-complex way to organize things. And explicit being a good thing is also a winner for me. In addition to my original arguments that bashing existing macro names in some spec files but not in all of them is a bad thing to force packagers to have to deal with.
tangentially, when speaking about long-ish, though, could we use something shorter than default_python? Maybe syspython to follow Nick's usage of "System Python"?
-Toshio
----- Original Message -----
I will. Do you have any idea who I should talk to regarding EPEL? Is it also Dennis?
Also dennis. Rel-eng is growing as a group so you could also put in a ticket/go to one of their weekly meetings but Dennis is probably still the one that knows the most about koji and buildroots.
Thanks, I'll talk to him.
I think this solution makes sense for *applications* that need to be built both in Fedora and in EPEL. Note that it'd also align with my EPEL-python3 proposal [1], in case such an app was to be moved to python3X in EPEL (%default_python would just be redefined to "python3X"). I also think that this approach should never be allowed for packaging "libraries", e.g. packages that have python-foo and python3-foo subpackages.
Thoughts?
If we were to use different macro names instead of overwriting currently well known ones I think this has merit.
For me, introducing yet another set of macros is unnecessary. I think that it'd introduce a long(-ish) new part of guidelines that'll make them even more complicated than they are right now (compared to one new macro function and rules on how/when to use it).
Nick's breakdown of the three desired states seems like a non-complex way to organize things. And explicit being a good thing is also a winner for me. In addition to my original arguments that bashing existing macro names in some spec files but not in all of them is a bad thing to force packagers to have to deal with.
tangentially, when speaking about long-ish, though, could we use something shorter than default_python? Maybe syspython to follow Nick's usage of "System Python"?
I don't really feel strongly about this, I agree that your solution has a merit (and syspython *is* better than default_python :)). I think I'll open an FPC ticket to ask FPC for their opinions on this and then I'll start taking concrete steps [1] ;) to get this done.
-Toshio
Slavek
[1] http://www.sevenstarconcrete.com/images/exposed_double_stair_case.jpg
----- Original Message -----
I don't really feel strongly about this, I agree that your solution has a merit (and syspython *is* better than default_python :)). I think I'll open an FPC ticket to ask FPC for their opinions on this and then I'll start taking concrete steps [1] ;) to get this done.
Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498
-Toshio
Slavek
[1] http://www.sevenstarconcrete.com/images/exposed_double_stair_case.jpg
Slavek
On 4 February 2015 at 21:01, Bohuslav Kabrda bkabrda@redhat.com wrote:
----- Original Message -----
I don't really feel strongly about this, I agree that your solution has a merit (and syspython *is* better than default_python :)). I think I'll open an FPC ticket to ask FPC for their opinions on this and then I'll start taking concrete steps [1] ;) to get this done.
Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498
In commenting on the ticket, I realised something fundamental: we really do need to keep the RPM macro and the binary symlink in sync. That way the unversioned shebang lines in any packaged scripts can match up with the unversioned macros in the spec file.
Which seems to bring the viable options down to just two:
* switch the symlink to Python 3 as well (I don't like this due to the impact on end users with custom scripts) * hold off on switching the default for the time being and instead focus on enabling explicitly opting in to Python 3 in EPEL
Cheers, Nick.
On 5 February 2015 at 17:48, Nick Coghlan ncoghlan@gmail.com wrote:
On 4 February 2015 at 21:01, Bohuslav Kabrda bkabrda@redhat.com wrote:
----- Original Message -----
I don't really feel strongly about this, I agree that your solution has a merit (and syspython *is* better than default_python :)). I think I'll open an FPC ticket to ask FPC for their opinions on this and then I'll start taking concrete steps [1] ;) to get this done.
Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498
In commenting on the ticket, I realised something fundamental: we really do need to keep the RPM macro and the binary symlink in sync. That way the unversioned shebang lines in any packaged scripts can match up with the unversioned macros in the spec file.
Which seems to bring the viable options down to just two:
- switch the symlink to Python 3 as well (I don't like this due to the
impact on end users with custom scripts)
- hold off on switching the default for the time being and instead
focus on enabling explicitly opting in to Python 3 in EPEL
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
I came back to this question myself due to a couple of different ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19
* How does the situation in Fedora change if the upstream PEP 494 recommendation to downstream Linux distros was to change in conjunction with the Python 3.5 release currently scheduled for September? That release (https://www.python.org/dev/peps/pep-0478/) amongst other things, automatically handles EINTR errors for syscalls, restores binary interpolation support and adds matrix multiplication support and os.scandir().
* With the default interpreter change postponed to Fedora 23, would it make sense to patch the system Python in Fedora 22 to emit Python 3 warnings by default when it was run using the unqualified "python" reference rather than being run as the qualified "python2"? And then switch the symlink along with the RPM macros in Fedora 23?
It's also worth noting that the change I am considering to the upstream recommendation would place a qualifier on the distro providing a C.UTF-8 locale, so the "C.UTF-8 in upstream glibc" RFE (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a key enabler for making the symlink switch in Fedora (associated Fedora RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)
Regards, Nick.
Nick Coghlan ncoghlan@gmail.com schrieb am Do., 26. Feb. 2015 um 00:14 Uhr:
On 5 February 2015 at 17:48, Nick Coghlan ncoghlan@gmail.com wrote:
On 4 February 2015 at 21:01, Bohuslav Kabrda bkabrda@redhat.com wrote:
----- Original Message -----
I don't really feel strongly about this, I agree that your solution
has a
merit (and syspython *is* better than default_python :)). I think I'll
open
an FPC ticket to ask FPC for their opinions on this and then I'll start taking concrete steps [1] ;) to get this done.
Here's the FPC ticket: https://fedorahosted.org/fpc/ticket/498
In commenting on the ticket, I realised something fundamental: we really do need to keep the RPM macro and the binary symlink in sync. That way the unversioned shebang lines in any packaged scripts can match up with the unversioned macros in the spec file.
Which seems to bring the viable options down to just two:
- switch the symlink to Python 3 as well (I don't like this due to the
impact on end users with custom scripts)
- hold off on switching the default for the time being and instead
focus on enabling explicitly opting in to Python 3 in EPEL
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
I came back to this question myself due to a couple of different ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19
- How does the situation in Fedora change if the upstream PEP 494
recommendation to downstream Linux distros was to change in conjunction with the Python 3.5 release currently scheduled for September? That release (https://www.python.org/dev/peps/pep-0478/) amongst other things, automatically handles EINTR errors for syscalls, restores binary interpolation support and adds matrix multiplication support and os.scandir().
- With the default interpreter change postponed to Fedora 23, would it
make sense to patch the system Python in Fedora 22 to emit Python 3 warnings by default when it was run using the unqualified "python" reference rather than being run as the qualified "python2"? And then switch the symlink along with the RPM macros in Fedora 23?
Isn't it to late for Fedora 22 to emit such warnings by default? Iiuc, the screen output would change, when using /usr/bin/python, which could possibly break quite some things. So instead of a really badly "impact on end users with custom scripts", it might be one? This would ease the transition at least a little, so it _could_ be done...
Do you think those warning could be harmfull in Fedora 22? I'm undecided if these warnings could break more things than ease the Python 3 transition. Is changing the recommendation without the warnings an option?
Nevertheless, it would be great for Fedora, if the upstream recommendation changes with Python 3.5. This would confirm that the unversioned python can be assumed to be python3 everywhere and Fedora could more easily follow that route.
It's also worth noting that the change I am considering to the
upstream recommendation would place a qualifier on the distro providing a C.UTF-8 locale, so the "C.UTF-8 in upstream glibc" RFE (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a key enabler for making the symlink switch in Fedora (associated Fedora RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)
How does this impact the symlink switch? Unfortunately, that's not clear to me... Am I overlooking something?
Regards, Thomas
On Feb 25, 2015 3:14 PM, "Nick Coghlan" ncoghlan@gmail.com wrote:
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
I came back to this question myself due to a couple of different ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19
- How does the situation in Fedora change if the upstream PEP 494
recommendation to downstream Linux distros was to change in conjunction with the Python 3.5 release currently scheduled for September? That release (https://www.python.org/dev/peps/pep-0478/) amongst other things, automatically handles EINTR errors for syscalls, restores binary interpolation support and adds matrix multiplication support and os.scandir().
I would be against making the switch to /usr/bin/python at this time but would do most of that fighting upstream. If the pep were updated then I'd at least want to see that the other major distros are committed to changing their /usr/bin/python at the same time.
Changing the behavior of a well known program like this is a bad idea. As it breaks compatibility: with people's home grown scripts, with their self installed programs, and between os's and os releases. The pep is palatable because the arguments in favor of someday changing the value revolve around someday there not being a /usr/bin/python on most systems. At that point it becomes reasonable to reallocate /usr/bin/python and let the systems where /usr/bin/python be declared legacy and the behavior of /usr/bin/python on those legacy systems is the quirk, not ours.
We could cut over sooner than this argument actually makes the case for but now is definitely not that day. Fedora, rhel, ubuntu, aren't yet at the point where /usr/bin/python isn't present on most of their installed systems in their latest version, let alone all of their versions.people are still pulling /usr/bin/python onto their systems through dependencies for common applications even if their os is advanced enough not to need it by default. We have quite a ways to go before /usr/bin/python can be switched.
- With the default interpreter change postponed to Fedora 23, would it
make sense to patch the system Python in Fedora 22 to emit Python 3 warnings by default when it was run using the unqualified "python" reference rather than being run as the qualified "python2"? And then switch the symlink along with the RPM macros in Fedora 23?
No to switching the value of /usr/bin/python and stated above. The test makes some sense. If your warning is restricted to warning not to use /usr/bin/python (use /usr/bin/python 2 instead) that sounds really good to me. (Your wording sounded like we should turn on warnings like python2 -3 does which I don't think is such a good idea for fedora 22 but might be good in the future... our perhaps, like the kernel does with extra kernel debugging, we should turn it on in rawhide and fedora.n+1 but turn it off before release.)
It's also worth noting that the change I am considering to the upstream recommendation would place a qualifier on the distro providing a C.UTF-8 locale, so the "C.UTF-8 in upstream glibc" RFE (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a key enabler for making the symlink switch in Fedora (associated Fedora RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)
Like tomspur I'm not sure I see the specific relevance of this to what /usr/bin/python invokes although I would welcome the change :-)
-Toshio
On 27 February 2015 at 11:02, Toshio Kuratomi a.badger@gmail.com wrote:
On Feb 25, 2015 3:14 PM, "Nick Coghlan" ncoghlan@gmail.com wrote:
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
I came back to this question myself due to a couple of different ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19
- How does the situation in Fedora change if the upstream PEP 494
recommendation to downstream Linux distros was to change in conjunction with the Python 3.5 release currently scheduled for September? That release (https://www.python.org/dev/peps/pep-0478/) amongst other things, automatically handles EINTR errors for syscalls, restores binary interpolation support and adds matrix multiplication support and os.scandir().
I would be against making the switch to /usr/bin/python at this time but would do most of that fighting upstream. If the pep were updated then I'd at least want to see that the other major distros are committed to changing their /usr/bin/python at the same time.
Changing the behavior of a well known program like this is a bad idea. As it breaks compatibility: with people's home grown scripts, with their self installed programs, and between os's and os releases. The pep is palatable because the arguments in favor of someday changing the value revolve around someday there not being a /usr/bin/python on most systems. At that point it becomes reasonable to reallocate /usr/bin/python and let the systems where /usr/bin/python be declared legacy and the behavior of /usr/bin/python on those legacy systems is the quirk, not ours.
We could cut over sooner than this argument actually makes the case for but now is definitely not that day. Fedora, rhel, ubuntu, aren't yet at the point where /usr/bin/python isn't present on most of their installed systems in their latest version, let alone all of their versions.people are still pulling /usr/bin/python onto their systems through dependencies for common applications even if their os is advanced enough not to need it by default. We have quite a ways to go before /usr/bin/python can be switched.
Yeah, that's fair - a staggered release with the distros switching first before end user scripts makes sense. That would make the likely target date for a PEP 394 update some time in early 2017 with Python 3.6.
We could *potentially* switch the recommendation some time in 2016 if all goes well with the distro migrations and significant libraries start dropping Python 2.7 support, but switching in conjunction with Python 3.5 would still be too soon.
- With the default interpreter change postponed to Fedora 23, would it
make sense to patch the system Python in Fedora 22 to emit Python 3 warnings by default when it was run using the unqualified "python" reference rather than being run as the qualified "python2"? And then switch the symlink along with the RPM macros in Fedora 23?
No to switching the value of /usr/bin/python and stated above. The test makes some sense. If your warning is restricted to warning not to use /usr/bin/python (use /usr/bin/python 2 instead) that sounds really good to me. (Your wording sounded like we should turn on warnings like python2 -3 does which I don't think is such a good idea for fedora 22 but might be good in the future... our perhaps, like the kernel does with extra kernel debugging, we should turn it on in rawhide and fedora.n+1 but turn it off before release.)
I did mean the latter (turning on -3 warnings), but I like your idea of only doing that in Rawhide and the Alpha releases for F23, and then switching to a simple "use a qualified Python reference" warning in the Betas and the actual release.
It's also worth noting that the change I am considering to the upstream recommendation would place a qualifier on the distro providing a C.UTF-8 locale, so the "C.UTF-8 in upstream glibc" RFE (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a key enabler for making the symlink switch in Fedora (associated Fedora RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)
Like tomspur I'm not sure I see the specific relevance of this to what /usr/bin/python invokes although I would welcome the change :-)
Being able to type "LANG=C.UTF-8" instead of "LANG=C" fixes some of the odd corner cases where the interpreter startup sequence gets confused. However, I remembered that subtle issues aren't the ones we're worried about here - it's the big noisy ones like legacy print and exec statements.
Cheers, Nick.
----- Original Message -----
On 27 February 2015 at 11:02, Toshio Kuratomi a.badger@gmail.com wrote:
On Feb 25, 2015 3:14 PM, "Nick Coghlan" ncoghlan@gmail.com wrote:
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
I came back to this question myself due to a couple of different ideas, discussed in https://fedorahosted.org/fpc/ticket/498#comment:19
- How does the situation in Fedora change if the upstream PEP 494
recommendation to downstream Linux distros was to change in conjunction with the Python 3.5 release currently scheduled for September? That release (https://www.python.org/dev/peps/pep-0478/) amongst other things, automatically handles EINTR errors for syscalls, restores binary interpolation support and adds matrix multiplication support and os.scandir().
I would be against making the switch to /usr/bin/python at this time but would do most of that fighting upstream. If the pep were updated then I'd at least want to see that the other major distros are committed to changing their /usr/bin/python at the same time.
Changing the behavior of a well known program like this is a bad idea. As it breaks compatibility: with people's home grown scripts, with their self installed programs, and between os's and os releases. The pep is palatable because the arguments in favor of someday changing the value revolve around someday there not being a /usr/bin/python on most systems. At that point it becomes reasonable to reallocate /usr/bin/python and let the systems where /usr/bin/python be declared legacy and the behavior of /usr/bin/python on those legacy systems is the quirk, not ours.
We could cut over sooner than this argument actually makes the case for but now is definitely not that day. Fedora, rhel, ubuntu, aren't yet at the point where /usr/bin/python isn't present on most of their installed systems in their latest version, let alone all of their versions.people are still pulling /usr/bin/python onto their systems through dependencies for common applications even if their os is advanced enough not to need it by default. We have quite a ways to go before /usr/bin/python can be switched.
Yeah, that's fair - a staggered release with the distros switching first before end user scripts makes sense. That would make the likely target date for a PEP 394 update some time in early 2017 with Python 3.6.
We could *potentially* switch the recommendation some time in 2016 if all goes well with the distro migrations and significant libraries start dropping Python 2.7 support, but switching in conjunction with Python 3.5 would still be too soon.
+1 to switch with Python 3.6. PEP 394 should however be made to reflect this ASAP - I mean it should be made to say that this change will happen with Python 3.6. The sooner it says that, the more time for people to notice it and more time for distros to prepare.
- With the default interpreter change postponed to Fedora 23, would it
make sense to patch the system Python in Fedora 22 to emit Python 3 warnings by default when it was run using the unqualified "python" reference rather than being run as the qualified "python2"? And then switch the symlink along with the RPM macros in Fedora 23?
No to switching the value of /usr/bin/python and stated above. The test makes some sense. If your warning is restricted to warning not to use /usr/bin/python (use /usr/bin/python 2 instead) that sounds really good to me. (Your wording sounded like we should turn on warnings like python2 -3 does which I don't think is such a good idea for fedora 22 but might be good in the future... our perhaps, like the kernel does with extra kernel debugging, we should turn it on in rawhide and fedora.n+1 but turn it off before release.)
I did mean the latter (turning on -3 warnings), but I like your idea of only doing that in Rawhide and the Alpha releases for F23, and then switching to a simple "use a qualified Python reference" warning in the Betas and the actual release.
I'm also +1 on emitting a warning about /usr/bin/python usage, although I'm almost sure that will break something. There are always apps that expect certain form of subprocess output etc - and this will break them. IMO this should only be done in F23, not in F22 which is already in alpha. I'm assuming that there is no builtin configure option to emit this warning and we'd have to patch this ourselves, is that right or have I just missed such option?
It's also worth noting that the change I am considering to the upstream recommendation would place a qualifier on the distro providing a C.UTF-8 locale, so the "C.UTF-8 in upstream glibc" RFE (https://sourceware.org/bugzilla/show_bug.cgi?id=17318) would become a key enabler for making the symlink switch in Fedora (associated Fedora RFE: https://bugzilla.redhat.com/show_bug.cgi?id=902094)
Like tomspur I'm not sure I see the specific relevance of this to what /usr/bin/python invokes although I would welcome the change :-)
Being able to type "LANG=C.UTF-8" instead of "LANG=C" fixes some of the odd corner cases where the interpreter startup sequence gets confused. However, I remembered that subtle issues aren't the ones we're worried about here - it's the big noisy ones like legacy print and exec statements.
FWIW, I would also welcome this change (and yeah, "I want my print statement back" is the most often complain that I've heard so far ;))
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks, Slavek
On 26.2.2015 00:13, Nick Coghlan wrote:
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
This thing is in Russian (while I guess it is actually about Python and I see some Fedora links in there)
On 27.2.2015 21:06, Miro Hrončok wrote:
On 26.2.2015 00:13, Nick Coghlan wrote:
For those not following along with the FPC ticket, Toshio and Tomspur have a nice write-up of the options at https://etherpad.mozilla.org/2Uqk0ikCll
This thing is in Russian (while I guess it is actually about Python and I see some Fedora links in there)
Oh it seems only in the latest revision someone took the courtesy of translating it all to Russian.
python-devel@lists.fedoraproject.org
-
Bohuslav Kabrda -
Konstantin Zemlyak -
Miro Hrončok -
Nick Coghlan -
Thomas Spura -
Toshio Kuratomi