Hello Python-SIG,
Currently I am working on some ideas regarding the python package.
Nothing is too concrete yet, however the technical side should be fairly simple.
The first thing is renaming python to python2.
Currently the python packaging guidelines, suggest that we use the python_provides macro, for both python2 and python3 subpackages of whatever python module someone might be packaging. The macro for python3 subacpakages, currently, does nothing however sometime in the future it would be great if everything could switch to python 3. When this is done we can flip the switch in the macros and the python3 subpackage will also provide the python-<module> namespace, meaning essentially that when we speak about python we talk about python3, and the python binary will be a symlink to python3. In order to slowly start the procedure, a first step would be to rename the python package to python2.
Here is a github repo with the current work so far: https://github.com/fedora-python/python2-spec/tree/py2rename
Would love to hear some feedback on that.
Regards,
Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat
On 31 August 2016 at 23:54, Charalampos Stratakis cstratak@redhat.com wrote:
Hello Python-SIG,
Currently I am working on some ideas regarding the python package.
Nothing is too concrete yet, however the technical side should be fairly simple.
The first thing is renaming python to python2.
Moving "python" to a virtual provides (currently provided by python2) makes sense to me as a starting point.
Currently the python packaging guidelines, suggest that we use the python_provides macro, for both python2 and python3 subpackages of whatever python module someone might be packaging. The macro for python3 subacpakages, currently, does nothing however sometime in the future it would be great if everything could switch to python 3. When this is done we can flip the switch in the macros and the python3 subpackage will also provide the python-<module> namespace, meaning essentially that when we speak about python we talk about python3, and the python binary will be a symlink to python3.
For /usr/bin/python, we had an upstream discussion about that at the 2015 language summit: https://lwn.net/Articles/640296/
The ideal point we'd like to get to is one where all distro provided scripts actually have the appropriate major version in their shebang lines, and the unqualifed "python" is something along the lines of a user-configurable launcher, akin to the "py" launcher for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows (see https://www.python.org/dev/peps/pep-0397/ for more details on that)
Debian's Geoffrey Thomas actually wrote a proof of concept launcher that implements something along those lines (described at https://ldpreload.com/blog/usr-bin-python-23 ), and as far as I know the main blocker to it going any further has been finding someone that has the time, energy, and inclination to pursue including it in a distro as the official implementation of "/usr/bin/python", as well as proposing it upstream as an amendment to the distro recommendations in PEP 394.
The nice thing about the design of pythonmux is that, if Python 2 is installed, it will use it automatically in non-interactive mode for maximum compatibility with existing scripts, but if only Python 3 is available, it will implicitly try that, rather than failing outright the way a missing symlink will.
Cheers, Nick.
On Thu, Sep 1, 2016 at 7:44 AM, Nick Coghlan ncoghlan@gmail.com wrote:
The ideal point we'd like to get to is one where all distro provided scripts actually have the appropriate major version in their shebang lines, and the unqualifed "python" is something along the lines of a user-configurable launcher, akin to the "py" launcher for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows (see https://www.python.org/dev/peps/pep-0397/ for more details on that)
While I see some benefit to something like this and could see use cases for including it in the shebang, it would be overkill for "/usr/bin/python". Windows needs something like that because it doesn't support symlinks.
Let's stop trying to reinvent the wheel. There is already a system for handling this, alternatives, which is used by other languages and included in a minimal Fedora install. It would only require adding to the %post and %postun scripts in the spec file and allows very easy testing to see what breaks when changing the default. When we are ready to change the default, only one value needs to be changed. And most importantly, it gives the end-user an easy way to change their system-wide default either from python 2 to 3 or when running a pre-release version of 3.x in parallel.
Avram
On Thu, Sep 1, 2016 at 8:04 AM, Avram Lubkin aviso@rockhopper.net wrote:
On Thu, Sep 1, 2016 at 7:44 AM, Nick Coghlan ncoghlan@gmail.com wrote:
The ideal point we'd like to get to is one where all distro provided scripts actually have the appropriate major version in their shebang lines, and the unqualifed "python" is something along the lines of a user-configurable launcher, akin to the "py" launcher for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows (see https://www.python.org/dev/peps/pep-0397/ for more details on that)
While I see some benefit to something like this and could see use cases for including it in the shebang, it would be overkill for "/usr/bin/python". Windows needs something like that because it doesn't support symlinks.
Let's stop trying to reinvent the wheel. There is already a system for handling this, alternatives, which is used by other languages and included in a minimal Fedora install. It would only require adding to the %post and %postun scripts in the spec file and allows very easy testing to see what breaks when changing the default. When we are ready to change the default, only one value needs to be changed. And most importantly, it gives the end-user an easy way to change their system-wide default either from python 2 to 3 or when running a pre-release version of 3.x in parallel.
Alternatives doesn't work in this case because Python 2.x and Python 3.x versions don't share resources, even with minor versions of the same series. Enabling alternatives for it is a recipe for disaster.
On Thu, Sep 1, 2016 at 8:14 AM, Neal Gompa ngompa13@gmail.com wrote:
Alternatives doesn't work in this case because Python 2.x and Python 3.x versions don't share resources, even with minor versions of the same series. Enabling alternatives for it is a recipe for disaster.
Could you elaborate? The use case we are talking about is what gets executed when you run /usr/bin/python. i.e. symlink management, which is what alternatives is intended for.
On Thu, Sep 1, 2016 at 8:33 AM, Avram Lubkin aviso@rockhopper.net wrote:
On Thu, Sep 1, 2016 at 8:14 AM, Neal Gompa ngompa13@gmail.com wrote:
Alternatives doesn't work in this case because Python 2.x and Python 3.x versions don't share resources, even with minor versions of the same series. Enabling alternatives for it is a recipe for disaster.
Could you elaborate? The use case we are talking about is what gets executed when you run /usr/bin/python. i.e. symlink management, which is what alternatives is intended for.
Sure, but those scripts may not actually work because modules that are supposed to be there, aren't. For example, if you depend on a non-standard lib module, then that means it needs to be installed for each python version supported. How do you expect to guarantee that?
Again, alternatives is only for things that operate functionally identically. That's not even true between Python 2.6 and Python 2.7 if they are installed in parallel because they can't use the same module path. Likewise for Python 3.5 and 3.6.
You're essentially asking for unpredictable breakage.
On Thu, Sep 1, 2016 at 8:40 AM, Neal Gompa ngompa13@gmail.com wrote:
Sure, but those scripts may not actually work because modules that are supposed to be there, aren't. For example, if you depend on a non-standard lib module, then that means it needs to be installed for each python version supported. How do you expect to guarantee that?
Again, alternatives is only for things that operate functionally identically. That's not even true between Python 2.6 and Python 2.7 if they are installed in parallel because they can't use the same module path. Likewise for Python 3.5 and 3.6.
You're essentially asking for unpredictable breakage.
I would argue what "/usr/bin/python" points to should be a user option and what is in the shebang line is a packager issues. As a user, I should be able to select whatever version of Python I want. As a packager, I should make sure my package will run with the correct version. This is easier when we're only talking about major versions, but we could add an rpm macro that would point to the minor version executable if there was a need.
Something like pythonmux might be useful in the shebang for this purpose, but "/usr/bin/python" shouldn't be linked to it.
On 1 September 2016 at 22:04, Avram Lubkin aviso@rockhopper.net wrote:
On Thu, Sep 1, 2016 at 7:44 AM, Nick Coghlan ncoghlan@gmail.com wrote:
The ideal point we'd like to get to is one where all distro provided scripts actually have the appropriate major version in their shebang lines, and the unqualifed "python" is something along the lines of a user-configurable launcher, akin to the "py" launcher for Windows: https://docs.python.org/3/using/windows.html#python-launcher-for-windows (see https://www.python.org/dev/peps/pep-0397/ for more details on that)
While I see some benefit to something like this and could see use cases for including it in the shebang, it would be overkill for "/usr/bin/python". Windows needs something like that because it doesn't support symlinks.
This isn't about Windows, this is about making Fedora a sufficiently user friendly platform for Python developers that they don't feel obliged to reach for third party runtime managers like pyenv and conda (which, make no mistake, a great many of them already do).
We (wearing my upstream CPython hat for this post) would like developer focused Linux distros to either put individual end users fully in control of what happens when they type "python" at the shell (or run a script with "#!/usr/bin/python" as its shebang line), or at least give system administrators the ability to configure that on a per-user basis. (There are some other commands like pypy and ipython that could use a similar treatment, but if we can solve this for CPython, then it becomes easier to apply a similar solution in other contexts)
One key advantage this approach offers is that system administrators can upgrade selected users to "Python 3 by default", while still keeping Python 2 as the default for their own accounts and for service accounts (e.g. Ansible's remote access). "You must upgrade entire systems to Python 3 all at once" simply doesn't give people enough granularity to do a properly managed transition (cf. the Ansible bootstrapping problems when Python 2 was first dropped from the Fedora base install).
Folks that don't want the behaviour of their scripts to be dependent on a per-user settting should be using the symlinks qualified with the major version rather than the unqualified path (which is already the recommendation in PEP 394, and has been for years, since the meaning of an unqualified Python reference can vary by distro).
Let's stop trying to reinvent the wheel. There is already a system for handling this, alternatives, which is used by other languages and included in a minimal Fedora install.
Unfortunately, the alternatives system also requires administrator access to configure and is also a global setting affecting the entire system, so it doesn't solve the fundamental usability problem associated with neither individual users nor system administrators being able to reliably configure different options for what particular users get when they type "python" into a shell session or run a script with "/usr/bin/python" in the shebang line. (Hence the line in https://lwn.net/Articles/640296/ about "The 'alternatives' mechanism is another possibility, but that makes more sense for things like choosing an editor than it does for choosing a version of Python.")
I'll acknowledge that putting `python` under alternatives would be *better* than the status quo of having it globally locked to a distro-defined version (since it would at least get us to "admins in control", even thought it doesn't get us all the way to putting the user in control of their own developer experience), but it still gets the user experience wrong, as a particular group of users shouldn't have to have a shared definition of what 'python' means just because they all have accounts on the same server, or run on the same Standard Operating Environment.
? It would only require adding to the %post and
%postun scripts in the spec file and allows very easy testing to see what breaks when changing the default. When we are ready to change the default, only one value needs to be changed. And most importantly, it gives the end-user an easy way to change their system-wide default either from python 2 to 3 or when running a pre-release version of 3.x in parallel.
Switching to putting /usr/bin/python under alternatives would definitely be a better option than unilaterally switching it to Python 3. However, I'm also confident that it wouldn't be a significant enough change to slow the rate of adoption of non-distro Python versions, such as those published for pyenv and conda (note that my assumption is that we have a shared goal of wanting Python developers to find value in using the Fedora curated Python runtime binaries over those provided by other distros, or by distro-independent publishers).
Cheers, Nick.
On 09/01/2016 01:44 PM, Nick Coghlan wrote:
For /usr/bin/python, we had an upstream discussion about that at the 2015 language summit: https://lwn.net/Articles/640296/
Whatever we do, I'd like it to be coordinated across multiple distrubutions, and blesed by a PEP like [PEP 394]. We don't want a solution specific to Fedora. Unless you don't agree with that, a good place to discuss these issues it Python's [linux-sig].
Now, the names of Fedora's packages are a different thing – I agree that "python" should be renamed to "python2".
[PEP 394]: https://www.python.org/dev/peps/pep-0394/ [linux-sig]: https://mail.python.org/mailman/listinfo/linux-sig
On Thu, Sep 1, 2016 at 10:22 AM, Petr Viktorin pviktori@redhat.com wrote:
Whatever we do, I'd like it to be coordinated across multiple distrubutions, and blesed by a PEP like [PEP 394]. We don't want a solution specific to Fedora.
Completely agree. Based on the suggestions offered in PEP 394, we would need to add idle2 and pydoc2 symlinks in the python spec.
The PEP also reiterates that the focus is on the end-user and not on the distribution packages. It's very clear that the shebang should be set upstream to the level of specificity required.
I think the best way to serve the end-user is to manage /usr.bin/python with alternatives, with slave entries for idle, pydoc, python-config, and the python man page. This is compliant with the suggestions of the PEP and gives control to the user. The symlinks should point to the major version links/files as minor versions may introduce too many issues to address at the moment.
On 2 September 2016 at 02:29, Avram Lubkin aviso@rockhopper.net wrote:
On Thu, Sep 1, 2016 at 10:22 AM, Petr Viktorin pviktori@redhat.com wrote:
Whatever we do, I'd like it to be coordinated across multiple distrubutions, and blesed by a PEP like [PEP 394]. We don't want a solution specific to Fedora.
We do need to figure out the scope of what would be acceptable within the context of Fedora though - otherwise we could reach consensus upstream on Python's linux-sig, and then find we couldn't get the proposal through FESCo downstream.
I think the best way to serve the end-user is to manage /usr.bin/python with alternatives, with slave entries for idle, pydoc, python-config, and the python man page. This is compliant with the suggestions of the PEP and gives control to the user. The symlinks should point to the major version links/files as minor versions may introduce too many issues to address at the moment.
Summarising my longer reply to your other email about this, I think alternatives is a good fallback position that should be acceptable cross-distro if we can't come up with (and agree on) anything better.
Unfortunately, alternatives doesn't actually meet the user needs we've had described upstream, which requests the ability to do *per-user* selective upgrades, allowing admins to initially restrict deployment of "Python 3 as default" to users that they're pretty sure can cope with the possible resulting incompatibilities without too much handholding, and then progressively roll it out to more users over time as the compatibility issues get ironed out for their particular environment.
There's also clear migration value in allowing administrators to say "These particular service accounts still need to live in a Python 2 only world, blissfully unaware that Python 3 is even a thing", and then whittle those away over time, rather than needing to migrate all their custom admin scripts before even one user gets upgraded to Python 3 as their default Python.
Putting it like that though, perhaps an upstream proposal for multi-tier configurability support would make sense?
* Tier 0: "the default Python" is determined at the distribution level. For the majority of current distributions, that means Python 2, but on a select few it means Python 3.
* Tier 1: "the default Python" is configurable on a per-system level. Distros pursuing this option (e.g. by moving /usr/bin/python under 'alternatives') are still advised to keep the out-of-the-box default as Python 2. This will be sufficient for at least some environments to start managed migrations from Python-2-as-default to Python-3-as-default, and further establishes the precedent that "/usr/bin/python will always be Python 2" is an incorrect assumption.
* Tier 2: "the default Python" is configurable on a per-user level. Sufficiently determined sysadmins can actually configure this today, so it would mainly be a matter of looking at what's involved in doing that, and making it simpler for people to set up.
* Tier 3 (highly experimental): "the default Python" is configurable on a per-script level, using something like pythonmux.
If we pitched something like that, then rather than distros having to agree on which level of configurability was appropriate, we'd just have to agree on "if we implement that tier, we're happy to abide by those guidelines", and then it would be up to individual distros to decide which level of configurability they wanted to offer (and they may make different choices when it came to interactive use vs non-interactive use).
Cheers, Nick.
On Sat, Sep 3, 2016 at 2:32 PM, Nick Coghlan ncoghlan@gmail.com wrote:
- Tier 0: "the default Python" is determined at the distribution
level. For the majority of current distributions, that means Python 2, but on a select few it means Python 3.
- Tier 1: "the default Python" is configurable on a per-system level.
Distros pursuing this option (e.g. by moving /usr/bin/python under 'alternatives') are still advised to keep the out-of-the-box default as Python 2. This will be sufficient for at least some environments to start managed migrations from Python-2-as-default to Python-3-as-default, and further establishes the precedent that "/usr/bin/python will always be Python 2" is an incorrect assumption.
- Tier 2: "the default Python" is configurable on a per-user level.
Sufficiently determined sysadmins can actually configure this today, so it would mainly be a matter of looking at what's involved in doing that, and making it simpler for people to set up.
- Tier 3 (highly experimental): "the default Python" is configurable
on a per-script level, using something like pythonmux.
I think that is a great approach!
For Teir 2:
Today, a user can alias python for direct execution. For scripts that use /use/bin/env, they can manipulate their $PATH (and create symlinks). Some of that could be automated.
For Teir 3:
Ideally this would be something that evaluated a list of preferred interpreters given on the shebang, but defaulted to the user/system default. It should also work across multiple python interpreters like pypy or jython and not just versions of Cpython.
This could be a non-Python-specific solution. What if there was a utility, like env, that searched python in $PATH, but could take multiple commands? Then you might have a shebang like:
#! /usr/bin/envmux pypy3 pypy python3.5 python3 python2.7 python
It would probably also need a way to pass options to specific interpreters, so maybe syntax like this?
#! /usr/bin/envmux python3 "[-b]" python2 python
It could even handle user and system defaults by allowing a keypair file in the user's home directory and in etc. Something like:
~/.envmux --------------- python : /usr/bin/python3.5
/etc/envmux ------------------ python : /usr/bin/python2.7
In this example the system default would be Python2.7, but the user default would be Python 3.5. To handle the python command, python could be aliased to "/usr/bin/envmux python".
I think a solution like this is more flexible than pythonmux and would also be valuable to non-Python implementations.
The downside is you can't directly say a script required Python 2.6+ or 3.3+, you would have to list out all the versions, but the solutions today don't handle that either, and that is probably better handled within the script to give more information to the user.
Avram
On 3 September 2016 at 14:32, Nick Coghlan ncoghlan@gmail.com wrote:
On 2 September 2016 at 02:29, Avram Lubkin aviso@rockhopper.net wrote:
On Thu, Sep 1, 2016 at 10:22 AM, Petr Viktorin pviktori@redhat.com wrote:
Whatever we do, I'd like it to be coordinated across multiple distrubutions, and blesed by a PEP like [PEP 394]. We don't want a solution specific to Fedora.
We do need to figure out the scope of what would be acceptable within the context of Fedora though - otherwise we could reach consensus upstream on Python's linux-sig, and then find we couldn't get the proposal through FESCo downstream.
I think the best way to serve the end-user is to manage /usr.bin/python with alternatives, with slave entries for idle, pydoc, python-config, and the python man page. This is compliant with the suggestions of the PEP and gives control to the user. The symlinks should point to the major version links/files as minor versions may introduce too many issues to address at the moment.
Summarising my longer reply to your other email about this, I think alternatives is a good fallback position that should be acceptable cross-distro if we can't come up with (and agree on) anything better.
Unfortunately, alternatives doesn't actually meet the user needs we've had described upstream, which requests the ability to do *per-user* selective upgrades, allowing admins to initially restrict deployment of "Python 3 as default" to users that they're pretty sure can cope with the possible resulting incompatibilities without too much handholding, and then progressively roll it out to more users over time as the compatibility issues get ironed out for their particular environment.
There's also clear migration value in allowing administrators to say "These particular service accounts still need to live in a Python 2 only world, blissfully unaware that Python 3 is even a thing", and then whittle those away over time, rather than needing to migrate all their custom admin scripts before even one user gets upgraded to Python 3 as their default Python.
Putting it like that though, perhaps an upstream proposal for multi-tier configurability support would make sense?
- Tier 0: "the default Python" is determined at the distribution
level. For the majority of current distributions, that means Python 2, but on a select few it means Python 3.
- Tier 1: "the default Python" is configurable on a per-system level.
Distros pursuing this option (e.g. by moving /usr/bin/python under 'alternatives') are still advised to keep the out-of-the-box default as Python 2. This will be sufficient for at least some environments to start managed migrations from Python-2-as-default to Python-3-as-default, and further establishes the precedent that "/usr/bin/python will always be Python 2" is an incorrect assumption.
- Tier 2: "the default Python" is configurable on a per-user level.
Sufficiently determined sysadmins can actually configure this today, so it would mainly be a matter of looking at what's involved in doing that, and making it simpler for people to set up.
So the general way that 'configurable on per-user level' would be to put to put links in a directory that over-rode the system path files.. eg
~/.userdefined_bin/:/usr/bin:/usr/local/bin:~/bin
This has generally been considered a security problem in that anything dropped in ~/.userdefined_bin can override defaults without the user knowing it. [The old ~/.userdefined_bin/mv -> /usr/bin/rm or a script which is called ls and then rms the files after showing them.] However I think we are in a world of such security nightmares just from the browser being used that this is just par for the course.
- Tier 3 (highly experimental): "the default Python" is configurable
on a per-script level, using something like pythonmux.
If we pitched something like that, then rather than distros having to agree on which level of configurability was appropriate, we'd just have to agree on "if we implement that tier, we're happy to abide by those guidelines", and then it would be up to individual distros to decide which level of configurability they wanted to offer (and they may make different choices when it came to interactive use vs non-interactive use).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject...
python-devel@lists.fedoraproject.org
-
Avram Lubkin -
Charalampos Stratakis -
Neal Gompa -
Nick Coghlan -
Petr Viktorin -
Stephen John Smoogen