Changes in Guidelines Connected to "Python 3 as Default" Change

[Bohuslav Kabrda]

Hi all,
as you are well aware, there is the "Python 3 as Default" change proposed for F22 [1]. There are several guideline changes that will need to be done and some that may be done in order to improve the situation. I'd like to make this thread a sort of brainstorming on my thoughts as well as possibility for anyone to add something that should be done as well. After that, I'm planning to open an FPC ticket with all the gathered and discussed changes that will come up on this thread.

So here are my proposals for changes in current guidelines [2]:
- In [3], it says "If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package." - this should be changed to prefer Python 3
- (This is not really related to the switch, but more of a general remark) In [4], it says that "python 3 version of the executable gains a python3- prefix". This is IMO bad, since upstream projects tend to name the versioned binaries "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one of these - I'm not really certain which one of them. I tried to discuss this several times on distutils-sig mailing list, but without reaching a consensus. Either way, prefixing with python3- doesn't make sense to me, because it's not similar to any upstream way and you don't find the binaries under their names using tab completion (e.g. foo<tab> doesn't tell you about python3-foo).
- As for binaries/scripts in /usr/bin (assuming there are both python2 and python3 versions), the unversioned files should point to python2 version. This aligns with /usr/bin/python still pointing to python2.
- Some time ago, I also put together a proposal for some larger changes in Python packaging [5], mostly in how the subpackages for different interpreters should be done. I opened an FPC ticket [6] to get some comments and it seemed that FPC was favorable. While I still think it'd be great to do this change, it'll require a significant effort and that is better spent helping upstreams to port to Python 3 ATM. So I'd also like to receive more comments on this proposal, although I think we should postpone it to F23 (or maybe even later).

Thanks for all your comments and further suggestions.

Slavek

[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default
[2] http://fedoraproject.org/wiki/Packaging:Python
[3] http://fedoraproject.org/wiki/Packaging:Python#Guidelines
[4] http://fedoraproject.org/wiki/Packaging:Python#Naming
[5] https://fedoraproject.org/wiki/User:Bkabrda/Py2to3GuidelineChanges
[6] https://fedorahosted.org/fpc/ticket/379

[Nick Coghlan]

On 4 Dec 2014 00:38, "Bohuslav Kabrda"  wrote:
> So here are my proposals for changes in current guidelines [2]:
> In [3], it says "If the executables provide the same functionality

independent of whether they are run on top of Python 2 or Python 3, then
only one version of the executable should be packaged. Currently it will be
the python 2 implementation, but once the Python 3 implementation is proven
to work, the executable can be retired from the python 2 build and enabled
in the python 3 package." - this should be changed to prefer Python 3

Agreed.
> (This is not really related to the switch, but more of a general

remark) In [4], it says that "python 3 version of the executable gains a
python3- prefix". This is IMO bad, since upstream projects tend to name the
versioned binaries "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one
of these - I'm not really certain which one of them. I tried to discuss
this several times on distutils-sig mailing list, but without reaching a
consensus. Either way, prefixing with python3- doesn't make sense to me,
because it's not similar to any upstream way and you don't find the
binaries under their names using tab completion (e.g. foo<tab> doesn't tell
you about python3-foo).

Agreed.

CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a
reason to use that convention by default. We may want a "unless upstream
does it differently" caveat though.
> As for binaries/scripts in /usr/bin (assuming there are both python2

and python3 versions), the unversioned files should point to python2
version. This aligns with /usr/bin/python still pointing to python2.

This also aligns with CPython & pip conventions. Between them, only
"pyvenv" runs under Python 3 by default, and that's only because it doesn't
exist in Python 2.
> Some time ago, I also put together a proposal for some larger changes

in Python packaging [5], mostly in how the subpackages for different
interpreters should be done. I opened an FPC ticket [6] to get some
comments and it seemed that FPC was favorable. While I still think it'd be
great to do this change, it'll require a significant effort and that is
better spent helping upstreams to port to Python 3 ATM. So I'd also like to
receive more comments on this proposal, although I think we should postpone
it to F23 (or maybe even later).

I haven't reviewed this part yet.

Regards,
Nick.
> Thanks for all your comments and further suggestions.
>
> Slavek
>
> [1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default
> [2] http://fedoraproject.org/wiki/Packaging:Python
> [3] http://fedoraproject.org/wiki/Packaging:Python#Guidelines
> [4] http://fedoraproject.org/wiki/Packaging:Python#Naming
> [5] https://fedoraproject.org/wiki/User:Bkabrda/Py2to3GuidelineChanges
> [6] https://fedorahosted.org/fpc/ticket/379
> _______________________________________________
> python-devel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/python-devel

[Donald Stufft]

> On Dec 3, 2014, at 9:51 AM, Nick Coghlan  wrote:
> > (This is not really related to the switch, but more of a general remark) In [4], it says that "python 3 version of the executable gains a python3- prefix". This is IMO bad, since upstream projects tend to name the versioned binaries "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one of these - I'm not really certain which one of them. I tried to discuss this several times on distutils-sig mailing list, but without reaching a consensus. Either way, prefixing with python3- doesn't make sense to me, because it's not similar to any upstream way and you don't find the binaries under their names using tab completion (e.g. foo<tab> doesn't tell you about python3-foo).
>
> Agreed.
>
> CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a reason to use that convention by default. We may want a "unless upstream does it differently" caveat though.

It doesn't really matter right now but long term I think python packaging should just natively support commands like this. Either just as a matter of fact, opt in, or by allowing templated command names. Either way I think the upstream tooling should and likely will follow python's lead for how these are written.

[Nick Coghlan]

On 4 Dec 2014 00:59, "Donald Stufft"  wrote:
> On Dec 3, 2014, at 9:51 AM, Nick Coghlan  wrote:
> > > (This is not really related to the switch, but more of a general

remark) In [4], it says that "python 3 version of the executable gains a
python3- prefix". This is IMO bad, since upstream projects tend to name the
versioned binaries "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one
of these - I'm not really certain which one of them. I tried to discuss
this several times on distutils-sig mailing list, but without reaching a
consensus. Either way, prefixing with python3- doesn't make sense to me,
because it's not similar to any upstream way and you don't find the
binaries under their names using tab completion (e.g. foo<tab> doesn't tell
you about python3-foo).
> > Agreed.
> >
> > CPython & pip use the "foo3.4, foo3" convention, so that seems enough of

a reason to use that convention by default. We may want a "unless upstream
does it differently" caveat though.

Oops, I just noticed an ambiguity in my caveat. I meant "unless the
particular upstream project being packaged does it differently".
> It doesn't really matter right now but long term I think python packaging

should just natively support commands like this. Either just as a matter of
fact, opt in, or by allowing templated command names. Either way I think
the upstream tooling should and likely will follow python's lead for how
these are written.

Yeah. It's somewhere on the list of the umpteen gazillion things that could
still use improving ;)

Cheers,
Nick.
> python-devel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/python-devel

[Bohuslav Kabrda]

----- Original Message -----
> On Dec 3, 2014, at 9:51 AM, Nick Coghlan < [e-mail] > wrote:
> > > (This is not really related to the switch, but more of a general
> > >
> > > remark)
> > > In [4], it says that "python 3 version of the executable gains a python3-
> > > prefix". This is IMO bad, since upstream projects tend to name the
> > > versioned binaries "foo-3.4, foo-3" or "foo3.4, foo3". We should accept
> > > one of these - I'm not really certain which one of them. I tried to
> > > discuss this several times on distutils-sig mailing list, but without
> > > reaching a consensus. Either way, prefixing with python3- doesn't make
> > > sense to me, because it's not similar to any upstream way and you don't
> > > find the binaries under their names using tab completion (e.g. foo<tab>
> > > doesn't tell you about python3-foo).
> > Agreed.
> > CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a
> > reason to use that convention by default. We may want a "unless upstream
> > does it differently" caveat though.
>
> It doesn't really matter right now but long term I think python packaging
> should just natively support commands like this. Either just as a matter of
> fact, opt in, or by allowing templated command names. Either way I think the
> upstream tooling should and likely will follow python's lead for how these
> are written.

Agreed. However from my experience, most upstreams use the foo-3.4 (i.e. with dash) way. I have to admit I like using the dash more. One more reason for this is that if we start building stacks for other interpreters, e.g. pypy, it would be good to have foo-pypy3 instead of foopypy3 (or foo3pypy? :)).

[Toshio Kuratomi]

On Thu, Dec 04, 2014 at 12:51:40AM +1000, Nick Coghlan wrote:
> On 4 Dec 2014 00:38, "Bohuslav Kabrda"  wrote:
> > So here are my proposals for changes in current guidelines [2]:
> > In [3], it says "If the executables provide the same functionality
>
> independent of whether they are run on top of Python 2 or Python 3, then only
> one version of the executable should be packaged. Currently it will be the
> python 2 implementation, but once the Python 3 implementation is proven to
> work, the executable can be retired from the python 2 build and enabled in the
> python 3 package." - this should be changed to prefer Python 3
>
> Agreed.

Definitely +1 as well.
> > (This is not really related to the switch, but more of a general remark) In
>
> [4], it says that "python 3 version of the executable gains a python3- prefix".
> This is IMO bad, since upstream projects tend to name the versioned binaries
> "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one of these - I'm not
> really certain which one of them. I tried to discuss this several times on
> distutils-sig mailing list, but without reaching a consensus. Either way,
> prefixing with python3- doesn't make sense to me, because it's not similar to
> any upstream way and you don't find the binaries under their names using tab
> completion (e.g. foo<tab> doesn't tell you about python3-foo).
>
> Agreed.
>
> CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a
> reason to use that convention by default. We may want a "unless upstream does
> it differently" caveat though.

Second caveat here is that currently we use version suffixes to denote
a command coming from a compat package.  If we make this change we need to
address that as well.  So, you might have sphinx-build, sphinx-build-2,
sphinx-build-2.7, sphinx-build-3, and sphinx-build-3.4 for the python
interpreter.  If you need a forwards or backwards compat package you might
also have an 0.9 and 1.1 that you need to tack on.  Possible solution here:
use a "v" prefix for the compat package's version.  So if the default
package is 1.1, you would have the python-sphinx0.9 and python3-sphinx0.9
packages provide:

* sphinx-build-v0.9
* sphinx-build-2-v0.9
* sphinx-build-2.7-v0.9
* sphinx-build-3-v0.9
* sphinx-build-3.7-v0.9

Also an addition: It would be great for us to always have a "major version number
only" script name.  Currently there's a few packages (python3-nose, I'm
looking at you) that only provide the MAJOR.MINOR form ie: nosetests-3.4.
That means scripts (user scripts as well as spec files) have to change
whenever we update python3 to a new major release.  Having the major only
form for all of these binaries will alleviate that.  Packagers can just
create a symlink if upstrean doesn't provide the -MAJOR version.
> > As for binaries/scripts in /usr/bin (assuming there are both python2 and
>
> python3 versions), the unversioned files should point to python2 version. This
> aligns with /usr/bin/python still pointing to python2.
>
> This also aligns with CPython & pip conventions. Between them, only "pyvenv"
> runs under Python 3 by default, and that's only because it doesn't exist in
> Python 2.

One exception to this, I think, should be 2to3.  Or perhaps python-tools
should stop shipping 2to3 and the other /bin/ scripts which python3-tools
ships and are not needed in two versions (Not sure if any of those scripts
actually need to be carried in two versions.. pygettext.py might need to be
tested to be sure the python3 version can handle all sorts of python2
strings correctly).

-Toshio

[Fedora-packaging] Changes in Guidelines Connected to "Python 3 as Default" Change

[Bohuslav Kabrda]

----- Original Message -----
> On Thu, Dec 04, 2014 at 12:51:40AM +1000, Nick Coghlan wrote:
> > On 4 Dec 2014 00:38, "Bohuslav Kabrda"  wrote:
> > > (This is not really related to the switch, but more of a general
> > >
> > > remark) In
> >
> > [4], it says that "python 3 version of the executable gains a python3-
> > prefix".
> > This is IMO bad, since upstream projects tend to name the versioned
> > binaries
> > "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one of these - I'm not
> > really certain which one of them. I tried to discuss this several times on
> > distutils-sig mailing list, but without reaching a consensus. Either way,
> > prefixing with python3- doesn't make sense to me, because it's not similar
> > to
> > any upstream way and you don't find the binaries under their names using
> > tab
> > completion (e.g. foo<tab> doesn't tell you about python3-foo).
> >
> > Agreed.
> >
> > CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a
> > reason to use that convention by default. We may want a "unless upstream
> > does
> > it differently" caveat though.
>
> Second caveat here is that currently we use version suffixes to denote
> a command coming from a compat package.  If we make this change we need to
> address that as well.  So, you might have sphinx-build, sphinx-build-2,
> sphinx-build-2.7, sphinx-build-3, and sphinx-build-3.4 for the python
> interpreter.  If you need a forwards or backwards compat package you might
> also have an 0.9 and 1.1 that you need to tack on.  Possible solution here:
> use a "v" prefix for the compat package's version.  So if the default
> package is 1.1, you would have the python-sphinx0.9 and python3-sphinx0.9
> packages provide:
> sphinx-build-v0.9
> sphinx-build-2-v0.9
> sphinx-build-2.7-v0.9
> sphinx-build-3-v0.9
> sphinx-build-3.7-v0.9

I'd rather see sphinx-build-v0.9-3.4. IMO keeping the Python version at the very end in every case is better. In other words, the binary would normally be "sphinx-build-0.9" and we just append "-3.4" to it.
Also, this makes me realize more arguments to append Python version with dash, not without it:
1) sphinx-build-v0.93.4 would be very confusing (I do understand that this is a downstream problem, but see the following point)
2) Similarly, if there is an upstream whose entry_point/script ends with a number (pep8 for example), it'd be highly confusing to use the notation without dash, it would yield pep83.4 assuming the upstream community would accept this scheme. This feels just wrong.
> Also an addition: It would be great for us to always have a "major version
> number
> only" script name.  Currently there's a few packages (python3-nose, I'm
> looking at you) that only provide the MAJOR.MINOR form ie: nosetests-3.4.
> That means scripts (user scripts as well as spec files) have to change
> whenever we update python3 to a new major release.  Having the major only
> form for all of these binaries will alleviate that.  Packagers can just
> create a symlink if upstrean doesn't provide the -MAJOR version.

I do agree that we should have that, although you can now use nosetests-%{python3_version}.
> > > As for binaries/scripts in /usr/bin (assuming there are both python2
> > >
> > > and
> >
> > python3 versions), the unversioned files should point to python2 version.
> > This
> > aligns with /usr/bin/python still pointing to python2.
> >
> > This also aligns with CPython & pip conventions. Between them, only
> > "pyvenv"
> > runs under Python 3 by default, and that's only because it doesn't exist in
> > Python 2.
>
> One exception to this, I think, should be 2to3.  Or perhaps python-tools
> should stop shipping 2to3 and the other /bin/ scripts which python3-tools
> ships and are not needed in two versions (Not sure if any of those scripts
> actually need to be carried in two versions.. pygettext.py might need to be
> tested to be sure the python3 version can handle all sorts of python2
> strings correctly).

Agreed, it's been on my todolist for quite some time to make 2to3 from python3-tools *the* 2to3.
> -Toshio

[Fedora-packaging] Changes in Guidelines Connected to "Python 3 as Default" Change

[Nick Coghlan]

On 4 December 2014 at 23:10, Bohuslav Kabrda  wrote:
> ----- Original Message -----
> > On Thu, Dec 04, 2014 at 12:51:40AM +1000, Nick Coghlan wrote:
> > > On 4 Dec 2014 00:38, "Bohuslav Kabrda"  wrote:
> > > > (This is not really related to the switch, but more of a general
> > > >
> > > > remark) In
> > >
> > > [4], it says that "python 3 version of the executable gains a python3-
> > > prefix".
> > > This is IMO bad, since upstream projects tend to name the versioned
> > > binaries
> > > "foo-3.4, foo-3" or "foo3.4, foo3". We should accept one of these - I'm not
> > > really certain which one of them. I tried to discuss this several times on
> > > distutils-sig mailing list, but without reaching a consensus. Either way,
> > > prefixing with python3- doesn't make sense to me, because it's not similar
> > > to
> > > any upstream way and you don't find the binaries under their names using
> > > tab
> > > completion (e.g. foo<tab> doesn't tell you about python3-foo).
> > >
> > > Agreed.
> > >
> > > CPython & pip use the "foo3.4, foo3" convention, so that seems enough of a
> > > reason to use that convention by default. We may want a "unless upstream
> > > does
> > > it differently" caveat though.
> >
> > Second caveat here is that currently we use version suffixes to denote
> > a command coming from a compat package.  If we make this change we need to
> > address that as well.  So, you might have sphinx-build, sphinx-build-2,
> > sphinx-build-2.7, sphinx-build-3, and sphinx-build-3.4 for the python
> > interpreter.  If you need a forwards or backwards compat package you might
> > also have an 0.9 and 1.1 that you need to tack on.  Possible solution here:
> > use a "v" prefix for the compat package's version.  So if the default
> > package is 1.1, you would have the python-sphinx0.9 and python3-sphinx0.9
> > packages provide:
> > sphinx-build-v0.9
> > sphinx-build-2-v0.9
> > sphinx-build-2.7-v0.9
> > sphinx-build-3-v0.9
> > sphinx-build-3.7-v0.9
>
> I'd rather see sphinx-build-v0.9-3.4. IMO keeping the Python version at the very end in every case is better. In other words, the binary would normally be "sphinx-build-0.9" and we just append "-3.4" to it.
> Also, this makes me realize more arguments to append Python version with dash, not without it:
> sphinx-build-v0.93.4 would be very confusing (I do understand that this is a downstream problem, but see the following point)
> Similarly, if there is an upstream whose entry_point/script ends with a number (pep8 for example), it'd be highly confusing to use the notation without dash, it would yield pep83.4 assuming the upstream community would accept this scheme. This feels just wrong.

I think these are good reasons to default to using the dash if its
Fedora adding it. The guideline could be something like "For Python
executables, also provide symlinks with a '-X' and '-X.Y' suffix,
unless upstream already provides appropriately versioned executables
without the dash. For compatibility packages, the Python version is
appended *after* the specific package version".

Cheers,
Nick.

[Fedora-packaging] Changes in Guidelines Connected to "Python 3 as Default" Change

[Toshio Kuratomi]

On Thu, Dec 04, 2014 at 08:10:39AM -0500, Bohuslav Kabrda wrote:
> > sphinx-build-v0.9
> > sphinx-build-2-v0.9
> > sphinx-build-2.7-v0.9
> > sphinx-build-3-v0.9
> > sphinx-build-3.7-v0.9
>
> I'd rather see sphinx-build-v0.9-3.4. IMO keeping the Python version at
> the very end in every case is better. In other words, the binary would
> normally be "sphinx-build-0.9" and we just append "-3.4" to it.

I'm not really attached to whether the v0.9 goes before or after the -3.4
however, the argument was made earlier that upstream naming and documentation,
and tab completion were important factors.  To remain consistent with that
it seems more appropriate to put any Fedora-added suffixes for
backwards/forwards compat packages at the very end.
> Also, this makes me realize more arguments to append Python version with dash, not without it:
> sphinx-build-v0.93.4 would be very confusing (I do understand that this
> is a downstream problem, but see the following point)
> Similarly, if there is an upstream whose entry_point/script ends with
> a number (pep8 for example), it'd be highly confusing to use the notation
> without dash, it would yield pep83.4 assuming the upstream community would
> accept this scheme. This feels just wrong.

I'm wholehearterdly in favor of dashes when we're creating these script
names too.

I also note that this line of reasoning also lends itself to putting any
backwards compat versions at the very end.  sphinx-build-v3.1-2 is more
ambiguous (especially to a sysadmin who is used to reading rpm NEVR's) than
sphinx-build-2-v3.1.  The "v" provides additional guidance to someone
looking at it that the number following it serves a separate purpose from
the number before it.

Anyhow, not really attached so I've stated the reasons I think the
fedora-local version suffix makes more sense at the very end and now I'll
shut about it ;-)
> > Also an addition: It would be great for us to always have a "major version
> > number
> > only" script name.  Currently there's a few packages (python3-nose, I'm
> > looking at you) that only provide the MAJOR.MINOR form ie: nosetests-3.4.
> > That means scripts (user scripts as well as spec files) have to change
> > whenever we update python3 to a new major release.  Having the major only
> > form for all of these binaries will alleviate that.  Packagers can just
> > create a symlink if upstrean doesn't provide the -MAJOR version.
>
> I do agree that we should have that, although you can now use nosetests-%{python3_version}.

<nod> -- yeah, that's what I use in spec files now.  Unfortunately, spec
macros aren't as helpful for the scripts that I have for building and
testing my own projects.

Even in spec files, it's also an annoyingly long way to write "3.4" ;-)

-Toshio

[Fedora-packaging] Changes in Guidelines Connected to "Python 3 as Default" Change

[Toshio Kuratomi]

On Thu, Dec 04, 2014 at 11:18:58PM +1000, Nick Coghlan wrote:
> I think these are good reasons to default to using the dash if its
> Fedora adding it. The guideline could be something like "For Python
> executables, also provide symlinks with a '-X' and '-X.Y' suffix,
> unless upstream already provides appropriately versioned executables
> without the dash. For compatibility packages, the Python version is
> appended *after* the specific package version".

Couple notes -- if we do go with putting the Python version after the Fedora
compat version, we'll want to phrase this so people know whether to do that
for all packages or only for those where upstream is not already putting the
version in the name.

Always putting the python version at the end means that users have to
remember that change when they switch to using a compat package:
nosetests-3.4 vs nosetests-v1.1-3.4

Only putting the Fedora version first when upstream doesn't provide the
names means that users have to remember the change on a package by package
basis:
sphinx-build-3.4-v1.1 vs nosetests-v1.1-3.4

We also want to mention the "v" addition to the version specified in the
compat version.  The "v" helps the user differentiate between the pyhton
version and the package version.  Most Fedora compat packages don't have
this matrix of $language_version + $package_version so they mark the compat
files with just a bare $package_version.  Since we have both, we need to let
packagers know that compat packages need to do a little bit more to
distinguish the two sets of numbers.

-Toshio

[Bohuslav Kabrda]

----- Original Message -----
> Hi all,
> as you are well aware, there is the "Python 3 as Default" change proposed for
> F22 [1]. There are several guideline changes that will need to be done and
> some that may be done in order to improve the situation. I'd like to make
> this thread a sort of brainstorming on my thoughts as well as possibility
> for anyone to add something that should be done as well. After that, I'm
> planning to open an FPC ticket with all the gathered and discussed changes
> that will come up on this thread.

<snip>

The FPC ticket is submitted at https://fedorahosted.org/fpc/ticket/475. Feel free to post any further comments there.

Thanks everyone for replies and suggestions in this thread!

Slavek