How best to handle tools which support multiple Python runtimes

[Avram Lubkin]

I'm a co-maintainer for the python-sphinx package. There's a bug [1] I'd
like to address, but I'd like some input on what I was thinking. There was
some discussion about this on FPC ticket [2], but nothing workable really
came out of that. I also attempted to get input [3] from the Sphinx
community, but didn't receive a response.

For background, python-sphinx is available for both Python2 and Python3. It
is primarily a library, but provides four commands for convenience:
sphinx-build, sphinx-quickstart, sphinx-apidoc, sphinx-autogen. Here's an
example of how these commands are currently made available to the user:

/usr/bin/sphinx-build -> sphinx-build-2
/usr/bin/sphinx-build-2 -> sphinx-build-2.7
/usr/bin/sphinx-build-2.7 (#!/usr/bin/python2)
/usr/bin/sphinx-build-3 -> sphinx-build-3.4
/usr/bin/sphinx-build-3.4 (#!/usr/bin/python3)

It should be noted, all of these commands are just used to call entry
points and don't have any of their own logic. Here are their equivalents:

sphinx-build            python -m sphinx
sphinx-quickstart    python -m sphinx.quickstart
sphinx-apidoc         python -m sphinx.apidoc
sphinx-autogen       python -m sphinx.ext.autosummary.generate

Sphinx itself isn't Python version specific and maintains compatibility
regardless of what Python version you are using or if you switch back and
forth. Potentially there are some Sphinx extensions that are Python version
dependent, but I am not aware of any.

Use cases for Sphinx are all over the place. For example, I rarely use the
commands and call sphinx through setuptools 99% of the time. Other people
use the commands exclusively, and still others use the makefile created by
sphinx-quickstart as their primary way to call it.

It's this last one is where we are seeing the biggest problem and is
described in the bug. Essentially, sphinx-quickstart-3 is used to create a
makefile which lists sphinx-build as the build command. But sphinx-build is
currently linked to the python2 version, which may not be expected, and
won't necessarily be installed.

It was suggested the makefile should be changed so the Python version that
was used for sphinx-quickstart should be used for sphinx-build.
Unfortunately, that would make the file less portable.

I have two possible fixes:

The first would be to decouple the commands from the libraries. Since the
package name sphinx is already taken by the search engine, something like
python-sphinx-bin may be appropriate. Once separated, the command would
default to python3 since the packaging guidelines say it Python 3 versions
should be preferred when equivalent.

The second option would be to retain the multiple versions, but to manage
the links with alternatives so the unversioned commands will default to
whatever version is installed and again, Python 3 would be preferred if
both were installed.

Does anyone have any preferences, thoughts or alternative approaches?

Avram

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1321413
[2] https://fedorahosted.org/fpc/ticket/612
[3] https://groups.google.com/forum/#!topic/sphinx-dev/vdFeOEj4EKg

[Miroslav Suchý]

Dne 13.7.2016 v 21:15 Avram Lubkin napsal(a):
>  Does anyone have any preferences, thoughts or alternative approaches?
I do not have the solution. But I want to point out similar case, which I recently
reported against pyp2rpm:
  https://github.com/fedora-python/pyp2rpm/issues/63

-- 
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

[Igor Gnatenko]

As someone pointed out somewhere on IRC - we should try to invent
"alternatives" to python packages like it was done in java packages in
Fedora.

On Thu, Jul 14, 2016 at 3:13 PM, Miroslav Suchý <msuchy(a)redhat.com&gt; wrote:
>  Dne 13.7.2016 v 21:15 Avram Lubkin napsal(a):
> > Does anyone have any preferences, thoughts or alternative approaches?
>
>  I do not have the solution. But I want to point out similar case, which I recently
> reported against pyp2rpm:
>    https://github.com/fedora-python/pyp2rpm/issues/63
>
>  --
>  Miroslav Suchy, RHCA
>  Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
>  --
>  devel mailing list
>  devel(a)lists.fedoraproject.org
>  https://lists.fedoraproject.org/admin/lists/[e-mail] 


-- 
-Igor Gnatenko

[Björn Persson]

Avram Lubkin wrote:
>  Potentially there are some Sphinx
>  extensions that are Python version dependent, but I am not aware of
>  any. 
It sounds like I should start trying to get sphinxcontrib-adadomain
ported to Python 3.

>  It was suggested the makefile should be changed so the Python
> version
>  that was used for sphinx-quickstart should be used for sphinx-build.
>  Unfortunately, that would make the file less portable. 
Yeah, that's not a particularly good idea. I suppose it would work for
those who generate the makefile and use it themselves on the same
workstation. For makefiles that are included in source tarballs it
would only make the problem worse.

>  I have two possible fixes:
>  
>  The first would be to decouple the commands from the libraries. Since
>  the package name sphinx is already taken by the search engine,
>  something like python-sphinx-bin may be appropriate. Once separated,
>  the command would default to python3 since the packaging guidelines
>  say it Python 3 versions should be preferred when equivalent. 
If this approach is chosen, will the commands still invoke the Python 2
version if only that one is installed?

>  The second option would be to retain the multiple versions, but to
>  manage the links with alternatives so the unversioned commands will
>  default to whatever version is installed and again, Python 3 would be
>  preferred if both were installed. 
This approach seems like it would work for my use case.

Ahven uses Sphinx to generate its documentation, using Sphinx's
generated makefile that invokes sphinx-build. I have ahven.spec
regenerate the documentation, but the makefile is generated upstream.
Until the Ada domain gets ported, the Ahven build will have to use the
Python 2 Sphinx, but I'd rather not encode that in ahven.spec. I want
to keep that dependency contained in python-sphinxcontrib-adadomain.
Ahven shouldn't need to even know that Sphinx is written in Python.

If I can have python-sphinxcontrib-adadomain depend on the Python 2
Sphinx specifically, and then have the ahven package require
python-sphinxcontrib-adadomain, plus python-sphinx or python-sphinx-bin
or whatever but not python2-sphinx, and invoke sphinx-build and know
that it will run on Python 2, then I'm satisfied.

Björn Persson

[Avram Lubkin]

On Sun, Jul 17, 2016 at 6:09 AM, Björn Persson <Bjorn(a)xn--rombobjrn-67a.se&gt; wrote:

>  It sounds like I should start trying to get sphinxcontrib-adadomain
>  ported to Python 3.
>  
Definitely!

>  I have two possible fixes:
>  >
>  > The first would be to decouple the commands from the libraries. Since
>  > the package name sphinx is already taken by the search engine,
>  > something like python-sphinx-bin may be appropriate. Once separated,
>  > the command would default to python3 since the packaging guidelines
>  > say it Python 3 versions should be preferred when equivalent.
>
>  If this approach is chosen, will the commands still invoke the Python 2
>  version if only that one is installed?
>  
No, the command would require the python3 version, the second option would
allow the python commands to be the default if it's the only one installed.


>  > The second option would be to retain the multiple versions, but to
>  > manage the links with alternatives so the unversioned commands will
>  > default to whatever version is installed and again, Python 3 would be
>  > preferred if both were installed.
>
>  This approach seems like it would work for my use case.
>
>  Ahven uses Sphinx to generate its documentation, using Sphinx's
>  generated makefile that invokes sphinx-build. I have ahven.spec
>  regenerate the documentation, but the makefile is generated upstream.
>  Until the Ada domain gets ported, the Ahven build will have to use the
>  Python 2 Sphinx, but I'd rather not encode that in ahven.spec. I want
>  to keep that dependency contained in python-sphinxcontrib-adadomain.
>  Ahven shouldn't need to even know that Sphinx is written in Python.
>
>  If I can have python-sphinxcontrib-adadomain depend on the Python 2
>  Sphinx specifically, and then have the ahven package require
>  python-sphinxcontrib-adadomain, plus python-sphinx or python-sphinx-bin
>  or whatever but not python2-sphinx, and invoke sphinx-build and know
>  that it will run on Python 2, then I'm satisfied.
>  
It sounds like the first option wouldn't work for your use case, since you
have a hard Python 2 dependency until sphinxcontrib-adadomain gets a Python
3 package. The only way it could work would be to modify the make file
during the build. The second option would work as long as you don't have
python3-sphinx installed during the package build. That should be fine in a
mock build, but if someone wanted to do a rebuild on their regular system
and had python3-sphinx installed, they'd have to set the sphinx version
with the alternatives command.


It seems like the second option, using alternatives, would be the preferred
method so as to not break things. My current thinking is to implement this
in the next package update, but to keep Python 2 as the default (if both
are installed) for Fedora 24, then make Python 3 the default in Rawhide and
Fedora 25.

Avram

[Björn Persson]

Avram Lubkin wrote:
>  The
>  second option would work as long as you don't have python3-sphinx
>  installed during the package build. That should be fine in a mock
>  build, but if someone wanted to do a rebuild on their regular system
>  and had python3-sphinx installed, they'd have to set the sphinx
>  version with the alternatives command. 
That's true. Well, I think we can live with that for a while and solve
it by porting sphinxcontrib-adadomain.

Björn Persson