6:16 a.m.
On 2019-10-26 17:01, Neal Gompa wrote:
On Sat, Sep 21, 2019 at 5:16 PM Miro Hrončok
<mhroncok(a)redhat.com> wrote:
>
> Hello,
>
> we've been recently approached by a colleague from Red Hat working on
> performance (CCed).
>
> According to their testing, Fedora Python performance could be improved by ~15%
> by building /usr/bin/python* statically with libpython*.a. That sounds like a
> worthy thing to do.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1749479
>
> Since Python 3.8 Python extension modules are no longer linked to libpython.so
> and we can do the following:
>
> * build /usr/bin/python3(.8) statically with libpython*.a
> * build and ship libpython3.8.so.1.0 for packages that "embed" Python
>
> The change in the python3 package is trivial:
>
> https://src.fedoraproject.org/rpms/python3/pull-request/133
>
> However it can have serious impact on Python extension modules that are linked
> to libpython3.8.so.1.0 by various "nonstandard" build mechanisms or by
compiling
> code for Python extension module and code that embeds Python into one file.
>
> We will likely propose a Fedora 32 Change for this, however I'm opening this
> topic for discussion before we do so.
>
> Testing the proposed Pull Request with your code is also helpful. Let me know
> how can we make that easier (e.g. if you want a Copr or a Fedora 30/31 python38
> package with this change).
>
> WDYT?
One major concern: We currently rely on the libpython shared object
linkage for non-standard builds to ensure we get everything for moving
to new Python versions. I've personally experienced quirks with trying
to force linking via the shared object when the Python interpreter
itself is not. For example, perl-Inline-Python determines how to build
itself by asking the interpreter. In Fedora, this works correctly
because the interpreter returns the shared object. In other
distributions where I've built the module, it doesn't and uses the
static link object, which makes it difficult to automatically generate
the link dependency to the system Python we are using.
In Mageia, we actually reverted Python's change to not link libpython
with extension modules so that we still have the dependency link for
binary objects:
https://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/link-...
Is there no other way to get better performance? Do we want to
potentially give up the easily trackable dependency web for that? Is
it worth breaking non-standard build mechanisms that interrogate the
interpreter for how to link to it?
I'll share a more high-level view, since I don't know too many details
about linking. Hope y'all won't mind such vague text.
Is it worth breaking non-standard build mechanisms that interrogate
the
interpreter for how to link to it?
There's a spectrum: on one side, there's forever supporting any hack
that happened to work the way some frustrated engineer put it in a cmake
file. On the other, there's a mythical future-proof "How to link to
Python" guide explaining the various use cases and the best ways to do
things, which everyone follows.
Both sides of the spectrum lead to stagnation. The middle would have
some best practices but also room to experiment.
I think interrogating the interpreter for how to link to it is an
important use case. But it currently seems like it's a bit of black
magic: Python exposes some of its internals in a bunch of different
ways, and that's what everyone uses. Nobody knows how to do it properly
(including authors of python-config). Things mostly work, but when the
internals need to change (for a 20% speedup), some stuff will fail.
I think Fedora is actually a great place to start changing that, because
this is an integration problem, and distros are all about integration.
Fedora has a lot of software with all kinds of needs and assumptions, we
have pretty strong ties to upstreams (certainly CPython, and hopefully
the other projects), so we can hopefully make changes that aren't
one-sided, but coordinated across the ecosystem (assuming that the
Fedora package collection is a reasonable sample of the ecosystem).
So let's, figure out what people need and what's the best way to solve
their problems. And if we can't answer all the questions, hit the revert
button.
Specifically:
- What should code that both embeds and extends Python do? Do we have a
concrete use case for this?
- Since perl-Inline-Python was mentioned, could we explicitly make sure
it works, or figure out what modifications it will need?