Handling --enable-experimental-jit in Python 3.13

[Miro Hrončok]

Hello Pythonistas,

Python 3.13 has an experimental JIT compiler:

https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler

Enabling it is a configure (hence build-time) option.

How do we handle this in Fedora?

- We can keep it disabled, as it is experimental.
- We can enable it, but be ready to revert if it causes problems.
- We can add yet another build variant, but we already have 4 of those 
(regular, debug, freethreading, freethreading-debug), so I'd rather not make it 
6 (or 8, if we include freethreading+jit combinations). I don't know yet if it 
would be co-installable.

Opinions?

[Frantisek Zatloukal]

On Wed, Apr 10, 2024 at 8:23 PM Miro Hrončok  wrote:
> Hello Pythonistas,
>
> Python 3.13 has an experimental JIT compiler:
>
> https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler
>
> Enabling it is a configure (hence build-time) option.
>
> How do we handle this in Fedora?
> We can keep it disabled, as it is experimental.
> We can enable it, but be ready to revert if it causes problems.
> We can add yet another build variant, but we already have 4 of those
>
> (regular, debug, freethreading, freethreading-debug), so I'd rather not
> make it
> 6 (or 8, if we include freethreading+jit combinations). I don't know yet
> if it
> would be co-installable.
>
> Opinions?

o/,
what about leaving it on in rawhide and reevaluating readiness for a proper
release around f42 branching based on how many bleeding (edge) kittens did
it eat?

[Neal Gompa]

On Wed, Apr 10, 2024 at 2:23 PM Miro Hrončok  wrote:
> Hello Pythonistas,
>
> Python 3.13 has an experimental JIT compiler:
>
> https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler
>
> Enabling it is a configure (hence build-time) option.
>
> How do we handle this in Fedora?
> We can keep it disabled, as it is experimental.
> We can enable it, but be ready to revert if it causes problems.
> We can add yet another build variant, but we already have 4 of those
>
> (regular, debug, freethreading, freethreading-debug), so I'd rather not make it
> 6 (or 8, if we include freethreading+jit combinations). I don't know yet if it
> would be co-installable.
>
> Opinions?

I lean toward just going ahead and shipping it and reverting if it
causes enough problems that cannot be fixed.

[Tomáš Orsava]

I don't think it's a good idea to enable an experimental feature on the 
main Python that runs all the system tools, that could wreak untold 
havoc in ways we can't tell yet. Especially since it's a brand new 
experimental feature.

If we really want to try it, adding it as a variant sounds best to me. 4 
vs 6 build variants is not such a big leap, and for any time-sensitive 
work on the Python spec file it can be turned off anyway for scratch builds.

Tomas

On 4/10/24 20:23, Miro Hrončok wrote:
> Hello Pythonistas,
>
> Python 3.13 has an experimental JIT compiler:
>
> https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler
>
> Enabling it is a configure (hence build-time) option.
>
> How do we handle this in Fedora?
> We can keep it disabled, as it is experimental.
> We can enable it, but be ready to revert if it causes problems.
> We can add yet another build variant, but we already have 4 of those
>
> (regular, debug, freethreading, freethreading-debug), so I'd rather 
> not make it 6 (or 8, if we include freethreading+jit combinations). I 
> don't know yet if it would be co-installable.
>
> Opinions?

[Victor Stinner]

On Wed, Apr 10, 2024 at 8:23 PM Miro Hrončok  wrote:
> Python 3.13 has an experimental JIT compiler:
> https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler
>
> Enabling it is a configure (hence build-time) option.
>
> How do we handle this in Fedora?
> We can keep it disabled, as it is experimental.

I concur with that.
> We can enable it, but be ready to revert if it causes problems.
> We can add yet another build variant, but we already have 4 of those
>
> (regular, debug, freethreading, freethreading-debug), so I'd rather not make it
> 6 (or 8, if we include freethreading+jit combinations). I don't know yet if it
> would be co-installable.

I don't think it's worth it.

Moreover, it doesn't build on Fedora 39 with clang-17, since Python
3.13 currently requires exactly clang-16.

Victor

[Miro Hrončok]

On 17. 04. 24 14:27, Victor Stinner wrote:
> On Wed, Apr 10, 2024 at 8:23 PM Miro Hrončok  wrote:
> > Python 3.13 has an experimental JIT compiler:
> > https://docs.python.org/dev/whatsnew/3.13.html#experimental-jit-compiler
> >
> > Enabling it is a configure (hence build-time) option.
> >
> > How do we handle this in Fedora?
> > We can keep it disabled, as it is experimental.
>
> I concur with that.
> > We can enable it, but be ready to revert if it causes problems.
> > We can add yet another build variant, but we already have 4 of those
> >
> > (regular, debug, freethreading, freethreading-debug), so I'd rather not make it
> > 6 (or 8, if we include freethreading+jit combinations). I don't know yet if it
> > would be co-installable.
>
> I don't think it's worth it.
>
> Moreover, it doesn't build on Fedora 39 with clang-17, since Python
> 3.13 currently requires exactly clang-16.

We have multiple clangs/llvms:

https://src.fedoraproject.org/rpms/llvm16
https://src.fedoraproject.org/rpms/clang16

Victor, do you think it would be possible to build in the JIT support but have 
a runtime opt-out/opt-in switch? That way, we can build it, but disable it by 
default, unless our users want to experiment with it.

[Victor Stinner]

On Wed, Apr 17, 2024 at 2:38 PM Miro Hrončok  wrote:
> Victor, do you think it would be possible to build in the JIT support but have
> a runtime opt-out/opt-in switch? That way, we can build it, but disable it by
> default, unless our users want to experiment with it.

PEP 744 "JIT Compilation" informal PEP is being discussed. I asked
your question there:
https://discuss.python.org/t/pep-744-jit-compilation/50756/33

Victor

[Miro Hrončok]

On 17. 04. 24 18:40, Victor Stinner wrote:
> On Wed, Apr 17, 2024 at 2:38 PM Miro Hrončok  wrote:
> > Victor, do you think it would be possible to build in the JIT support but have
> > a runtime opt-out/opt-in switch? That way, we can build it, but disable it by
> > default, unless our users want to experiment with it.
>
> PEP 744 "JIT Compilation" informal PEP is being discussed. I asked
> your question there:
> https://discuss.python.org/t/pep-744-jit-compilation/50756/33

The JIT support was built on x86_64+aarch64, on Fedora 40+

Set the PYTHON_JIT environment variable to 1 to enable it on runtime.

F41 (stable) https://bodhi.fedoraproject.org/updates/FEDORA-2024-ecfa04265a
F40 (pending) https://bodhi.fedoraproject.org/updates/FEDORA-2024-a5a0be41ba

[Fabio Valentini]

On Mon, Jun 24, 2024 at 3:53 PM Miro Hrončok  wrote:
> On 17. 04. 24 18:40, Victor Stinner wrote:
> > On Wed, Apr 17, 2024 at 2:38 PM Miro Hrončok  wrote:
> > > Victor, do you think it would be possible to build in the JIT support but have
> > > a runtime opt-out/opt-in switch? That way, we can build it, but disable it by
> > > default, unless our users want to experiment with it.
> >
> > PEP 744 "JIT Compilation" informal PEP is being discussed. I asked
> > your question there:
> > https://discuss.python.org/t/pep-744-jit-compilation/50756/33
>
> The JIT support was built on x86_64+aarch64, on Fedora 40+
>
> Set the PYTHON_JIT environment variable to 1 to enable it on runtime.

So this means that the JIT-enabled Python is not, as initially
expected, a separate build, but something that can be enabled in
existing builds by passing this environment variable instead? Nice!

Fabio

[Miro Hrončok]

On 24. 06. 24 23:08, Fabio Valentini wrote:
> On Mon, Jun 24, 2024 at 3:53 PM Miro Hrončok  wrote:
> > On 17. 04. 24 18:40, Victor Stinner wrote:
> > > On Wed, Apr 17, 2024 at 2:38 PM Miro Hrončok  wrote:
> > > > Victor, do you think it would be possible to build in the JIT support but have
> > > > a runtime opt-out/opt-in switch? That way, we can build it, but disable it by
> > > > default, unless our users want to experiment with it.
> > >
> > > PEP 744 "JIT Compilation" informal PEP is being discussed. I asked
> > > your question there:
> > > https://discuss.python.org/t/pep-744-jit-compilation/50756/33
> >
> > The JIT support was built on x86_64+aarch64, on Fedora 40+
> >
> > Set the PYTHON_JIT environment variable to 1 to enable it on runtime.
>
> So this means that the JIT-enabled Python is not, as initially
> expected, a separate build, but something that can be enabled in
> existing builds by passing this environment variable instead? Nice!

Exactly. The fact that it is built in might still create some regressions, but 
if the opt-in mechanism works correctly, only those who want the JIT will have it.