Is there a reason to do Python runtime flatpackages anymore?

[Neal Gompa]

Hey all,

Things have changed in Python runtime packaging since we started
introducing alternative Python versions years ago. For one, we now
always have fully versioned source packages, and now we have a flag
for whether the packages are "main runtime" vs "alternate runtime".
Another is that RHEL now offers multiple Python runtimes that you can
build packages from.

I'm wondering if it makes sense to continue having the logic in the
Python runtime packaging for "flatpackage" when we can now just have
them build as alternative runtimes. This doesn't get rid of the
concept of a "main runtime" that is generally supported by the macros,
but it brings us closer in line with what our downstreams are doing.
This could also ease Python transitions in the future, since we
wouldn't have the Python runtime ripped out from under us for DNF as
we rebootstrap the whole environment to a new Python version default.

At least for me, it would also make it easier for me to trivially
rebuild packages in COPR against an alternate Python version for
specific purposes, too, since the only required change to switch to an
alternate runtime would be setting %__python/%__python3 and making
sure the subpackage has the fully qualified Python version name in it.

What do you all think? Is this crazy talk or something we might want
to think about?

-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miro Hrončok]

On 22. 03. 21 15:50, Neal Gompa wrote:
>  Hey all, 
Hey Neal, thanks for bringing this up.

>  Things have changed in Python runtime packaging since we started
>  introducing alternative Python versions years ago. For one, we now
>  always have fully versioned source packages, and now we have a flag
>  for whether the packages are "main runtime" vs "alternate runtime".
>  Another is that RHEL now offers multiple Python runtimes that you can
>  build packages from. 
Yes, everything is prepared on the source-level to work nicely. RHEL 9 will show 
if this is indeed the case and we'll correct the stuff that's broken.

>  I'm wondering if it makes sense to continue having the logic in
> the
>  Python runtime packaging for "flatpackage" when we can now just have
>  them build as alternative runtimes. This doesn't get rid of the
>  concept of a "main runtime" that is generally supported by the macros,
>  but it brings us closer in line with what our downstreams are doing. 
This is possible. The reason we are not doing it is simple: We don't have the 
capacity to support this in Fedora. Taking care of one Python stack is an 
enormous amount of work. Taking care of another one can be simpler (because it 
might be smaller), but will bring other challenges (how long do we keep an 
alternate stack hanging around before we deprecate and remove it, how many 
parallel stacks do we allow t the same time, how do we handle building 
subpackages for multiple stacks form on source). Not to mention the maintenance 
cost fo the interpreters themselves. Quite often we close security bugzillas as 
WONTFIX or UPSTREAM, saying something like: "This package exists in Fedora only 
so that Python developers could run their test suites e.g. with tox." Once we 
make it possible to run actual Fedora packages on top of this, we cannot longer 
do that.

>  This could also ease Python transitions in the future, since we
>  wouldn't have the Python runtime ripped out from under us for DNF as
>  we rebootstrap the whole environment to a new Python version default. 
This would only make things easier if we package the entire dnf dependency tree 
for the next stack before we proceed with the upgrade. We tried a similar thing 
once (for "system-python") and it was enormously time- and energy-consumig. I 
don't want to do this myself ever again, so this would only possibly get easier 
if *all* the maintainers of the packages from the stack are actively 
participating (hint: usually, only some are).

I am also afraid that this would actually be more complicated at first, because 
there would be new challenges and new problems, before it gets easier. I am a 
bit pessimistic about the "return on investment" here.

>  At least for me, it would also make it easier for me to trivially
>  rebuild packages in COPR against an alternate Python version for
>  specific purposes, too, since the only required change to switch to an
>  alternate runtime would be setting %__python/%__python3 and making
>  sure the subpackage has the fully qualified Python version name in it. 
That I actually consider "supported". Flip the bcond in Python X.Y, rebuild it 
and start rebuilding packages atop. That's what RHEL 9 is going to do.

>  What do you all think? Is this crazy talk or something we might want
>  to think about? 
It is actually a very nice thought and I could support it, if only I wouldn't 
see all the work behind this. Give me 3 more full time people dedicated to this 
long-term and I might even drive this myself (I probably shouldn't, to avoid a 
burn-out).

----

I also have some reservations about how would Fedora's content actually look 
like if we allow package (as in apps, tools, services) maintainers to hand-pick 
a Python stack to use. Currently, we don't plan to do this in RHEL 9 (I imagine 
the "alternate" stacks as "leaf-clusters"), but this might change
after couple 
years of RHEL 9 life-time - e.g. in EPEL or even in some internal stuff.

-- 
Miro Hrončok
--
Phone: [phone]
IRC: mhroncok