On 14. 06. 21 14:40, Alexander Bokovoy wrote:
On ma, 14 kesä 2021, Miro Hrončok wrote:
> On 14. 06. 21 13:40, Alexander Bokovoy wrote:
>> On ma, 14 kesä 2021, Victor Stinner wrote:
>>> Congrats, that's amazing! :-) Let's fix remaining broken packages!
>> Right now the biggest broken package to us (FreeIPA) is mod_wsgi which
>> relied on Python C API which was removed in Python 3.10. The same API
>> was used by uWSGI as well, so it would really be great if Python core
>> developers and Python maintainers team could have helped mod_wsgi/uWSGI
>> upstreams to migrate from the old API.
>> Sadly, this was only found after the fact when a side-tag was already
>> merged, so FreeIPA right now is broken in Fedora Rawhide.
> The problem in mod_wsgi was actually found and reported in November 2020 ,
> which is exactly why we test with each pre-release of Python: To report
> problems *early* and give maintainers time to fix them, so we can land the
> new version with limited negative impact.
> Unfortunately, the mod_wsgi maintainer did not respond to the problem :(
> While we try really hard, we cannot report each failure to upstreams directly.
> The fact that this blocks FreeIPA was indeed only discovered by chance while
> the side tag rebuild was already in progress (and about to be merged). I
> wonder haw can we improve the process to ensure problems like this are known
> to FreeIPA maintainers since the beginning and prioritized accordingly.
> (Ideally, the process would not only be improved for FreeIPA but the entire
Well, this is a dependency problem in the first place. When an ABI
change happens, like a Python ABI change with 3.10 mass-rebuild, it
should be assumed and expected that until all previously successfully
built packages were to be rebuilt, there will be broken dependencies.
Yes, however getting "all previously successfully built packages rebuilt" is an
impossible goal. Instead we strive to get "most of them rebuilt" and "the
important ones rebuilt". Getting most of them rebuilt is not an easy task but
at least is is quite easy to quantify: 3333 are rebuilt, 293 to go, that is
~91%. OTOH knowing what is "important" is hard. There is the "critical
thing (but mod_wsgi is not in it and the list seems heavily outdated).
Perhaps, we could extend our existing checks for a broken compose to
done on a side-tag on demand? This way mass-rebuilders could ask for
such a run one they consider to be ready to merge and see how that
side-tag merge would affect the distribution. I don't think we have a
better way to detect it before the merge.
We try hard not to break the compose. It is not fully automated and having a
way to run a test compose would be awesome.
However, this particular problem does not break the compose, the compose is
done now even when mod_wsgi is not installable.
We have also validated the following groups are installable before merging the
Because we parsed them from the hopefully compsoe-blocking media kickstarts.
What I do not see as acceptable is what Python core team did in
2020 when this issue was made clear in the original PR to remove the
'old' parser that broke mod_wsgi/uWSGI/unbound and some more packages.
Instead of following up in providing a supported API function, the
comment from Victor was practically ignored.
I am not happy either that Python upstream keeps breaking things and often the
migration path is "go figure". Whenever and wherever I can, I argue against that
> I assume that opening bugzillas for all the dependent packages
> failure would be considered as spam by many.
It depends on what that bug opening would mean. If you are doing it
early together with an upstream that breaks its ABI/API, helping them to
see the issues early is one of your goals -- as you stated above.
Realizing through reverse dependencies a damage such breakage could
cause is something you (as Python maintainers in Fedora) could
definitely do and raise the issue earlier also with the package
maintainers/upstreams. In this case FreeIPA is affected but we missed it
completely in November 2020 as nobody told us mod_wsgi would not work
This is a chicken-egg problem: Until we measure the impact, we don't know it.
And reporting many dependent-packages bugzillas early seems like a waste of
effort if the bug is getting fixed soon. What we need to detect (and we don't
know how to automate that yet) is:
- a bugzilla for a problem is stalled AND
- the package is dependent on by "important stuff" or by a large stack
Even when not automated, we manually rise priorities for bugzillas that block
the builds of large stacks.
What would be particularity helpful would be if *all* the package maintainers
triaged their bugzillas and responded to them in timely manner, notify
upstreams and ask for help if they need it. However, that is not realistic.
> In the meantime, I've marked mod_wsgi as "blocker"
for the Python 3.11
> rebuild next year, but that is not a systematic solution.
>> I saw that there is some discussion and an effort to help mod_wsgi
>> upstream already, thank you!
> There is a proposed fix  but we don't know enough about Apache to say if
> it is ideal. In case this blocks you very much, I suggest you work with the
> Fedora's mod_wsgi maintainer and backport this patch to rawhide, keeping an
> eye on it in case it breaks your use cases, and report problems to us, so we
> can adapt it. At this point of the Fedora 35 development schedule, I think
> this patch should be good enough for it.
I already nominated mod_wsgi bug as Fedora 35 beta release blocker
because it violates Fedora Server release criteria. I am not a
maintainer for mod_wsgi myself and have no rights to that package.
I know and I voted +1 for a blocker. I am not a maintainer for mod_wsgi either.