On Tue, 22 Mar 2022 11:48:57 -0700
Adam Williamson <adamwill(a)fedoraproject.org> wrote:
I found quite a big mess today, caused by an attempt to bump perl to
5.34.1 in Fedora 36:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4
Because some packages depend on the exact perl interpreter version,
the maintainer made a buildroot override for perl:
https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36
and rebuilt them. Okay.
The problem with this is, we have lots of perl modules. People build
them all the time. And buildroot overrides apply to *everything*.
So, while the buildroot override has been valid, multiple *other* perl
modules have been unwittingly built against it:
perl-Scalar-List-Utils:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732
perl-Parallel-Pipes:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527
perl-PPIx-Regexp:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522
perl-Crypt-SSLeay:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521
perl-Crypt-OpenSSL-PKCS10:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471
...and those are just the ones I found so far. Because they were built
against 5.34.1, all of those builds require
"perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1
from updates-testing. They cannot be installed with perl-5.34.0 from
stable. But the maintainers likely had no idea about this - buildroot
overrides are not at all discoverable, there is zero indication your
package is building against one when you build it - and have created
separate updates for those packages:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2
https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901
https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f
https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964
https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e
Users testing those updates will likely not notice the problem,
because they'll have *all* of updates-testing enabled when they
update, and perl-5.34.1 will be pulled in. But if any of those
updates is pushed stable before the perl update is pushed stable, it
will fail to install for anyone who does not have updates-testing
enabled.
This is quite a mess. I'm going to try and clean it up, but it'll be a
bit ugly (there are a couple of ways I can do it, but both will
involve doing bumps-and-rebuilds of the affected packages with no
changes).
I've been worried about this sort of thing happening for a while due
to the undesirable properties of buildroot overrides. This is the
biggest real-world case I've seen, but it could certainly happen
again. It might even have happened without anyone noticing exactly
what was going on (just mysterious dependency issues that magically
went away after a bit).
So: now we have convenient self-service side tags, *please use them*.
Especially for something as major as a bump of perl that changes
dependencies of packages built against it like this. Side tags avoid
this mess entirely. Using the mechanism to produce an update from a
side tag also makes it easier to make sure all the right packages are
in the update and they don't get incorrectly submitted as separate
updates.
Thanks folks!
OK, so this is largely my fault. Whilst I didn't do the initial perl
5.34.1 build and update, I did set up the buildroot override and the
builds of the two packages (perl-PAR-Packer and polymake) that have
hard dependencies on the specific perl version they were built against.
Unfortunately the polymake build took over 24 hours on armv7hl but
after that I could have and should have expired the buildroot override
to prevent the follow-up issues affecting other perl-related builds.
The issue was already known about and it was already planned to do the
forthcoming update for f35 to perl 5.34.1 in a side tag
(
https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5).
In mitigation, my thinking was that since the f36 beta freeze is still
ongoing, the perl update and its hard dependencies would almost
certainly have been pushed to stable at the same time anyway. In
addition, since those updates were created prior to the ones for
the other modules that were incidentally built against 5.34.1, perl
itself would have been stable before the later updates arrived. So in
practice there probably wouldn't have been any mysterious dependency
issues in this particular case. And whilst I could have added the
delayed polymake build to the perl+perl-PAR-Packer update, I chose not
to so as not to reset the timer on the perl+perl-PAR-Packer update,
which would have set it back by 2 days and potentially resulted in
other modules being pushed to stable before the underlying perl update.
Anyway, won't happen again.
Regards, Paul.