Please use side tags for backwards-incompatible bumps of major packages, not buildroot overrides

[Adam Williamson]

I found quite a big mess today, caused by an attempt to bump perl to
5.34.1 in Fedora 36:

https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4

Because some packages depend on the exact perl interpreter version, the
maintainer made a buildroot override for perl:

https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36

and rebuilt them. Okay.

The problem with this is, we have lots of perl modules. People build
them all the time. And buildroot overrides apply to *everything*.

So, while the buildroot override has been valid, multiple *other* perl
modules have been unwittingly built against it:

perl-Scalar-List-Utils:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732
perl-Parallel-Pipes:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527
perl-PPIx-Regexp:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522
perl-Crypt-SSLeay:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521
perl-Crypt-OpenSSL-PKCS10:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471

...and those are just the ones I found so far. Because they were built
against 5.34.1, all of those builds require
"perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1
from updates-testing. They cannot be installed with perl-5.34.0 from
stable. But the maintainers likely had no idea about this - buildroot
overrides are not at all discoverable, there is zero indication your
package is building against one when you build it - and have created
separate updates for those packages:

https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2
https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901
https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f
https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964
https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e

Users testing those updates will likely not notice the problem, because
they'll have *all* of updates-testing enabled when they update, and
perl-5.34.1 will be pulled in. But if any of those updates is pushed
stable before the perl update is pushed stable, it will fail to install
for anyone who does not have updates-testing enabled.

This is quite a mess. I'm going to try and clean it up, but it'll be a
bit ugly (there are a couple of ways I can do it, but both will involve
doing bumps-and-rebuilds of the affected packages with no changes).

I've been worried about this sort of thing happening for a while due to
the undesirable properties of buildroot overrides. This is the biggest
real-world case I've seen, but it could certainly happen again. It
might even have happened without anyone noticing exactly what was going
on (just mysterious dependency issues that magically went away after a
bit).

So: now we have convenient self-service side tags, *please use them*.
Especially for something as major as a bump of perl that changes
dependencies of packages built against it like this. Side tags avoid
this mess entirely. Using the mechanism to produce an update from a
side tag also makes it easier to make sure all the right packages are
in the update and they don't get incorrectly submitted as separate
updates.

Thanks folks!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Paul Howarth]

On Tue, 22 Mar 2022 11:48:57 -0700
Adam Williamson <adamwill(a)fedoraproject.org&gt; wrote:

>  I found quite a big mess today, caused by an attempt to bump perl to
>  5.34.1 in Fedora 36:
>  
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4
>  
>  Because some packages depend on the exact perl interpreter version,
>  the maintainer made a buildroot override for perl:
>  
>  https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36
>  
>  and rebuilt them. Okay.
>  
>  The problem with this is, we have lots of perl modules. People build
>  them all the time. And buildroot overrides apply to *everything*.
>  
>  So, while the buildroot override has been valid, multiple *other* perl
>  modules have been unwittingly built against it:
>  
>  perl-Scalar-List-Utils:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732
>  perl-Parallel-Pipes:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527
>  perl-PPIx-Regexp:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522
>  perl-Crypt-SSLeay:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521
>  perl-Crypt-OpenSSL-PKCS10:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471
>  
>  ...and those are just the ones I found so far. Because they were built
>  against 5.34.1, all of those builds require
>  "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1
>  from updates-testing. They cannot be installed with perl-5.34.0 from
>  stable. But the maintainers likely had no idea about this - buildroot
>  overrides are not at all discoverable, there is zero indication your
>  package is building against one when you build it - and have created
>  separate updates for those packages:
>  
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e
>  
>  Users testing those updates will likely not notice the problem,
>  because they'll have *all* of updates-testing enabled when they
>  update, and perl-5.34.1 will be pulled in. But if any of those
>  updates is pushed stable before the perl update is pushed stable, it
>  will fail to install for anyone who does not have updates-testing
>  enabled.
>  
>  This is quite a mess. I'm going to try and clean it up, but it'll be a
>  bit ugly (there are a couple of ways I can do it, but both will
>  involve doing bumps-and-rebuilds of the affected packages with no
>  changes).
>  
>  I've been worried about this sort of thing happening for a while due
>  to the undesirable properties of buildroot overrides. This is the
>  biggest real-world case I've seen, but it could certainly happen
>  again. It might even have happened without anyone noticing exactly
>  what was going on (just mysterious dependency issues that magically
>  went away after a bit).
>  
>  So: now we have convenient self-service side tags, *please use them*.
>  Especially for something as major as a bump of perl that changes
>  dependencies of packages built against it like this. Side tags avoid
>  this mess entirely. Using the mechanism to produce an update from a
>  side tag also makes it easier to make sure all the right packages are
>  in the update and they don't get incorrectly submitted as separate
>  updates.
>  
>  Thanks folks! 
OK, so this is largely my fault. Whilst I didn't do the initial perl
5.34.1 build and update, I did set up the buildroot override and the
builds of the two packages (perl-PAR-Packer and polymake) that have
hard dependencies on the specific perl version they were built against.
Unfortunately the polymake build took over 24 hours on armv7hl but
after that I could have and should have expired the buildroot override
to prevent the follow-up issues affecting other perl-related builds.
The issue was already known about and it was already planned to do the
forthcoming update for f35 to perl 5.34.1 in a side tag
(https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5).

In mitigation, my thinking was that since the f36 beta freeze is still
ongoing, the perl update and its hard dependencies would almost
certainly have been pushed to stable at the same time anyway. In
addition, since those updates were created prior to the ones for
the other modules that were incidentally built against 5.34.1, perl
itself would have been stable before the later updates arrived. So in
practice there probably wouldn't have been any mysterious dependency
issues in this particular case. And whilst I could have added the
delayed polymake build to the perl+perl-PAR-Packer update, I chose not
to so as not to reset the timer on the perl+perl-PAR-Packer update,
which would have set it back by 2 days and potentially resulted in
other modules being pushed to stable before the underlying perl update.

Anyway, won't happen again.

Regards, Paul.

[Adam Williamson]

On Wed, 2022-03-23 at 08:39 +0000, Paul Howarth wrote:
>  
>  In mitigation, my thinking was that since the f36 beta freeze is still
>  ongoing, the perl update and its hard dependencies would almost
>  certainly have been pushed to stable at the same time anyway. In
>  addition, since those updates were created prior to the ones for
>  the other modules that were incidentally built against 5.34.1, perl
>  itself would have been stable before the later updates arrived. So in
>  practice there probably wouldn't have been any mysterious dependency
>  issues in this particular case. And whilst I could have added the
>  delayed polymake build to the perl+perl-PAR-Packer update, I chose not
>  to so as not to reset the timer on the perl+perl-PAR-Packer update,
>  which would have set it back by 2 days and potentially resulted in
>  other modules being pushed to stable before the underlying perl update.
>   
Thanks Paul! So yeah, it's true about the freeze and the ordering, but
it's still possible for undesirable things to happen, thanks to
autopush. All the updates seem to have default autopush settings - +3
karma, 14 days time. So if one of the dependent updates happened to get
+3 karma before the perl update, and the perl update wasn't submitted
manually, they could still have been submitted first.

The freeze could potentially be lifted quite soon, since go/no-go is
tomorrow, so it wouldn't necessarily save us.

And finally, since polymake depends on *exactly* the version of perl
(it's not a >= situation), I believe polymake and perl really have to
be submitted *exactly together* in order not to break anything. AIUI,
if perl gets pushed before polymake, the polymake in stable will no
longer be installable with the perl in stable. This is why I think it'd
still be best to edit polymake into the perl update, even though the
build does take forever (my rebuild is still running :|)

What might be safest overall is to edit my polymake rebuild into the
perl update when it's done, then make sure the perl update gets +1 or
+2 karma (whichever it needs) and is submitted for stable manually
ASAP.

Thanks again!
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Adam Williamson]

On Wed, 2022-03-23 at 08:39 +0000, Paul Howarth wrote:
>  
>  OK, so this is largely my fault. Whilst I didn't do the initial perl
>  5.34.1 build and update, I did set up the buildroot override and the
>  builds of the two packages (perl-PAR-Packer and polymake) that have
>  hard dependencies on the specific perl version they were built against.
>  Unfortunately the polymake build took over 24 hours on armv7hl but
>  after that I could have and should have expired the buildroot override
>  to prevent the follow-up issues affecting other perl-related builds.
>  The issue was already known about and it was already planned to do the
>  forthcoming update for f35 to perl 5.34.1 in a side tag
>  (https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5). 
Oh sorry, forgot to mention a couple of other things:

1) Neat trick: I'm pretty sure the buildroot override only needs to be
valid until all the build dependencies have been installed. For my
polymake rebuild, I put the override back in place, fired the polymake
build, waited till all the build tasks for the different arch had
installed build dependencies, then expired the override again. It
doesn't need to stay valid for the whole time the actual compilation
stage is happening.

2) Just to note what I wound up doing here - aside from the special
polymake case, I found (I hope) all the packages that got built against
5.34.1, bumped and rebuilt them against 5.34.0, and edited the
standalone updates to have the new builds, which will work with both
5.34.0 and 5.34.1, so whatever order they get pushed in things should
be OK.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Kevin Fenzi]

I wonder... should we stop allowing buildroot overrides?

Or at the very least add a admon to adding a new one in bodhi,
explaining that you should probibly use a side tag, etc?

kevin

[Miro Hrončok]

On 23. 03. 22 18:41, Kevin Fenzi wrote:
>  I wonder... should we stop allowing buildroot overrides? 
I wondered this for a long time. Unfortunately I still find usecases for 
buildroot overrides. E.g. when we ship new versions of some macro packages etc. 
and we want them available even before the update reaches stable (e.g. to see 
them not breaking other packages in Kochei).

They should certainly be discouraged as a mean to update interdependent packages.

>  Or at the very least add a admon to adding a new one in bodhi,
>  explaining that you should probibly use a side tag, etc? 
+1

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Paul Howarth]

On Wed, 23 Mar 2022 10:41:52 -0700
Kevin Fenzi <kevin(a)scrye.com&gt; wrote:

>  I wonder... should we stop allowing buildroot overrides?
>  
>  Or at the very least add a admon to adding a new one in bodhi,
>  explaining that you should probibly use a side tag, etc? 
They're still very useful when bringing up new EPEL releases. When you
have things like the perl stack with lots of interdependencies I think
side tags would be quite unwieldy. They got lots of use (not just by
me!) when bringing up EPEL-8 and EPEL-9 in particular.

Paul.

[Elliott Sales de Andrade]

On Wed, Mar 23, 2022 at 12:16 PM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Wed, 2022-03-23 at 08:39 +0000, Paul Howarth wrote:
>  >
>  > OK, so this is largely my fault. Whilst I didn't do the initial perl
>  > 5.34.1 build and update, I did set up the buildroot override and the
>  > builds of the two packages (perl-PAR-Packer and polymake) that have
>  > hard dependencies on the specific perl version they were built against.
>  > Unfortunately the polymake build took over 24 hours on armv7hl but
>  > after that I could have and should have expired the buildroot override
>  > to prevent the follow-up issues affecting other perl-related builds.
>  > The issue was already known about and it was already planned to do the
>  > forthcoming update for f35 to perl 5.34.1 in a side tag
>  > (https://bugzilla.redhat.com/show_bug.cgi?id=2064808#c5).
>
>  Oh sorry, forgot to mention a couple of other things:
>
>  1) Neat trick: I'm pretty sure the buildroot override only needs to be
>  valid until all the build dependencies have been installed. For my
>  polymake rebuild, I put the override back in place, fired the polymake
>  build, waited till all the build tasks for the different arch had
>  installed build dependencies, then expired the override again. It
>  doesn't need to stay valid for the whole time the actual compilation
>  stage is happening.
>  
Note, this override isn't strictly needed either. You can create a
side tag, and tag in _any_ build you need to fix things. Pretty sure
you can even tag in older versions of things if necessary. You just
have to remember to untag the extra builds before creating the update
in Bodhi, if you're creating it from the side tag.

>  2) Just to note what I wound up doing here - aside from the special
>  polymake case, I found (I hope) all the packages that got built against
>  5.34.1, bumped and rebuilt them against 5.34.0, and edited the
>  standalone updates to have the new builds, which will work with both
>  5.34.0 and 5.34.1, so whatever order they get pushed in things should
>  be OK.
>  --
>  Adam Williamson
>  Fedora QA
>  IRC: adamw | Twitter: adamw_ha
>  https://www.happyassassin.net
>  
-- 
Elliott

[Adam Williamson]

On Wed, 2022-03-23 at 18:13 -0400, Elliott Sales de Andrade wrote:
>  > 
>  > 1) Neat trick: I'm pretty sure the buildroot override only needs to be
>  > valid until all the build dependencies have been installed. For my
>  > polymake rebuild, I put the override back in place, fired the polymake
>  > build, waited till all the build tasks for the different arch had
>  > installed build dependencies, then expired the override again. It
>  > doesn't need to stay valid for the whole time the actual compilation
>  > stage is happening.
>  > 
>  
>  Note, this override isn't strictly needed either. You can create a
>  side tag, and tag in _any_ build you need to fix things. Pretty sure
>  you can even tag in older versions of things if necessary. You just
>  have to remember to untag the extra builds before creating the update
>  in Bodhi, if you're creating it from the side tag. 
Yeah, I could've made a new side tag for this, but at this point the
override existed and I had it open in a browser tab so I could
basically turn it off and on like a light switch, so I just went with
that. :D I was just noting this as a detail about how buildroot
overrides work, if you're really determined to use one.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Jerry James]

On Wed, Mar 23, 2022 at 10:16 AM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  2) Just to note what I wound up doing here - aside from the special
>  polymake case, I found (I hope) all the packages that got built against
>  5.34.1, bumped and rebuilt them against 5.34.0, and edited the
>  standalone updates to have the new builds, which will work with both
>  5.34.0 and 5.34.1, so whatever order they get pushed in things should
>  be OK. 
It's probably time for me to chime in about polymake again.  The
polymake package has relatively few Fedora users, and takes a long
time to build.  Perl, on the other hand, has a large number of users.
I don't think we should hold up perl releases for a polymake build.
It's okay with me to just go ahead and update perl and let polymake be
broken.  I'll notice it is broken and do the necessary builds.  The
few polymake users will be annoyed that they can't update to the
latest perl and will send me nastygrams.  I'll respond with a canned
email I have sitting around because exactly this situation has
happened many times in the past.

Of course, I am also trying to give polymake away, so maybe I should
not put my (purely hypothetical so far) successor on the hook like
that. :-)

The other possibility is that somebody who knows more about perl than
me breaks polymake's hard dependency on the exact version it was built
with.  I think upstream is concerned about that because polymake
inserts its tentacles into all corners of perl's C innards, so even a
small change could require a rebuild.
-- 
Jerry James
http://www.jamezone.org/

[Miro Hrončok]

On 22. 03. 22 19:48, Adam Williamson wrote:
>  I found quite a big mess today, caused by an attempt to bump perl to
>  5.34.1 in Fedora 36:
>  
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4
>  
>  Because some packages depend on the exact perl interpreter version, the
>  maintainer made a buildroot override for perl:
>  
>  https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36
>  
>  and rebuilt them. Okay.
>  
>  The problem with this is, we have lots of perl modules. People build
>  them all the time. And buildroot overrides apply to *everything*.
>  
>  So, while the buildroot override has been valid, multiple *other* perl
>  modules have been unwittingly built against it:
>  
>  perl-Scalar-List-Utils:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732
>  perl-Parallel-Pipes:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527
>  perl-PPIx-Regexp:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522
>  perl-Crypt-SSLeay:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521
>  perl-Crypt-OpenSSL-PKCS10:
>  https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471
>  
>  ...and those are just the ones I found so far. Because they were built
>  against 5.34.1, all of those builds require
>  "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1
>  from updates-testing. They cannot be installed with perl-5.34.0 from
>  stable. But the maintainers likely had no idea about this - buildroot
>  overrides are not at all discoverable, there is zero indication your
>  package is building against one when you build it - and have created
>  separate updates for those packages:
>  
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964
>  https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e
>  
>  Users testing those updates will likely not notice the problem, because
>  they'll have *all* of updates-testing enabled when they update, and
>  perl-5.34.1 will be pulled in. But if any of those updates is pushed
>  stable before the perl update is pushed stable, it will fail to install
>  for anyone who does not have updates-testing enabled.
>  
>  This is quite a mess. I'm going to try and clean it up, but it'll be a
>  bit ugly (there are a couple of ways I can do it, but both will involve
>  doing bumps-and-rebuilds of the affected packages with no changes).
>  
>  I've been worried about this sort of thing happening for a while due to
>  the undesirable properties of buildroot overrides. This is the biggest
>  real-world case I've seen, but it could certainly happen again. It
>  might even have happened without anyone noticing exactly what was going
>  on (just mysterious dependency issues that magically went away after a
>  bit).
>  
>  So: now we have convenient self-service side tags, *please use them*.
>  Especially for something as major as a bump of perl that changes
>  dependencies of packages built against it like this. Side tags avoid
>  this mess entirely. Using the mechanism to produce an update from a
>  side tag also makes it easier to make sure all the right packages are
>  in the update and they don't get incorrectly submitted as separate
>  updates.
>  
>  Thanks folks! 
Thanks for this, Adam.

I will add one more reason:

When the automatic "fails to install" reports run, they use the Koji buidlroot.

Every now and then, a bugzilla is reported like this one:

https://bugzilla.redhat.com/show_bug.cgi?id=2066728

It is not a false positive, the dependency problem exists, but it only exists 
because there is an override:

https://bodhi.fedoraproject.org/overrides/R-4.1.3-1.fc36

Please, use side tags, not overrides, when updating inter-dependent packages, 
it avoid (even temporary) breakage:

https://docs.fedoraproject.org/en-US/fesco/Updates_Policy/#updating-inter...

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Mattia Verga]

So, now that we have side-tags to perform this kind of builds, does the
buildroot override existence still make sense? Is there any use case
that still requires BR overrides and cannot be done with side-tags?

Mattia

[Miro Hrončok]

On 23. 03. 22 18:40, Mattia Verga via devel wrote:
>  So, now that we have side-tags to perform this kind of builds, does
> the
>  buildroot override existence still make sense? Is there any use case
>  that still requires BR overrides and cannot be done with side-tags? 
As I've said elsewhere in the thread: 
https://lists.fedoraproject.org/archives/list/[e-mail].o...

Happy to elaborate and seek for alternate solutions.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Petr Pisar]

V Wed, Mar 23, 2022 at 05:40:28PM +0000, Mattia Verga via devel napsal(a):
>  So, now that we have side-tags to perform this kind of builds, does
> the
>  buildroot override existence still make sense? Is there any use case
>  that still requires BR overrides and cannot be done with side-tags?
>   
I use overrides pretty extensively when populating new EPEL. If you have
a deep dependency tree and the packages are scattered among many maintainers,
it's much faster to use an override than to block depending packages for
a week. Side tags also do not work there because every new package would reset
the testing period. And if I'm not mistaken non-proven packagers cannot edit
other's updates. And I don't count communication overhead. People would end up
mixing various side tags and crosstagging builds.

-- Petr

[Mattia Verga]

Il 24/03/22 09:12, Petr Pisar ha scritto:
>  V Wed, Mar 23, 2022 at 05:40:28PM +0000, Mattia Verga via devel
> napsal(a):
> > So, now that we have side-tags to perform this kind of builds, does the
> > buildroot override existence still make sense? Is there any use case
> > that still requires BR overrides and cannot be done with side-tags?
> >
>  I use overrides pretty extensively when populating new EPEL. If you have
>  a deep dependency tree and the packages are scattered among many maintainers,
>  it's much faster to use an override than to block depending packages for
>  a week. Side tags also do not work there because every new package would reset
>  the testing period. And if I'm not mistaken non-proven packagers cannot edit
>  other's updates. And I don't count communication overhead. People would end up
>  mixing various side tags and crosstagging builds.
>  
Maybe BR overrides usage should be restricted only to users with special
needs (users in provenpackager or releng groups), while "normal" users
should be forced to take the side-tag way?

Mattia

[Petr Pisar]

V Tue, Mar 29, 2022 at 05:04:29PM +0000, Mattia Verga via devel napsal(a):
>  Il 24/03/22 09:12, Petr Pisar ha scritto:
>  > V Wed, Mar 23, 2022 at 05:40:28PM +0000, Mattia Verga via devel napsal(a):
>  >> So, now that we have side-tags to perform this kind of builds, does the
>  >> buildroot override existence still make sense? Is there any use case
>  >> that still requires BR overrides and cannot be done with side-tags?
>  >>
>  > I use overrides pretty extensively when populating new EPEL. If you have
>  > a deep dependency tree and the packages are scattered among many maintainers,
>  > it's much faster to use an override than to block depending packages for
>  > a week. Side tags also do not work there because every new package would reset
>  > the testing period. And if I'm not mistaken non-proven packagers cannot edit
>  > other's updates. And I don't count communication overhead. People would end
> up
>  > mixing various side tags and crosstagging builds.
>  >
>  Maybe BR overrides usage should be restricted only to users with special
>  needs (users in provenpackager or releng groups), while "normal" users
>  should be forced to take the side-tag way?
>   
I'm not sure. There are more people in Perl stack who do it and are not
proven
packagers. Maybe EPEL in its initial phase should work more like Rawhide, i.e.
without updates-testing repo.

-- Petr

[Gary Buhrmaster]

On Tue, Mar 29, 2022 at 5:05 PM Mattia Verga via devel
<devel(a)lists.fedoraproject.org&gt; wrote:

>  Maybe BR overrides usage should be restricted only to users with
> special
>  needs (users in provenpackager or releng groups), while "normal" users
>  should be forced to take the side-tag way? 
As always, there are special cases.  I certainly
would not like to see packagers giving up on
packaging because the impediments placed
in their way to accomplish what they want
(and need) to do are too onerous.

Before taking any next steps I would like to see
if Adam's request to reconsider BR overrides
(and typically move to side tags) will reduce
usage.

[Miro Hrončok]

On 22. 03. 22 19:48, Adam Williamson wrote:
>  now we have convenient self-service side tags,*please use them*.
>  Especially for something as major as a bump of perl that changes
>  dependencies of packages built against it like this. Side tags avoid
>  this mess entirely. Using the mechanism to produce an update from a
>  side tag also makes it easier to make sure all the right packages are
>  in the update and they don't get incorrectly submitted as separate
>  updates.
>  
>  Thanks folks! 
BTW it seems our packaging documentation does not recognize side tags for 
anything else than rawhide an early branched: 
https://pagure.io/fedora-docs/package-maintainer-docs/issue/72

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok