Some reasons I really dislike buildroot overrides and would like us to get rid of them soon

[Adam Williamson]

So there's this CI ticket ATM[0] about whether the environment in which
CI tests are run should or should not include and update from the
'buildroot' repo, which contains both:

1. Packages that have been pushed stable since the last time a compose
succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
Branched compose, for stable releases it's an updates compose)
2. Packages that have active buildroot overrides

Thinking about this reminded me again why buildroot overrides are such
a bad idea:
https://pagure.io/fedora-ci/general/issue/376#comment-830638

Buildroot overrides have unpredictable consequences for builds, updates
*and* tests. I really feel like we should consider disallowing them and
requiring all rebases to be done using side tags. Side tags are a
*much* better design that avoids the problem of packages unexpectedly
being built against a buildroot override somebody else submitted, and
means test systems aren't stuck in a bind of not really knowing for
sure what other packages should be installed when testing any given
package.

What does everyone else think? Has the time come? Or is there more we
need to do to make side tags usable for all cases before getting rid of
overrides?

[0]: https://pagure.io/fedora-ci/general/issue/376
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Richard Shaw]

On Fri, Dec 2, 2022 at 3:30 PM Adam Williamson <adamwill(a)fedoraproject.org&gt;
wrote:

>  What does everyone else think? Has the time come? Or is there more we
>  need to do to make side tags usable for all cases before getting rid of
>  overrides?
>  
I wasn't aware of the CI issues so that seems like the last nail in the
coffin. The only time I've used one recently is because I forgot about a
package dependency and it was easier to create a buildroot override than
look up how to re-tag the build into a side tag.

Thanks,
Richard

[Fabio Valentini]

On Fri, Dec 2, 2022 at 11:13 PM Richard Shaw <hobbes1069(a)gmail.com&gt; wrote:
>  On Fri, Dec 2, 2022 at 3:30 PM Adam Williamson <adamwill(a)fedoraproject.org&gt; wrote:
> >
> >
> > What does everyone else think? Has the time come? Or is there more we
> > need to do to make side tags usable for all cases before getting rid of
> > overrides?
>
>
>  I wasn't aware of the CI issues so that seems like the last nail in the coffin. The
> only time I've used one recently is because I forgot about a package dependency and it
> was easier to create a buildroot override than look up how to re-tag the build into a side
> tag. 
That has been my experience as well: Use on-demand side-tags for
everything, and use buildroot overrides only when using a side-tag
would be very inconvenient (i.e. tagging in builds that were already
submitted as another update into another side tag). But I don't think
buildroot overrides are really *necessary* for any workflows anymore,
either ...

Not sure if we should turn them off entirely, but we could restrict
their use somewhat? For example, only allow people in "releng" or
"qa"
groups to file them.

Fabio

[Maxwell G]

On Fri Dec 2, 2022 at 13:30 -0800, Adam Williamson wrote:
>  What does everyone else think? Has the time come? Or is there more
> we
>  need to do to make side tags usable for all cases before getting rid of
>  overrides? 
-1. For many use cases, side tags are the correct solution and buildroot
overrides should not be used. However, there are occasions where they are still
useful. If people are improperly using buildroot overrides and breaking the
buildroot, the correct solution is education and documentation, not banning
them entirely. Examples:

1. There is a Go compiler release that has patches for a CVE (i.e. almost every
   Go release ). I create buildroot overrides so all new builds are able to
   immediately start using it.

2. The same thing above but with other Go libraries that are statically linked
   into Go binaries.

3. There is a bug in macros that break builds. The fix should be available in
   the buildroot for all new builds immediately.

4. Some other buildtime only dependency update provides a bugfix that is needed
   for packages that BuildRequire it.


On Fri Dec 2, 2022 at 23:21 +0100, Fabio Valentini wrote:
>  Not sure if we should turn them off entirely, but we could restrict
>  their use somewhat? For example, only allow people in "releng" or
> "qa"
>  groups to file them. 
I'd prefer not to have to bother releng every time I need to do this.

--
Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Kalev Lember]

On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson <adamwill(a)fedoraproject.org&gt;
wrote:

>  So there's this CI ticket ATM[0] about whether the environment in
> which
>  CI tests are run should or should not include and update from the
>  'buildroot' repo, which contains both:
>
>  1. Packages that have been pushed stable since the last time a compose
>  succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>  Branched compose, for stable releases it's an updates compose)
>  2. Packages that have active buildroot overrides
>
>  Thinking about this reminded me again why buildroot overrides are such
>  a bad idea:
>  https://pagure.io/fedora-ci/general/issue/376#comment-830638
>
>  Buildroot overrides have unpredictable consequences for builds, updates
>  *and* tests. I really feel like we should consider disallowing them and
>  requiring all rebases to be done using side tags. Side tags are a
>  *much* better design that avoids the problem of packages unexpectedly
>  being built against a buildroot override somebody else submitted, and
>  means test systems aren't stuck in a bind of not really knowing for
>  sure what other packages should be installed when testing any given
>  package.
>
>  What does everyone else think? Has the time come? Or is there more we
>  need to do to make side tags usable for all cases before getting rid of
>  overrides?
>  
I would say that side tags are almost always the correct tool for ABI
rebuilds. I imagine people who submit global buildroot overrides to handle
a library soname bump are almost always doing it because they haven't
learned the "new" ways of doing it.

I'd go as far as to say that anyone who does ABI rebuilds using global
buildroot overrides are doing it wrong.

However, having said that, I also find buildroot overrides useful. Some
examples:

1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new
version through the freeze, but that includes a library soname bump.

What I would do in that case is submit the GNOME megaupdate to Bodhi, and
also submit the library as a buildroot override to ensure that nothing can
build against the old soname -- I am fairly confident that the GNOME
megaupdate, together with the soname bump makes it to stable first.

2) I need to do a container build and include a new CVE fix (as it's
critical and we need to get fixes out ASAP), but that package build to
include in the container is only in updates-testing.

What I'd do in this case is to submit a buildroot override because
everything that's overridden gets included in container builds. After my
container build is done I'd expire the buildroot override.

3) We've had some "fun" issues with sysprof symbols leaking out from the
GTK stack into other libraries consuming it. This has caused subtle ABI
issues and when working on a fix and to make sure no package can build
against the "wrong" GTK version, I've used buildroot overrides.

4) Compiler issues, with compilers producing broken code.

To test the fixes and make sure packages start using the new fixed versions
ASAP, a buildroot override is often useful.

I could continue with the list but I think you get my point that there are
some cases where it's useful :) However I'd never use buildroot overrides
for soname bump rebuilds; that's what side tags are good for.

Maybe doing occasional blog posts and teaching maintainers how to do multi
package updates would be helpful? Not sure what else we could do to help
improve the situation here.

-- 
Kalev

[Miro Hrončok]

On 03. 12. 22 0:14, Kalev Lember wrote:
>  On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson
> <adamwill(a)fedoraproject.org 
>  <mailto:[e-mail]>> wrote:
>  
>      So there's this CI ticket ATM[0] about whether the environment in which
>      CI tests are run should or should not include and update from the
>      'buildroot' repo, which contains both:
>  
>      1. Packages that have been pushed stable since the last time a compose
>      succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>      Branched compose, for stable releases it's an updates compose)
>      2. Packages that have active buildroot overrides
>  
>      Thinking about this reminded me again why buildroot overrides are such
>      a bad idea:
>      https://pagure.io/fedora-ci/general/issue/376#comment-830638
>      <https://pagure.io/fedora-ci/general/issue/376#comment-830638>
>  
>      Buildroot overrides have unpredictable consequences for builds, updates
>      *and* tests. I really feel like we should consider disallowing them and
>      requiring all rebases to be done using side tags. Side tags are a
>      *much* better design that avoids the problem of packages unexpectedly
>      being built against a buildroot override somebody else submitted, and
>      means test systems aren't stuck in a bind of not really knowing for
>      sure what other packages should be installed when testing any given
>      package.
>  
>      What does everyone else think? Has the time come? Or is there more we
>      need to do to make side tags usable for all cases before getting rid of
>      overrides?
>  
>  
>  I would say that side tags are almost always the correct tool for ABI rebuilds. 
>  I imagine people who submit global buildroot overrides to handle a library 
>  soname bump are almost always doing it because they haven't learned the
> "new" 
>  ways of doing it.
>  
>  I'd go as far as to say that anyone who does ABI rebuilds using global 
>  buildroot overrides are doing it wrong.
>  
>  However, having said that, I also find buildroot overrides useful. Some examples:
>  
>  1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new 
>  version through the freeze, but that includes a library soname bump.
>  
>  What I would do in that case is submit the GNOME megaupdate to Bodhi, and also 
>  submit the library as a buildroot override to ensure that nothing can build 
>  against the old soname -- I am fairly confident that the GNOME megaupdate, 
>  together with the soname bump makes it to stable first.
>  
>  2) I need to do a container build and include a new CVE fix (as it's critical 
>  and we need to get fixes out ASAP), but that package build to include in the 
>  container is only in updates-testing.
>  
>  What I'd do in this case is to submit a buildroot override because everything 
>  that's overridden gets included in container builds. After my container build 
>  is done I'd expire the buildroot override.
>  
>  3) We've had some "fun" issues with sysprof symbols leaking out from the
> GTK 
>  stack into other libraries consuming it. This has caused subtle ABI issues and 
>  when working on a fix and to make sure no package can build against the "wrong"
>
>  GTK version, I've used buildroot overrides.
>  
>  4) Compiler issues, with compilers producing broken code.
>  
>  To test the fixes and make sure packages start using the new fixed versions 
>  ASAP, a buildroot override is often useful.
>  
>  I could continue with the list but I think you get my point that there are some 
>  cases where it's useful :) However I'd never use buildroot overrides for soname 
>  bump rebuilds; that's what side tags are good for.
>  
>  Maybe doing occasional blog posts and teaching maintainers how to do multi 
>  package updates would be helpful? Not sure what else we could do to help 
>  improve the situation here. 
I am pretty much with Kalev on this.

The only way to land ABI (or similar) rebuilds should be side tags, but I still 
use overrides for other stuff.

E.g. when I need a change in an RPM macro package to make other packages build 
properly. I see no reason for creating a single bodhi update with the RPM macro 
package and couple packages that were blocked on that update.

My latest buidlroot overrides were:

https://bodhi.fedoraproject.org/overrides/epel-rpm-macros-8-34
See 
https://src.fedoraproject.org/rpms/epel-rpm-macros/pull-request/54#commen...

https://bodhi.fedoraproject.org/overrides/python-setuptools-59.6.0-3.fc36
https://bodhi.fedoraproject.org/overrides/python-pip-21.3.1-4.fc36
https://bodhi.fedoraproject.org/overrides/python-rpm-macros-3.11-5.fc37
To workaround https://pagure.io/fedora-ci/general/issue/240

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Adam Williamson]

On Sat, 2022-12-03 at 00:14 +0100, Kalev Lember wrote:
>  On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson
> <adamwill(a)fedoraproject.org&gt;
>  wrote:
>  
>  > So there's this CI ticket ATM[0] about whether the environment in which
>  > CI tests are run should or should not include and update from the
>  > 'buildroot' repo, which contains both:
>  > 
>  > 1. Packages that have been pushed stable since the last time a compose
>  > succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>  > Branched compose, for stable releases it's an updates compose)
>  > 2. Packages that have active buildroot overrides
>  > 
>  > Thinking about this reminded me again why buildroot overrides are such
>  > a bad idea:
>  > https://pagure.io/fedora-ci/general/issue/376#comment-830638
>  > 
>  > Buildroot overrides have unpredictable consequences for builds, updates
>  > *and* tests. I really feel like we should consider disallowing them and
>  > requiring all rebases to be done using side tags. Side tags are a
>  > *much* better design that avoids the problem of packages unexpectedly
>  > being built against a buildroot override somebody else submitted, and
>  > means test systems aren't stuck in a bind of not really knowing for
>  > sure what other packages should be installed when testing any given
>  > package.
>  > 
>  > What does everyone else think? Has the time come? Or is there more we
>  > need to do to make side tags usable for all cases before getting rid of
>  > overrides?
>  > 
>  
>  I would say that side tags are almost always the correct tool for ABI
>  rebuilds. I imagine people who submit global buildroot overrides to handle
>  a library soname bump are almost always doing it because they haven't
>  learned the "new" ways of doing it.
>  
>  I'd go as far as to say that anyone who does ABI rebuilds using global
>  buildroot overrides are doing it wrong.
>  
>  However, having said that, I also find buildroot overrides useful. Some
>  examples:
>  
>  1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new
>  version through the freeze, but that includes a library soname bump.
>  
>  What I would do in that case is submit the GNOME megaupdate to Bodhi, and
>  also submit the library as a buildroot override to ensure that nothing can
>  build against the old soname -- I am fairly confident that the GNOME
>  megaupdate, together with the soname bump makes it to stable first.
>  
>  2) I need to do a container build and include a new CVE fix (as it's
>  critical and we need to get fixes out ASAP), but that package build to
>  include in the container is only in updates-testing.
>  
>  What I'd do in this case is to submit a buildroot override because
>  everything that's overridden gets included in container builds. After my
>  container build is done I'd expire the buildroot override.
>  
>  3) We've had some "fun" issues with sysprof symbols leaking out from the
>  GTK stack into other libraries consuming it. This has caused subtle ABI
>  issues and when working on a fix and to make sure no package can build
>  against the "wrong" GTK version, I've used buildroot overrides.
>  
>  4) Compiler issues, with compilers producing broken code.
>  
>  To test the fixes and make sure packages start using the new fixed versions
>  ASAP, a buildroot override is often useful.
>  
>  I could continue with the list but I think you get my point that there are
>  some cases where it's useful :) However I'd never use buildroot overrides
>  for soname bump rebuilds; that's what side tags are good for. 
Well, hmm.

The examples you provide are definitely interesting. They all
essentially boil down to, well, "I know exactly how this process works
and I'm gonna take advantage of that to achieve the right outcome
behind the scenes". Which, you know, we have a lot of stuff like that,
but it can be difficult to work with in ways like this.

I do see the value in your use of the buildroot in those cases, but I
can't help thinking there must be a *better* way. But, I'm not sure
I've thought of a practical one yet. All the ones that came off the top
of my head turn out to be silly after thirty seconds thinking about it.
Ugh.

Hum. I suppose if there really were only uses of the buildroot like the
above...then...it should be *fairly* safe for test systems to always
test against the buildroot repo contents. Uh. Maybe. I don't love it,
but I don't know what's better.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Vitaly Zaitsev]

On 02/12/2022 22:30, Adam Williamson wrote:
>  1. Packages that have been pushed stable since the last time a
> compose
>  succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>  Branched compose, for stable releases it's an updates compose)
>  2. Packages that have active buildroot overrides 
I use buildroot overrides because generating side tags for stable 
releases of Fedora is a very slow process.

Two days ago I created F36, F37 and F38 side tags. F38 side tag was 
ready in 10 minutes, and F36+F37 wasn't generated even after 6 hours. 
That's why I created a new buildroot override and built everything directly.

Before disabling buildroot overrides this issue must be fixed. Waiting 
for 6+ hours is unacceptable.

Also I think we should limit the existing side tags for 1 week. After 
this date they should be removed automatically.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Vitaly Zaitsev]

On 02/12/2022 23:21, Fabio Valentini wrote:
>  Not sure if we should turn them off entirely, but we could restrict
>  their use somewhat? For example, only allow people in "releng" or
> "qa"
>  groups to file them. 
Strongly -1.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Zbigniew Jędrzejewski-Szmek]

On Fri, Dec 02, 2022 at 03:35:31PM -0800, Adam Williamson wrote:
>  On Sat, 2022-12-03 at 00:14 +0100, Kalev Lember wrote:
>  > On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson <adamwill(a)fedoraproject.org&gt;
>  > wrote:
>  > 
>  > > So there's this CI ticket ATM[0] about whether the environment in which
>  > > CI tests are run should or should not include and update from the
>  > > 'buildroot' repo, which contains both:
>  > > 
>  > > 1. Packages that have been pushed stable since the last time a compose
>  > > succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>  > > Branched compose, for stable releases it's an updates compose)
>  > > 2. Packages that have active buildroot overrides
>  > > 
>  > > Thinking about this reminded me again why buildroot overrides are such
>  > > a bad idea:
>  > > https://pagure.io/fedora-ci/general/issue/376#comment-830638
>  > > 
>  > > Buildroot overrides have unpredictable consequences for builds, updates
>  > > *and* tests. I really feel like we should consider disallowing them and
>  > > requiring all rebases to be done using side tags. Side tags are a
>  > > *much* better design that avoids the problem of packages unexpectedly
>  > > being built against a buildroot override somebody else submitted, and
>  > > means test systems aren't stuck in a bind of not really knowing for
>  > > sure what other packages should be installed when testing any given
>  > > package.
>  > > 
>  > > What does everyone else think? Has the time come? Or is there more we
>  > > need to do to make side tags usable for all cases before getting rid of
>  > > overrides?
>  > > 
>  > 
>  > I would say that side tags are almost always the correct tool for ABI
>  > rebuilds. I imagine people who submit global buildroot overrides to handle
>  > a library soname bump are almost always doing it because they haven't
>  > learned the "new" ways of doing it.
>  > 
>  > I'd go as far as to say that anyone who does ABI rebuilds using global
>  > buildroot overrides are doing it wrong.
>  > 
>  > However, having said that, I also find buildroot overrides useful. Some
>  > examples:
>  > 
>  > 1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new
>  > version through the freeze, but that includes a library soname bump.
>  > 
>  > What I would do in that case is submit the GNOME megaupdate to Bodhi, and
>  > also submit the library as a buildroot override to ensure that nothing can
>  > build against the old soname -- I am fairly confident that the GNOME
>  > megaupdate, together with the soname bump makes it to stable first.
>  > 
>  > 2) I need to do a container build and include a new CVE fix (as it's
>  > critical and we need to get fixes out ASAP), but that package build to
>  > include in the container is only in updates-testing.
>  > 
>  > What I'd do in this case is to submit a buildroot override because
>  > everything that's overridden gets included in container builds. After my
>  > container build is done I'd expire the buildroot override.
>  > 
>  > 3) We've had some "fun" issues with sysprof symbols leaking out from
> the
>  > GTK stack into other libraries consuming it. This has caused subtle ABI
>  > issues and when working on a fix and to make sure no package can build
>  > against the "wrong" GTK version, I've used buildroot overrides.
>  > 
>  > 4) Compiler issues, with compilers producing broken code.
>  > 
>  > To test the fixes and make sure packages start using the new fixed versions
>  > ASAP, a buildroot override is often useful.
>  > 
>  > I could continue with the list but I think you get my point that there are
>  > some cases where it's useful :) However I'd never use buildroot overrides
>  > for soname bump rebuilds; that's what side tags are good for.
>  
>  Well, hmm.
>  
>  The examples you provide are definitely interesting. They all
>  essentially boil down to, well, "I know exactly how this process works
>  and I'm gonna take advantage of that to achieve the right outcome
>  behind the scenes". 
Yeah, those four examples are very good. But they all can be summed up as
"we need this update that is in updates-testing (or possibly will soon be there)
to apply to all subsequent builds". Thus, maybe it would be enough to replace
buildroot overrides with a single switch that says "make this update visible
in the buildroot *now*".

Zbyszek

[Neal Gompa]

On Sun, Dec 4, 2022 at 2:28 PM Zbigniew Jędrzejewski-Szmek
<zbyszek(a)in.waw.pl&gt; wrote:
>  On Fri, Dec 02, 2022 at 03:35:31PM -0800, Adam Williamson wrote:
>  > On Sat, 2022-12-03 at 00:14 +0100, Kalev Lember wrote:
>  > > On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson
> <adamwill(a)fedoraproject.org&gt;
>  > > wrote:
>  > >
>  > > > So there's this CI ticket ATM[0] about whether the environment in
> which
>  > > > CI tests are run should or should not include and update from the
>  > > > 'buildroot' repo, which contains both:
>  > > >
>  > > > 1. Packages that have been pushed stable since the last time a compose
>  > > > succeeded (for Rawhide that's a Rawhide compose, for Branched it's
> a
>  > > > Branched compose, for stable releases it's an updates compose)
>  > > > 2. Packages that have active buildroot overrides
>  > > >
>  > > > Thinking about this reminded me again why buildroot overrides are such
>  > > > a bad idea:
>  > > > https://pagure.io/fedora-ci/general/issue/376#comment-830638
>  > > >
>  > > > Buildroot overrides have unpredictable consequences for builds, updates
>  > > > *and* tests. I really feel like we should consider disallowing them and
>  > > > requiring all rebases to be done using side tags. Side tags are a
>  > > > *much* better design that avoids the problem of packages unexpectedly
>  > > > being built against a buildroot override somebody else submitted, and
>  > > > means test systems aren't stuck in a bind of not really knowing for
>  > > > sure what other packages should be installed when testing any given
>  > > > package.
>  > > >
>  > > > What does everyone else think? Has the time come? Or is there more we
>  > > > need to do to make side tags usable for all cases before getting rid of
>  > > > overrides?
>  > > >
>  > >
>  > > I would say that side tags are almost always the correct tool for ABI
>  > > rebuilds. I imagine people who submit global buildroot overrides to handle
>  > > a library soname bump are almost always doing it because they haven't
>  > > learned the "new" ways of doing it.
>  > >
>  > > I'd go as far as to say that anyone who does ABI rebuilds using global
>  > > buildroot overrides are doing it wrong.
>  > >
>  > > However, having said that, I also find buildroot overrides useful. Some
>  > > examples:
>  > >
>  > > 1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new
>  > > version through the freeze, but that includes a library soname bump.
>  > >
>  > > What I would do in that case is submit the GNOME megaupdate to Bodhi, and
>  > > also submit the library as a buildroot override to ensure that nothing can
>  > > build against the old soname -- I am fairly confident that the GNOME
>  > > megaupdate, together with the soname bump makes it to stable first.
>  > >
>  > > 2) I need to do a container build and include a new CVE fix (as it's
>  > > critical and we need to get fixes out ASAP), but that package build to
>  > > include in the container is only in updates-testing.
>  > >
>  > > What I'd do in this case is to submit a buildroot override because
>  > > everything that's overridden gets included in container builds. After my
>  > > container build is done I'd expire the buildroot override.
>  > >
>  > > 3) We've had some "fun" issues with sysprof symbols leaking out
> from the
>  > > GTK stack into other libraries consuming it. This has caused subtle ABI
>  > > issues and when working on a fix and to make sure no package can build
>  > > against the "wrong" GTK version, I've used buildroot overrides.
>  > >
>  > > 4) Compiler issues, with compilers producing broken code.
>  > >
>  > > To test the fixes and make sure packages start using the new fixed versions
>  > > ASAP, a buildroot override is often useful.
>  > >
>  > > I could continue with the list but I think you get my point that there are
>  > > some cases where it's useful :) However I'd never use buildroot
> overrides
>  > > for soname bump rebuilds; that's what side tags are good for.
>  >
>  > Well, hmm.
>  >
>  > The examples you provide are definitely interesting. They all
>  > essentially boil down to, well, "I know exactly how this process works
>  > and I'm gonna take advantage of that to achieve the right outcome
>  > behind the scenes".
>
>  Yeah, those four examples are very good. But they all can be summed up as
>  "we need this update that is in updates-testing (or possibly will soon be there)
>  to apply to all subsequent builds". Thus, maybe it would be enough to replace
>  buildroot overrides with a single switch that says "make this update visible
>  in the buildroot *now*".
>  
I would prefer that *every* build would automatically generate a side
tag, even if it's a side-tag of one package. Or at least provide an
option to do that. That would provide opportunities for server-side
automation for populating side-tags with updated builds against a
package.

But one thing I like about the non-side-tag workflow is that I can
submit one update for multiple releases and Bodhi will auto-split it
for me. It speeds up my ability to ship updates a fair bit. I'd use
side tags for everything if I could do that with it too.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Kevin Fenzi]

On Sat, Dec 03, 2022 at 12:04:18PM +0100, Vitaly Zaitsev via devel wrote:
>  On 02/12/2022 22:30, Adam Williamson wrote:
>  > 1. Packages that have been pushed stable since the last time a compose
>  > succeeded (for Rawhide that's a Rawhide compose, for Branched it's a
>  > Branched compose, for stable releases it's an updates compose)
>  > 2. Packages that have active buildroot overrides
>  
>  I use buildroot overrides because generating side tags for stable releases
>  of Fedora is a very slow process.
>  
>  Two days ago I created F36, F37 and F38 side tags. F38 side tag was ready in
>  10 minutes, and F36+F37 wasn't generated even after 6 hours. That's why I
>  created a new buildroot override and built everything directly.
>  
>  Before disabling buildroot overrides this issue must be fixed. Waiting for
>  6+ hours is unacceptable. 
I don't think thats at all normal. We had mass updates/reboots and there
was an issue with some ppc64le vm's not having the needed nfs mounts,
causing them to fail newrepo tasks. This would have been the timeframe
for that issue. 

Can you try again now and see if the stable release sidetags are not
ready in ~10min? 

kevin

[Kevin Kofler]

Neal Gompa wrote:
>  I would prefer that *every* build would automatically generate a
> side
>  tag, even if it's a side-tag of one package. Or at least provide an
>  option to do that. That would provide opportunities for server-side
>  automation for populating side-tags with updated builds against a
>  package. 
But it is not practical given the performance concerns around side tags.

        Kevin Kofler

[Demi Marie Obenour]

On 12/4/22 16:29, Kevin Kofler via devel wrote:
>  Neal Gompa wrote:
> > I would prefer that *every* build would automatically generate a side
> > tag, even if it's a side-tag of one package. Or at least provide an
> > option to do that. That would provide opportunities for server-side
> > automation for populating side-tags with updated builds against a
> > package.
>  
>  But it is not practical given the performance concerns around side tags. 
Can those
be fixed?
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[Adam Williamson]

On Sun, 2022-12-04 at 19:27 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > 
>  > The examples you provide are definitely interesting. They all
>  > essentially boil down to, well, "I know exactly how this process works
>  > and I'm gonna take advantage of that to achieve the right outcome
>  > behind the scenes".
>  
>  Yeah, those four examples are very good. But they all can be summed up as
>  "we need this update that is in updates-testing (or possibly will soon be there)
>  to apply to all subsequent builds". Thus, maybe it would be enough to replace
>  buildroot overrides with a single switch that says "make this update visible
>  in the buildroot *now*". 
Isn't that...what a buildroot override *is*, though?
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Neal Gompa]

On Mon, Dec 5, 2022 at 12:02 AM Adam Williamson
<adamwill(a)fedoraproject.org&gt; wrote:
>  On Sun, 2022-12-04 at 19:27 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > >
>  > > The examples you provide are definitely interesting. They all
>  > > essentially boil down to, well, "I know exactly how this process works
>  > > and I'm gonna take advantage of that to achieve the right outcome
>  > > behind the scenes".
>  >
>  > Yeah, those four examples are very good. But they all can be summed up as
>  > "we need this update that is in updates-testing (or possibly will soon be
> there)
>  > to apply to all subsequent builds". Thus, maybe it would be enough to replace
>  > buildroot overrides with a single switch that says "make this update visible
>  > in the buildroot *now*".
>
>  Isn't that...what a buildroot override *is*, though? 
Yes, but I guess the idea is that the workflow is simplified to allow
you to submit an update and an override in one go.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Vitaly Zaitsev]

On 04/12/2022 20:48, Neal Gompa wrote:
>  I would prefer that*every*  build would automatically generate a
> side
>  tag, even if it's a side-tag of one package. 
No, please. Koji is already dying with 100 side tags regeneration (check 
my first message in this thread). When there will be 1000+ we will wait 
forever for our builds.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Vitaly Zaitsev]

On 05/12/2022 02:36, Demi Marie Obenour wrote:
>  Can those be fixed? 
I doubt. All side tags must be regenerated after each build in it or 
after each global buildroot override, which is a very slow process.

-- 
Sincerely,
   Vitaly Zaitsev (vitaly(a)easycoding.org)

[Zbigniew Jędrzejewski-Szmek]

On Mon, Dec 05, 2022 at 12:05:59AM -0500, Neal Gompa wrote:
>  On Mon, Dec 5, 2022 at 12:02 AM Adam Williamson
>  <adamwill(a)fedoraproject.org&gt; wrote:
>  >
>  > On Sun, 2022-12-04 at 19:27 +0000, Zbigniew Jędrzejewski-Szmek wrote:
>  > > >
>  > > > The examples you provide are definitely interesting. They all
>  > > > essentially boil down to, well, "I know exactly how this process
> works
>  > > > and I'm gonna take advantage of that to achieve the right outcome
>  > > > behind the scenes".
>  > >
>  > > Yeah, those four examples are very good. But they all can be summed up as
>  > > "we need this update that is in updates-testing (or possibly will soon be
> there)
>  > > to apply to all subsequent builds". Thus, maybe it would be enough to
> replace
>  > > buildroot overrides with a single switch that says "make this update
> visible
>  > > in the buildroot *now*".
>  >
>  > Isn't that...what a buildroot override *is*, though?
>  
>  Yes, but I guess the idea is that the workflow is simplified to allow
>  you to submit an update and an override in one go. 
Yeah. And also, by "single switch" I really mean "1 bit of
information".
I.e. all the additional metadata: description, expiration date, owner, separate name
would be gone, replaced with the boolean change that the update is *also* visible
in the buildroot as long as its on the way to stable.

Zbyszek

[Vít Ondruch]

Dne 04. 12. 22 v 20:27 Zbigniew Jędrzejewski-Szmek napsal(a):
>  On Fri, Dec 02, 2022 at 03:35:31PM -0800, Adam Williamson wrote:
> > On Sat, 2022-12-03 at 00:14 +0100, Kalev Lember wrote:
> >> On Fri, Dec 2, 2022 at 10:30 PM Adam Williamson
> <adamwill(a)fedoraproject.org&gt;
> >> wrote:
> >>
> >>> So there's this CI ticket ATM[0] about whether the environment in which
> >>> CI tests are run should or should not include and update from the
> >>> 'buildroot' repo, which contains both:
> >>>
> >>> 1. Packages that have been pushed stable since the last time a compose
> >>> succeeded (for Rawhide that's a Rawhide compose, for Branched it's
> a
> >>> Branched compose, for stable releases it's an updates compose)
> >>> 2. Packages that have active buildroot overrides
> >>>
> >>> Thinking about this reminded me again why buildroot overrides are such
> >>> a bad idea:
> >>> https://pagure.io/fedora-ci/general/issue/376#comment-830638
> >>>
> >>> Buildroot overrides have unpredictable consequences for builds, updates
> >>> *and* tests. I really feel like we should consider disallowing them and
> >>> requiring all rebases to be done using side tags. Side tags are a
> >>> *much* better design that avoids the problem of packages unexpectedly
> >>> being built against a buildroot override somebody else submitted, and
> >>> means test systems aren't stuck in a bind of not really knowing for
> >>> sure what other packages should be installed when testing any given
> >>> package.
> >>>
> >>> What does everyone else think? Has the time come? Or is there more we
> >>> need to do to make side tags usable for all cases before getting rid of
> >>> overrides?
> >>>
> >> I would say that side tags are almost always the correct tool for ABI
> >> rebuilds. I imagine people who submit global buildroot overrides to handle
> >> a library soname bump are almost always doing it because they haven't
> >> learned the "new" ways of doing it.
> >>
> >> I'd go as far as to say that anyone who does ABI rebuilds using global
> >> buildroot overrides are doing it wrong.
> >>
> >> However, having said that, I also find buildroot overrides useful. Some
> >> examples:
> >>
> >> 1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new
> >> version through the freeze, but that includes a library soname bump.
> >>
> >> What I would do in that case is submit the GNOME megaupdate to Bodhi, and
> >> also submit the library as a buildroot override to ensure that nothing can
> >> build against the old soname -- I am fairly confident that the GNOME
> >> megaupdate, together with the soname bump makes it to stable first.
> >>
> >> 2) I need to do a container build and include a new CVE fix (as it's
> >> critical and we need to get fixes out ASAP), but that package build to
> >> include in the container is only in updates-testing.
> >>
> >> What I'd do in this case is to submit a buildroot override because
> >> everything that's overridden gets included in container builds. After my
> >> container build is done I'd expire the buildroot override.
> >>
> >> 3) We've had some "fun" issues with sysprof symbols leaking out
> from the
> >> GTK stack into other libraries consuming it. This has caused subtle ABI
> >> issues and when working on a fix and to make sure no package can build
> >> against the "wrong" GTK version, I've used buildroot overrides.
> >>
> >> 4) Compiler issues, with compilers producing broken code.
> >>
> >> To test the fixes and make sure packages start using the new fixed versions
> >> ASAP, a buildroot override is often useful.
> >>
> >> I could continue with the list but I think you get my point that there are
> >> some cases where it's useful :) However I'd never use buildroot
> overrides
> >> for soname bump rebuilds; that's what side tags are good for.
> > Well, hmm.
> >
> > The examples you provide are definitely interesting. They all
> > essentially boil down to, well, "I know exactly how this process works
> > and I'm gonna take advantage of that to achieve the right outcome
> > behind the scenes".
>  Yeah, those four examples are very good. But they all can be summed up as
>  "we need this update that is in updates-testing (or possibly will soon be there)
>  to apply to all subsequent builds".
>  

It seems that all the mentioned issues are about the (upcoming) stable 
releases. But I think Rawhide is different and maybe overrides does not 
make a sense there.



Vít


>    Thus, maybe it would be enough to replace
>  buildroot overrides with a single switch that says "make this update visible
>  in the buildroot *now*".
>
>  Zbyszek
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>

[Miro Hrončok]

On 05. 12. 22 11:03, Vít Ondruch wrote:
>  It seems that all the mentioned issues are about the (upcoming)
> stable 
>  releases. But I think Rawhide is different and maybe overrides does not make a 
>  sense there. 
There are no buildroot overrides for rawhide.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Mattia Verga]

IMO, buildroot overrides are devil, because may have impact on builds performed by other
unaware users. The reported examples may be handled by sidetags or requesting exceptions
to releng:

Il 03/12/22 00:14, Kalev Lember ha scritto:

>  However, having said that, I also find buildroot overrides useful.
> Some examples:
>
>  1) Fedora is in freeze. GNOME has gotten a Freeze Exception to pull a new version through
> the freeze, but that includes a library soname bump.
>
>  What I would do in that case is submit the GNOME megaupdate to Bodhi, and also submit the
> library as a buildroot override to ensure that nothing can build against the old soname --
> I am fairly confident that the GNOME megaupdate, together with the soname bump makes it to
> stable first. 
The premise here is that a library got a soname bump. That means that the megaupdate
should include all packages which depends on that library.

Say, in the megaupdate there's foo-1.0.0-1 which was built against libbar.so.2. If any
other user makes a build of foo-1.0.0-2 against the old libbar.so.1, we can have Bodhi
stop the user when they try to create the update, since there's already a pending
multiple update containing foo. After the megaupdate is pushed to stable, if the user
tries to push again foo-1.0.0-2, QA tests should see it depends on the old soname and
again block the update.

>  2) I need to do a container build and include a new CVE fix (as
> it's critical and we need to get fixes out ASAP), but that package build to include in
> the container is only in updates-testing.
>
>  What I'd do in this case is to submit a buildroot override because everything
> that's overridden gets included in container builds. After my container build is done
> I'd expire the buildroot override. 
If the CVE fix is so urgent, it doesn't make sense to push it out only for container.
We should have a policy for asking Releng exemptions about the testing period and push the
update out for everything.

>  3) We've had some "fun" issues with sysprof symbols
> leaking out from the GTK stack into other libraries consuming it. This has caused subtle
> ABI issues and when working on a fix and to make sure no package can build against the
> "wrong" GTK version, I've used buildroot overrides. 
Same as above. Broken builds which causes breakage of other dependencies should be removed
or the update should be pushed ASAP by Releng.

>  4) Compiler issues, with compilers producing broken code.
>
>  To test the fixes and make sure packages start using the new fixed versions ASAP, a
> buildroot override is often useful. 
Same as above.

The purpose we have an update system is to avoid not only distribution breakages for end
users, but also buildroot breakages. Buildroot overrides just overrides everything, so I
think they must be a Releng right only.

Or let's just get rid of Bodhi and trust all packagers to "know exactly what they
are doing with their package".

Mattia

[Kevin Kofler]

Mattia Verga via devel wrote:
>  Or let's just get rid of Bodhi and trust all packagers to
> "know exactly
>  what they are doing with their package". 
Yes please!

        Kevin Kofler

[Neal Gompa]

On Mon, Dec 5, 2022 at 4:44 PM Kevin Kofler via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  Mattia Verga via devel wrote:
>  > Or let's just get rid of Bodhi and trust all packagers to "know exactly
>  > what they are doing with their package".
>
>  Yes please!
>  
NO! That way is madness! It penalizes fresh packagers too much in
favor of experts, and raises the bar for things too high.



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Adam Williamson]

On Mon, 2022-12-05 at 22:44 +0100, Kevin Kofler via devel wrote:
>  Mattia Verga via devel wrote:
>  > Or let's just get rid of Bodhi and trust all packagers to "know exactly
>  > what they are doing with their package".
>  
>  Yes please! 
Exhibits 1 through 2636 for why this is a bad idea:
https://bodhi.fedoraproject.org/updates/?search=&status=unpushed&...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Mattia Verga]

Il 06/12/22 00:08, Adam Williamson ha scritto:
>  On Mon, 2022-12-05 at 22:44 +0100, Kevin Kofler via devel wrote:
> > Mattia Verga via devel wrote:
> >> Or let's just get rid of Bodhi and trust all packagers to "know exactly
> >> what they are doing with their package".
> > Yes please!
>  Exhibits 1 through 2636 for why this is a bad idea:
>  https://bodhi.fedoraproject.org/updates/?search=&status=unpushed&... 
Obviously, I was sarcastic about getting rid of Bodhi (otherwise, how
can I better spend 2 hours per day on average of my free time other than
trying to fix things in it?.... :-p ).

That's why I think that, having a tool to prevent (as much as possible)
things to break, we should not permit every user to bypass the workflow.

Mattia

[Kevin Kofler]

Adam Williamson wrote:

>  On Mon, 2022-12-05 at 22:44 +0100, Kevin Kofler via devel wrote:
> > Mattia Verga via devel wrote:
> > > Or let's just get rid of Bodhi and trust all packagers to "know
> exactly
> > > what they are doing with their package".
> > 
> > Yes please!
>  
>  Exhibits 1 through 2636 for why this is a bad idea:
>  https://bodhi.fedoraproject.org/updates/?search=&status=unpushed&... 
In my ideal world, it would still be possible for a maintainer and for rel-
eng to unpush bad updates. IMHO, even from stable updates, since there is 
not really a technical reason to not allow that. It could be as simple as 
"koji untag-pkg fnn-updates foo".

        Kevin Kofler