Shall we add a limit to number of packages in a Bodhi update?

[Mattia Verga]

Bodhi is a nice interface to create multi-builds update from side-tags,
but it is not really optimized for massive updates. In the past we had
some troubles pushing updates with a lot of builds. There are too many
things done in the background that can break while processing a massive
update and making Bodhi more reliable is not so simple.

I wonder if it would be safe to set a cap to the number of builds a
single Bodhi update can carry. I think **really** massive updates (say
more than 99 builds?) should be handled manually asking Releng to merge
the side-tag manually.

I'd like to hear someone else opinion, especially from FESCo and Releng
members. Meanwhile I'll keep an eye on the recent massive Golang update
(which carries 315 builds...) to see if it shows any hiccups. (I'm quite
surprised it didn't screw up already)

Mattia

[Adam Williamson]

On Wed, 2022-07-13 at 15:38 +0000, Mattia Verga via devel wrote:
>  Bodhi is a nice interface to create multi-builds update from
> side-tags,
>  but it is not really optimized for massive updates. In the past we had
>  some troubles pushing updates with a lot of builds. There are too many
>  things done in the background that can break while processing a massive
>  update and making Bodhi more reliable is not so simple.
>  
>  I wonder if it would be safe to set a cap to the number of builds a
>  single Bodhi update can carry. I think **really** massive updates (say
>  more than 99 builds?) should be handled manually asking Releng to merge
>  the side-tag manually.
>  
>  I'd like to hear someone else opinion, especially from FESCo and Releng
>  members. Meanwhile I'll keep an eye on the recent massive Golang update
>  (which carries 315 builds...) to see if it shows any hiccups. (I'm quite
>  surprised it didn't screw up already) 
Well, one awkward thing about this is that Bodhi is where we've
implemented quite a lot of CI/test automation integration. Fedora CI
runs tests at the package build level, but the results are primarily
shown - and gating is potentially done - at the Bodhi update level.
openQA runs tests at the Bodhi update level; if there isn't an update,
it will not automatically run tests.

It's also where we usually route *manual* testing feedback. If people
can't comment and karma a Bodhi update, where can they test this big
and very-potentially-destabilizing change?

99+-package updates are exactly the kind of case where we'd *want* to
consider automated test results before deciding to merge, usually.

If Bodhi can't be improved to handle this, I'd really want the
alternative path to have proper consideration for integrating test
feedback, both manual and automated.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Maxwell G]

On 22/07/13 08:58AM, Adam Williamson wrote:
>  It's also where we usually route *manual* testing feedback. If
> people
>  can't comment and karma a Bodhi update, where can they test this big
>  and very-potentially-destabilizing change? 
The go rebuilds in question are not very destabilizing, and they don't
have breaking changes. The packages are simply getting rebuilt against
the new go minor version with fixes for the CVE(s). I've already had to
waive the tests, as they timed out when trying to download the 300+
packages in the update.

-- 
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Fabio Valentini]

On Wed, Jul 13, 2022 at 7:32 PM Maxwell G via devel
<devel(a)lists.fedoraproject.org&gt; wrote:
>  On 22/07/13 08:58AM, Adam Williamson wrote:
>  > It's also where we usually route *manual* testing feedback. If people
>  > can't comment and karma a Bodhi update, where can they test this big
>  > and very-potentially-destabilizing change?
>
>  The go rebuilds in question are not very destabilizing, and they don't
>  have breaking changes. The packages are simply getting rebuilt against
>  the new go minor version with fixes for the CVE(s). I've already had to
>  waive the tests, as they timed out when trying to download the 300+
>  packages in the update. 
I wonder if it would have made sense to have submitted those 300+
builds in separate bodhi updates (at least in several smaller batches,
if not individually)?
At least in this case, that would've been a little bit more work, but
would have caused less of a chance to break bodhi.
As far as I can tell, there's no reason the builds need to be handled
together, as the only thing that ties these builds together is the
*reason* why they were rebuilt, but they don't necessary need to be
pushed to testing or stable as a single unit.

Fabio

[Maxwell G]

On 22/07/13 07:49PM, Fabio Valentini wrote:
>  I wonder if it would have made sense to have submitted those 300+
>  builds in separate bodhi updates (at least in several smaller batches,
>  if not individually)? 
>  At least in this case, that would've been a little bit more work,
> but
>  would have caused less of a chance to break bodhi.
>  As far as I can tell, there's no reason the builds need to be handled
>  together, as the only thing that ties these builds together is the
>  *reason* why they were rebuilt, but they don't necessary need to be
>  pushed to testing or stable as a single unit. 
You're right. They don't have to be rebuilt together as long as the
patched version of golang/the libraries with CVEs are in the buildroot.
I decided to handle them as a single update to make it easier to
manage/organize. I don't want to have to manage 300+ different updates
and have my Fedora mailbox flooded with notifications from them. The RH
prodsec team already does a good enough job at flooding my inbox :(.

It probably wouldn't be too much effort to split them into multiple
batches, though.

---

Also, there was a new golang version released today that has fixes for 9
CVEs, so I will probably have to do another rebuild in F36 and Rawhide.
It would be helpful if we could come to a conclusion about how to handle
this properly sooner rather than later.

-- 
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Adam Williamson]

On Wed, 2022-07-13 at 19:49 +0200, Fabio Valentini wrote:
>  On Wed, Jul 13, 2022 at 7:32 PM Maxwell G via devel
>  <devel(a)lists.fedoraproject.org&gt; wrote:
>  > 
>  > On 22/07/13 08:58AM, Adam Williamson wrote:
>  > > It's also where we usually route *manual* testing feedback. If people
>  > > can't comment and karma a Bodhi update, where can they test this big
>  > > and very-potentially-destabilizing change?
>  > 
>  > The go rebuilds in question are not very destabilizing, and they don't
>  > have breaking changes. The packages are simply getting rebuilt against
>  > the new go minor version with fixes for the CVE(s). I've already had to
>  > waive the tests, as they timed out when trying to download the 300+
>  > packages in the update.
>  
>  I wonder if it would have made sense to have submitted those 300+
>  builds in separate bodhi updates (at least in several smaller batches,
>  if not individually)?
>  At least in this case, that would've been a little bit more work, but
>  would have caused less of a chance to break bodhi.
>  As far as I can tell, there's no reason the builds need to be handled
>  together, as the only thing that ties these builds together is the
>  *reason* why they were rebuilt, but they don't necessary need to be
>  pushed to testing or stable as a single unit. 
If they need to be rebuilt for an API/ABI change, then by policy they
should be grouped together. We do not want a situation where the
API/ABI change gets pushed stable but some of the rebuilds do not, or
vice versa.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Kevin Fenzi]

My 2 cents... we should try and get bodhi (and the rest of the pipeline)
to work with these large updates if at all possible. 

Having releng merge tags works of course, but as Adam pointed out, it
skips our CI and feedback testing area. Additionally, it's more work to
releng folks and more waiting around time for requesters. 

We also have some KDE updates that were over 300 builds. 
(like
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-NEXT-2022-a6c0f04770
)
I'm sure there have been others. 

kevin

[Mattia Verga]

Il 13/07/22 21:11, Kevin Fenzi ha scritto:
>  My 2 cents... we should try and get bodhi (and the rest of the
> pipeline)
>  to work with these large updates if at all possible.
>
>  Having releng merge tags works of course, but as Adam pointed out, it
>  skips our CI and feedback testing area. Additionally, it's more work to
>  releng folks and more waiting around time for requesters.
>
>  We also have some KDE updates that were over 300 builds.
>  (like
>  https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-NEXT-2022-a6c0f04770
>  )
>  I'm sure there have been others.
>
>  kevin 
Yep, maybe my concerns are unfounded. It seems that lately Bodhi is much
better in handling these massive updates.

Let's see how if there are any more problem in future.

Mattia.

[Maxwell G]

Jul 13, 2022 1:13:38 PM Adam Williamson <adamwill(a)fedoraproject.org&gt;:

>  If they need to be rebuilt for an API/ABI change, then by policy
> they
>  should be grouped together. We do not want a situation where the
>  API/ABI change gets pushed stable but some of the rebuilds do not, or
>  vice versa. 
We're not talking about ABI/API changes. This about the recent
rebuilds we've done to fix CVEs in golang/the surrounding libraries. golang/the
libraries are only needed at buildtime and the binaries are statically linked, so it
doesn't matter if everything is pushed at exactly the same time.
--
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Maxwell G]

Jul 13, 2022 10:45:02 AM Mattia Verga via devel <devel(a)lists.fedoraproject.org&gt;:

>  Meanwhile I'll keep an eye on the recent massive Golang update
> (which
>  carries 315 builds...) to see if it shows any hiccups. (I'm quite
>  surprised it didn't screw up already) 
The recent 315 build F35 Bodhi update is not the first one that I've
done. I've done the same thing for F36 and F37 with minimal hiccups,
including the gating tests timing out due to the amount of packages and
needing to be waived. The other rebuilds that I did not perform had
issues for reasons unrelated to the amount of packages in them.

I've also written a script to check for new builds that involve the
packages in the rebuild so I can warn the packagers not to push those
until the sidetag is merged to avoid the problem we had with snapd last
time.

>  I wonder if it would be safe to set a cap to the number of builds a
>  single Bodhi update can carry. I think **really** massive updates (say
>  more than 99 builds?) should be handled manually asking Releng to merge
>  the side-tag manually. 
That's fine with me. I'm happy to work with releng to make this process
smoother in the future. The Python rebuilds are already handled like
this. However, they have thousands of packages instead of hundreds.

Would those still have to wait 7 days, though? The nice thing about
submitting through Bodhi is that the updates can get karma and proceed
to stable sooner, which is desirable for security updates.

--
Thanks,

Maxwell G (@gotmax23)
Pronouns: He/Him/His

[Richard W.M. Jones]

On Wed, Jul 13, 2022 at 03:38:29PM +0000, Mattia Verga via devel wrote:
>  Bodhi is a nice interface to create multi-builds update from
> side-tags,
>  but it is not really optimized for massive updates. In the past we had
>  some troubles pushing updates with a lot of builds. There are too many
>  things done in the background that can break while processing a massive
>  update and making Bodhi more reliable is not so simple.
>  
>  I wonder if it would be safe to set a cap to the number of builds a
>  single Bodhi update can carry. I think **really** massive updates (say
>  more than 99 builds?) should be handled manually asking Releng to merge
>  the side-tag manually. 
I'd prefer you didn't limit them to such a low number.  OCaml updates
have ~ 180 packages
(https://bodhi.fedoraproject.org/updates/FEDORA-2022-d3e15b41f5) and
it's a lot easier to do the push myself rather than have to ask
someone.  And I will have to remember each time _how_ to ask someone,
since I do this infrequently enough that I'll forget each time.

What is the actual problem and could it be fixed instead?

>  I'd like to hear someone else opinion, especially from FESCo and
> Releng
>  members. Meanwhile I'll keep an eye on the recent massive Golang update
>  (which carries 315 builds...) to see if it shows any hiccups. (I'm quite
>  surprised it didn't screw up already) 
315 builds doesn't strike me as being that massive TBH.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top