Speed improvements in moving packages through bodhi

[Stewart Hardie]

Hi there

Apologies if this is not the right place to post. If not, forwarding to the appropriate
people would be useful.

As a Fedora end user, this is a suggestion about how package processing through bodhi
could be improved.

The time taken for a package to enter the testing or stable repos is sometimes days after
it has been submitted to that repo. Presumably signing is part of that delay. It would be
worthwhile to consider if this part of the bodhi process could be made faster, or capped
in time by general policy. This would be primarily and more importantly done for security
updates.

Delays in packages entering repos after submission mean that feedback is missing on
testing done by those who wait for the convenience and signing security of having the
package being available in the testing repo. And it allows important updates to get into
stable more rapidly, slicing off hours or days in the process. For security updates, this
is only likely to be more important in the future.

As an explanation....

When bodhi says:
"This update has been submitted for testing by XXX."

then the following bodhi message must appear within 1 hour:
"This update has been pushed to testing."

Naturally a package must then wait for appropriate feedback and karma.........

But then, when bodhi says:
"This update has been submitted for stable by bodhi."

then the following bodhi message must appear within 1 hour:
"This update has been pushed to stable."

It may be that 1 hour is not currently feasible. If not, would 2, 6 or 12 hours be
reasonable??

Ofcourse this only applies when the general process is running smoothly, not when servers
and software are misbehaving.

Regards
Stewart

[Peter Robinson]

On Mon, Jan 30, 2017 at 10:49 PM, Stewart Hardie
<stewart.r.hardie(a)gmail.com&gt; wrote:
>  Hi there
>
>  Apologies if this is not the right place to post. If not, forwarding to the appropriate
> people would be useful.
>
>  As a Fedora end user, this is a suggestion about how package processing through bodhi
> could be improved.
>
>  The time taken for a package to enter the testing or stable repos is sometimes days after
> it has been submitted to that repo. Presumably signing is part of that delay. It would be
> worthwhile to consider if this part of the bodhi process could be made faster, or capped
> in time by general policy. This would be primarily and more importantly done for security
> updates. 
Signing is not part of the delay. When an update is submitted it goes
into a queue for autosigning and is generally done in seconds, even
for a big update it's mostly just minutes. It's not part of the bodhi
process, it's automatically run based on actions that generally come
from the bodhi process but is completely independent.

>  Delays in packages entering repos after submission mean that feedback
> is missing on testing done by those who wait for the convenience and signing security of
> having the package being available in the testing repo. And it allows important updates to
> get into stable more rapidly, slicing off hours or days in the process. For security
> updates, this is only likely to be more important in the future.
>
>  As an explanation....
>
>  When bodhi says:
>  "This update has been submitted for testing by XXX."
>
>  then the following bodhi message must appear within 1 hour:
>  "This update has been pushed to testing."
>
>  Naturally a package must then wait for appropriate feedback and karma.........
>
>  But then, when bodhi says:
>  "This update has been submitted for stable by bodhi."
>
>  then the following bodhi message must appear within 1 hour:
>  "This update has been pushed to stable."
>
>  It may be that 1 hour is not currently feasible. If not, would 2, 6 or 12 hours be
> reasonable??
>
>  Ofcourse this only applies when the general process is running smoothly, not when servers
> and software are misbehaving. 
There's a lost of must in there. The bodhi updates process appears
slow for four reasons 1) it must be initialised by a human 2) there is
a LOT of work that gets done, if any part "misbehaves" it needs to be
debugged and potentially teams on different timezones consulted 3)
it's very IO intensive 4) it needs to get out onto the mirrors which
is completely out of our control as the mirrors pull from us so it's
up to them.

Things like the pushed to stable within an hour isn't going to happen
currently as the updates process takes a lot longer than that and it
also adds a lot of churn to the mirrors process and we try to be
respectful of the mirrors as they provide us an essential service.

You'll have to trust me when I say that rel-eng has looked extensively
at how to speed up the process (it's a lot more painful for us than
you believe me!) and there's always ongoing improvements (and change
and additions which can affect time taken too). EG the signing use to
be manual until not too long ago.

Peter

[Kevin Fenzi]

yeah, we would all love for things to be faster, but just declaring
them so does not make it happen. ;) 

There's some things that have been in process for a long time that may
well help when they finally land: 

* Koji signed repos. This would allow bodhi to use koji builders to do
  all the repo mashing and composing. It would spread out the i/o load
  to a bunch of builders and we could basically do all them at once. 

* mash (the thing bodhi uses now to make repos) uses createrepo. The
  above change might get us to using createrepo_c which is faster, but
  failing that, someone could work on moving mash to createrepo_c

* createrepo_c could be improved to do parallel drpms. That would speed
  things up a lot. 

There's probibly more things we could look into again. 

kevin

[Stewart Hardie]

Hi Peter, Kevin

Thanks for the informative replies. Its good to learn more detail about the current Fedora
processes and issues.

Regards
Stewart