benchmarking koji

[Ken Dreyer]

If we had a "koji benchmark" tool, what would you like it to do?

I've written an Ansible playbook that automatically sets up Koji
according to our documentation, and I want to exercise the hub to see
how it performs. We can use a benchmark tool to understand changes
that could impact performance. As one example, we probably want to
compare CentOS 7, 8, and Fedora.

What scalability problems have you encountered with Koji in the wild?

I'm thinking of things like:
- Tagging thousands of builds quickly
- Contention on the sessions table
- Maintaining thousands of builders
- Ensuring kojira runs in a timely manner

When we have test data, even if the benchmarks are fairly synthetic,
it's easier to see the impact of code changes before we ship.

(As an aside, I started writing a really simple "echo" and "login"
benchmark tool using the txkoji library, but I found that Twisted's
built-in XML-RPC client only supports HTTP/1.0, so no keepalives!)

- Ken

[Miroslav Suchý]

Dne 05. 06. 20 v 22:12 Ken Dreyer napsal(a):
> What scalability problems have you encountered with Koji in the wild?

* How long it takes to generate SRPM? - when I hit that, it was in fact issue in (dist-)git.
* How long it takes from building package A until I can submit build of package B which build-requires that package A?
(real life issue with mergerepo).
* How long it takes to run Mock with current distribution config?

[Alex]

Hi,

On 6/5/20 10:12 PM, Ken Dreyer wrote:
> What scalability problems have you encountered with Koji in the wild?

* How long does it take to build a docker image/VM?

This goes beyond Koji, I guess, but we're seeing a slowdown of qemu 
processes over time and builds start timing out until you reboot the 
builder. We've been unable to track this issue down so far.

Cheers,
Alex

[Tomas Kopecek]

* How long run of koji-gc takes
* How quickly are +- no-op tasks consumed (tagNotification)
* What is the impact of queue size to task waiting time (There is an
observation that koji doesn't scale well if there are thousands of tasks in
the queue)

po 8. 6. 2020 v 15:35 odesílatel Alex  napsal:
> Hi,
>
> On 6/5/20 10:12 PM, Ken Dreyer wrote:
> > What scalability problems have you encountered with Koji in the wild?
> How long does it take to build a docker image/VM?
>
> This goes beyond Koji, I guess, but we're seeing a slowdown of qemu
> processes over time and builds start timing out until you reboot the
> builder. We've been unable to track this issue down so far.
>
> Cheers,
> Alex
> _______________________________________________
> koji-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/[e-mail].o...

[Kevin Fenzi]

On Fri, Jun 05, 2020 at 02:12:51PM -0600, Ken Dreyer wrote:
> If we had a "koji benchmark" tool, what would you like it to do?
>
> I've written an Ansible playbook that automatically sets up Koji
> according to our documentation, and I want to exercise the hub to see
> how it performs. We can use a benchmark tool to understand changes
> that could impact performance. As one example, we probably want to
> compare CentOS 7, 8, and Fedora.

Are you looking to just test the hub? Or also builders?

And a lot of this will depend on the database... and the builders and
the storage speed, etc.

The fedora koji db is public, so you're welcome to use that, but 
not sure it's a 'common' case. :)
> What scalability problems have you encountered with Koji in the wild?
>
> I'm thinking of things like:
> Tagging thousands of builds quickly
> Contention on the sessions table
> Maintaining thousands of builders
> Ensuring kojira runs in a timely manner

- tons of side tags, how long to newrepo them all
- patalogical packages like texlive hitting timeouts uploading, etc.
- writing out signatures of a ton of packages. 
- response time for various web pages on the hub?
> When we have test data, even if the benchmarks are fairly synthetic,
> it's easier to see the impact of code changes before we ship.

Yeah.

kevin

[Ken Dreyer]

On Sat, Jun 13, 2020 at 6:14 PM Kevin Fenzi  wrote:
> On Fri, Jun 05, 2020 at 02:12:51PM -0600, Ken Dreyer wrote:
> > If we had a "koji benchmark" tool, what would you like it to do?
> >
> > I've written an Ansible playbook that automatically sets up Koji
> > according to our documentation, and I want to exercise the hub to see
> > how it performs. We can use a benchmark tool to understand changes
> > that could impact performance. As one example, we probably want to
> > compare CentOS 7, 8, and Fedora.
>
> Are you looking to just test the hub? Or also builders?
>
> And a lot of this will depend on the database... and the builders and
> the storage speed, etc.

You're right. Thanks to everyone who responded so far! This made me
realize I should think a little bigger here. Originally I was just
thinking about benchmarking the hub alone because there are a lot of
things to measure and improve there (like mass-tagging), but several
of the responses made me think it makes sense to track more processes
end-to-end.

For example, SRPM generation would require dist-git and it's related
to how efficiently we perform Git operations. Off the top of my head,
the Koji SCM class simply runs "git clone -n", and we could probably
change that to always run shallow clones instead so that we're not
fetching so much history and unrelated branches.

Image generation is another important one, because it would implicitly
test compatibility and performance of qemu and docker (eventually
buildah).

- Ken