[releng] Issue #6786: Kojira cannot handle number of targets it used to handle month ago.

[Jan Kaluža]

jkaluza reported a new issue against the project: `releng` that you are following:
``
Hi,

let me first say something about the current situation and steps we have taken to mitigate
it:

1) MBS used to create 2 targets per module build (one classic build target and one -repo
target to have the repository with the final tag available in order to allow Copr or
taskotron or local builds to get the built RPMs without the need to download all of them
from koji directly). This has been done since the very first date of MBS deployment and
over the time, the number of targets have grown to 670. In the same time, kojira started
having problems generating the repositories and after discussion on #fedora-releng, we
have been advised to stop generating so many tags.

2) On Thursday May 4th, we have removed 504 old targets
(https://pagure.io/releng/issue/6774) in a hope things will get to normal and we will have
a time to fix the MBS to not require any target when building module. I want to note here
that we left just 170 targets in Koji - this is number of targets it used to normally
handle in the previous months. We did not get to 670 targets over the night and there were
whole month in which kojira worked just fine with more than 170 targets. Even with 170
targets left, kojira had the same issues regenerating the repositories.

3) On Thursday May 11, we have decided to remove all modular targets (not only the old
ones) (https://pagure.io/releng/issue/6784). That day, we started with no modular target
and it used to work well for some time, but after we hit around 40 modular targets, kojira
started to be slow again and tasks time-outed on waitrepo. Note that this decision has
broken modular builds in Copr and signing of modules in robosignatory which is another
complication in Fedora 26 Modular release.

4) On Friday May 12, we have deployed new MBS which creates only single target per module
build and which deletes the targets right after the module build finishes, we are now on
22 modular targets and this number will go down to more or less 10 targets in any time
(because there are still old builds in progress which created 2 targets instead of 1).
After an hour after the removal of targets to get to 22 modular targets, we still see 7
tasks waiting on waitrepo.


Now the question. Why Kojira cannot handle the load it used to handle one month ago? Why
it cannot handle even the much lower load currently?



``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Tomas Tomecek]

ttomecek added a new comment to an issue you are following:
``
@ausil @kevin we would really appreciate some feedback here
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Petr Hracek]

phracek added a new comment to an issue you are following:
``
@ausil @kevin This definitely blocks Modularity world. Please send us a feedback. Why is
it so complicated?
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Ralph Bean]

ralph added a new comment to an issue you are following:
``
Copying from IRC:

>  its possible we have storage issues going on that started recently.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
I was away at a hackfest last week, so I didn't have time to respond here. 

Yes, we are seeing a storage performance issue and have been since around May 1st. We have
an open ticket with storage folks to investigate, and I have just tried to move it forward
some. 

Additionally, the koji config was not quite right and when I tried to up it to do 10
concurrent newrepos, it was only doing 8. @ausil fixed this last week I think and it
should be up to 15 now. 


``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Jan Kaluža]

jkaluza added a new comment to an issue you are following:
``
Thanks for your response Kevin.

Just to update this issue with current situation. We have less than 20 modularity targets
all the time, most of the time around 10. When we get closer to 20, waitrepo is starting
to take a long time. Sometimes it cannot manage to create a repo before the 120 minutes
timeout, so the situation is still not good for us.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Jan Kaluža]

jkaluza added a new comment to an issue you are following:
``
Still a problem.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
ok. Might you be able to ping me when you are seeing this happen so I can look at what the
bottleneck might be? I guess after builds you don't need the repos anymore and they
are inactive?
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/6786