On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý <msuchy(a)redhat.com> wrote:
Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
> - builds failing due to failure to download packages from official
> Fedora mirror
dl.fedoraproject.org
This is not first time I hear this. So I will open discussion to (again) alter
builder's mock config to point to PHX
location, which is just rack away.
Wouldn't it make sense to maintain a local mirror of the distributions
enabled in COPR and just have all the mock configs point to that? That
eliminates all the network pressure and the issues we keep seeing
there when trying to use it.
> - builds failing to import to COPR distgit (was not a problem
before
> dist-git was introduced)
This is actually new to me. Can you point me to some report?
I can't say I've experienced this issue. However, I do wish we could
interact with COPR Dist-Git the same way we do with the official
Fedora Dist-Git, with a Pagure instance and everything...
> - outages caused by read-only filesystem after reboot - copr is
> unusable until someone remounts it rw
This was actually recently fixed.
> - multi-day long outages caused by out of disk space
> - multi-hour or even day- long build queues
That is because I actually care about the resources. It is very easy to submit a proposal
and state that "it will run in
Fedora Cloud", but the cloud is not infinite. I can easily increase the soft limits
in Cloud dashboard. But we are
either hitting hard limits (storage) or we would drastically limit other users using the
same cloud (vCPU in case more
builders).
Copr is not designed for spikes, it is designed for the average traffic. Which is
handling pretty well. You can check it
here:
https://copr.fedorainfracloud.org/status/stats/
If you submit 300+ build today (like happened today) you will simply have to wait some
time. At the same time, it should
not affect (too much) other users. This is IMO big difference from Koji which is IIRC
strictly FIFO.
Would it be possible to extend COPR to support multiple locations for
builders? For example, in addition to an OpenStack system, builders
could be hosted on an oVirt system, or AWS, or GCP, and so on? That
way it can support on-premise deployments and cloudy deployments too.
Perhaps supporting even some limited form of auto-scaling for when it
needs to "burst" to support more build traffic using clouds or
hypervisors? For example, you could use AWS spot instances to do it on
the cheap for the time a builder needs to run, and then have it go
away afterward. I've done builds this way with buildbot and you can do
thousands of builds for really cheap (on the order of hundreds of
dollars each year).
> - outdated SOP preventing non-COPR people like me from fixing COPR outages
Feel free to raise any issue. Last update in this document was in October 2018. Checking
it right now I see outdated
part about ppc builders, but otherwise it should be ok.
Thou, I will review that document after weekend.
I don't know anything about this...
--
真実はいつも一つ!/ Always, there's only one truth!