disable networking in Copr by default
by Pavel Raiskup
Just to be more pedantic and probably bit more secure, what about making
networking in copr opt-in (at least for new projects)?
If build of my package used internet arbitrarily, I would be warned very
early..
Also, in Koji we also do not have networking, so we can faster teach
(future) Fedora packagers.
Thanks for considering,
Pavel
7 years, 11 months
New release of Copr
by Miroslav Suchý
I just deployed in production new version of Copr.
It is mostly bugfix release.
It contains one new feature, but it has some performance issues so I'm not going to announce it yet. If you discover it,
then feel free to test it, but do not use it too much :)
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years
Post-mortem
by Miroslav Suchý
Last two days we had problem processing the queue. This is post-mortem of what happened.
Mmraka sent several thousands of builds to Copr - that is fine, it was discussed in advance with me and in fact I
encourage such tests and rebuilds. However this triggered one bug: this users was unable to get list of builds as we
have inefficient SQL query on that page [1]. As result of this Michal (and very likely somebody else too) tried to
delete several hundreds of builds at once.
This resulted in bad JobGrabber behaviour where it fetched few dozen tasks and then stopped without any error.
When I debug it (on production (!) because it did not happen in stage) it processed first round of builds, first action
and then stopped.
I then learned that JobGrabber waited for lock, which was hanging there from previously killed JobGrabber. After I
removed it, I found there is that big number of tasks to be executed. And our code in JobGrabber looks like:
while True:
if some builds:
put builds in queue
if some tasks:
execute them immediately
That is because previously users send only few tasks at once and those operation are basically very cheap (usually just
unlink, followed by quick createrepo_c --update).
However repositories to which belong those actions are big (several GBs) and even createrepo_c run for more than minute.
So it effectively blocked next fetch of builds from frontend for several hours.
Right now the task queue is empty so builds are processed in timely manner and our code in master is already changed to
be resistant to such behaviour.
I am really sorry if you had to wait for your build in past two days.
We learned a lesson from this massive usage of Copr and we identified some other potential performance issue and it will
result in even better service in upcoming days.
[1] Adam is fixing the code right now.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years
New release of Copr
by Miroslav Suchý
It is my pleasure to announce new version of Copr.
https://copr.fedoraproject.org/
What's new? Group projects!
When you log in. You can see in right column link "My Groups". There you can see list of your FAS groups and you can
activate them. In fact, you actually create alias for them. E.g. FAS group "gitmock" can be named "mock" in Copr.
You can click on Copr group in right column. That navigate you to url in format:
https://copr.fedoraproject.org/groups/g/mock/coprs/
Where are listed all projects of this group. And you can create new group project there too.
Every member of that FAS group can administer it and build there. Due to the OpenID limitations, you have to
log-out/log-in to see your groups (or whenever you join new group in FAS).
Since copr-cli-1.45 (in updates-testing right now) you can submit package to group project like:
copr-cli build @GROUP/PROJECT
e.g
copr-cli build @mock/mock-dev
Creation of group project is not possible from command line right now due OpenID restriction and you have to use WebUI.
This may change in near future.
There is no API method in APIv1. But we provide APIv2
http://copr-rest-api.readthedocs.org/en/latest/
which has methods to manipulate with groups.
To create new group in FAS, just follow:
https://admin.fedoraproject.org/accounts/group/new
If you have some personal project and you want to change it to group project, then please let me know and we change it
manually.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years
COPR service announcement: builds failed last night
by Patrick Uiterwijk
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi all,
Due to issues with the metalink for f22 and f23 updates on ppc64 last night,
a lot of COPR tasks have failed their ppc64le chroot builds.
If you see that the x86_64 builds of of your task have succeeded, but the
ppc64le builds failed, please resubmit you job to COPR.
End of service announcement.
With kind regards,
Patrick Uiterwijk
Fedora Infra
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCgAGBQJWQV7NAAoJEIZXmA2atR5QXY0QAK8i8kjYCvsNUNbqE1A3/Slr
Jw0I+x4C9bzmTwUrRojCsknOQV1+Ym/8M4zyCJ+vwwaPxOe+/yVynK5geA/pCgq+
WJAzxco2Uad9B1/ETMF7224bwPsIvZsbGQiQPklPWi7rB6wXmJd7GNKyUpvhPiwR
HPhgW6RjZ5Y0kA0Aeo7okFCxigdenpRDMHppOJlsvl/JFe2gQAZ4kylTKN9aJE60
8x/vVDb4dSyTu1/0DixLyn+O282el7UJo4hzyLocyHcpWlmeBQ5RHcdx2H3PrV1i
P5LG8r6PqCmM6AXPGGxK0WCEuYZyXsR8FfuvuHOwBdRnPwhDPmi9N+7rKa5PPAce
1bLcqgJunzCX00oNYfzd2sBURapHthByIY3ldtcZe+XfC420QDr2R3vFh/rpHcVa
Jdw6SQXzeRX8cCumDz+PtabXEp67aAPSy7Skxynz4z//tq1zWm0Z3OUDKBPlRd/u
iKdEZ7DYhniC5oQfkE4mLjxCc23epkgXsH973Vkod5s7V1I8OMSqCOPMamrRCwsn
Onuip/V0xWeQuvzZGWIL38iqR2EJs1FD1yeHr5/10ydM0ynYiXCBjQxJv5c31JCn
c9B1Cqso2x7U2WIuk2NODv9vpKvAXzYmzfsyRCDXzzo21cMaMR9eRFPyDIAweIfz
r2k8uClm5snsV9/1lSXn
=cu1n
-----END PGP SIGNATURE-----
8 years
COPR infrastructure issues caused build failures
by Patrick Uiterwijk
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi everyone,
Throughout the day of October 31st, 2015there have been multiple build failures in COPR
because of problems with the COPR infrastructure.
If your build failed with only a mockchain.log.gz file which contains the following line:
"Failed to obtain srpm from dist-git", or something alike that, that was caused by this
infrastructure issue.
We have just found and fixed the root cause of this problem, so any builds from now on
should be successful.
If you have any builds that failed and only produces a mockchain.log.gz, please resubmit
your build and it should go through correctly.
If you have any build failures where you are unsure whether they're caused by this issue,
or have any other questions, feel free to email me or come by on IRC (#fedora-buildsys).
With kind regads,
Patrick Uiterwijk
Fedora Infra
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAEBCgAGBQJWNWelAAoJEIZXmA2atR5Q6q0P/jDaMtcMbudD5TNCGuSbjkCk
GKb5E0ax1Oux/HKqxyWrOknlc7aXWcc5AB3SWQUUk3ByhEnwfJIkYb+rr98fNxYL
AM83cTrVaOEiQ6nouXrQdoorEpwo4v9/jC1mwhyGyq7IGytYeIzW9XkOBIvtI3pg
ramiezOjQRWMDGswVuqGYwiF94+GaDv8sCS/OUQsCQ+6XVcyw8QnEiUcZRDM2Xun
FEGWKUGb12UdCw32iCPptlXvC9npuWZyFMwPyaHHnqGhYjpFrvnUClaGh5LmwqiY
PQPnAbKwrLYelGKDGVv9qpJg2LdjuCAplajDiEnu6GM5z+V3xYhoq5pVfAAOqao3
DyHBKh0sSC+rRB1pdGaDNsMBcTFDy9QN+NDKmEPJzg13LaOxtx8SKfmzpfTDLba/
9oMSPOWSPv3dMm2dlYDwOIJY6bcjFYcWTLMfy4kyE7E5Kfwb20bVr+LLXFT+TvE5
z1ELeu/IiYYk4seGNrMKQx0PgpZRhe13pfpburO1IQDvfWn7dPzJRG+7hzGOOUPG
SEDEsHejWHO5eVJGpbWokmIdojNzzXfcSB8yoOWy8CWMXg6Sjowt5ex136RdW1QG
VLDqMGNUE5qGJx+qslJcgl1AjUIkLyuNQeK6PtTMTPEqoqUajD+ZctbqnZUZpc0P
Al11cfVxLFPM0bhwu8oi
=YgO+
-----END PGP SIGNATURE-----
8 years, 1 month
