1:10 p.m.
Hi all,
I've not had any builds be scheduled for a couple of hours, is copr
stuck / crashed? Thanks.
Richard.
2:04 p.m.
New subject: COPRdown?
On Thu, 20 Feb 2014 19:10:27 +0000
Richard Hughes <hughsient(a)gmail.com> wrote:
Hi all,
I've not had any builds be scheduled for a couple of hours, is copr
stuck / crashed? Thanks.
The backend blew up, and I rebooted it...
I'll see if I can unstick processing. ;(
kevin
3:39 p.m.
New subject: COPRdown?
ok. It's finally processing again I hope. ;)
The problem seemed to be some kind of doom when it was set to 8 workers
(so 8 builds at a time). It would launch them all off fine, but then
when they returned something would go south and the instance wouldn't
be responsive at all. It might be that they all finished at the same
time and caused a storm of createrepos or something.
I set it back down to 1 worker (so things will be slow for a bit).
I'll up that to 4 or something tomorrow... and we can see if we can
isolate where it is hitting problems.
kevin
3:52 p.m.
New subject: COPRdown?
On 20 February 2014 21:39, Kevin Fenzi <kevin(a)scrye.com> wrote:
ok. It's finally processing again I hope. ;)
Great, thanks.
I'll up that to 4 or something tomorrow... and we can see if we
can
isolate where it is hitting problems.
I submitted 27 srpms at one time; was this antisocial?
Richard
3:53 p.m.
New subject: COPRdown?
On Thu, 20 Feb 2014 21:52:04 +0000
Richard Hughes <hughsient(a)gmail.com> wrote:
On 20 February 2014 21:39, Kevin Fenzi <kevin(a)scrye.com>
wrote:
> ok. It's finally processing again I hope. ;)
Great, thanks.
> I'll up that to 4 or something tomorrow... and we can see if we can
> isolate where it is hitting problems.
I submitted 27 srpms at one time; was this antisocial?
Naw, it shouldn't be, IMHO. ;)
The backend should cope with that kind of thing...
kevin
2:40 a.m.
New subject: COPRdown?
On 20 February 2014 21:39, Kevin Fenzi <kevin(a)scrye.com> wrote:
ok. It's finally processing again I hope. ;)
Down again I think:
https://copr.fedoraproject.org/coprs/rhughes/f19-gnome-3-12/builds/
Sorry to be the bearer of bad news....
Richard
3:39 a.m.
New subject: COPRdown?
On Fri, Feb 21, 2014 at 5:35 PM, Richard Hughes <hughsient(a)gmail.com> wrote:
On 21 February 2014 09:13, Christopher Meng
<cickumqt(a)gmail.com> wrote:
> Only SSL certificate issue.
Are you sure? I've had builds pending for 8h+...
Sorry... I misunderstood your comment...
3:45 a.m.
New subject: COPRdown?
On Fri, Feb 21, 2014 at 12:53 AM, Kevin Fenzi <kevin(a)scrye.com> wrote:
On Thu, 20 Feb 2014 21:52:04 +0000
Richard Hughes <hughsient(a)gmail.com> wrote:
> On 20 February 2014 21:39, Kevin Fenzi <kevin(a)scrye.com> wrote:
> > ok. It's finally processing again I hope. ;)
>
> Great, thanks.
>
> > I'll up that to 4 or something tomorrow... and we can see if we can
> > isolate where it is hitting problems.
>
> I submitted 27 srpms at one time; was this antisocial?
Naw, it shouldn't be, IMHO. ;)
The backend should cope with that kind of thing...
I think a ring/queue should be written to handle this.
I however have not looked into this for sometime.
I will bring this up in the channel.
Kind Regards,
kevin
_______________________________________________
copr-devel mailing list
copr-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/copr-devel
--
Skype: Frankie.Onuonga
twitter: Frankieonuonga
irc #freenode: Frankieonuonga
4:45 a.m.
New subject: COPRdown?
On 21 February 2014 09:45, Frankie Onuonga <frankie.onuonga(a)gmail.com> wrote:
I think a ring/queue should be written to handle this.
And now http://copr-be.cloud.fedoraproject.org/ seems down too. I'm
not having much luck this week.
Richard
10:12 a.m.
New subject: COPRdown?
On Sat, 22 Feb 2014 10:45:06 +0000
Richard Hughes <hughsient(a)gmail.com> wrote:
On 21 February 2014 09:45, Frankie Onuonga
<frankie.onuonga(a)gmail.com> wrote:
> I think a ring/queue should be written to handle this.
And now http://copr-be.cloud.fedoraproject.org/ seems down too. I'm
not having much luck this week.
Additionally Mirek is on vacation this week. ;(
Anyhow, the problem is that this backend instance keeps going
unresponsive. :(
I've destroyed it and brought it back up.
Hopefully we can track down why it's doing this.
kevin
12:25 p.m.
New subject: COPRdown?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/20/2014 10:39 PM, Kevin Fenzi wrote:
ok. It's finally processing again I hope. ;)
The problem seemed to be some kind of doom when it was set to 8
workers (so 8 builds at a time). It would launch them all off fine,
but then when they returned something would go south and the
instance wouldn't be responsive at all. It might be that they all
finished at the same time and caused a storm of createrepos or
something.
I set it back down to 1 worker (so things will be slow for a bit).
I'll up that to 4 or something tomorrow... and we can see if we
can isolate where it is hitting problems.
I think that the problem was that several threads crashed and there
was ~10 instances in Fedora Cloud running without parent so FC may run
out of resources.
I terminated them and set # workers back to 6.
I will try to read logs (if there are some) tomorrow.
I will have to write some maintainaince script which will clean up VM
which are not tracked by workers...
Mirek
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTCOuFAAoJEODXORnxaDVAA2gP/AuNVvnYRWUqRA/9uMTTOyPL
b6WOZDzDmKg8WLRRLdf4LnQjmgMy0zx85Yg9RhAgQQB2wbn4lNXCEYPz8NBhFZ6r
GI/pQlal5oVoq9RJ765EmnpNP9ZbZfPW6IsmcXHwVBwtkN8BLn2Ba9K5ybbD8jd9
55FL3giFVfto5DB+W2Jxlv/QtEtmHFzXjFpXHk6PxBHvbcDIQAHFVqvszCjPakKh
4BE6JGfQqB5r0YSlH4WMikEvrQihWfxtokdybyR0N/xFYkBxSdHd3GAOodEM3EUl
t1sCOxcvKXmIBccHZHEMrD32jbqMw8QBejj4H3+w2dIm/e/C6Ql6oOkeD1sMymGL
zSaICtpSgQb8KIfdu8qNLrKk+gHApTYTnx094ttGIIOmGowbQRV8EDWlBjfQnN13
o94m9RAE56o7a5nHE0bOIFdwkzZ49lcm5AiVAE9+T9wjckD+bFj09nmBl0fqdezE
y7UZwfslIr6ZtgcmjiD0UPfH/O5cSYXgBQ/Emm/AXpgAiOtgx7I2iLuI8KSZQt4s
2xqdIHf0K5VZSO/MRtH5pEmnbJWmh89pcERBePV2OaxPl4Ux5fCjRJXjCZ9sPZ0x
XucWMK01yYH1MRuGGtlGBcChxu3sO/pkDQ3buhe3BXFGNWMyRQi2WUFSMC3mXaE9
P+lgOLUQmqQh7CAUgWfn
=hMuJ
-----END PGP SIGNATURE-----
1:13 p.m.
New subject: COPRdown?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/20/2014 09:04 PM, Kevin Fenzi wrote:
The backend blew up, and I rebooted it...
Seems to be blew up again. :((
$ ssh root(a)copr-be.cloud.fedoraproject.org
ssh: connect to host copr-be.cloud.fedoraproject.org port 22: No route
to host
And even from Fedora Cloud:
$ ping 172.16.5.4
PING 172.16.5.4 (172.16.5.4) 56(84) bytes of data.
- From 172.16.5.5 icmp_seq=1 Destination Host Unreachable
Kevin, can you get to the console? And check why this happened? And
restart it again please?
Mirek
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBAgAGBQJTCPbKAAoJEODXORnxaDVAZFIP/ijL4EoNYOOKj8B2hZ3dFUl4
y4OAhf9OYWB/e9bfbCFWDDm85auUwOBetR1773v94zqefU/1zUVCKL21dd444To/
mW2zZnooHnykr/bMsah1Y7COh+AJt2fJCIaYFaNwIv4XGxyaR8vUqEld9ukS5rct
WAwz8fvlfgVtN2F+2srVREpH4l9+XYwNXXHZQ/WvUix0wN9WBlti4OPkvmMQyb4B
ROEW67f9PIoN9+GYa9GA+1yxcYld/GpPGAxICnqX66Cr5TGT5tm0sUyDOMH4np4C
UYZZzSu5sHxqllKhz9kihr4qph0EBjLLvYHDJuvJMPKfqQEVT7eiS7YkmKI+lSbK
/NZnfJgWU+EEfeETsuZQ2NB6Sxo6icV9dmPV1bULg8NLGODtFRHbhDS6nmZoQPhw
D7k1k6zrIybN1peCLzVuULotGg2joxMuNig5RZl/pZLq+qijiKfLJN0QzoAyJNp8
fdW3tedY5YweU9Aknz0AGxL6UcuI7iQyRPyfY3TeXtwK1m+4X57DisPCWKCeapqp
GnIF2URXhBmNXBbEIan4bmo7XOVVUXTGAJa+iQlqWgMr5kgaXt/nB33yrHwEfGji
YRIwKiLYVKQSjMYb7MPr0S83ejul5oYQwmur2ozwqwGoyC5+Lm8B2P1EN3vKOJPi
JCqo7bg1u6HRYFgmPRRW
=6owI
-----END PGP SIGNATURE-----
1:23 p.m.
New subject: COPRdown?
On Sat, 22 Feb 2014 20:13:15 +0100
Miroslav Suchy <msuchy(a)redhat.com> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 02/20/2014 09:04 PM, Kevin Fenzi wrote:
> The backend blew up, and I rebooted it...
Seems to be blew up again. :((
$ ssh root(a)copr-be.cloud.fedoraproject.org
ssh: connect to host copr-be.cloud.fedoraproject.org port 22: No route
to host
And even from Fedora Cloud:
$ ping 172.16.5.4
PING 172.16.5.4 (172.16.5.4) 56(84) bytes of data.
- From 172.16.5.5 icmp_seq=1 Destination Host Unreachable
Kevin, can you get to the console? And check why this happened? And
restart it again please?
The console is completely blank.
The logs have a bunch of OOM's from createrepo_c...
Restarted it.
kevin
3716
days inactive
3718
days old
copr-devel@lists.fedorahosted.org
14 comments
5 participants
participants (5)
-
Christopher Meng -
Frankie -
Kevin Fenzi -
Miroslav Suchý -
Richard Hughes