10:46 p.m.
Hi,
I've had a koji build running for over 6 hours, and the s390x build
still hasn't started. Are the s390x builders offline?
--
Susi Lehtola
Fedora Project Contributor
jussilehtola(a)fedoraproject.org
11:40 p.m.
On Wed, Mar 31, 2021 at 11:46:15PM -0400, Susi Lehtola wrote:
Hi,
I've had a koji build running for over 6 hours, and the s390x build
still hasn't started. Are the s390x builders offline?
Nope. However, a bunch of builds landed earlier today and all the big
ones are taking up the s390x builders. Additionally, the LPAR is very
slow due to other tasks on the same mainframe. ;(
It should process through them, but it's going to take a while. ;(
kevin
1:09 a.m.
Hi,
On 4/1/21 6:40 AM, Kevin Fenzi wrote:
On Wed, Mar 31, 2021 at 11:46:15PM -0400, Susi Lehtola wrote:
> Hi,
>
>
> I've had a koji build running for over 6 hours, and the s390x build
> still hasn't started. Are the s390x builders offline?
Nope. However, a bunch of builds landed earlier today and all the big
ones are taking up the s390x builders. Additionally, the LPAR is very
slow due to other tasks on the same mainframe. ;(
Very very slow is the right description, I just was watching build.log
for a task which is actually running on a s390x build and building a
single object file takes like a minute.
Can you perhaps poke the admin of the mainframe and have them take
a look? Maybe another LPAR has some runaway tasks ? This certainly
does not seem normal.
Regards,
Hans
7:52 a.m.
On Wed, 31 Mar 2021 21:40:21 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
On Wed, Mar 31, 2021 at 11:46:15PM -0400, Susi Lehtola wrote:
> Hi,
>
>
> I've had a koji build running for over 6 hours, and the s390x build
> still hasn't started. Are the s390x builders offline?
Nope. However, a bunch of builds landed earlier today and all the big
ones are taking up the s390x builders. Additionally, the LPAR is very
slow due to other tasks on the same mainframe. ;(
It should process through them, but it's going to take a while. ;(
and I can see a few of misbehaving builds (tasks being restarted after a
timeout for few days) that also take some s390x capacity (but other
arches are affected as well).
Dan
7:32 p.m.
On Thu, Apr 01, 2021 at 02:52:44PM +0200, Dan Horák wrote:
On Wed, 31 Mar 2021 21:40:21 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
> On Wed, Mar 31, 2021 at 11:46:15PM -0400, Susi Lehtola wrote:
> > Hi,
> >
> >
> > I've had a koji build running for over 6 hours, and the s390x build
> > still hasn't started. Are the s390x builders offline?
>
> Nope. However, a bunch of builds landed earlier today and all the big
> ones are taking up the s390x builders. Additionally, the LPAR is very
> slow due to other tasks on the same mainframe. ;(
>
> It should process through them, but it's going to take a while. ;(
and I can see a few of misbehaving builds (tasks being restarted after a
timeout for few days) that also take some s390x capacity (but other
arches are affected as well).
Yeah, those are usually the case of the build taking up all memory and
OOM killer kills kojid, so the build gets restarted by koji.
I think we may want to try and set kojid as exempt from OOM and see if
that helps it kill the right thing.
kevin
1135
days inactive
1136
days old
4 comments
4 participants
participants (4)
-
Dan Horák -
Hans de Goede -
Kevin Fenzi -
Susi Lehtola