Copr ppc64le builds hanging for two days

[Roman Mohr]

Hi,

I noticed some builds for ppc64 [1] which are building for two days and it
seems that they are blocking all ppc64le slots.

Best Regards,
Roman

[1] https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/169353/

[Roman Mohr]

CC johnmh

On Tue, Mar 22, 2016 at 1:31 PM, Roman Mohr  wrote:
> Hi,
>
> I noticed some builds for ppc64 [1] which are building for two days and it
> seems that they are blocking all ppc64le slots.
>
> Best Regards,
> Roman
>
> [1] https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/169353/

[Kevin Fenzi]

On Tue, 22 Mar 2016 16:43:58 +0100
Roman Mohr  wrote:
> CC johnmh
>
> On Tue, Mar 22, 2016 at 1:31 PM, Roman Mohr 
> wrote:
> > Hi,
> >
> > I noticed some builds for ppc64 [1] which are building for two days
> > and it seems that they are blocking all ppc64le slots.
> >
> > Best Regards,
> > Roman
> >
> > [1]
> > https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/169353/

I've restarted things so it should start processing them soon.

Note that things should be a lot better before too long in this area as
we have some power8 machines we are hoping to bring up locally to copr
and run the builds there (instead of remotely).

kevin

[Sérgio Basto]

On Ter, 2016-03-22 at 11:16 -0600, Kevin Fenzi wrote:
> On Tue, 22 Mar 2016 16:43:58 +0100
> Roman Mohr  wrote:
> > CC johnmh
> >
> > On Tue, Mar 22, 2016 at 1:31 PM, Roman Mohr 
> > wrote:
> > > Hi,
> > >
> > > I noticed some builds for ppc64 [1] which are building for two
> > > days
> > > and it seems that they are blocking all ppc64le slots.
> > >
> > > Best Regards,
> > > Roman
> > >
> > > [1]
> > > https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/16
> > > 9353/
>
> I've restarted things so it should start processing them soon.
>
> Note that things should be a lot better before too long in this area
> as
> we have some power8 machines we are hoping to bring up locally to
> copr
> and run the builds there (instead of remotely).

Anyway, usually is lantw44/vim-latest that hangs ppc64le, he makes
builds, of 6 in 6 hours [1]. I don't understand the goal , but if last
build doesn't finish , shouldn't start another one ... , and I give up
try to build for ppc64le, because my builds hang for days (waiting for
 lantw44/vim-latest ...)

[1] https://copr.fedorainfracloud.org/coprs/lantw44/vim-latest/builds/

Best regards,

[Roman Mohr]

Kevin Fenzi  schrieb am Di., 22. März 2016 um 18:19 Uhr:
> On Tue, 22 Mar 2016 16:43:58 +0100
> Roman Mohr  wrote:
> > CC johnmh
> >
> > On Tue, Mar 22, 2016 at 1:31 PM, Roman Mohr 
> > wrote:
> > > Hi,
> > >
> > > I noticed some builds for ppc64 [1] which are building for two days
> > > and it seems that they are blocking all ppc64le slots.
> > >
> > > Best Regards,
> > > Roman
> > >
> > > [1]
> > > https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/169353/
>
> I've restarted things so it should start processing them soon.
>
> Note that things should be a lot better before too long in this area as
> we have some power8 machines we are hoping to bring up locally to copr
> and run the builds there (instead of remotely).

Thanks, worked!
> kevin
>
> copr-devel mailing list
> [e-mail]
>
> https://lists.fedorahosted.org/admin/lists/[e-mail]

[Ting-Wei Lan]

於 二，2016-03-22 於 18:46 +0000，Sérgio Basto 提到：
> On Ter, 2016-03-22 at 11:16 -0600, Kevin Fenzi wrote:
> > On Tue, 22 Mar 2016 16:43:58 +0100
> > Roman Mohr  wrote:
> > > CC johnmh
> > >
> > > On Tue, Mar 22, 2016 at 1:31 PM, Roman Mohr 
> > > wrote:
> > > > Hi,
> > > >
> > > > I noticed some builds for ppc64 [1] which are building for two
> > > > days
> > > > and it seems that they are blocking all ppc64le slots.
> > > >
> > > > Best Regards,
> > > > Roman
> > > >
> > > > [1]
> > > > https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/
> > > > 16
> > > > 9353/
> >
> > I've restarted things so it should start processing them soon.
> >
> > Note that things should be a lot better before too long in this
> > area
> > as
> > we have some power8 machines we are hoping to bring up locally to
> > copr
> > and run the builds there (instead of remotely).
>
> Anyway, usually is lantw44/vim-latest that hangs ppc64le, he makes
> builds, of 6 in 6 hours [1].

I am sorry for causing problems. I have modified my script to reduce
the build frequency from 4 times a day to 1 time a week for ppc64le. It
seems that ppc64le builders are not able to process builds as fast as
x86_64 and i386 builders, and they usually hang for days.
> I don't understand the goal , but if last
> build doesn't finish , shouldn't start another one ... ,

As vim upstream releases updates much faster than before, I no longer
manually submit builds for lantw44/vim-latest. These builds are
automatically submitted by my crontab every 6 hours when an update is
found. My script doesn't check whether the old build is finished before
submitting a new build, and it hardly causes problems for x86_64 and
i386 because these builds usually finish in 10 minutes.

I use 'copr-cli build -r' separate ppc64le builds and non-ppc64le
builds now. Non-ppc64le builds are still submitted when new updates are
released, but ppc64le builds are only submitted once a week. I hope it
is much less possible for my lantw44/vim-latest builds to block other
builds.
> and I give up
> try to build for ppc64le, because my builds hang for days (waiting
> for
>  lantw44/vim-latest ...)
>
> [1] https://copr.fedorainfracloud.org/coprs/lantw44/vim-latest/builds
> /
>
> Best regards,

[Miroslav Suchý]

Dne 23.3.2016 v 14:06 Ting-Wei Lan napsal(a):
> I use 'copr-cli build -r' separate ppc64le builds and non-ppc64le
> builds now. Non-ppc64le builds are still submitted when new updates are
> released, but ppc64le builds are only submitted once a week. I hope it
> is much less possible for my lantw44/vim-latest builds to block other
> builds.

That is perfect. We have quite a lot of resources (and reliable) for intel machines, but ppc64 is still little bit tricky.

[Sérgio Basto]

Hello,

On Qua, 2016-03-23 at 21:06 +0800, Ting-Wei Lan wrote:
> I am sorry for causing problems. I have modified my script to reduce
> the build frequency from 4 times a day to 1 time a week for ppc64le.
> It
> seems that ppc64le builders are not able to process builds as fast as
> x86_64 and i386 builders, and they usually hang for days.

I don't know what to say, sorry for my bad mood, sometimes is a way to
write things quickly, but be rewarded feels good. 
Thanks for reply and explaining what happened.
Anyway, to be clear, I don't expect all ppc64le problems be solved but
it will help. Builders, now, are running very well ...
> > I don't understand the goal , but if last
> > build doesn't finish , shouldn't start another one ... ,
>
> As vim upstream releases updates much faster than before, I no longer
> manually submit builds for lantw44/vim-latest. These builds are
> automatically submitted by my crontab every 6 hours when an update is
> found. My script doesn't check whether the old build is finished
> before
> submitting a new build, and it hardly causes problems for x86_64 and
> i386 because these builds usually finish in 10 minutes.
>
> I use 'copr-cli build -r' separate ppc64le builds and non-ppc64le
> builds now. Non-ppc64le builds are still submitted when new updates
> are
> released, but ppc64le builds are only submitted once a week. I hope
> it
> is much less possible for my lantw44/vim-latest builds to block other
> builds.

Looks a good idea (submit builds automatically :) ) , you could, easily
, watch the page https://copr.fedorainfracloud.org/status/waiting/%C2%A0and
with a web scraper, see if you have running jobs and if you have, you
may delay the next update ...

Web scraper example:
wget https://copr.fedorainfracloud.org/status/waiting/ -qO - | grep lantw44/vim-latest

Best regards,

[Sérgio Basto]

Hi,
Today we have the same problem ...
https://copr.fedorainfracloud.org/status/waiting/

What is the best and quickest place to report this kind of issues ?

Thanks.

On Ter, 2016-03-22 at 12:31 +0000, Roman Mohr wrote:
> Hi,
>
> I noticed some builds for ppc64 [1] which are building for two days
> and it seems that they are blocking all ppc64le slots.
>
> Best Regards,
> Roman
>
> [1] https://copr.fedorainfracloud.org/coprs/johnmh/libraknet/build/16
> 9353/
>
> copr-devel mailing list
> [e-mail]
> https://lists.fedorahosted.org/admin/lists/[e-mail]
> ted.org

[Miroslav Suchý]

Dne 19.4.2016 v 20:41 Sérgio Basto napsal(a):
> What is the best and quickest place to report this kind of issues ?

I poked it.

IRC, mailing list, infra ticket. (this order for speed, for reliability
in reverse order)

Mirek