PSA: koji armv7hl builders locking up

[Fabio Valentini]

Hi all,

TL;DR: It seems like armv7hl koji builders are locking up, most often
when running dnf to install the buildroot or to install dependencies.

I've seen many people getting hit by this issue:
https://pagure.io/fedora-infrastructure/issue/10833

So, if your build takes unusually long time on armv7hl, look at the
task's logs in koji, and if they're stuck at an early stage of the
build, you will need to cancel the task and resubmit it (or ask releng
to run "koji free-task NNNNNNN" for the stuck armv7hl task).

Fabio

[Kevin Fenzi]

On Sun, Aug 07, 2022 at 12:10:44PM +0200, Fabio Valentini wrote:
>  Hi all,
>  
>  TL;DR: It seems like armv7hl koji builders are locking up, most often
>  when running dnf to install the buildroot or to install dependencies. 
Yeah. ;( 

>  I've seen many people getting hit by this issue:
>  https://pagure.io/fedora-infrastructure/issue/10833
>  
>  So, if your build takes unusually long time on armv7hl, look at the
>  task's logs in koji, and if they're stuck at an early stage of the
>  build, you will need to cancel the task and resubmit it (or ask releng
>  to run "koji free-task NNNNNNN" for the stuck armv7hl task). 
Or just wait... I have been checking them a few times a day and
rebooting any that are locked up. :( 

I've been trying to isolate it, but it's not been easy. 
I've tried 5.18 and 5.17 kernels. I've tweaked the virt settings. 
Still trying to figure out a way to get them stable. 

Sorry for the hassle, and thanks for noting it.

kevin

[Steve Dickson]

On 8/7/22 3:49 PM, Kevin Fenzi wrote:
>  Or just wait... I have been checking them a few times a day and
>  rebooting any that are locked up. 
How long do we have to wait? [1] has been stuck
since
Mon, 08 Aug 2022 14:17:44 UTC

steved.

[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=90612742

[Kevin Fenzi]

On Tue, Aug 09, 2022 at 09:14:17AM -0400, Steve Dickson wrote:
>  
>  
>  On 8/7/22 3:49 PM, Kevin Fenzi wrote:
>  > Or just wait... I have been checking them a few times a day and
>  > rebooting any that are locked up.
>  How long do we have to wait? [1] has been stuck since
>  Mon, 08 Aug 2022 14:17:44 UTC
>  
>  steved.
>  
>  [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=90612742 
Thats one I rebooted this morning after I woke up (and your build is
finished). I am not sure why I didn't see that one as stuck yesterday,
perhaps there's two failure modes here. ;( 

kevin