[releng] Issue #8299: ppc64le imagebuilds are failing for F29 Atomic Host Cloud Image

[Sinny Kumari]

sinnykumari reported a new issue against the project: `releng` that you are following:
``
Latest ppc64le imagebuilds are failing on ppc64le. By looking at failure from [F29
AtomicHost cloud Image
20190422.0](https://koji.fedoraproject.org/koji/taskinfo?taskID=34346463) , it appears
that vm install failed during boot time.

Note that ppc64le imagebuilds are failing since 20190415 onward compose for F29 updates
and updates-testing. Also, similar failure were seen for F30 ppc64le cloud-base images
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Sinny Kumari]

sinnykumari added a new comment to an issue you are following:
``
Snippet from IRC conversation on #fedora-releng

```
<mboddu> ksinny: Nothing has changed on the builders side
<nirik> yeah, nothing I know of off hand.
<ksinny> weird :/
<ksinny> nirik: imagebuilders areon Power9 now, right?
<nirik> there is a fair number of stuck vm's... I can clean all those up.
<nirik> yes
<ksinny> ok
<ksinny> let's see if stuck vms clean-up helps with next compose
<ksinny> nirik: imagebuild of F29 AH cloud Image went fine for me on Power 9 vm
locally(on F30), looks like builder specific issue. Did you get a chance to clean-up stuck
builds?
<nirik> yes, I did that right after I mentioned it yesterday.
<nirik> could be a kernel issue? or host...
<ksinny> nirik: P9 vm I am using has kernel-5.0.7-300.fc30.ppc64le, what kernel
version our koji imagebuilders are on?
<mboddu> ksinny: 5.0.6-200.fc29.ppc64le is the kernel version on the builders
```
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Sinny Kumari]

sinnykumari added a new comment to an issue you are following:
``
Good to see that ppc64le AH cloudImage built fine for F29
[updates](https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-upd...
and
[updates-testing](https://kojipkgs.fedoraproject.org/compose/updates/Fedor...
compose from 20190429.0 . Not sure what fixed it as I don't see any package update in
ostree since last compose from 20190427.0 . Any possible changes has been made on or after
20190427 which could possibly have fixed it?
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
I spent several hours yesterday looking into this and finally tracked it down to firewall
rules. The iptables firewall had been restarted, but libvirtd wasn't so it didn't
insert it's nat rules in. This meant that guest vm's couldn't get out to the
net for the initramfs/kernel to boot. 


``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Kevin Fenzi]

The status of the issue: `ppc64le imagebuilds are failing for F29 Atomic Host Cloud Image`
of project: `releng` has been updated to: Closed as Fixed by kevin.

https://pagure.io/releng/issue/8299

[Kevin Fenzi]

kevin added a new comment to an issue you are following:
``
So, this is hopefully all fixed now. 

``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Sinny Kumari]

sinnykumari added a new comment to an issue you are following:
``
Wow, that was really a tricky fix. Thanks a lot @kevin for fixing it!
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Mohan Boddu]

mohanboddu added a new comment to an issue you are following:
``
@kevin Thats some fix. Thanks for the help.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Dan Horák]

sharkcz added a new comment to an issue you are following:
``
Thanks @kevin, I've spend some time with it last week and couldn't reproduce the
problem.
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299

[Dusty Mabe]

dustymabe added a new comment to an issue you are following:
``
Thanks @kevin - We appreciate you looking into this! 
``

To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8299