8:07 p.m.
dustymabe reported a new issue against the project: `releng` that you are following:
``
I'm not sure if it's because of an issue with a builder of because of the software
content in rawhide itself but we need to investigate this so we can get ppc64le media
building again.
#####here are koji tasks from Rawhide-20180217.n.2:
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25123798
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25123819
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25123781
##### Rawhide-20180216.n.0
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25095465
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25095487
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25095448
etc...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
8:11 p.m.
dustymabe added a new comment to an issue you are following:
``
Note @kevin looked earlier in the day today and did see this in the logs:
```
13:41:43 +nirik | [Sat Feb 17 15:19:25 2018] Couldn't emulate instruction
0x7c00491c (op 31 xop 142)
13:41:43 +nirik | [Sat Feb 17 15:19:25 2018] kvmppc_exit_pr_progint: emulation at
c00000000004e0b0 failed (7c00491c)
13:41:43 +nirik | [Sat Feb 17 15:19:25 2018] Couldn't emulate instruction
0x7c00491c (op 31 xop 142)
13:41:43 +nirik | [Sat Feb 17 15:19:25 2018] kvmppc_exit_pr_progint: emulation at
c00000000004e0b0 failed (7c00491c)
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
8:16 p.m.
dustymabe added a new comment to an issue you are following:
``
hey @sinnykumari - can you look at this bug on monday and see if there is an issue with
running an f28 ppc KVM guest on top of an F27 ppc hypervisor?
Try to do an anaconda install for f28 on top of f27 hypervisor and if that doesn't
work then try to do an anaconda install for f28 on top of f28 hypervisor.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
3:44 a.m.
sinnykumari added a new comment to an issue you are following:
``
Anaconda text based install of F28 server dvd from Rawhide-20180218 nightly on F27
hypervisor happened successfully for me. Also, it boots fine.
qemu version - qemu-2.10.1-2.fc27.ppc64le
kernel - 4.15.3-300.fc27.ppc64le
Will try to run imagefactory locally and see if I can figure out something.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
7:35 a.m.
dustymabe added a new comment to an issue you are following:
``
Will try to run imagefactory locally and see if I can figure out
something.
thanks!
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
5:53 a.m.
sinnykumari added a new comment to an issue you are following:
``
Filed a bug against qemu for this issue -
https://bugzilla.redhat.com/show_bug.cgi?id=1547467
Let's see what is their thought on this.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
8:37 a.m.
sinnykumari added a new comment to an issue you are following:
``
There was at least a successful Atomic cloudImage build in rawhide from
Fedora-Atomic-Rawhide-20180221.n.2 compose
https://koji.fedoraproject.org/koji/taskinfo?taskID=25224905 .
qemu still crashes in my local system, so not sure what caused this successful
imagefactory run in koji builder.
I will wait for today's nightly compose to finish in rawhide and f28 and will observe
the result.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9:09 a.m.
dustymabe added a new comment to an issue you are following:
``
still failing. from last night's run:
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25278703
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25278700
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25278689
- https://koji.fedoraproject.org/koji/taskinfo?taskID=25278680
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
3:24 a.m.
sinnykumari added a new comment to an issue you are following:
``
Now, I am getting successful imagefactory run for F29 Atomic cloud and container image by
using required artifacts from 20180224.n.0 rawhide compose . Host machine I am using is
latest Fedora 27, ppc64le. I still see problem if host machine used is rawhide.
May I know which Fedora version we are running on ppc64le koji builders where these failed
composes are running?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
4:06 a.m.
pbrobinson added a new comment to an issue you are following:
``
May I know which Fedora version we are running on ppc64le koji
builders where these failed composes are running?
Fedora 27 with a 4.15.3-300.fc27.ppc64le kernel
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
4:09 a.m.
sinnykumari added a new comment to an issue you are following:
``
May I know which Fedora version we are running on ppc64le koji builders where these
failed composes are running?
Fedora 27 with a 4.15.3-300.fc27.ppc64le kernel
Hmm, even I have same kernel version.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
8:49 a.m.
sinnykumari added a new comment to an issue you are following:
``
So, it looks like concurrency issue when some jobs are running in parallel on Power box.
For imageafctory run on our builders, we define cpus=2 in oz config [defined in ansible
script](https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/ro....
If we just use single cpu core on power box, imagefactory runs fine. Ran locally
imagefactory for cloud(f28), container(f28) and atomic (f29) builds and they all
succeeded.
So, until we know what has caused this problem on power for F28 and onward image builds,
maybe we can update oz.cfg with cpus=1 for power builders used for building images?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9 a.m.
ausil added a new comment to an issue you are following:
``
oz defaults to 1 cpu, we had to add more due to a bug in anaconda years ago. we can look
at changing it back to 1, do we know why 2 cores is failing and 1 is not? I ask because I
expect that users will typically have more than one core, are we just delaying the issues
to them?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9:29 a.m.
sinnykumari added a new comment to an issue you are following:
``
oz defaults to 1 cpu, we had to add more due to a bug in anaconda
years ago. we can look at changing it back to 1, do we know why 2 cores is failing and 1
is not? I ask because I expect that users will typically have more than one core, are we
just delaying the issues to them?
No, I don't know why this is failing when multiple cores used. I filed a [bug against
qemu](https://bugzilla.redhat.com/show_bug.cgi?id=1547467) but we don't have much
information yet that why install vm crashes in between the image build process.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
10:44 a.m.
sharkcz added a new comment to an issue you are following:
``
as my investigation show the issue is a "nested virt with a multi-CPU guest"
problem, so having a workaround by changing the oz.cfg to cpus=1 in the infra would be
useful
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
6:34 p.m.
kevin added a new comment to an issue you are following:
``
I've pushed a change for now that will set it to 1 on ppc64{le} builders.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
8:55 a.m.
sharkcz added a new comment to an issue you are following:
``
And it helped, there are ppc64le images today. Thanks.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
10:59 a.m.
dustymabe added a new comment to an issue you are following:
``
I'm glad we got this worked around for now. @sharkcz are we still working to find the
root cause and get that fixed (via the BZ) ?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
2:14 a.m.
sharkcz added a new comment to an issue you are following:
``
@dustymabe , yes, that's the plan, AFAICT the root cause is known, now working on
getting it fixed.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9:56 a.m.
syeghiay added a new comment to an issue you are following:
``
@ksinny , is this still an issue?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
2:30 a.m.
sinnykumari added a new comment to an issue you are following:
``
This issue should be fixed long time back. We have various successful imagebuild run
including cloud and containe rimage in F29 release and various F29 TwoWeek releases
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
2:30 a.m.
sinnykumari added a new comment to an issue you are following:
``
This issue should be fixed long time back. We have various successful
imagebuild run including cloud and container image in F29 release and various F29 TwoWeek
releases
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9:04 a.m.
The status of the issue: `ppc image build tasks are failing for f28/rawhide` of project:
`releng` has been updated to: Closed as Fixed by mohanboddu.
https://pagure.io/releng/issue/7326
9:04 a.m.
mohanboddu added a new comment to an issue you are following:
``
ppc64 (not le) cloud images have been failing, but we removed them for building.
Just FYI.
Closing this ticket as its not needed any more.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
9:26 a.m.
dustymabe added a new comment to an issue you are following:
``
@mohanboddu can we just remove all f28 building that is happening now? the only builds
that may still be wanted are container builds maybe. We definitely want to drop cloud
builds.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7326
1856
days inactive
2252
days old
rel-eng@lists.fedoraproject.org
24 comments
8 participants
participants (8)
-
Dan Horák -
Dennis Gilmore
-
Dusty Mabe
-
Kevin Fenzi
-
Mohan Boddu
-
Peter Robinson
-
Sinny Kumari
-
Suzanne Yeghiayan