[releng] Issue #7703: ppc builders seem broken: KVM implementation does not support Transactional Memory
10:51 a.m.
dustymabe reported a new issue against the project: `releng` that you are following:
``
* Describe the issue
koji image build tasks are failing with `qemu-system-ppc64: KVM implementation does not
support Transactional Memory, try cap-htm=off`.
See:
- https://pagure.io/dusty/failed-composes/issue/673
- https://pagure.io/dusty/failed-composes/issue/674
- https://pagure.io/dusty/failed-composes/issue/675
* When do you need this? (YYYY/MM/DD)
soon
* When is this no longer needed or useful? (YYYY/MM/DD)
never
* If we cannot complete your request, what is the impact?
ppc users sad
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
1:40 p.m.
kevin added a new comment to an issue you are following:
``
So, I am not sure what exactly is causing this. I did see that both the affected builders
were running some old stuck compose vm's, so perhaps it was a resource issue?
In any case, I updated them, killed the old compose jobs and rebooted them.
If that doesn't fix it, at least we have a recent kernel/base to start debugging with.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
3:09 p.m.
sharkcz added a new comment to an issue you are following:
``
@kevin, what's hypervisor on the builders? RHV?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
3:45 p.m.
kevin added a new comment to an issue you are following:
``
The builders are running as kvm vm's on a rhel7-alt installed host.
I have not yet tried to update the host kernel/libvirt/qemu stack. Perhaps thats worth a
shot?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
3:53 p.m.
kevin added a new comment to an issue you are following:
``
I have now rebooted them with 'ppc_tm=off' to see if that helps any...
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
4:01 p.m.
pbrobinson added a new comment to an issue you are following:
``
The things that come to mind:
* issue with Transactional Memory on host or guest
* regression in nested virt on either RHEL-Alt or Fedora (could be kernel/libvirt/qemu)
It would be useful to know if there was any patching of either host or build vm around the
time it started to fail
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
3:14 a.m.
sharkcz added a new comment to an issue you are following:
``
yes, knowing the history of updates (host, builder vm) would be helpful, handling HTM is
new feature in libvirt 4.6.0
disabling HTM from the bottom up might be a good workaround
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
11:19 a.m.
dustymabe added a new comment to an issue you are following:
``
this is still breaking basically all of our composes that run nightly for
container/cloud/atomic can we look closer at it?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
8:25 p.m.
kevin added a new comment to an issue you are following:
``
The builders are fedora 28. This started happening on the 25th.
On the 25th, they applied the following security updates:
```
qemu-kvm-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:16 AM UTC
qemu-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:16 AM UTC
kernel-modules-4.17.17-200.fc28.ppc64le Sat 25 Aug 2018 04:46:11 AM UTC
kernel-core-4.17.17-200.fc28.ppc64le Sat 25 Aug 2018 04:46:09 AM UTC
qemu-system-xtensa-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:05 AM UTC
qemu-system-xtensa-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:05 AM UTC
qemu-img-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:05 AM UTC
qemu-system-x86-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:04 AM UTC
qemu-system-x86-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:04 AM UTC
qemu-system-unicore32-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:03 AM UTC
qemu-system-unicore32-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:03 AM UTC
qemu-system-tricore-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:02 AM UTC
qemu-system-tricore-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:02 AM UTC
qemu-system-sparc-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:02 AM UTC
qemu-system-sparc-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:02 AM UTC
qemu-system-sh4-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:01 AM UTC
qemu-system-sh4-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:01 AM UTC
qemu-system-s390x-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:00 AM UTC
qemu-system-s390x-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:46:00 AM UTC
qemu-system-ppc-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:59 AM UTC
qemu-system-ppc-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:59 AM UTC
qemu-system-or1k-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:58 AM UTC
qemu-system-or1k-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:57 AM UTC
qemu-system-nios2-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:57 AM UTC
qemu-system-nios2-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:57 AM UTC
qemu-system-moxie-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:57 AM UTC
qemu-system-moxie-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:56 AM UTC
qemu-system-mips-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:56 AM UTC
qemu-system-mips-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:56 AM UTC
qemu-system-microblaze-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:54 AM UTC
qemu-system-microblaze-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:54 AM UTC
qemu-system-m68k-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:53 AM UTC
qemu-system-m68k-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:53 AM UTC
qemu-system-lm32-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:52 AM UTC
qemu-system-lm32-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:52 AM UTC
qemu-system-cris-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:52 AM UTC
qemu-system-cris-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:52 AM UTC
qemu-system-arm-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:51 AM UTC
qemu-system-arm-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:51 AM UTC
qemu-system-alpha-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:50 AM UTC
qemu-system-alpha-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:50 AM UTC
qemu-system-aarch64-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:50 AM UTC
qemu-user-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:49 AM UTC
qemu-system-aarch64-core-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:49 AM UTC
qemu-block-ssh-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:46 AM UTC
qemu-block-rbd-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:45 AM UTC
qemu-block-nfs-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:45 AM UTC
qemu-block-iscsi-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:45 AM UTC
qemu-block-gluster-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:45 AM UTC
qemu-block-dmg-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:45 AM UTC
qemu-common-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:44 AM UTC
qemu-block-curl-2.11.2-2.fc28.ppc64le Sat 25 Aug 2018 04:45:44 AM UTC
```
This lead me to:
https://bugzilla.redhat.com/show_bug.cgi?id=1614948
which is this same bug I am pretty sure... they are looking at reverting it now there.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
3:40 a.m.
pbrobinson added a new comment to an issue you are following:
``
Good catch, it looks like the culprit to me.
I suppose the question is do we downgrade qemu on just those builders building
images/containers to allow the fix to settle through while we get beta freeze out of the
way?
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
2:24 p.m.
dustymabe added a new comment to an issue you are following:
``
@sayanchowdhury FYI: we need to monitor the status of this ticket. Currently we can't
create ppc64le cloud images so we won't be able to release FAH next week. @kevin,
we'll monitor this ticket, but let us know if we can help you with anything.
cc @sinnykumari.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
7:49 p.m.
kevin added a new comment to an issue you are following:
``
I've upgraded the builders to the 'fixed' qemu... so this should be fixed now.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
8:20 p.m.
dustymabe added a new comment to an issue you are following:
``
:fireworks:
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
6:27 a.m.
sinnykumari added a new comment to an issue you are following:
``
It seems there is still some issue (different issue) after upgrading F28 builders to
qemu-2.11.2-3.fc28. We have imagebuild failing from 20180831.n.0 composes.
https://pagure.io/dusty/failed-composes/issue/730
https://pagure.io/dusty/failed-composes/issue/733
I can reproduce the problem locally but didn't find the cause yet.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
8:19 a.m.
dustymabe added a new comment to an issue you are following:
``
I see this in the logs:
```
2018-08-31T06:15:45.899543Z qemu-system-ppc64: warning: Failed to set KVM's VSMT mode
to 8 (errno -22)
2018-08-31T06:15:45.979785Z qemu-system-ppc64: KVM implementation does not support
Transactional Memory, try cap-htm=off
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
8:37 a.m.
sinnykumari added a new comment to an issue you are following:
``
I see this in the logs:
2018-08-31T06:15:45.899543Z qemu-system-ppc64: warning: Failed to set KVM's VSMT mode
to 8 (errno -22)
2018-08-31T06:15:45.979785Z qemu-system-ppc64: KVM implementation does not support
Transactional Memory, try cap-htm=off
I think these log messages are from the qemu which introduced the bug.
[qemu-2.11.2-3.fc28](https://koji.fedoraproject.org/koji/buildinfo?buildID... was
built to fix this issue and is available under f28-infra tag and I believe @kevin upgraded
power builders with qemu-2.11.2-3.fc28 .
From today's compose e.g.
[f28-container](https://koji.fedoraproject.org/koji/taskinfo?taskID=29390408), we see
different error now:
```
ApplianceError: Image status is FAILED: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.
```
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
2:15 p.m.
dustymabe added a new comment to an issue you are following:
``
any more ideas? this is going to block FAH releases unfortunately if we don't get it
fixed before next week.
@kevin could we downgrade as @pbrobinson
[suggested](https://pagure.io/releng/issue/7703#comment-528815)
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
2:36 p.m.
sinnykumari added a new comment to an issue you are following:
``
I am not very sure that downgrading qemu version is going to help much. I later did a
fresh install of F28 on ppc64le vm, installed imagebuild dependencies from F28 stable
repo. Ran imagefactory, worked fine. Updated only qemu packages to 2.11.2-2 version and
ran again imagefactory. Got transaction memory issue as expected. Updated only qemu
packages to 2.11.2-3, and ran imagefactory. It worked fine.
So, I suspect that current error could be coming from some other package update. I
didn't get much time to update rest of the packages individually to latest and see
what causes the issue. I might try this over weekend if not then on Monday.
I maybe wrong, so feel free to downgrade qemu to last working version and see if it helps.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
4:31 p.m.
kevin added a new comment to an issue you are following:
``
yeah, just downgrading qemu doesn't do it. ;(
I'm trying to isolate the problem package, but it's pretty daunting. ;(
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
10:30 p.m.
kevin added a new comment to an issue you are following:
``
I can't seem to isolate it. ;(
When everything is updated I do see in the libvirt logs:
2018-09-01T03:20:46.049548Z qemu-system-ppc64: warning: Failed to set KVM's VSMT mode
to 8 (errno -22)
2018-09-01T03:20:49.443963Z qemu-system-ppc64: error creating device tree:
(fdt_begin_node(fdt_skel, "")): FDT_ERR_BADSTATE
https://bugzilla.redhat.com/show_bug.cgi?id=1624539
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
2:08 a.m.
sinnykumari added a new comment to an issue you are following:
``
Found the package which might be causing the issue, it is libfdt package update to
1.4.7-1. I downgraded it to 1.4.6-5 and problem seems to be gone.
libfdt package is part of dtc, maybe let's try again with downgrading to
[libfdt-1.4.6-5](https://koji.fedoraproject.org/koji/buildinfo?buildID=109... on power
image builders?
Image build succeeded for me locally with downgraded libfdt-1.4.6-5 and latest
qemu-2.11.2-3.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
11:04 a.m.
dustymabe added a new comment to an issue you are following:
``
Found the package which might be causing the issue
:heart:
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
11:57 a.m.
pbrobinson added a new comment to an issue you are following:
``
Interesting, wouldn't have thought, be good to get that problem reported to the
maintainer upstream and I can pull in a fix
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
12:47 p.m.
kevin added a new comment to an issue you are following:
``
You rock @sinnykumari ! That does indeed seem to be it. ;)
I'll move my bug to that package...and I have downgraded the image builders.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
7:32 a.m.
sinnykumari added a new comment to an issue you are following:
``
Awesome!
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
9:09 a.m.
The status of the issue: `ppc builders seem broken: KVM implementation does not support
Transactional Memory` of project: `releng` has been updated to: Closed as Fixed by
mohanboddu.
https://pagure.io/releng/issue/7703
9:09 a.m.
mohanboddu added a new comment to an issue you are following:
``
@sinnykumari You are awesome.
Thanks for figuring it out.
Closing this ticket.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7703
2079
days inactive
2092
days old
rel-eng@lists.fedoraproject.org
26 comments
6 participants
participants (6)
-
Dan Horák -
Dusty Mabe
-
Kevin Fenzi
-
Mohan Boddu
-
Peter Robinson
-
Sinny Kumari