Meta question: What is the best upstream list for discussing ARM & KVM issues? Or ARM & virt issues in general?
I'm trying to get stock qemu from git or qemu 1.6 to boot a guest using KVM.
My host has a 3.11.0 LPAE kernel, and it exposes a /dev/kvm device. qemu can open this device, it gets as far as creating a KVM guest, and that is all apparently successful. But the ioctl KVM_RUN returns -ENXIO.
ENXIO is not a documented return from the KVM_RUN API but I've looked at the kernel code and I can't see where KVM_RUN would return ENXIO.
All places in general where ENXIO might be returned are related to the function "irqchip_in_kernel" and the mysterious "VGIC" hardware (which I guess is doing interrupt routing like APIC on PCs?)
Do I have to add one of these devices to the emulation perhaps?
Does anyone have a qemu command line which reliably boots a guest using KVM?
Rich.
/usr/bin/qemu-system-arm \ -global virtio-blk-device.scsi=off \ -nodefconfig \ -nodefaults \ -nographic \ -M vexpress-a9 \ -machine accel=kvm:tcg \ -m 500 \ -no-reboot \ -kernel /home/rjones/d/libguestfs/tmp/.guestfs-1001/kernel.3268 \ -dtb /home/rjones/d/libguestfs/tmp/.guestfs-1001/dtb.3268 \ -initrd /home/rjones/d/libguestfs/tmp/.guestfs-1001/initrd.3268 \ -device virtio-scsi-device,id=scsi \ -drive file=/home/rjones/d/libguestfs/tmp/libguestfsFZio45/scratch.1,cache=unsafe,format=raw,id=hd0,if=none \ -device scsi-hd,drive=hd0 \ -drive file=/home/rjones/d/libguestfs/tmp/.guestfs-1001/root.3268,snapshot=on,id=appliance,cache=unsafe,if=none \ -device scsi-hd,drive=appliance \ -device virtio-serial-device \ -serial stdio \ -chardev socket,path=/home/rjones/d/libguestfs/tmp/libguestfsFZio45/guestfsd.sock,id=channel0 \ -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \ -append 'panic=1 console=ttyAMA0 udevtimeout=600 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color' error: kvm run failed No such device or address
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
All places in general where ENXIO might be returned are related to the function "irqchip_in_kernel" and the mysterious "VGIC" hardware (which I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem to be getting there, slowly ...
Rich.
On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
All places in general where ENXIO might be returned are related to the function "irqchip_in_kernel" and the mysterious "VGIC" hardware (which I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem to be getting there, slowly ...
Rich.
Hi Rich,
Thanks for the tip above. I was able to get virtualization running on my Samsung ARM chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a patch).
https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB files are needed for newer kernels. Is there a particular DTB file that would be best to use? There seems to be several vexpress*.dtb files to choose from.
-Will
On Tue, Sep 10, 2013 at 12:27:48PM -0400, William Cohen wrote:
On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
All places in general where ENXIO might be returned are related to the function "irqchip_in_kernel" and the mysterious "VGIC" hardware (which I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem to be getting there, slowly ...
Rich.
Hi Rich,
Thanks for the tip above. I was able to get virtualization running on my Samsung ARM chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a patch).
Interesting .. which/what is "nv-uboot" and where did you get it from?
https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB files are needed for newer kernels. Is there a particular DTB file that would be best to use? There seems to be several vexpress*.dtb files to choose from.
I used:
qemu-system-arm \ -M vexpress-a9 \ -dtb /boot/dtb-....../vexpress-v2p-ca9.dtb \ -machine accel=kvm:tcg,kernel_irqchip=off \ [etc]
You can find the full command line that libguestfs passes here:
https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L24...
Rich.
On 09/10/2013 12:41 PM, Richard W.M. Jones wrote:
On Tue, Sep 10, 2013 at 12:27:48PM -0400, William Cohen wrote:
On 09/07/2013 08:16 AM, Richard W.M. Jones wrote:
On Sat, Sep 07, 2013 at 01:02:39PM +0100, Richard W.M. Jones wrote:
All places in general where ENXIO might be returned are related to the function "irqchip_in_kernel" and the mysterious "VGIC" hardware (which I guess is doing interrupt routing like APIC on PCs?)
The solution was some creative grepping of the qemu sources:
-machine accel=kvm:tcg,kernel_irqchip=off
Works great (except of course virtio-serial is still broken). I seem to be getting there, slowly ...
Rich.
Hi Rich,
Thanks for the tip above. I was able to get virtualization running on my Samsung ARM chromebook (using a nv-uboot that turns on HYP and a locally built 3.11 kernel with a patch).
Interesting .. which/what is "nv-uboot" and where did you get it from?
Hi Rich,
I used the u-boot described in http://www.virtualopensystems.com/media/chromebook/chromebook.pdf:
git://github.com/virtualopensystems/u-boot.git
This u-boot doesn't seem to understand initramfs, so the kernel I am using has some things compiled in rather than modules. For the linux-3.11 kernel I also had to use a patch based on:
https://github.com/virtualopensystems/linux-kvm-arm/commit/32682304c9935dd0a...
Peter Robinson mentioned there are some other patches submitted for review that are suppose to accomplished the same.
https://fedoraproject.org/wiki/Architectures/ARM/F18/Versatile_Express
This particular image uses a 3.6.10 linux kernel. The URL above mentions that DTB files are needed for newer kernels. Is there a particular DTB file that would be best to use? There seems to be several vexpress*.dtb files to choose from.
I used:
qemu-system-arm \ -M vexpress-a9 \ -dtb /boot/dtb-....../vexpress-v2p-ca9.dtb \ -machine accel=kvm:tcg,kernel_irqchip=off \ [etc]
Okay I will try "-M vexpress-a9 -dtb ..vexpress-v2p-ca9.dtb"
You can find the full command line that libguestfs passes here:
https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L24...
Rich.
I tried to run a couple of the systemtap virtualization examples on the host (https://sourceware.org/systemtap/examples/keyword-index.html#VIRTUALIZATION). However, it looks like the some of the kernel tracepoint that are available for x86: kernel.trace("kvm_entry") and kernel.trace("kvm_exit") are not available on my kernel. However, I do see the trace points defined on: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/trace.h#L12 And look to be used in the 3.11 kernel.
trace_kvm_entry: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/arm.c#L568 trace_kvm_exit: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/arm.c#L577
Maybe the kernel doesn't have some config set correctly. There are some kernel.trace("kvm_*") that are listed on the host kernel:
$ sudo stap -L 'kernel.trace("kvm_*")' kernel.trace("kvm_ack_irq") $irqchip:unsigned int $pin:unsigned int kernel.trace("kvm_age_page") $hva:ulong $slot:struct kvm_memory_slot* $ref:int kernel.trace("kvm_fpu") $load:int kernel.trace("kvm_mmio") $type:int $len:int $gpa:u64 $val:u64 kernel.trace("kvm_set_irq") $gsi:unsigned int $level:int $irq_source_id:int kernel.trace("kvm_userspace_exit") $reason:__u32 $errno:int
The kvm trace event that were found are defined in http://lxr.linux.no/#linux+v3.11/include/trace/events/kvm.h
-Will
On Tue, Sep 10, 2013 at 02:32:55PM -0400, William Cohen wrote: [...]
Thanks for the links. I will get back to ARM once I've got libguestfs working on PPC64 ...
In the meantime:
I tried to run a couple of the systemtap virtualization examples on the host (https://sourceware.org/systemtap/examples/keyword-index.html#VIRTUALIZATION). However, it looks like the some of the kernel tracepoint that are available for x86: kernel.trace("kvm_entry") and kernel.trace("kvm_exit") are not available on my kernel. However, I do see the trace points defined on: http://lxr.linux.no/#linux+v3.11/arch/arm/kvm/trace.h#L12 And look to be used in the 3.11 kernel.
ARM KVM seems to be completely different from x86 KVM (and incidentally different from PPC's 2 different types of KVM). They all seem to just share a rough outline of userspace API.
Anyway, I was going to suggest that you use libguestfs to test KVM on ARM. It's extremely easy to fire up a VM (eg: 'libguestfs-test-tool' or 'guestfish run ...').
Rich.
-
Richard W.M. Jones -
William Cohen