Dear all, does anybody know if it is currently possible to do GPU passthrough in kvm? I have access to a dual Quadro 4000 setup and I would like to pass the second one to kvm for use in a VM.
I read a lot on internet and it seems feasible [1, 2, 3] and there is even a reported success case [5].
However when I start the VM the host reboots with PCIe error 928: Completion timeout.
My current setup is:
* Fedora 18 with 3.9.4-200 x86_64 * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled * nvidia driver installed from rpmfusion * I use vfio assignment as per [4]
Thanks and regards,
Mario
[1] http://www.linux-kvm.org/page/VGA_device_assignment [2] https://bbs.archlinux.org/viewtopic.php?id=162768 [3] http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-nvi... [4] https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Ass... [5] http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-and.h...
On 06/20/2013 03:30 AM, Mario Ceresa wrote:
Dear all, does anybody know if it is currently possible to do GPU passthrough in kvm? I have access to a dual Quadro 4000 setup and I would like to pass the second one to kvm for use in a VM.
I read a lot on internet and it seems feasible [1, 2, 3] and there is even a reported success case [5].
However when I start the VM the host reboots with PCIe error 928: Completion timeout.
My current setup is:
- Fedora 18 with 3.9.4-200 x86_64
- dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
- nvidia driver installed from rpmfusion
- I use vfio assignment as per [4]
Thanks and regards,
Mario
[1] http://www.linux-kvm.org/page/VGA_device_assignment [2] https://bbs.archlinux.org/viewtopic.php?id=162768 [3] http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-nvi... [4] https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Ass... [5] http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-and.h...
CC-ing Alex (the VFIO guy)
What qemu and libvirt versions are you using?
- Cole
Hi Cole, thanks for your mail.
I use:
* qemu-2:1.2.2-11.fc18 * ibvirt 0.10.2.5-1.fc18
My second GPU is 04:00.0 and the associated sound device is 04:00.1
So I do:
sudo chmod 666 /dev/vfio/vfio echo "10de 06dd" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:04:00.0 > /sys/bus/pci/devices/0000:04:00.0/driver/unbind echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:04:00.1 > /sys/bus/pci/devices/0000:04:00.1/driver/unbind echo "1" > /sys/bus/pci/devices/0000:04:00.1/remove
Then I follow instructions in https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Ass... assign the PCI slot.
Mario
On 20 June 2013 16:12, Cole Robinson crobinso@redhat.com wrote:
On 06/20/2013 03:30 AM, Mario Ceresa wrote:
Dear all, does anybody know if it is currently possible to do GPU passthrough in
kvm? I
have access to a dual Quadro 4000 setup and I would like to pass the
second
one to kvm for use in a VM.
I read a lot on internet and it seems feasible [1, 2, 3] and there is
even a
reported success case [5].
However when I start the VM the host reboots with PCIe error 928:
Completion
timeout.
My current setup is:
- Fedora 18 with 3.9.4-200 x86_64
- dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
- nvidia driver installed from rpmfusion
- I use vfio assignment as per [4]
Thanks and regards,
Mario
[1] http://www.linux-kvm.org/page/VGA_device_assignment [2] https://bbs.archlinux.org/viewtopic.php?id=162768 [3]
http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-nvi...
[4]
https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Ass...
[5]
http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-and.h...
CC-ing Alex (the VFIO guy)
What qemu and libvirt versions are you using?
- Cole
On 06/20/2013 10:24 AM, Mario Ceresa wrote:
Hi Cole, thanks for your mail.
I use:
- qemu-2:1.2.2-11.fc18
- ibvirt 0.10.2.5-1.fc18
My second GPU is 04:00.0 and the associated sound device is 04:00.1
So I do:
sudo chmod 666 /dev/vfio/vfio echo "10de 06dd" > /sys/bus/pci/drivers/pci-stub/new_id echo 0000:04:00.0 > /sys/bus/pci/devices/0000:04:00.0/driver/unbind echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind echo 0000:04:00.1 > /sys/bus/pci/devices/0000:04:00.1/driver/unbind echo "1" > /sys/bus/pci/devices/0000:04:00.1/remove
Then I follow instructions in https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Ass... to assign the PCI slot.
You might want to try with newer libvirt and qemu, like from the virt-preview repo:
http://fedoraproject.org/wiki/Virtualization_Preview_Repository
On 20 June 2013 16:12, Cole Robinson <crobinso@redhat.com mailto:crobinso@redhat.com> wrote:
On 06/20/2013 03:30 AM, Mario Ceresa wrote: > Dear all, > does anybody know if it is currently possible to do GPU passthrough in kvm? I > have access to a dual Quadro 4000 setup and I would like to pass the second > one to kvm for use in a VM. > > I read a lot on internet and it seems feasible [1, 2, 3] and there is even a > reported success case [5]. > > However when I start the VM the host reboots with PCIe error 928: Completion > timeout. > > My current setup is: > > * Fedora 18 with 3.9.4-200 x86_64 > * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled > * nvidia driver installed from rpmfusion > * I use vfio assignment as per [4] > > Thanks and regards, > > Mario > > > [1] http://www.linux-kvm.org/page/VGA_device_assignment > [2] https://bbs.archlinux.org/viewtopic.php?id=162768 > [3] > http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-nvidia-quadro-2000/ > [4] > https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_Assignment > [5] > http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-and.html > CC-ing Alex (the VFIO guy) What qemu and libvirt versions are you using? - Cole
On Thu, 2013-06-20 at 09:30 +0200, Mario Ceresa wrote:
Dear all, does anybody know if it is currently possible to do GPU passthrough in kvm? I have access to a dual Quadro 4000 setup and I would like to pass the second one to kvm for use in a VM.
I read a lot on internet and it seems feasible [1, 2, 3] and there is even a reported success case [5].
However when I start the VM the host reboots with PCIe error 928: Completion timeout.
My current setup is:
- Fedora 18 with 3.9.4-200 x86_64
- dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
- nvidia driver installed from rpmfusion
- I use vfio assignment as per [4]
GPUs are not well supported yet and you're probably not going to get very far unless you're willing to compile a kernel and qemu and use qemu from the commandline instead of through libvirt. Some people have luck assigning a GPU as a secondary graphics device to the emulated video, others do not. Sometimes I suspect this has more to do with addresses assigned to the physical device possibly matching the virtual device... so it really is luck.
I've posted branches of qemu and kernel that seem to work well for several people (this is what was pointed to in the Arch Linux thread you footnoted). That code attempts to use the assigned GPU as the primary graphics head for the guest. I have one Quadro card on hand and it never works as well as the GeForce cards, YMMV. Graphics cards have all sorts of interesting backdoors to get to PCI config space and I suspect the Quadro cards are not as well supported by the Nouveau project, which we've used for reverse engineered documentation of some of these quirks.
There's still some work to do before that code can be posted upstream, in particular I've seen cases where a PCI bus won't come back from reset without resetting the host system. We need to make it more stable before pushing to get it upstream. Thanks,
Alex
Thanks Alex, for your kind and detailed response: please keep up with the good work!
I'm interested in learning more about this and I have access to Quadro 4000 and 5000 cards, so please tell me if I can help with testing. I do not mind compiling from sources or doing some debug, but I have almost zero practical knowledge on GPU cards so I'll need some pointers :)
Best,
Mario
On 20 June 2013 23:19, Alex Williamson alex.williamson@redhat.com wrote:
On Thu, 2013-06-20 at 09:30 +0200, Mario Ceresa wrote:
Dear all, does anybody know if it is currently possible to do GPU passthrough in
kvm?
I have access to a dual Quadro 4000 setup and I would like to pass the second one to kvm for use in a VM.
I read a lot on internet and it seems feasible [1, 2, 3] and there is
even
a reported success case [5].
However when I start the VM the host reboots with PCIe error 928: Completion timeout.
My current setup is:
- Fedora 18 with 3.9.4-200 x86_64
- dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
- nvidia driver installed from rpmfusion
- I use vfio assignment as per [4]
GPUs are not well supported yet and you're probably not going to get very far unless you're willing to compile a kernel and qemu and use qemu from the commandline instead of through libvirt. Some people have luck assigning a GPU as a secondary graphics device to the emulated video, others do not. Sometimes I suspect this has more to do with addresses assigned to the physical device possibly matching the virtual device... so it really is luck.
I've posted branches of qemu and kernel that seem to work well for several people (this is what was pointed to in the Arch Linux thread you footnoted). That code attempts to use the assigned GPU as the primary graphics head for the guest. I have one Quadro card on hand and it never works as well as the GeForce cards, YMMV. Graphics cards have all sorts of interesting backdoors to get to PCI config space and I suspect the Quadro cards are not as well supported by the Nouveau project, which we've used for reverse engineered documentation of some of these quirks.
There's still some work to do before that code can be posted upstream, in particular I've seen cases where a PCI bus won't come back from reset without resetting the host system. We need to make it more stable before pushing to get it upstream. Thanks,
Alex
-
Alex Williamson -
Cole Robinson -
Mario Ceresa