Fatal PCIe error 928 on KVM GPU Passthrough

[Mario Ceresa]

Dear all,
does anybody know if it is currently possible to do GPU passthrough in kvm?
I have access to a dual Quadro 4000 setup and I would like to pass the
second one to kvm for use in a VM.

I read a lot on internet and it seems feasible [1, 2, 3] and there is even
a reported success case [5].

However when I start the VM the host reboots with PCIe error 928:
Completion timeout.

My current setup is:

* Fedora 18 with 3.9.4-200 x86_64
* dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
* nvidia driver installed from rpmfusion
* I use vfio assignment as per [4]

Thanks and regards,

Mario


[1] http://www.linux-kvm.org/page/VGA_device_assignment
[2] https://bbs.archlinux.org/viewtopic.php?id=162768
[3]
http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-...
[4]
https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
[5]
http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-an...

[Cole Robinson]

On 06/20/2013 03:30 AM, Mario Ceresa wrote:
>  Dear all,
>  does anybody know if it is currently possible to do GPU passthrough in kvm? I
>  have access to a dual Quadro 4000 setup and I would like to pass the second
>  one to kvm for use in a VM.
>  
>  I read a lot on internet and it seems feasible [1, 2, 3] and there is even a
>  reported success case [5].
>  
>  However when I start the VM the host reboots with PCIe error 928: Completion
>  timeout.
>  
>  My current setup is:
>  
>  * Fedora 18 with 3.9.4-200 x86_64
>  * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
>  * nvidia driver installed from rpmfusion
>  * I use vfio assignment as per [4]
>  
>  Thanks and regards,
>  
>  Mario
>  
>  
>  [1] http://www.linux-kvm.org/page/VGA_device_assignment
>  [2] https://bbs.archlinux.org/viewtopic.php?id=162768
>  [3]
>
> http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-...
>  [4]
>  https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
>  [5]
>  http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-an...
>   
CC-ing Alex (the VFIO guy)

What qemu and libvirt versions are you using?

- Cole

[Mario Ceresa]

Hi Cole, thanks for your mail.

I use:

* qemu-2:1.2.2-11.fc18
* ibvirt 0.10.2.5-1.fc18

My second GPU is 04:00.0 and the associated sound device is 04:00.1

So I do:

sudo chmod 666 /dev/vfio/vfio
echo "10de 06dd" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:04:00.0 > /sys/bus/pci/devices/0000\:04\:00.0/driver/unbind
echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind
echo 0000:04:00.1 > /sys/bus/pci/devices/0000\:04\:00.1/driver/unbind
echo "1" > /sys/bus/pci/devices/0000:04:00.1/remove

Then I follow instructions in
https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
assign the PCI slot.

Mario


On 20 June 2013 16:12, Cole Robinson <crobinso(a)redhat.com&gt; wrote:

>  On 06/20/2013 03:30 AM, Mario Ceresa wrote:
>  > Dear all,
>  > does anybody know if it is currently possible to do GPU passthrough in
>  kvm? I
>  > have access to a dual Quadro 4000 setup and I would like to pass the
>  second
>  > one to kvm for use in a VM.
>  >
>  > I read a lot on internet and it seems feasible [1, 2, 3] and there is
>  even a
>  > reported success case [5].
>  >
>  > However when I start the VM the host reboots with PCIe error 928:
>  Completion
>  > timeout.
>  >
>  > My current setup is:
>  >
>  > * Fedora 18 with 3.9.4-200 x86_64
>  > * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
>  > * nvidia driver installed from rpmfusion
>  > * I use vfio assignment as per [4]
>  >
>  > Thanks and regards,
>  >
>  > Mario
>  >
>  >
>  > [1] http://www.linux-kvm.org/page/VGA_device_assignment
>  > [2] https://bbs.archlinux.org/viewtopic.php?id=162768
>  > [3]
>  >
>
> http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-...
>  > [4]
>  >
>  https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
>  > [5]
>  >
>  http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-an...
>  >
>
>  CC-ing Alex (the VFIO guy)
>
>  What qemu and libvirt versions are you using?
>
>  - Cole
>

[Cole Robinson]

On 06/20/2013 10:24 AM, Mario Ceresa wrote:
>  Hi Cole, thanks for your mail.
>  
>  I use:
>  
>  * qemu-2:1.2.2-11.fc18
>  * ibvirt 0.10.2.5-1.fc18
>  
>  My second GPU is 04:00.0 and the associated sound device is 04:00.1
>  
>  So I do:
>  
>  sudo chmod 666 /dev/vfio/vfio
>  echo "10de 06dd" > /sys/bus/pci/drivers/pci-stub/new_id
>  echo 0000:04:00.0 > /sys/bus/pci/devices/0000\:04\:00.0/driver/unbind
>  echo 0000:04:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>  echo 0000:04:00.1 > /sys/bus/pci/devices/0000\:04\:00.1/driver/unbind
>  echo "1" > /sys/bus/pci/devices/0000:04:00.1/remove
>  
>  Then I follow instructions in
>  https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
>  to assign the PCI slot.
>   
You might want to try with newer libvirt and qemu, like from the virt-preview
repo:

http://fedoraproject.org/wiki/Virtualization_Preview_Repository

>  
>  On 20 June 2013 16:12, Cole Robinson <crobinso(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>  
>      On 06/20/2013 03:30 AM, Mario Ceresa wrote:
>      > Dear all,
>      > does anybody know if it is currently possible to do GPU passthrough in
>      kvm? I
>      > have access to a dual Quadro 4000 setup and I would like to pass the second
>      > one to kvm for use in a VM.
>      >
>      > I read a lot on internet and it seems feasible [1, 2, 3] and there is even a
>      > reported success case [5].
>      >
>      > However when I start the VM the host reboots with PCIe error 928: Completion
>      > timeout.
>      >
>      > My current setup is:
>      >
>      > * Fedora 18 with 3.9.4-200 x86_64
>      > * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
>      > * nvidia driver installed from rpmfusion
>      > * I use vfio assignment as per [4]
>      >
>      > Thanks and regards,
>      >
>      > Mario
>      >
>      >
>      > [1] http://www.linux-kvm.org/page/VGA_device_assignment
>      > [2] https://bbs.archlinux.org/viewtopic.php?id=162768
>      > [3]
>      >
>     
> http://samos-it.com/removedetach-a-pci-device-linux-kvm-vga-passthrgough-...
>      > [4]
>      >
>      https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
>      > [5]
>      >
>      http://tavi-tech.blogspot.com.es/2012/05/vga-passthrough-kvm-fedora-17-an...
>      >
>  
>      CC-ing Alex (the VFIO guy)
>  
>      What qemu and libvirt versions are you using?
>  
>      - Cole
>  
>

[Alex Williamson]

On Thu, 2013-06-20 at 09:30 +0200, Mario Ceresa wrote:
>  Dear all,
>  does anybody know if it is currently possible to do GPU passthrough in kvm?
>  I have access to a dual Quadro 4000 setup and I would like to pass the
>  second one to kvm for use in a VM.
>  
>  I read a lot on internet and it seems feasible [1, 2, 3] and there is even
>  a reported success case [5].
>  
>  However when I start the VM the host reboots with PCIe error 928:
>  Completion timeout.
>  
>  My current setup is:
>  
>  * Fedora 18 with 3.9.4-200 x86_64
>  * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
>  * nvidia driver installed from rpmfusion
>  * I use vfio assignment as per [4] 
GPUs are not well supported yet and you're probably not going to get
very far unless you're willing to compile a kernel and qemu and use qemu
from the commandline instead of through libvirt.  Some people have luck
assigning a GPU as a secondary graphics device to the emulated video,
others do not.  Sometimes I suspect this has more to do with addresses
assigned to the physical device possibly matching the virtual device...
so it really is luck.

I've posted branches of qemu and kernel that seem to work well for
several people (this is what was pointed to in the Arch Linux thread you
footnoted).  That code attempts to use the assigned GPU as the primary
graphics head for the guest.  I have one Quadro card on hand and it
never works as well as the GeForce cards, YMMV.  Graphics cards have all
sorts of interesting backdoors to get to PCI config space and I suspect
the Quadro cards are not as well supported by the Nouveau project, which
we've used for reverse engineered documentation of some of these quirks.

There's still some work to do before that code can be posted upstream,
in particular I've seen cases where a PCI bus won't come back from reset
without resetting the host system.  We need to make it more stable
before pushing to get it upstream.  Thanks,

Alex

[Mario Ceresa]

Thanks Alex, for your kind and detailed response: please keep up with the
good work!

I'm interested in learning more about this and I have access to Quadro 4000
and 5000 cards, so please tell me if I can help with testing. I do not mind
compiling from sources or doing some debug, but I have almost zero
practical knowledge on GPU cards so I'll need some pointers :)

Best,

Mario


On 20 June 2013 23:19, Alex Williamson <alex.williamson(a)redhat.com&gt; wrote:

>  On Thu, 2013-06-20 at 09:30 +0200, Mario Ceresa wrote:
>  > Dear all,
>  > does anybody know if it is currently possible to do GPU passthrough in
>  kvm?
>  > I have access to a dual Quadro 4000 setup and I would like to pass the
>  > second one to kvm for use in a VM.
>  >
>  > I read a lot on internet and it seems feasible [1, 2, 3] and there is
>  even
>  > a reported success case [5].
>  >
>  > However when I start the VM the host reboots with PCIe error 928:
>  > Completion timeout.
>  >
>  > My current setup is:
>  >
>  > * Fedora 18 with 3.9.4-200 x86_64
>  > * dmesg | grep -e DMAR -e IOMMU shows that they are both enabled
>  > * nvidia driver installed from rpmfusion
>  > * I use vfio assignment as per [4]
>
>  GPUs are not well supported yet and you're probably not going to get
>  very far unless you're willing to compile a kernel and qemu and use qemu
>  from the commandline instead of through libvirt.  Some people have luck
>  assigning a GPU as a secondary graphics device to the emulated video,
>  others do not.  Sometimes I suspect this has more to do with addresses
>  assigned to the physical device possibly matching the virtual device...
>  so it really is luck.
>
>  I've posted branches of qemu and kernel that seem to work well for
>  several people (this is what was pointed to in the Arch Linux thread you
>  footnoted).  That code attempts to use the assigned GPU as the primary
>  graphics head for the guest.  I have one Quadro card on hand and it
>  never works as well as the GeForce cards, YMMV.  Graphics cards have all
>  sorts of interesting backdoors to get to PCI config space and I suspect
>  the Quadro cards are not as well supported by the Nouveau project, which
>  we've used for reverse engineered documentation of some of these quirks.
>
>  There's still some work to do before that code can be posted upstream,
>  in particular I've seen cases where a PCI bus won't come back from reset
>  without resetting the host system.  We need to make it more stable
>  before pushing to get it upstream.  Thanks,
>
>  Alex
>
>