Issues with VGA passthrough using KVM and VFIO

[Stewart Adam]

Hi,

I am trying to get VGA passthrough working using KVM and VFIO like described 
on this [1] thread and ran into some issues I was hoping someone could shed 
some light on. I have tried various configurations, but can't seem to get it 
working correctly. Either I get "Code 10" device error in Windows, or I get 
a BSOD 0x00000116 (VIDEO_TDR_ ERROR: attempt to reset the display driver and 
recover from a timeout failed) when booting the machine after installing the 
Catalyst 3D drivers. Other users on the Arch Linux forum have also reported 
similar issues.

I have included my hardware, software, and QEMU command-line flags at the 
end of this message for reference.

I tried using virt-manager to create the VM and modified the libvirt 
configuration per the Fedora test case [2] as well as various Q35 machine 
configurations by launching QEMU directly and all resulted in a "Code 10" on 
the GPU device except for when disabling emulated graphics entirely and 
passing x-vga=on. This results in a monitor signal on the card and it seems 
to work very well enough - with kernel 3.12.5 I could even restart the VM 
several times and it all works well *until* I install the Catalyst drivers. 
Then I get the BSOD 0x116 right after the Windows splash-screen. Safe mode 
and system repair still work correctly.

There are no errors on the console or in dmesg, and I can provide that 
output in a pastebin if required. Is this indicative of a hardware problem, 
or a bug in the Catalyst drivers? Is there anything I can do to help debug 
this? I'm happy to provide any information necessary.

Here's some background information about my software configuration:

  * Host: Fedora 20 with virt-preview repo
      o kernel-3.12.5-301.fc20.x86_64 (from updates-testing)
      o kernel-3.12.5-302.fc20.x86_64 (built with fedpkg from git, f20
        branch w/ patches from comment #23 in [3] to fix
        pci_find_upstream_pcie_bridge bug)
      o qemu-kvm-1.7.0-1.fc20.x86_64
          + also tried latest qemu-vfio from git
      o libvirt-daemon-1.2.0-1.fc20.x86_64
  * Guest: Windows 7 SP1 x64

And now the hardware:

  * Xeon E3-1225 v3 (has VT-d, VT-x)
  * SuperMicro MBD-X10SAE-O
  * Sapphire Radeon R9 270X

This is the QEMU command-line to boot the machine:
     /usr/bin/qemu-system-x86_64 -machine accel=kvm -name win7 -S -M q35 -m 
2048 -cpu host \
     -rtc base=localtime -realtime mlock=off \
     -smp 4,sockets=1,cores=4,threads=1 \
     -bios /usr/share/qemu/bios.bin -vga none \
     -device 
ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
     -device 
vfio-pci,host=01:00.0,bus=root.1,addr=0x00,multifunction=on,x-vga=on,romfile=/home/sysadmin/vm/R9270X.rom

\
     -device vfio-pci,host=00:19.0,bus=pcie.0 \
     -device ahci,bus=pcie.0,id=ahci \
     -drive file=/home/VMs/win7-q35-ahci.img,id=disk0,format=qcow2 -device 
ide-hd,bus=ahci.0,drive=disk0 \
     -boot menu=on

I have omitted passing through the HDMI audio device, although including it 
and/or moving it to pcie.0 as suggested by some users in the Arch Linux 
thread didn't make any difference. Tried with and without romfile, also no 
difference. I did clean Windows installs with each attempt. PCIe 00:19:00 is 
a network controller.

Thanks,
Stewart

[1] https://bbs.archlinux.org/viewtopic.php?pid=1361071
[2] 
https://fedoraproject.org/wiki/QA:Testcase_Virtualization_KVM_PCI_Device_...
[3] https://bugzilla.kernel.org/show_bug.cgi?id=44881

[Alex Williamson]

On Thu, 2013-12-19 at 01:48 -0500, Stewart Adam wrote:
>  Hi,
>  
>  I am trying to get VGA passthrough working using KVM and VFIO like
>  described on this [1] thread and ran into some issues I was hoping
>  someone could shed some light on. I have tried various configurations,
>  but can't seem to get it working correctly. Either I get "Code 10"
>  device error in Windows, or I get a BSOD 0x00000116 (VIDEO_TDR_ ERROR:
>  attempt to reset the display driver and recover from a timeout failed)
>  when booting the machine after installing the Catalyst 3D drivers.
>  Other users on the Arch Linux forum have also reported similar issues.
>  
>  I have included my hardware, software, and QEMU command-line flags at
>  the end of this message for reference.
>  
>  I tried using virt-manager to create the VM and modified the libvirt
>  configuration per the Fedora test case [2] as well as various Q35
>  machine configurations by launching QEMU directly and all resulted in
>  a "Code 10" on the GPU device except for when disabling emulated
>  graphics entirely and passing x-vga=on. This results in a monitor
>  signal on the card and it seems to work very well enough - with kernel
>  3.12.5 I could even restart the VM several times and it all works well
>  *until* I install the Catalyst drivers. Then I get the BSOD 0x116
>  right after the Windows splash-screen. Safe mode and system repair
>  still work correctly.
>  
>  There are no errors on the console or in dmesg, and I can provide that
>  output in a pastebin if required. Is this indicative of a hardware
>  problem, or a bug in the Catalyst drivers? Is there anything I can do
>  to help debug this? I'm happy to provide any information necessary.
>  
>  Here's some background information about my software configuration:
>        * Host: Fedora 20 with virt-preview repo
>                * kernel-3.12.5-301.fc20.x86_64 (from updates-testing)
>                * kernel-3.12.5-302.fc20.x86_64 (built with fedpkg from
>                  git, f20 branch w/ patches from comment #23 in [3] to
>                  fix pci_find_upstream_pcie_bridge bug)
>                * qemu-kvm-1.7.0-1.fc20.x86_64
>                        * also tried latest qemu-vfio from git
>                * libvirt-daemon-1.2.0-1.fc20.x86_64
>        * Guest: Windows 7 SP1 x64
>  
>  And now the hardware:
>  
>  
>        * Xeon E3-1225 v3 (has VT-d, VT-x)
>        * SuperMicro MBD-X10SAE-O
>        * Sapphire Radeon R9 270X
>  This is the QEMU command-line to boot the machine:
>      /usr/bin/qemu-system-x86_64 -machine accel=kvm -name win7 -S -M
>  q35 -m 2048 -cpu host \
>      -rtc base=localtime -realtime mlock=off \
>      -smp 4,sockets=1,cores=4,threads=1 \
>      -bios /usr/share/qemu/bios.bin -vga none \
>      -device
>  ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
>      -device
>
> vfio-pci,host=01:00.0,bus=root.1,addr=0x00,multifunction=on,x-vga=on,romfile=/home/sysadmin/vm/R9270X.rom
> \
>      -device vfio-pci,host=00:19.0,bus=pcie.0 \
>      -device ahci,bus=pcie.0,id=ahci \
>      -drive file=/home/VMs/win7-q35-ahci.img,id=disk0,format=qcow2
>  -device ide-hd,bus=ahci.0,drive=disk0 \
>      -boot menu=on
>  
>  I have omitted passing through the HDMI audio device, although
>  including it and/or moving it to pcie.0 as suggested by some users in
>  the Arch Linux thread didn't make any difference. Tried with and
>  without romfile, also no difference. I did clean Windows installs with
>  each attempt. PCIe 00:19:00 is a network controller. 
Hi Stewart,

The "x" in the x-vga option is for eXperimental.  It works in some
cases, not others.  The archlinux forum thread is the best place to
either get help or commiserate with others having the same problem.  The
advice there is largely not distribution specific.  The 0x116 BSOD is a
common problem with assigned Radeon graphics, we don't have a solution
yet.  Also note that this work is still under development, for the
current "state of the art" you need a 3.13-rc kernel and qemu.git, and
you may need to patch in some Intel graphics fixes for VGA arbitration.
Even then, I have no reason to suspect the 0x116 BSOD is fixed.  More
users seem to be having success with Nvidia cards, so if that's an
option for you, it may be the quickest path to results.  Thanks,

Alex

[Stewart Adam]

On 2013-12-19 10:19 AM, Alex Williamson wrote:
>  On Thu, 2013-12-19 at 01:48 -0500, Stewart Adam wrote:
> > Hi,
> >
> > I am trying to get VGA passthrough working using KVM and VFIO like
> > described on this [1] thread and ran into some issues I was hoping
> > someone could shed some light on. I have tried various configurations,
> > but can't seem to get it working correctly. Either I get "Code 10"
> > device error in Windows, or I get a BSOD 0x00000116 (VIDEO_TDR_ ERROR:
> > attempt to reset the display driver and recover from a timeout failed)
> > when booting the machine after installing the Catalyst 3D drivers.
> > Other users on the Arch Linux forum have also reported similar issues.
> >
> > <cut>
>
>  Hi Stewart,
>
>  The "x" in the x-vga option is for eXperimental.  It works in some
>  cases, not others.  The archlinux forum thread is the best place to
>  either get help or commiserate with others having the same problem.  The
>  advice there is largely not distribution specific.  The 0x116 BSOD is a
>  common problem with assigned Radeon graphics, we don't have a solution
>  yet.  Also note that this work is still under development, for the
>  current "state of the art" you need a 3.13-rc kernel and qemu.git, and
>  you may need to patch in some Intel graphics fixes for VGA arbitration.
>  Even then, I have no reason to suspect the 0x116 BSOD is fixed.  More
>  users seem to be having success with Nvidia cards, so if that's an
>  option for you, it may be the quickest path to results.  Thanks,
>
>  Alex 
Hi Alex,

Thanks for replying so quickly. Ironically, I normally buy nVidia and 
purchased this Radeon card because I had read online they tend to do 
passthrough better (with Xen at least)... I didn't have much luck when I 
tried a Xen VM either though. I do have an nVidia card handy and will try 
that later tonight.

Would the VGA arbitration patches relate to the Code 10 errors? Also, I will 
try again with 3.13-rc later this week but should I use qemu-vfio.git or 
qemu.git?

I realize this is all bleeding edge and thank you for all of your work on 
it. Is there anything I can do to help you debug the 0x116 problem?

Lastly, out of curiosity - is the 0x116 error dependent on particular piece 
of hardware or it's a problem with the emulation layer + certain GPUs?

Regards,
Stewart

[Alex Williamson]

On Thu, 2013-12-19 at 12:41 -0500, Stewart Adam wrote:
>  On 2013-12-19 10:19 AM, Alex Williamson wrote:
>  > On Thu, 2013-12-19 at 01:48 -0500, Stewart Adam wrote:
>  >> Hi,
>  >>
>  >> I am trying to get VGA passthrough working using KVM and VFIO like
>  >> described on this [1] thread and ran into some issues I was hoping
>  >> someone could shed some light on. I have tried various configurations,
>  >> but can't seem to get it working correctly. Either I get "Code
> 10"
>  >> device error in Windows, or I get a BSOD 0x00000116 (VIDEO_TDR_ ERROR:
>  >> attempt to reset the display driver and recover from a timeout failed)
>  >> when booting the machine after installing the Catalyst 3D drivers.
>  >> Other users on the Arch Linux forum have also reported similar issues.
>  >>
>  >> <cut>
>  >
>  > Hi Stewart,
>  >
>  > The "x" in the x-vga option is for eXperimental.  It works in some
>  > cases, not others.  The archlinux forum thread is the best place to
>  > either get help or commiserate with others having the same problem.  The
>  > advice there is largely not distribution specific.  The 0x116 BSOD is a
>  > common problem with assigned Radeon graphics, we don't have a solution
>  > yet.  Also note that this work is still under development, for the
>  > current "state of the art" you need a 3.13-rc kernel and qemu.git, and
>  > you may need to patch in some Intel graphics fixes for VGA arbitration.
>  > Even then, I have no reason to suspect the 0x116 BSOD is fixed.  More
>  > users seem to be having success with Nvidia cards, so if that's an
>  > option for you, it may be the quickest path to results.  Thanks,
>  >
>  > Alex
>  
>  Hi Alex,
>  
>  Thanks for replying so quickly. Ironically, I normally buy nVidia and 
>  purchased this Radeon card because I had read online they tend to do 
>  passthrough better (with Xen at least)... I didn't have much luck when I 
>  tried a Xen VM either though. I do have an nVidia card handy and will try 
>  that later tonight.
>  
>  Would the VGA arbitration patches relate to the Code 10 errors? Also, I will 
>  try again with 3.13-rc later this week but should I use qemu-vfio.git or 
>  qemu.git? 
Probably not related to the Code 10, but I'm no expert in Windows error
codes.

>  I realize this is all bleeding edge and thank you for all of your
> work on 
>  it. Is there anything I can do to help you debug the 0x116 problem? 
vfio has plenty of debug support for tracing accesses to the card, it
simply needs to be turned on in the source file.  The issue is
identifying the problem among all the output.  With a closed source
guest and driver, and lack of hardware documentation, it's quite a lot
of guesswork.

>  Lastly, out of curiosity - is the 0x116 error dependent on particular
> piece 
>  of hardware or it's a problem with the emulation layer + certain GPUs? 
No idea.  Thanks,

Alex

[Stewart Adam]

On 2013-12-20 12:06 AM, Alex Williamson wrote:
>  On Thu, 2013-12-19 at 12:41 -0500, Stewart Adam wrote:
> > On 2013-12-19 10:19 AM, Alex Williamson wrote:
> >> On Thu, 2013-12-19 at 01:48 -0500, Stewart Adam wrote:
> >>> Hi,
> >>>
> >>> I am trying to get VGA passthrough working using KVM and VFIO like
> >>> described on this [1] thread and ran into some issues I was hoping
> >>> someone could shed some light on. I have tried various configurations,
> >>> but can't seem to get it working correctly. Either I get "Code
> 10"
> >>> device error in Windows, or I get a BSOD 0x00000116 (VIDEO_TDR_ ERROR:
> >>> attempt to reset the display driver and recover from a timeout failed)
> >>> when booting the machine after installing the Catalyst 3D drivers.
> >>> Other users on the Arch Linux forum have also reported similar issues.
> >>>
> >>> <cut>
> >>
> >> Hi Stewart,
> >>
> >> The "x" in the x-vga option is for eXperimental.  It works in some
> >> cases, not others.  The archlinux forum thread is the best place to
> >> either get help or commiserate with others having the same problem.  The
> >> advice there is largely not distribution specific.  The 0x116 BSOD is a
> >> common problem with assigned Radeon graphics, we don't have a solution
> >> yet.  Also note that this work is still under development, for the
> >> current "state of the art" you need a 3.13-rc kernel and qemu.git, and
> >> you may need to patch in some Intel graphics fixes for VGA arbitration.
> >> Even then, I have no reason to suspect the 0x116 BSOD is fixed.  More
> >> users seem to be having success with Nvidia cards, so if that's an
> >> option for you, it may be the quickest path to results.  Thanks,
> >>
> >> Alex
> >
> > Hi Alex,
> >
> > Thanks for replying so quickly. Ironically, I normally buy nVidia and
> > purchased this Radeon card because I had read online they tend to do
> > passthrough better (with Xen at least)... I didn't have much luck when I
> > tried a Xen VM either though. I do have an nVidia card handy and will try
> > that later tonight.
> >
> > Would the VGA arbitration patches relate to the Code 10 errors? Also, I will
> > try again with 3.13-rc later this week but should I use qemu-vfio.git or
> > qemu.git?
>
>  Probably not related to the Code 10, but I'm no expert in Windows error
>  codes.
>
> > I realize this is all bleeding edge and thank you for all of your work on
> > it. Is there anything I can do to help you debug the 0x116 problem?
>
>  vfio has plenty of debug support for tracing accesses to the card, it
>  simply needs to be turned on in the source file.  The issue is
>  identifying the problem among all the output.  With a closed source
>  guest and driver, and lack of hardware documentation, it's quite a lot
>  of guesswork.
>
> > Lastly, out of curiosity - is the 0x116 error dependent on particular piece
> > of hardware or it's a problem with the emulation layer + certain GPUs?
>
>  No idea.  Thanks,
>
>  Alex 
Hi Alex,

Yesterday I did a bit more reading and based on your suggestions above, 
recompiled rawhide kernel 3.13.0-0.rc4.git4.1.fc21 from Fedora's build 
system with a few additional patches applied to fix the problematic 
PCIE-to-PCI bridge mentioned in my original post as well as your patches for 
the i915 VGA arbitration with Haswell.

I then compiled QEMU from latest git with the NoSnoop patch applied, as I 
checked 'lspci -vv' and my card was showing NoSnoop+.

The combination worked perfectly, and I had my monitor displaying at 1080p 
with Catalyst Control Center fully functional!

When I have more time I'm going to apply the patches one at a time and also 
try using QEMU 1.7.0 to see which changes, exactly, made the difference. 
I'll post back once I have more info.

Thanks,
Stewart

[Alex Williamson]

On Fri, 2013-12-20 at 12:14 -0500, Stewart Adam wrote:
>  On 2013-12-20 12:06 AM, Alex Williamson wrote:
>  > On Thu, 2013-12-19 at 12:41 -0500, Stewart Adam wrote:
>  >> On 2013-12-19 10:19 AM, Alex Williamson wrote:
>  >>> On Thu, 2013-12-19 at 01:48 -0500, Stewart Adam wrote:
>  >>>> Hi,
>  >>>>
>  >>>> I am trying to get VGA passthrough working using KVM and VFIO like
>  >>>> described on this [1] thread and ran into some issues I was hoping
>  >>>> someone could shed some light on. I have tried various configurations,
>  >>>> but can't seem to get it working correctly. Either I get "Code
> 10"
>  >>>> device error in Windows, or I get a BSOD 0x00000116 (VIDEO_TDR_ ERROR:
>  >>>> attempt to reset the display driver and recover from a timeout failed)
>  >>>> when booting the machine after installing the Catalyst 3D drivers.
>  >>>> Other users on the Arch Linux forum have also reported similar issues.
>  >>>>
>  >>>> <cut>
>  >>>
>  >>> Hi Stewart,
>  >>>
>  >>> The "x" in the x-vga option is for eXperimental.  It works in
> some
>  >>> cases, not others.  The archlinux forum thread is the best place to
>  >>> either get help or commiserate with others having the same problem.  The
>  >>> advice there is largely not distribution specific.  The 0x116 BSOD is a
>  >>> common problem with assigned Radeon graphics, we don't have a solution
>  >>> yet.  Also note that this work is still under development, for the
>  >>> current "state of the art" you need a 3.13-rc kernel and qemu.git,
> and
>  >>> you may need to patch in some Intel graphics fixes for VGA arbitration.
>  >>> Even then, I have no reason to suspect the 0x116 BSOD is fixed.  More
>  >>> users seem to be having success with Nvidia cards, so if that's an
>  >>> option for you, it may be the quickest path to results.  Thanks,
>  >>>
>  >>> Alex
>  >>
>  >> Hi Alex,
>  >>
>  >> Thanks for replying so quickly. Ironically, I normally buy nVidia and
>  >> purchased this Radeon card because I had read online they tend to do
>  >> passthrough better (with Xen at least)... I didn't have much luck when I
>  >> tried a Xen VM either though. I do have an nVidia card handy and will try
>  >> that later tonight.
>  >>
>  >> Would the VGA arbitration patches relate to the Code 10 errors? Also, I will
>  >> try again with 3.13-rc later this week but should I use qemu-vfio.git or
>  >> qemu.git?
>  >
>  > Probably not related to the Code 10, but I'm no expert in Windows error
>  > codes.
>  >
>  >> I realize this is all bleeding edge and thank you for all of your work on
>  >> it. Is there anything I can do to help you debug the 0x116 problem?
>  >
>  > vfio has plenty of debug support for tracing accesses to the card, it
>  > simply needs to be turned on in the source file.  The issue is
>  > identifying the problem among all the output.  With a closed source
>  > guest and driver, and lack of hardware documentation, it's quite a lot
>  > of guesswork.
>  >
>  >> Lastly, out of curiosity - is the 0x116 error dependent on particular piece
>  >> of hardware or it's a problem with the emulation layer + certain GPUs?
>  >
>  > No idea.  Thanks,
>  >
>  > Alex
>  
>  Hi Alex,
>  
>  Yesterday I did a bit more reading and based on your suggestions above, 
>  recompiled rawhide kernel 3.13.0-0.rc4.git4.1.fc21 from Fedora's build 
>  system with a few additional patches applied to fix the problematic 
>  PCIE-to-PCI bridge mentioned in my original post as well as your patches for 
>  the i915 VGA arbitration with Haswell.
>  
>  I then compiled QEMU from latest git with the NoSnoop patch applied, as I 
>  checked 'lspci -vv' and my card was showing NoSnoop+.
>  
>  The combination worked perfectly, and I had my monitor displaying at 1080p 
>  with Catalyst Control Center fully functional!
>  
>  When I have more time I'm going to apply the patches one at a time and also 
>  try using QEMU 1.7.0 to see which changes, exactly, made the difference. 
>  I'll post back once I have more info. 
Great!  The NoSnoop patch should not be necessary with 3.13-rc and
qemu.git, we've taken a different approach with that upstream, it should
be handled by:

kernel:
ec53500fae421e07c5d035918ca454a429732ef4
d96eb2c6f480769bff32054e78b964860dae4d56
e0f0bbc527f6e9c0261f1d16b2a0b47612b7f235
qemu:
bf63839ffa2d0eebb1eb1706022f46e93b6fec08
5b49ab188ff0339aa3097ce7f5309f1306092f9e

linux 3.12 + above + i915 patches and qemu 1.7 + above would hopefully
work as well.  Thanks,

Alex