[fedora-virt] Problem assigning an NVIDIA Quadro K2000 to a guest OS

[storri]

Can I use an installed version of Windows 7 on a real partition with virt? I wanted to get something fully working so I made a Windows 7 installation on a uefi drive last night. That way i can directly boot into windows or via virt-manager.

Stephen


Sent from my Verizon Wireless 4G LTE smartphone

<div>-------- Original message --------</div><div>From: Alex Williamson <alex.williamson at redhat.com> </div><div>Date:03/08/2015  10:13 PM  (GMT-05:00) </div><div>To: Stephen Torri <storri at torri.org> </div><div>Cc: virt <virt at lists.fedoraproject.org> </div><div>Subject: Re: [fedora-virt] Problem assigning an NVIDIA Quadro K2000 to a
  guest OS </div><div>
</div>On Sun, 2015-03-08 at 17:30 -0400, Stephen Torri wrote:
> I followed a presentation
> (www.linux-kvm.org/wiki/images/b/b4/2012-forum-VFIO.pdf) to try to
> enable VGA passthrough for the guest OS. This is my first attempt at
> using QEMU+KVM to install a guest OS. My intention is to have the guest
> OS have direct access to the GPU for 3d gaming.
> 
> Problem: Not sure if it is setup right. Launching the VM appears to be
> ok. I can open windows and such but if I try to change the resolution
> from 800x600 to something higher I get a lot garbage on the screen.
> Moving the mouse pointer causes the background image to draw over the
> task bar.
> 
> My current setup:
> 
> GPU1: NVIDIA GeForce 780 Ti (Two monitors currently connected)
> GPU2: NVIDIA Quadro K2000 (No monitors connected)
> - NVIDIA 346.47 properitary drivers installed.
> - Xinerama is disabled due to possible GDM bug.
> 
> Goal:
> 
> (Not using VM): Linux desktop stretched across both monitors
> (Using VM): Guest OS has full screen rendering on one montior while
> linux is one the other.

This is not the way GPU assignment works.  GPU assignment of Quadro
cards is very much like assignment of a NIC.  When the NIC is assigned
to the VM, the network cable attached to the NIC is used by the VM.
When a GPU is assigned, the accelerated graphics channel is through the
video outputs of the GPU card.  Somehow the idea of rendering into a
Window on the host is a common misconception.  There is some opportunity
to due this, but not with Windows 7 as a guest and at a huge cost to
performance.  You can of course use the input selection button on you
monitor to select an input channel connected to the assigned GPU.

> Steps done so far:
> 1. Used virt-manager to create a storage device on a SSD (50GB size)
> 2. Installed Windows 7 in VM
> 3. Updated Windows 7
> 4. Enable virtualization support and added PCI 0000:03:00.0 as a
> physical PCI device to the VM.
> 5. Thinking I was not done (due to ignorance) I followed the
> presentation I linked above to find device to assign:
> 
> $ sudo lspci -nn | grep NVIDIA
> 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK110B
> [GeForce GTX 780 Ti] [10de:100a] (rev a1)
> 02:00.1 Audio device [0403]: NVIDIA Corporation GK110 HDMI Audio
> [10de:0e1a] (rev a1)
> 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL
> [Quadro K2000] [10de:0ffe] (rev a1)
> 03:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio
> Controller [10de:0e1b] (rev a1)
> 
> 6. Worked on finding the group:
> 
> $ sudo readlink /sys/bus/pci/devices/0000\:03\:00.0/iommu_group
> ../../../../kernel/iommu_groups/17
> 
> 7. Now that I have the group I worked to find the devices in the group:
> 
> $ sudo ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices
> 0000:03:00.0  0000:03:00.1
> 
> *** Ok. So two devices need to dealt with here.
> 
> 8. I unbinded each from the device driver
> 
> $ echo 0000:03:00.0 | sudo tee \
> /sys/bus/pci/devices/0000:03:00.0/driver/unbind
> 
> $ echo 0000:03:00.1 | sudo tee \
> /sys/bus/pci/devices/0000:03:00.1/driver/unbind
> 
> 9. Found the vendor and device ID for each:
> 
> $ sudo lspci -n -s 03:00.0
> 03:00.0 0300: 10de:0ffe (rev a1)
> $ sudo lspci -n -s 03:00.1
> 03:00.1 0403: 10de:0e1b (rev a1)
> 
> 10. Now I bind them to vfio-pci
> $ echo 10de 0ffe | sudo tee \
> /sys/bus/pci/drivers/vfio-pci/new_id
> $ echo 10de 01eb | sudo tee \
> /sys/bus/pci/drivers/vfio-pci/new_id
> 
> ** Not sure about this. I am not sure I have bounded them correctly.
> 
> 11. Now checking them I see:
> $ ls /dev/vfio
> 17 vfio

It's good that you verified the IOMMU group only contains the GPU itself
and audio function, but beyond that, these steps aren't necessary.  So
long as the hostdev device in your domain XML is set to managed, which
is the default when using virt-manager, the binding of the assigned
device to vfio is handled by libvirt.  However, it's generally
recommended to prevent host drivers from binding to the GPU (they're not
as well practiced at releasing devices as NIC drivers), so we'll want to
configure pci-stub to claim it rather than nouveau.  We also need to
deal with the audio function since it's part of the IOMMU group, so
we'll handle it the same way.

To do this, take the PCI vendor and device IDs that you show above and
add the following to your kernel commandline:

pci-stub.ids=10de:0ffe,10de:0e1b

You can do this by editing /etc/sysconfig/grub and re-running
grub2-mkconfig and rebooting.  After reboot, if you run "lspci -ks 3:",
pci-stub should show as the driver in use for both GPU and audio
functions.

Now, attach a monitor to the Quadro card and you should find that when
the Nvidia driver is initialized in the guest, the console window in
virt-manager freezes (Windows7 disables that graphics controller) and
the monitor initializes.  If you were to use Windows 8, the guest would
be able to use both the virt-manager window and the physical monitor
simultaneously as if it were a dual-monitor setup.  In this mode you can
even get 3D graphics in the virt-manager window, but *at greatly reduced
performance*.  Hope that helps,

Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/virt/attachments/20150309/8f7be36c/attachment-0001.html>