No iommu_group links in /sys/bus/pci/devices/

[Mario Ceresa]

Dear all,
following Cole's suggestions I installed the fedora-virt-preview repo and
tried again with VGA
passthrough from [1]

For this setup i use a nvidia card as the primary one and a radeon hd 7850
as the passthrough device.

However, there are no iommu_group links neither under
/sys/bus/pci/devices/0000:05:00.0/ (which is my radeon card) nor under any
of the other devices.

AFAIG it should be the vfio module which creates these links so I guess
it's not working for me.

The driver loads correctly: there are no errors in dmesg and lsmod shows:
vfio_pci               32063  0
vfio                   19210  1 vfio_pci

I added iommu=pt iommu=1 to grub

My system currently includes:

* libvrit 1.0.6
* qemu 1.5.0
* kernel 3.9.6

Maybe the kernel is outdated?

I know the functionality is experimental so it's just to know if I'm doing
some obvious mistakes.

Thanks for your help,

Best regards,

Mario

[1] https://bbs.archlinux.org/viewtopic.php?id=162768

[Alex Williamson]

On Tue, 2013-06-25 at 18:46 +0200, Mario Ceresa wrote:
> Dear all,
> following Cole's suggestions I installed the fedora-virt-preview repo and
> tried again with VGA
> passthrough from [1]
>
> For this setup i use a nvidia card as the primary one and a radeon hd 7850
> as the passthrough device.
>
> However, there are no iommu_group links neither under
> /sys/bus/pci/devices/0000:05:00.0/ (which is my radeon card) nor under any
> of the other devices.
>
> AFAIG it should be the vfio module which creates these links so I guess
> it's not working for me.
>
> The driver loads correctly: there are no errors in dmesg and lsmod shows:
> vfio_pci               32063  0
> vfio                   19210  1 vfio_pci
>
> I added iommu=pt iommu=1 to grub

iommu=1 is not a valid option.  The user in archlinux thread is using an
AMD system, which enables the iommu by default.  On an Intel system, use
intel_iommu=on
> My system currently includes:
> libvrit 1.0.6
> qemu 1.5.0
> kernel 3.9.6
>
> Maybe the kernel is outdated?
>
> I know the functionality is experimental so it's just to know if I'm doing
> some obvious mistakes.
>
> Thanks for your help,
>
> Best regards,
>
> Mario
>
> [1] https://bbs.archlinux.org/viewtopic.php?id=162768
> _______________________________________________
> virt mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/virt

[Mario Ceresa]

Thanks Alex, now the driver is working fine.

I spent the last two days making tests and the results are as folows:

1. from qemu command line I never get past the "This VM has no graphic
card" messages even if I change the number of the pci id to the one I bound
to vfio-pci (05:00). I'm still trying to learn how to control quemu by
command line so most likely I'm using the wrong spell here :)

qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
\
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1

2. With virt-manager I'm able to go a bit further once I add the two ids as
PCI passthrough devices:

2.1 I can run a copy of Fedora 18 that loads the open source radeon driver.
It doesn't seem to be really accelerated though but I suppose it's working
because when I reset/poweroff the host resets :)
However I have the following errors in the log:

* Expecting atombios for cayman GPU
* Fatal error during GPU init
* probe of <pci id here> failed with -22

2.2 I'm not able to use radeon binary drivers from rpmfusion on this VM. X
always dies with "No recognized devices"

2.3 I created a new winxp test machine and installed radeon binary drivers
on it but the card is not detected. It does appear as "Unrecognized VGA
adapter" so it seems that it is exposed, at least partially.

I'm a bit out of my wits by now, is there any way to see how virt-manager
calls qemu?

Do I have to remove nvidia binary driver or patch them? (the primary gpu is
a quadro 4000)

Thanks and regards,

Mario

On 25 June 2013 20:04, Alex Williamson  wrote:
> On Tue, 2013-06-25 at 18:46 +0200, Mario Ceresa wrote:
> > Dear all,
> > following Cole's suggestions I installed the fedora-virt-preview repo and
> > tried again with VGA
> > passthrough from [1]
> >
> > For this setup i use a nvidia card as the primary one and a radeon hd
>
> 7850
> > as the passthrough device.
> >
> > However, there are no iommu_group links neither under
> > /sys/bus/pci/devices/0000:05:00.0/ (which is my radeon card) nor under
>
> any
> > of the other devices.
> >
> > AFAIG it should be the vfio module which creates these links so I guess
> > it's not working for me.
> >
> > The driver loads correctly: there are no errors in dmesg and lsmod shows:
> > vfio_pci               32063  0
> > vfio                   19210  1 vfio_pci
> >
> > I added iommu=pt iommu=1 to grub
>
> iommu=1 is not a valid option.  The user in archlinux thread is using an
> AMD system, which enables the iommu by default.  On an Intel system, use
> intel_iommu=on
> > My system currently includes:
> > libvrit 1.0.6
> > qemu 1.5.0
> > kernel 3.9.6
> >
> > Maybe the kernel is outdated?
> >
> > I know the functionality is experimental so it's just to know if I'm
>
> doing
> > some obvious mistakes.
> >
> > Thanks for your help,
> >
> > Best regards,
> >
> > Mario
> >
> > [1] https://bbs.archlinux.org/viewtopic.php?id=162768
> > _______________________________________________
> > virt mailing list
> > [e-mail]
> > https://admin.fedoraproject.org/mailman/listinfo/virt

[Alex Williamson]

On Thu, 2013-06-27 at 16:35 +0200, Mario Ceresa wrote:
> Thanks Alex, now the driver is working fine.
>
> I spent the last two days making tests and the results are as folows:
> from qemu command line I never get past the "This VM has no graphic
>
> card" messages even if I change the number of the pci id to the one I bound
> to vfio-pci (05:00). I'm still trying to learn how to control quemu by
> command line so most likely I'm using the wrong spell here :)
>
> qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host \
> -smp 6,sockets=1,cores=6,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
> \
> -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1

Looks like you're still trying to create an SDL window since I believe
that's on by default, add -nographic
> With virt-manager I'm able to go a bit further once I add the two ids as
>
> PCI passthrough devices:
>
> 2.1 I can run a copy of Fedora 18 that loads the open source radeon driver.
> It doesn't seem to be really accelerated though but I suppose it's working
> because when I reset/poweroff the host resets :)

Not good.  This is using the vfio-vga-reset branches?
> However I have the following errors in the log:
> Expecting atombios for cayman GPU
> Fatal error during GPU init
> probe of <pci id here> failed with -22

Cayman, is this a Radeon HD 6xxx series?  Are there any messages in the
host log file or dmesg about retrieving the BIOS from the device?
> 2.2 I'm not able to use radeon binary drivers from rpmfusion on this VM. X
> always dies with "No recognized devices"
>
> 2.3 I created a new winxp test machine and installed radeon binary drivers
> on it but the card is not detected. It does appear as "Unrecognized VGA
> adapter" so it seems that it is exposed, at least partially.

libvirt is going to be using 440fx, unless there's a q35 option for it
now.  The older chipset is not tested as well and libvirt will put the
emulated and passthrough graphics on the same bus, which makes things
more difficult.
> I'm a bit out of my wits by now, is there any way to see how virt-manager
> calls qemu?

/var/log/libvirt/qemu/$VM
> Do I have to remove nvidia binary driver or patch them? (the primary gpu is
> a quadro 4000)

Remove or patch them for what?  If you want to assign that device, yes,
the nvidia binary driver needs to be unbound from the device and you
must then use the vfio-vga-reset branches to get the device into a fresh
state to be assigned.  Note that assigning a primary GPU from the host
still has issues with removing the host VGA console from using it.
Thanks,

Alex

[Mario Ceresa]

Thanks Alex for your prompt reply! I'll gather all the information you
requested, make some more tests with command line qemu and then write back.
In the meantime, I'll briefly comment on same of your lines:
> Looks like you're still trying to create an SDL window since I believe

that's on by default, add -nographic

I'll try that. However, I thought that the purpose of this line (in the
arch tutorial at [1]) was to show a running VM with seabios. How can I see
what's going on in the VM with no graphic?
> Not good.  This is using the vfio-vga-reset branches?

No. I'm just using the fedora-virt-preview repository [2]. I would like to
test that branch. Is it:

git://github.com/awilliam/linux-vfio.git
git://github.com/awilliam/qemu-vfio.git

with vfio-vga-reset, right?

* Are there any recommended build options apart of:

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

* This is not the version in fedora-virt-preview repo so I need to
recompile qemu as well, right?
* Should I stop using the virt-preview repo completely? It also ships
libvirtd and seabios packages I think...
> Cayman, is this a Radeon HD 6xxx [...] retrieving the BIOS from the

device?

It is a radeon HD 7850. I would say there are no rom-related failures on
dmesg but I'll check it again and report.
> Remove or patch them for what?

I have two gpu: nvidia quadro 4000 which is the primary and the radeon
which is the one to pass through. I read in [3, 4] that someone needed to
patch the nvidia binary driver. I can also get rid of that and nouveau
altogether, but then, how can I check what the VM is doing without X?

Best,

Mario

[1] https://bbs.archlinux.org/viewtopic.php?id=162768&p=1
[2] http://fedoraproject.org/wiki/Virtualization_Preview_Repository
[3] http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg04296.html
[4] http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg04300.html

On 27 June 2013 17:03, Alex Williamson  wrote:
> On Thu, 2013-06-27 at 16:35 +0200, Mario Ceresa wrote:
> > Thanks Alex, now the driver is working fine.
> >
> > I spent the last two days making tests and the results are as folows:
> > from qemu command line I never get past the "This VM has no graphic
> >
> > card" messages even if I change the number of the pci id to the one I
>
> bound
> > to vfio-pci (05:00). I'm still trying to learn how to control quemu by
> > command line so most likely I'm using the wrong spell here :)
> >
> > qemu-system-x86_64 --enable-kvm -M q35 -m 1024 -cpu host \
> > -smp 6,sockets=1,cores=6,threads=1 \
> > -bios /usr/share/qemu/bios.bin -vga none \
> > -device
>
> ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1
> > \
> > -device
>
> vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> > -device vfio-pci,host=05:00.1,bus=root.1,addr=00.1
>
> Looks like you're still trying to create an SDL window since I believe
> that's on by default, add -nographic
> > With virt-manager I'm able to go a bit further once I add the two ids
>
> as
> > PCI passthrough devices:
> >
> > 2.1 I can run a copy of Fedora 18 that loads the open source radeon
>
> driver.
> > It doesn't seem to be really accelerated though but I suppose it's
>
> working
> > because when I reset/poweroff the host resets :)
>
> Not good.  This is using the vfio-vga-reset branches?
> > However I have the following errors in the log:
> > Expecting atombios for cayman GPU
> > Fatal error during GPU init
> > probe of <pci id here> failed with -22
>
> Cayman, is this a Radeon HD 6xxx series?  Are there any messages in the
> host log file or dmesg about retrieving the BIOS from the device?
> > 2.2 I'm not able to use radeon binary drivers from rpmfusion on this VM.
>
> X
> > always dies with "No recognized devices"
> >
> > 2.3 I created a new winxp test machine and installed radeon binary
>
> drivers
> > on it but the card is not detected. It does appear as "Unrecognized VGA
> > adapter" so it seems that it is exposed, at least partially.
>
> libvirt is going to be using 440fx, unless there's a q35 option for it
> now.  The older chipset is not tested as well and libvirt will put the
> emulated and passthrough graphics on the same bus, which makes things
> more difficult.
> > I'm a bit out of my wits by now, is there any way to see how virt-manager
> > calls qemu?
>
> /var/log/libvirt/qemu/$VM
> > Do I have to remove nvidia binary driver or patch them? (the primary gpu
>
> is
> > a quadro 4000)
>
> Remove or patch them for what?  If you want to assign that device, yes,
> the nvidia binary driver needs to be unbound from the device and you
> must then use the vfio-vga-reset branches to get the device into a fresh
> state to be assigned.  Note that assigning a primary GPU from the host
> still has issues with removing the host VGA console from using it.
> Thanks,
>
> Alex

[Alex Williamson]

On Thu, 2013-06-27 at 17:42 +0200, Mario Ceresa wrote:
> Thanks Alex for your prompt reply! I'll gather all the information you
> requested, make some more tests with command line qemu and then write back.
> In the meantime, I'll briefly comment on same of your lines:
> > Looks like you're still trying to create an SDL window since I believe
>
> that's on by default, add -nographic
>
> I'll try that. However, I thought that the purpose of this line (in the
> arch tutorial at [1]) was to show a running VM with seabios. How can I see
> what's going on in the VM with no graphic?

It should initialize the VGA device and you should get a seabios splash
there (though depending on the card, it may already be gone by the time
the monitor wakes up).

BTW, see this message for some tips I learned setting this up:
 http://lists.nongnu.org/archive/html/qemu-devel/2013-05/msg01172.html
> > Not good.  This is using the vfio-vga-reset branches?
>
> No. I'm just using the fedora-virt-preview repository [2]. I would like to
> test that branch. Is it:
>
> git://github.com/awilliam/linux-vfio.git
> git://github.com/awilliam/qemu-vfio.git
>
> with vfio-vga-reset, right?

Yes.  Current upstream/fedora-virt-preview is missing some things, like
the bus reset that is both the key to making it more reproducible and
the thorn that's keeping it from upstream.
> Are there any recommended build options apart of:
>
> CONFIG_VFIO_IOMMU_TYPE1=y
> CONFIG_VFIO=y
> CONFIG_VFIO_PCI=y
> CONFIG_VFIO_PCI_VGA=y

That's it.  I tend to run with a pretty lean host config, so you may
still run into problems with unbinding host drivers like radeon,
nouveau, nvidia.  Some of them are not very robust in their unbind code
so you need to blacklist them to avoid them altogether.
> This is not the version in fedora-virt-preview repo so I need to
>
> recompile qemu as well, right?

Correct
> Should I stop using the virt-preview repo completely? It also ships
>
> libvirtd and seabios packages I think...

For this, yes.  It's still in development upstream.
> > Cayman, is this a Radeon HD 6xxx [...] retrieving the BIOS from the
>
> device?
>
> It is a radeon HD 7850. I would say there are no rom-related failures on
> dmesg but I'll check it again and report.

I should have known that, I have one of those ;)
> > Remove or patch them for what?
>
> I have two gpu: nvidia quadro 4000 which is the primary and the radeon
> which is the one to pass through. I read in [3, 4] that someone needed to
> patch the nvidia binary driver. I can also get rid of that and nouveau
> altogether, but then, how can I check what the VM is doing without X?

Oh right, yes.  IIRC, the nvidia driver uses the VGA arbiter
incorrectly, acquiring the VGA lock and never releasing it.  That means
a guest hangs on the first attempt to try to use VGA space.  Yes, if you
intend to continue using the nvidia binary in the host, you'll need to
patch it.  Thanks,

Alex