10:03 p.m.
I'm having a decent amount of trouble getting pciback to behave the way I'd
like it to.
Originally I found xen-pciback was compiled as a module and not directly
into the kernel, ruling out use of the boot-time argument
xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback
directly into the kernel.
I added my pciback.hide arguments to the GRUB_CMDLINE_XEN in
/etc/default/grub and ran a grub-mkconfig updating /boot/grub2/grub.cfg
The devices were not excluded from Dom0.
So I tried adding the same arguments to the regular GRUB_CMDLINE and the
devices are still not hidden.
Can someone point me in the right direction? Did I mess up something?
System is FC17 on an x86_64
here's a snippet of my kernel config ... I just noticed that
PCIDEV_FRONTEND=m ... maybe that needs to be set to Y along with
PCIDEV_BACKEND ?
[root@xenhost linux-3.6.1-1.glmxen1.fc17.x86_64]# grep XEN .config
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=500
CONFIG_XEN_SAVE_RESTORE=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_FRONTEND=m
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_XEN_WDT=m
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SELFBALLOONING=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=y
CONFIG_XEN_PCIDEV_BACKEND=y
CONFIG_XEN_PRIVCMD=m
CONFIG_XEN_ACPI_PROCESSOR=m
here's the grub default
[root@xenhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap
KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
pciback.hide=(06:00.0)(02:00.0)(02:00.1)(07:00.0)(08:00.0)(00:1a.0)"
GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M loglvl=all guest_loglvl=all
pciback.hide=(06:00.0)(02:00.0)(02:00.1)(07:00.0)(08:00.0)(00:1a.0)"
#GRUB_THEME="/boot/grub2/themes/system/theme.txt"
5:12 p.m.
On Sat, 13 Oct 2012, Gordon McLellan wrote:
I'm having a decent amount of trouble getting pciback to behave
the way I'd
like it to.
Originally I found xen-pciback was compiled as a module and not directly
into the kernel, ruling out use of the boot-time argument
xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback
directly into the kernel.
Could you pass the option to the module by creating a file in
/etc/modprobe.d/ to supply the options to the module (the modprobe.conf
explains the format)? I think those files get copied over to the initramfs
when you install a kernel so they should still be used even if the module
is loaded early in the boot process.
Incidentally, you seem to be using xen-pciback in one place and pciback in
another. With a recent Fedora based kernel I would expect you would need
to use xen-pciback throughout.
Michael Young
3:02 p.m.
On Sat, Oct 13, 2012 at 6:12 PM, M A Young <m.a.young(a)durham.ac.uk> wrote:
On Sat, 13 Oct 2012, Gordon McLellan wrote:
I'm having a decent amount of trouble getting pciback to behave the way
> I'd
> like it to.
> Originally I found xen-pciback was compiled as a module and not directly
> into the kernel, ruling out use of the boot-time argument
> xen-pciback.hide=(blah) ... so I recompiled my kernel, incluging pciback
> directly into the kernel.
>
Could you pass the option to the module by creating a file in
/etc/modprobe.d/ to supply the options to the module (the modprobe.conf
explains the format)? I think those files get copied over to the initramfs
when you install a kernel so they should still be used even if the module
is loaded early in the boot process.
Incidentally, you seem to be using xen-pciback in one place and pciback in
another. With a recent Fedora based kernel I would expect you would need to
use xen-pciback throughout.
Michael Young
Michael,
Thank you for the tips. I made sure I specified xen-pciback instead of the
older pciback ... not really sure which line in the configuration it goes
on in /etc/default/grub - leaving it on the XEN line for now. I created
/etc/modprobe.d/xen-pciback.conf to also reflect the same:
options xen-pciback
hide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1)(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted.
After reboot, none of the devices are hidden from dom0. I can forcibly
unbind them from dom0 with a script, but that seems messy and some of the
pci devices don't seem to like it (sata controller for example)
Any suggestions?
Gordon
3:08 p.m.
On Sun, 14 Oct 2012, Gordon McLellan wrote:
Thank you for the tips. I made sure I specified xen-pciback instead
of the
older pciback ... not really sure which line in the configuration it goes on
in /etc/default/grub - leaving it on the XEN line for now. I created
/etc/modprobe.d/xen-pciback.conf to also reflect the same:
options
xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1
)(0000:00:0b.0)'
after making these changes, I updated the grub config file and rebooted.
After reboot, none of the devices are hidden from dom0. I can forcibly
unbind them from dom0 with a script, but that seems messy and some of the
pci devices don't seem to like it (sata controller for example)
Any suggestions?
Could you try without the leading 0000: in one or both places. I have been
looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples
there don't have the leading 0000: .
Michael Young
9:22 p.m.
On Sun, Oct 14, 2012 at 4:08 PM, M A Young <m.a.young(a)durham.ac.uk> wrote:
On Sun, 14 Oct 2012, Gordon McLellan wrote:
> Thank you for the tips. I made sure I specified xen-pciback instead of
> the
> older pciback ... not really sure which line in the configuration it goes
> on
> in /etc/default/grub - leaving it on the XEN line for now. I created
> /etc/modprobe.d/xen-pciback.conf to also reflect the same:
>
> options
>
xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1
>
> )(0000:00:0b.0)'
>
> after making these changes, I updated the grub config file and rebooted.
> After reboot, none of the devices are hidden from dom0. I can forcibly
> unbind them from dom0 with a script, but that seems messy and some of the
> pci devices don't seem to like it (sata controller for example)
>
> Any suggestions?
Could you try without the leading 0000: in one or both places. I have been
looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples
there don't have the leading 0000: .
Michael Young
Looks like I have some problem with grub passing the arguments
correctly. I see them properly when I view the "dmesg" output, but
when modprobe tries to load the xen-pciback module, it claims invalid
argument 0) ... like the rest of the line has been lost or something.
I tried single and doubel quotes, and with / without the extra 0's.
I've gone back to dynamic unbinding via script and I'll wait for 4.2
to come along :)
Thanks a bunch for the help.
Gordon
8:03 a.m.
On Sun, Oct 14, 2012 at 10:22:43PM -0400, Gordon McLellan wrote:
On Sun, Oct 14, 2012 at 4:08 PM, M A Young
<m.a.young(a)durham.ac.uk> wrote:
> On Sun, 14 Oct 2012, Gordon McLellan wrote:
>
>> Thank you for the tips. I made sure I specified xen-pciback instead of
>> the
>> older pciback ... not really sure which line in the configuration it goes
>> on
>> in /etc/default/grub - leaving it on the XEN line for now. I created
>> /etc/modprobe.d/xen-pciback.conf to also reflect the same:
>>
>> options
>>
xen-pcibackhide='(0000:00:1a.0)(0000:00:1b.0)(0000:00:1d.0)(0000:01:00.0)(0000:01:00.1
>>
>> )(0000:00:0b.0)'
>>
>> after making these changes, I updated the grub config file and rebooted.
>> After reboot, none of the devices are hidden from dom0. I can forcibly
>> unbind them from dom0 with a script, but that seems messy and some of the
>> pci devices don't seem to like it (sata controller for example)
>>
>> Any suggestions?
>
>
> Could you try without the leading 0000: in one or both places. I have been
> looking at http://wiki.xen.org/wiki/Xen_PCI_Passthrough and the examples
> there don't have the leading 0000: .
>
> Michael Young
Looks like I have some problem with grub passing the arguments
correctly. I see them properly when I view the "dmesg" output, but
when modprobe tries to load the xen-pciback module, it claims invalid
argument 0) ... like the rest of the line has been lost or something.
I tried single and doubel quotes, and with / without the extra 0's.
I've gone back to dynamic unbinding via script and I'll wait for 4.2
to come along :)
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Xen 4.1"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8
SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0
xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)"
GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
8:50 a.m.
On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk
<konrad.wilk(a)oracle.com> wrote:
Huh? Why? This is what I've in my /etc/defalt/grub
[root@phenom konrad]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Xen 4.1"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8
SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0
xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)"
GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
And it works quite well for me and I am using the stock Fedora kernel.
Konrad,
I'm not sure... it is probably something I have done wrong, and
probably something simple so I am overlooking it. I have gone back to
stock FC17 kernel and slimmed down my hide list to see if that helps:
[root@xenhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap
KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
xen-pciback.hide=(0b:00.0)(09:00.0)"
GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2
dom0_vcpus_pin loglvl=all guest_loglvl=all"
however the xen-pciback module is not loading at boot, and when I try
to load it manually:
[root@xenhost ~]# modprobe -vv xen-pciback
insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
0)
libkmod: kmod_module_insert_module: Failed to insert module
'/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko':
No such file or directory
ERROR: could not insert 'xen_pciback': No such file or directory
It is claiming file not found, but if I run the command on the first
line (insmod) it claims a different error, it does not like the "0)"
[root@xenhost ~]# insmod
/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
0)
-bash: syntax error near unexpected token `)'
8:51 a.m.
On Mon, Oct 15, 2012 at 09:50:31AM -0400, Gordon McLellan wrote:
On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk
<konrad.wilk(a)oracle.com> wrote:
>
> Huh? Why? This is what I've in my /etc/defalt/grub
>
> [root@phenom konrad]# cat /etc/default/grub
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR="Xen 4.1"
> GRUB_DEFAULT=saved
> GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8
SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0
xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)"
> GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
>
> And it works quite well for me and I am using the stock Fedora kernel.
>
Konrad,
I'm not sure... it is probably something I have done wrong, and
probably something simple so I am overlooking it. I have gone back to
stock FC17 kernel and slimmed down my hide list to see if that helps:
Hm, just realized that the machine is running F16, not F17!
[root@xenhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap
KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
xen-pciback.hide=(0b:00.0)(09:00.0)"
GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2
dom0_vcpus_pin loglvl=all guest_loglvl=all"
however the xen-pciback module is not loading at boot, and when I try
to load it manually:
[root@xenhost ~]# modprobe -vv xen-pciback
insmod /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
0)
libkmod: kmod_module_insert_module: Failed to insert module
'/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko':
No such file or directory
ERROR: could not insert 'xen_pciback': No such file or directory
That would imply it is simply gone. And does
cat /boot/config-3.6.1-1* | grep BACKEND
tell you whether the Xen PCI backend is compiled in (=y) or is a module (=m)?
It is claiming file not found, but if I run the command on the first
line (insmod) it claims a different error, it does not like the "0)"
[root@xenhost ~]# insmod
/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
0)
-bash: syntax error near unexpected token `)'
What is the "0)' from?
9:14 a.m.
On Mon, Oct 15, 2012 at 9:51 AM, Konrad Rzeszutek Wilk
<konrad.wilk(a)oracle.com> wrote:
On Mon, Oct 15, 2012 at 09:50:31AM -0400, Gordon McLellan wrote:
> On Mon, Oct 15, 2012 at 9:03 AM, Konrad Rzeszutek Wilk
> <konrad.wilk(a)oracle.com> wrote:
> >
> > Huh? Why? This is what I've in my /etc/defalt/grub
> >
> > [root@phenom konrad]# cat /etc/default/grub
> > GRUB_TIMEOUT=5
> > GRUB_DISTRIBUTOR="Xen 4.1"
> > GRUB_DEFAULT=saved
> > GRUB_CMDLINE_LINUX="rd.md=0 rd.lvm=0 rd.dm=0 KEYTABLE=us debug loglevel=8
SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 radeon.modeset=0
xen-pciback.hide=(01:00.0)(01:00.1)(02:00.0)"
> > GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all iommu=verbose"
> >
> > And it works quite well for me and I am using the stock Fedora kernel.
> >
>
> Konrad,
>
> I'm not sure... it is probably something I have done wrong, and
> probably something simple so I am overlooking it. I have gone back to
> stock FC17 kernel and slimmed down my hide list to see if that helps:
Hm, just realized that the machine is running F16, not F17!
> [root@xenhost ~]# cat /etc/default/grub
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR="Fedora"
> GRUB_DEFAULT=saved
> GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 SYSFONT=True rd.lvm.lv=vg0/swap
> KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
> xen-pciback.hide=(0b:00.0)(09:00.0)"
> GRUB_CMDLINE_XEN="dom0_mem=4096M,max:4096M dom0_max_vcpus=2
> dom0_vcpus_pin loglvl=all guest_loglvl=all"
>
> however the xen-pciback module is not loading at boot, and when I try
> to load it manually:
> [root@xenhost ~]# modprobe -vv xen-pciback
> insmod
/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
> 0)
> libkmod: kmod_module_insert_module: Failed to insert module
>
'/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko':
> No such file or directory
> ERROR: could not insert 'xen_pciback': No such file or directory
That would imply it is simply gone. And does
cat /boot/config-3.6.1-1* | grep BACKEND
tell you whether the Xen PCI backend is compiled in (=y) or is a module (=m)?
>
> It is claiming file not found, but if I run the command on the first
> line (insmod) it claims a different error, it does not like the "0)"
> [root@xenhost ~]# insmod
> /lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
> 0)
> -bash: syntax error near unexpected token `)'
What is the "0)' from?
pciback is compiled as a module:
[root@xenhost ~]# grep BACKEND /boot/config-3.6.1-1.fc17.x86_64
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_XEN_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND=m
and the file does exist:
[root@xenhost ~]# ls -lah
/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/
total 108K
drwxr-xr-x 2 root root 4.0K Oct 12 10:19 .
drwxr-xr-x 4 root root 4.0K Oct 12 10:19 ..
-rwxr--r-- 1 root root 100K Oct 10 08:41 xen-pciback.ko
Near as I can tell the 0) is coming from the kernel command line but
it is getting mangled somewhere along the way. If I remove the
xen-pciback.hide directive from the grub config file, the kernel is
able to load the pciback module without issue, and without any
configuration. I tried moving the xen-pciback.hide directive to the
front of the kernel command line, it did not help.
From dmesg:
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.6.1-1.fc17.x86_64 (mockbuild@) (gcc
version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) #1 SMP Wed Oct 10
12:13:05 UTC 2012
[ 0.000000] Command line: placeholder root=/dev/mapper/vg0-root ro
xen-pciback.hide=(0b:00.0)(09:00.0) rd.md=0 rd.dm=0 rd.lvm.lv=vg0/swap
KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
9:20 a.m.
On Mon, 15 Oct 2012, Gordon McLellan wrote:
pciback is compiled as a module:
[root@xenhost ~]# grep BACKEND /boot/config-3.6.1-1.fc17.x86_64
CONFIG_XEN_BLKDEV_BACKEND=m
CONFIG_XEN_NETDEV_BACKEND=m
CONFIG_XEN_BACKEND=y
CONFIG_XEN_PCIDEV_BACKEND=m
and the file does exist:
[root@xenhost ~]# ls -lah
/lib/modules/3.6.1-1.fc17.x86_64/kernel/drivers/xen/xen-pciback/
total 108K
drwxr-xr-x 2 root root 4.0K Oct 12 10:19 .
drwxr-xr-x 4 root root 4.0K Oct 12 10:19 ..
-rwxr--r-- 1 root root 100K Oct 10 08:41 xen-pciback.ko
Near as I can tell the 0) is coming from the kernel command line but
it is getting mangled somewhere along the way. If I remove the
xen-pciback.hide directive from the grub config file, the kernel is
able to load the pciback module without issue, and without any
configuration. I tried moving the xen-pciback.hide directive to the
front of the kernel command line, it did not help.
From dmesg:
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Linux version 3.6.1-1.fc17.x86_64 (mockbuild@) (gcc
version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) #1 SMP Wed Oct 10
12:13:05 UTC 2012
[ 0.000000] Command line: placeholder root=/dev/mapper/vg0-root ro
xen-pciback.hide=(0b:00.0)(09:00.0) rd.md=0 rd.dm=0 rd.lvm.lv=vg0/swap
KEYTABLE=us rd.lvm.lv=vg0/root rd.luks=0 LANG=en_US.UTF-8
If you haven't already done so I suggest you get rid of anything in
/etc/modprobe.d/ you added, just in case that is getting in the way.
Michael Young
12:07 p.m.
On Mon, Oct 15, 2012 at 10:20 AM, M A Young <m.a.young(a)durham.ac.uk> wrote:
If you haven't already done so I suggest you get rid of anything in
/etc/modprobe.d/ you added, just in case that is getting in the way.
Michael Young
I double checked, nothing about xen-pciback in /etc/modprobe.d and
/etc/modprobe.conf is empty
These files appear to be stock:
[root@xenhost ~]# ls /etc/modprobe.d
blacklist.conf openfwwf.conf udlfb.conf
I also tried booting the 3.5.6-1 fedora kernel, same difference.
modprobe xen-pciback fails, this is logged to /var/log/messages
Oct 15 13:05:59 xenhost kernel: [ 178.996253] xen_pciback: Unknown
parameter `0)'
4212
days inactive
4214
days old
10 comments
3 participants
participants (3)
-
Gordon McLellan -
Konrad Rzeszutek Wilk -
M A Young