Openstack and file injection

[Belmiro Moreira]

Hi,
I'm using openstack packages available on EPEL (Essex) to deploy openstack on CERN
Scientific Linux 6.
Openstack is failing to mount images to inject data because "nbd" module is not
available in RHEL.

Stderr: 'libguestfs: error: guestfs_launch failed, see earlier error messages\n'
from (pid=26216) mount /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:205
2012-11-12 08:44:35 WARNING nova.virt.libvirt.connection
[req-651b1576-15a3-46b5-ba85-a5651d4719b7 b7aa0805440f41bfa69b000bb475a0eb
237745f6e81d4a8494eea1b168d73610] [in
stance: 6045ff89-b323-448b-9001-3232e4c42e83] Ignoring error injecting data into image
e7aa813f-a6b8-4fbd-b960-f6f5009cc33d (
--
nbd unavailable: module not loaded
--

qemu-nbd is also not available.
Should I grab this from Fedora and rebuild it or I'm missing some configuration in my
system to be able to inject data in VMs.

thanks for any help,
Belmiro

[Pádraig Brady]

On 11/12/2012 08:04 AM, Belmiro Moreira wrote:
>  Hi,
>  I'm using openstack packages available on EPEL (Essex) to deploy openstack on CERN
> Scientific Linux 6.
>  Openstack is failing to mount images to inject data because "nbd" module is not
> available in RHEL.
>
>  Stderr: 'libguestfs: error: guestfs_launch failed, see earlier error messages\n'
> from (pid=26216) mount /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:205
>  2012-11-12 08:44:35 WARNING nova.virt.libvirt.connection
> [req-651b1576-15a3-46b5-ba85-a5651d4719b7 b7aa0805440f41bfa69b000bb475a0eb
> 237745f6e81d4a8494eea1b168d73610] [in
>  stance: 6045ff89-b323-448b-9001-3232e4c42e83] Ignoring error injecting data into image
> e7aa813f-a6b8-4fbd-b960-f6f5009cc33d (
>  --
>  nbd unavailable: module not loaded
>  --
>
>  qemu-nbd is also not available.
>  Should I grab this from Fedora and rebuild it or I'm missing some configuration in my
> system to be able to inject data in VMs.
>
>  thanks for any help,
>  Belmiro 
So there are 3 methods tried to inject data into a system image.
   loop, nbd and libguestfs.

loop suffices for raw images, nbd handles qcow2 format but
is not supported by RHEL based kernels as you've seen.
You might be able to compile separately, but I'd suggest
that you try to get libguestfs working, which is the most functional/flexible.

So things to try:

1. Ensure libvirt_inject_partition = -1 in /etc/nova/nova.conf
This will ensure that libvirt will go looking for the partition to
inject to, rather than requiring a specific partition.
Do you have a partitioned or flat guest image BTW?
What is the guest image?

2. Try the command that nova runs to mount the image:
   guestmount --rw -a file.img -i /tmp/mount_test
Note also the --verbose and --trace options.
If the problem is still not apparent from the console or logs,
you can enable extra debugging to see what's happening exactly with:
export LIBGUESTFS_DEBUG=1

thanks,
Pádraig.

[Belmiro Moreira]

Hi Pádraig,
thank you for your quick answer and suggestions.

After some investigation I discovered that guestfs fails to mount the image because
selinux:

type=AVC msg=audit(1352816002.979:249317): avc:  denied  { read } for pid=2806
comm="qemu-kvm" name="disk" dev=dm-3 ino=656740
scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1352816002.979:249317): arch=c000003e syscall=2 success=no exit=-13
a0=7fae966dbc20 a1=800 a2=0 a3=65636e6174736e69 items=0 ppid=2797 pid=2806 auid=0 uid=0
gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=19511
comm="qemu-kvm" exe="/usr/libexec/qemu-kvm"
subj=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1352816002.980:249318): avc:  denied  { getattr } for  pid=2806
comm="qemu-kvm" path="/var/lib/nova/instances/instance-000000a7/disk"
dev=dm-3 ino=656740 scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1352816002.980:249318): arch=c000003e syscall=4 success=no exit=-13
a0=7fae966dbc20 a1=7fffedb37730 a2=7fffedb37730 a3=65636e6174736e69 items=0 ppid=2797
pid=2806 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none)
ses=19511 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm"
subj=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1352816002.980:249319): avc:  denied  { read } for pid=2806
comm="qemu-kvm" name="disk" dev=dm-3 ino=656740
scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file

Disabling selinux it works fine.

Should I open a bug for this?

thank you,
Belmiro
CERN

On Nov 12, 2012, at 10:57 AM, Pádraig Brady <P(a)draigBrady.com&gt; wrote:

>  On 11/12/2012 08:04 AM, Belmiro Moreira wrote:
> > Hi,
> > I'm using openstack packages available on EPEL (Essex) to deploy openstack on
> CERN Scientific Linux 6.
> > Openstack is failing to mount images to inject data because "nbd" module is
> not available in RHEL.
> > 
> > Stderr: 'libguestfs: error: guestfs_launch failed, see earlier error
> messages\n' from (pid=26216) mount
> /usr/lib/python2.6/site-packages/nova/virt/disk/api.py:205
> > 2012-11-12 08:44:35 WARNING nova.virt.libvirt.connection
> [req-651b1576-15a3-46b5-ba85-a5651d4719b7 b7aa0805440f41bfa69b000bb475a0eb
> 237745f6e81d4a8494eea1b168d73610] [in
> > stance: 6045ff89-b323-448b-9001-3232e4c42e83] Ignoring error injecting data into
> image e7aa813f-a6b8-4fbd-b960-f6f5009cc33d (
> > --
> > nbd unavailable: module not loaded
> > --
> > 
> > qemu-nbd is also not available.
> > Should I grab this from Fedora and rebuild it or I'm missing some configuration
> in my system to be able to inject data in VMs.
> > 
> > thanks for any help,
> > Belmiro
>  
>  So there are 3 methods tried to inject data into a system image.
>   loop, nbd and libguestfs.
>  
>  loop suffices for raw images, nbd handles qcow2 format but
>  is not supported by RHEL based kernels as you've seen.
>  You might be able to compile separately, but I'd suggest
>  that you try to get libguestfs working, which is the most functional/flexible.
>  
>  So things to try:
>  
>  1. Ensure libvirt_inject_partition = -1 in /etc/nova/nova.conf
>  This will ensure that libvirt will go looking for the partition to
>  inject to, rather than requiring a specific partition.
>  Do you have a partitioned or flat guest image BTW?
>  What is the guest image?
>  
>  2. Try the command that nova runs to mount the image:
>   guestmount --rw -a file.img -i /tmp/mount_test
>  Note also the --verbose and --trace options.
>  If the problem is still not apparent from the console or logs,
>  you can enable extra debugging to see what's happening exactly with:
>  export LIBGUESTFS_DEBUG=1
>  
>  thanks,
>  Pádraig.

[Pádraig Brady]

On 11/13/2012 02:28 PM, Belmiro Moreira wrote:
>  Hi Pádraig,
>  thank you for your quick answer and suggestions.
>
>  After some investigation I discovered that guestfs fails to mount the image because
> selinux:
>
>  type=AVC msg=audit(1352816002.979:249317): avc:  denied  { read } for pid=2806
> comm="qemu-kvm" name="disk" dev=dm-3 ino=656740
> scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
> tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
>  type=SYSCALL msg=audit(1352816002.979:249317): arch=c000003e syscall=2 success=no
> exit=-13 a0=7fae966dbc20 a1=800 a2=0 a3=65636e6174736e69 items=0 ppid=2797 pid=2806 auid=0
> uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=19511
> comm="qemu-kvm" exe="/usr/libexec/qemu-kvm"
> subj=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 key=(null)
>  type=AVC msg=audit(1352816002.980:249318): avc:  denied  { getattr } for  pid=2806
> comm="qemu-kvm" path="/var/lib/nova/instances/instance-000000a7/disk"
> dev=dm-3 ino=656740 scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
> tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
>  type=SYSCALL msg=audit(1352816002.980:249318): arch=c000003e syscall=4 success=no
> exit=-13 a0=7fae966dbc20 a1=7fffedb37730 a2=7fffedb37730 a3=65636e6174736e69 items=0
> ppid=2797 pid=2806 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0
> tty=(none) ses=19511 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm"
> subj=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023 key=(null)
>  type=AVC msg=audit(1352816002.980:249319): avc:  denied  { read } for pid=2806
> comm="qemu-kvm" name="disk" dev=dm-3 ino=656740
> scontext=unconfined_u:system_r:qemu_t:s0-s0:c0.c1023
> tcontext=unconfined_u:object_r:nova_var_lib_t:s0 tclass=file
>
>  Disabling selinux it works fine.
>
>  Should I open a bug for this? 
Probably. Let's get details first.
Can you renable SElinux, and then:

   setsebool -P allow_unconfined_qemu_transition 0

Does that minimal SELinux relaxation allow libguestfs to work?

thanks,
Pǽdraig.