Nested virtualization on Intel with F16 already tested?

[Gianluca Cecchi]

Hello,
I'm going to test nested virtualization on Intel with F16 host.
I have used it successfully with Amd, where it is enabled by default
in its kvm-amd module.
Based on
https://github.com/torvalds/linux/blob/master/Documentation/virtual/kvm/n...
and
F16 having now
[root@f16 ~]# uname -r
3.1.0-7.fc16.x86_64
and the same confirmations in its kernel-doc
/usr/share/doc/kernel-doc-3.1.0/Documentation/virtual/kvm/nested-vmx.txt
file

In F15:
# uname -r
2.6.40.6-0.fc15.x86_64
# modinfo kvm-intel
...
parm:           bypass_guest_pf:bool
parm:           vpid:bool
parm:           flexpriority:bool
parm:           ept:bool
parm:           unrestricted_guest:bool
parm:           emulate_invalid_guest_state:bool
parm:           vmm_exclusive:bool
parm:           yield_on_hlt:bool
parm:           ple_gap:int
parm:           ple_window:int

In F16 indeed we have now:
# modinfo kvm-intel
...
parm:           vpid:bool
parm:           flexpriority:bool
parm:           ept:bool
parm:           unrestricted_guest:bool
parm:           emulate_invalid_guest_state:bool
parm:           vmm_exclusive:bool
parm:           yield_on_hlt:bool
parm:           nested:bool
parm:           ple_gap:int
parm:           ple_window:int

Based on docs, the "nested" option is disabled by default on Intel
(and probably not changed in Fedora?)
Did anyone already enable it and test nested virtualization
functionalities on Intel during F16 Beta/RC cycle and has now results
to share?
thanks in advance,
Gianluca

[Kashyap Chamarthy]

On 11/04/2011 07:36 PM, Gianluca Cecchi wrote:
>  Hello,
>  I'm going to test nested virtualization on Intel with F16 host.
>  I have used it successfully with Amd, where it is enabled by default
>  in its kvm-amd module.
>  Based on
>  https://github.com/torvalds/linux/blob/master/Documentation/virtual/kvm/n...
>  and
>  F16 having now
>  [root@f16 ~]# uname -r
>  3.1.0-7.fc16.x86_64
>  and the same confirmations in its kernel-doc
>  /usr/share/doc/kernel-doc-3.1.0/Documentation/virtual/kvm/nested-vmx.txt
>  file
>  
>  In F15:
>  # uname -r
>  2.6.40.6-0.fc15.x86_64
>  # modinfo kvm-intel
>  ...
>  parm:           bypass_guest_pf:bool
>  parm:           vpid:bool
>  parm:           flexpriority:bool
>  parm:           ept:bool
>  parm:           unrestricted_guest:bool
>  parm:           emulate_invalid_guest_state:bool
>  parm:           vmm_exclusive:bool
>  parm:           yield_on_hlt:bool
>  parm:           ple_gap:int
>  parm:           ple_window:int
>  
>  In F16 indeed we have now:
>  # modinfo kvm-intel
>  ...
>  parm:           vpid:bool
>  parm:           flexpriority:bool
>  parm:           ept:bool
>  parm:           unrestricted_guest:bool
>  parm:           emulate_invalid_guest_state:bool
>  parm:           vmm_exclusive:bool
>  parm:           yield_on_hlt:bool
>  parm:           nested:bool
>  parm:           ple_gap:int
>  parm:           ple_window:int
>  
>  Based on docs, the "nested" option is disabled by default on Intel
>  (and probably not changed in Fedora?)
>  Did anyone already enable it and test nested virtualization
>  functionalities on Intel during F16 Beta/RC cycle and has now results
>  to share? 
Not that I know of. But it be interesting to test it. Also would be nice if you could
share your results/experience.

Will give this a shot sometime over the next week.

/kashyap


>  thanks in advance,
>  Gianluca
>  _______________________________________________
>  virt mailing list
>  virt(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/virt

[Gianluca Cecchi]

Just made some tests this morning with latest F16 packages
kernel-3.1.0-7.fc16.x86_64
qemu-kvm-0.15.1-1.fc16.x86_64
virt-manager-0.9.0-7.fc16.noarch

host is Asus u36sd laptop with 8Gb of ram and an ssd disk
its cpu is:
model name	: Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz

on host:
$ sudo systool -m kvm_intel -v|grep nested
[sudo] password for g.cecchi:
    nested              = "Y"

Preliminary results are not so good.
I created an F16 guest (f16vm), with default options
I then put its virito disk in cache mode = none and I/O=native
I then selected "copy to guest" as cpu and it was put to "nehalem".

Inside the guest I create a windows xp with default values proposed by
virt-manager
(cd iso is winxp sp3)
Until now not able to complete installation
Better results if I choose "core2duo" as the cpu of f16vm.
But installations blocks in different points never arriving at the end
of the first "copy files" of windows xp install.
This because f16vm freezes (no more network, no more console...)
I have t power off it

I set up f16vm with serial console to see if something appears.. but
nothing appears when f16vm freezes

host run its F16 guest (f16vm) with this command:
# ps -ef|grep qemu
qemu     18638     1 85 15:39 ?        00:03:45 /usr/bin/qemu-kvm -S
-M pc-0.14 -cpu
core2duo,+rdtscp,+x2apic,+xtpr,+tm2,+est,+vmx,+ds_cpl,+pbe,+tm,+ht,+ss,+acpi,+ds
-enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name f16 -uuid
690251ac-b691-f320-1eba-9f7f61c2e0d3 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/f16.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=/var/lib/libvirt/images/f16.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:62:12:4a,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-usb -device usb-tablet,id=input0 -spice
port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global
qxl-vga.vram_size=67108864 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


F16 guest (f16vm) runs its windows xp install guest with this command:

qemu      1355     1 92 15:40 ?        00:03:38 /usr/bin/qemu-kvm -S
-M pc-0.14 -cpu
core2duo,+lahf_lm,+rdtscp,+hypervisor,+x2apic,+cx16,+vmx,+ss,-monitor
-enable-kvm -m 768 -smp 1,sockets=1,cores=1,threads=1 -name winxp
-uuid 9e7ed89e-1bab-b354-cccb-b545d0cc2c29 -nodefconfig -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/winxp.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
-drive file=/var/lib/libvirt/images/winxp.img,if=none,id=drive-ide0-0-0,format=raw
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
-drive
file=/var/lib/libvirt/images/winxp_sp3.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:91:6b:e6,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga std -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
root      1602  1251  0 15:44 pts/0    00:00:00 grep --color=auto qemu

F16 guest (f16vm) serial console output:
Fedora release 16 (Verne)
Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)

f16vm login:
Fedora release 16 (Verne)
Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)

f16vm login: [    0.443443] ioremap error for 0x7ffff000-0x80000000,
requested 0x10, got 0x0
[    3.885697] microcode: CPU0 update to revision 0xba failed
Started LSB: daemon for libvirt virtualization API.
Starting LSB: suspend/resume libvirt guests on shutdown/boot...
Started NFS file locking service..
Started Sendmail Mail Transport Client.
Started LSB: suspend/resume libvirt guests on shutdown/boot.
Started LSB: Starts and stops login and scanning of iSCSI devices..

Fedora release 16 (Verne)
Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)

f16vm login:
---> no more output until it then freezes

/var/log/messages of f16vm
[root@f16vm ~]# tail -f /var/log/messages
Nov  9 15:40:33 f16vm dbus-daemon[783]: ** Message: No devices in use, exit
Nov  9 15:40:42 f16vm kernel: [   50.520366] device vnet0 entered
promiscuous mode
Nov  9 15:40:42 f16vm kernel: [   50.521677] virbr1: topology change
detected, propagating
Nov  9 15:40:42 f16vm kernel: [   50.521680] virbr1: port 2(vnet0)
entering forwarding state
Nov  9 15:40:42 f16vm kernel: [   50.521687] virbr1: port 2(vnet0)
entering forwarding state
Nov  9 15:40:42 f16vm kernel: [   50.521773] ADDRCONF(NETDEV_CHANGE):
virbr1: link becomes ready
Nov  9 15:40:42 f16vm NetworkManager[752]: <warn>
/sys/devices/virtual/net/vnet0: couldn't determine device driver;
ignoring...
Nov  9 15:40:42 f16vm NetworkManager[752]: NetworkManager[752]: <warn>
/sys/devices/virtual/net/vnet0: couldn't determine device driver;
ignoring...
Nov  9 15:40:42 f16vm qemu-kvm: Could not find keytab file:
/etc/qemu/krb5.tab: No such file or directory
Nov  9 15:40:43 f16vm avahi-daemon[755]: Registering new address
record for fe80::fc54:ff:fe91:6be6 on vnet0.*.

---> no more until it freezes

more tests to come....
Probably needs to adjust particular features to enable/disable about
cpu to pass to f16vm guest...

I'm going try f16 host --> f16vm --> f16vmvm
and see what happens...

Gianluca

[Dor Laor]

On 11/09/2011 05:00 PM, Gianluca Cecchi wrote:
>  Just made some tests this morning with latest F16 packages
>  kernel-3.1.0-7.fc16.x86_64
>  qemu-kvm-0.15.1-1.fc16.x86_64
>  virt-manager-0.9.0-7.fc16.noarch
>
>  host is Asus u36sd laptop with 8Gb of ram and an ssd disk
>  its cpu is:
>  model name	: Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz
>
>  on host:
>  $ sudo systool -m kvm_intel -v|grep nested
>  [sudo] password for g.cecchi:
>       nested              = "Y" 
As nested is pretty experimental, better report that to kvm-devel 
mailing list
>  Preliminary results are not so good.
>  I created an F16 guest (f16vm), with default options
>  I then put its virito disk in cache mode = none and I/O=native
>  I then selected "copy to guest" as cpu and it was put to "nehalem".
>
>  Inside the guest I create a windows xp with default values proposed by
>  virt-manager
>  (cd iso is winxp sp3)
>  Until now not able to complete installation
>  Better results if I choose "core2duo" as the cpu of f16vm.
>  But installations blocks in different points never arriving at the end
>  of the first "copy files" of windows xp install.
>  This because f16vm freezes (no more network, no more console...)
>  I have t power off it
>
>  I set up f16vm with serial console to see if something appears.. but
>  nothing appears when f16vm freezes
>
>  host run its F16 guest (f16vm) with this command:
>  # ps -ef|grep qemu
>  qemu     18638     1 85 15:39 ?        00:03:45 /usr/bin/qemu-kvm -S
>  -M pc-0.14 -cpu
>  core2duo,+rdtscp,+x2apic,+xtpr,+tm2,+est,+vmx,+ds_cpl,+pbe,+tm,+ht,+ss,+acpi,+ds
>  -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name f16 -uuid
>  690251ac-b691-f320-1eba-9f7f61c2e0d3 -nodefconfig -nodefaults -chardev
>  socket,id=charmonitor,path=/var/lib/libvirt/qemu/f16.monitor,server,nowait
>  -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
>  -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
>
> file=/var/lib/libvirt/images/f16.img,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
>  -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
>  -netdev tap,fd=23,id=hostnet0 -device
>  virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:62:12:4a,bus=pci.0,addr=0x3
>  -chardev pty,id=charserial0 -device
>  isa-serial,chardev=charserial0,id=serial0 -chardev
>  spicevmc,id=charchannel0,name=vdagent -device
>
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
>  -usb -device usb-tablet,id=input0 -spice
>  port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global
>  qxl-vga.vram_size=67108864 -device
>  virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>
>
>  F16 guest (f16vm) runs its windows xp install guest with this command:
>
>  qemu      1355     1 92 15:40 ?        00:03:38 /usr/bin/qemu-kvm -S
>  -M pc-0.14 -cpu
>  core2duo,+lahf_lm,+rdtscp,+hypervisor,+x2apic,+cx16,+vmx,+ss,-monitor
>  -enable-kvm -m 768 -smp 1,sockets=1,cores=1,threads=1 -name winxp
>  -uuid 9e7ed89e-1bab-b354-cccb-b545d0cc2c29 -nodefconfig -nodefaults
>  -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/winxp.monitor,server,nowait
>  -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
>  -drive file=/var/lib/libvirt/images/winxp.img,if=none,id=drive-ide0-0-0,format=raw
>  -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2
>  -drive
> file=/var/lib/libvirt/images/winxp_sp3.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>  -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>  -netdev tap,fd=24,id=hostnet0 -device
>  rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:91:6b:e6,bus=pci.0,addr=0x3
>  -chardev pty,id=charserial0 -device
>  isa-serial,chardev=charserial0,id=serial0 -usb -device
>  usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga std -device
>  intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
>  hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
>  virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>  root      1602  1251  0 15:44 pts/0    00:00:00 grep --color=auto qemu
>
>  F16 guest (f16vm) serial console output:
>  Fedora release 16 (Verne)
>  Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)
>
>  f16vm login:
>  Fedora release 16 (Verne)
>  Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)
>
>  f16vm login: [    0.443443] ioremap error for 0x7ffff000-0x80000000,
>  requested 0x10, got 0x0
>  [    3.885697] microcode: CPU0 update to revision 0xba failed
>  Started LSB: daemon for libvirt virtualization API.
>  Starting LSB: suspend/resume libvirt guests on shutdown/boot...
>  Started NFS file locking service..
>  Started Sendmail Mail Transport Client.
>  Started LSB: suspend/resume libvirt guests on shutdown/boot.
>  Started LSB: Starts and stops login and scanning of iSCSI devices..
>
>  Fedora release 16 (Verne)
>  Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)
>
>  f16vm login:
>  --->  no more output until it then freezes
>
>  /var/log/messages of f16vm
>  [root@f16vm ~]# tail -f /var/log/messages
>  Nov  9 15:40:33 f16vm dbus-daemon[783]: ** Message: No devices in use, exit
>  Nov  9 15:40:42 f16vm kernel: [   50.520366] device vnet0 entered
>  promiscuous mode
>  Nov  9 15:40:42 f16vm kernel: [   50.521677] virbr1: topology change
>  detected, propagating
>  Nov  9 15:40:42 f16vm kernel: [   50.521680] virbr1: port 2(vnet0)
>  entering forwarding state
>  Nov  9 15:40:42 f16vm kernel: [   50.521687] virbr1: port 2(vnet0)
>  entering forwarding state
>  Nov  9 15:40:42 f16vm kernel: [   50.521773] ADDRCONF(NETDEV_CHANGE):
>  virbr1: link becomes ready
>  Nov  9 15:40:42 f16vm NetworkManager[752]:<warn>
>  /sys/devices/virtual/net/vnet0: couldn't determine device driver;
>  ignoring...
>  Nov  9 15:40:42 f16vm NetworkManager[752]: NetworkManager[752]:<warn>
>  /sys/devices/virtual/net/vnet0: couldn't determine device driver;
>  ignoring...
>  Nov  9 15:40:42 f16vm qemu-kvm: Could not find keytab file:
>  /etc/qemu/krb5.tab: No such file or directory
>  Nov  9 15:40:43 f16vm avahi-daemon[755]: Registering new address
>  record for fe80::fc54:ff:fe91:6be6 on vnet0.*.
>
>  --->  no more until it freezes
>
>  more tests to come....
>  Probably needs to adjust particular features to enable/disable about
>  cpu to pass to f16vm guest...
>
>  I'm going try f16 host -->  f16vm -->  f16vmvm
>  and see what happens...
>
>  Gianluca
>  _______________________________________________
>  virt mailing list
>  virt(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/virt

[Richard W.M. Jones]

On Wed, Nov 09, 2011 at 04:00:47PM +0100, Gianluca Cecchi wrote:
>  I set up f16vm with serial console to see if something appears.. but
>  nothing appears when f16vm freezes 
Try using virt-dmesg from the host to see if the F16 guest
gives any kernel errors before it freezes.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw

[Gianluca Cecchi]

On Thu, Nov 10, 2011 at 6:38 PM, Richard W.M. Jones wrote:
>  On Wed, Nov 09, 2011 at 04:00:47PM +0100, Gianluca Cecchi wrote:
> > I set up f16vm with serial console to see if something appears.. but
> > nothing appears when f16vm freezes
>
>  Try using virt-dmesg from the host to see if the F16 guest
>  gives any kernel errors before it freezes.
>  
Trying to reproduce, where the vm is already defined (because
installation failed but definition remained) now I get the L2 vm that
goes into paused mode and not able to have it exit from this state.

L1 f6 is f16vm
L2 f16 is f16vmvm

Tried with these L2 vms:
- winxp configured to start from cd iso
- f16vmvm configured to start form cd iso where I have CentOS 5.6 live

For example starting the f16vmvm (so the L2 one) I see it after a few
seconds in paused state in virt-manager of f16vm (L1 one).

[root@f16vm ~]# virsh list
 Id Name                 State
----------------------------------
  1 f16vmvm              paused

[root@f16vm ~]# virsh domstate f16vmvm --reason
paused (unknown)

Donna how to (try to) unpause it... clicking the pause button in
virt-manager does nothing...

Latest rows of the command
virt-dmesg f16vm
run from my host:
<30>[    2.136430] systemd[1]: Relabelled /dev and /run in 47ms 174us.
<7>[    2.139503] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.139830] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.140322] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.140511] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.140693] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.140889] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.141165] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<7>[    2.141374] SELinux: initialized (dev cgroup, type cgroup), uses
genfs_contexts
<30>[    2.141420] systemd[1]: systemd 36 running in system mode.
(+PAM +LIBWRAP +AUDIT +SELINUX +SYSVINIT +LIBCRYPTSETUP; fedora)
<30>[    2.145967] systemd[1]: Set hostname to <f16vm.test.org>.
<7>[    2.380982] SELinux: initialized (dev autofs, type autofs), uses
genfs_contexts
<30>[    2.396144] systemd-readahead-collect[427]: Disabling readahead
collector due to execution in virtualized environment.
<7>[    2.437212] SELinux: initialized (dev tmpfs, type tmpfs), uses
transition SIDs
<7>[    2.474475] SELinux: initialized (dev hugetlbfs, type
hugetlbfs), uses transition SIDs
<6>[    2.499487] RPC: Registered named UNIX socket transport module.
<6>[    2.499489] RPC: Registered udp transport module.
<6>[    2.499490] RPC: Registered tcp transport module.
<6>[    2.499492] RPC: Registered tcp NFSv4.1 backchannel transport module.
<7>[    2.499697] SELinux: initialized (dev rpc_pipefs, type
rpc_pipefs), uses genfs_contexts


I will try to reproduce the freeze during new vm install too...

[Gianluca Cecchi]

On Fri, Nov 11, 2011 at 12:34 AM, Gianluca Cecchi  wrote:
>  On Thu, Nov 10, 2011 at 6:38 PM, Richard W.M. Jones wrote:
> > On Wed, Nov 09, 2011 at 04:00:47PM +0100, Gianluca Cecchi wrote:
> >> I set up f16vm with serial console to see if something appears.. but
> >> nothing appears when f16vm freezes
> >
> > Try using virt-dmesg from the host to see if the F16 guest
> > gives any kernel errors before it freezes.
> >
>
>  Trying to reproduce, where the vm is already defined (because
>  installation failed but definition remained) now I get the L2 vm that
>  goes into paused mode and not able to have it exit from this state.
>
>  L1 f6 is f16vm
>  L2 f16 is f16vmvm
>
>  Tried with these L2 vms:
>  - winxp configured to start from cd iso
>  - f16vmvm configured to start form cd iso where I have CentOS 5.6 live
>
>  For example starting the f16vmvm (so the L2 one) I see it after a few
>  seconds in paused state in virt-manager of f16vm (L1 one).
>
>  [root@f16vm ~]# virsh list
>   Id Name                 State
>  ----------------------------------
>   1 f16vmvm              paused
>
>  [root@f16vm ~]# virsh domstate f16vmvm --reason
>  paused (unknown) 

Update on paused state of L2 guest:
inside L1 guest (f16vm) I create L2 guest named "c56" as red hat linux
5.4 or later and use iso of CentOS 56 live for installing

I get the pause state and the file /var/log/libvirt/qemu/c56.log contains
2011-11-11 18:04:27.903: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp
1,sockets=1,cores=1,threads=1 -name c56 -uuid
69857dbc-43b6-1f67-2576-16b9a0079a54 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/c56.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc
-no-reboot -drive
file=/storage1/c56L2.img,if=none,id=drive-virtio-disk0,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2
-drive
file=/var/lib/libvirt/images/CentOS-5.6-x86_64-LiveCD.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=24,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:4e:fd:00,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga
cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/1
KVM: entry failed, hardware error 0x80000021

If you're runnning a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest maybe running in big real mode
which is not supported on less recent Intel processors.

RAX=0000000000000000 RBX=ffffffff8006bdb5 RCX=0000000000000000
RDX=0000000000000000
RSI=0000000000000001 RDI=ffffffff80312738 RBP=0000000000023000
RSP=ffffffff80457f90
R8 =ffffffff80456000 R9 =000000000000003f R10=ffff81003fa20008
R11=0000000000000246
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000
R15=0000000000000000
RIP=ffffffff8006bdde RFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0000 0000000000000000 000fffff 00000000
GS =0000 ffffffff80425000 000fffff 00000000
LDT=0000 0000000000000000 000fffff 00000000
TR =0040 ffff810001733000 0000206f 00008b00 DPL=0 TSS64-busy
GDT=     ffffffff80458000 00000080
IDT=     ffffffff804ac000 00000fff
CR0=8005003b CR2=0000000001dda720 CR3=000000003fbdb000 CR4=000006e0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d01
Code=48 8b 04 25 10 00 00 00 8b 80 38 e0 ff ff a8 08 75 04 fb f4 <eb>
01 fb 65 48 8b 04 25 10 00 00 00 8b 80 38 e0 ff ff a8 08 74 d4 65 48
8b 04 25 10 00 00

BTW: my cpu infos

1)
On physical host
processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 42
model name	: Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz
stepping	: 7
cpu MHz		: 800.000
cache size	: 4096 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 2
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est
tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 x2apic popcnt xsave avx lahf_lm
ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept
vpid
bogomips	: 5387.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual

2)
The L1 guest f16vm is configued with cpu configuration as "core2duo"
[root@f16vm ~]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz
stepping	: 11
cpu MHz		: 2693.880
cache size	: 4096 KB
fpu		: yes
fpu_exception	: yes
cpuid level	: 10
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm
constant_tsc up nopl pni vmx ssse3 cx16 x2apic hypervisor lahf_lm
bogomips	: 5387.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: