Nested KVM help

[James Slagle]

Hi,

When I try to use nested KVM with Fedora 19, I'm experiencing a hang of the L1
vm.  I eventually have to do a "Force Off" from virt-manager on L0 since the L1
vm is completely unresponsive.

Both my L0 and L1 are F19, x86_64.  L2 doesn't really seem to matter as I can't
even get the guest to start before L1 completely locks up.  Even just
attempting to define a L2 guest that uses kvm virtualization locks up L1
completely.

Here are some more details.

On L0 (i've rebooted L0 after enabling the nested parameter):
[root@dublin ~]# uname -a
Linux dublin 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45 UTC 2013 x86_64
x86_64 x86_64 GNU/Linux
[root@dublin ~]# cat /sys/module/kvm_intel/parameters/nested 
Y
[root@dublin ~]# rpm -q libvirt
libvirt-1.0.5.5-1.fc19.x86_64
[root@dublin ~]# lsmod | grep kvm
kvm_intel             138528  3 
kvm                   422809  1 kvm_intel

cpu section of libvirt xml for L1 vm:
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
    <feature policy='require' name='vmx'/>
  </cpu>


On L1:
[root@localhost jslagle]# uname -a
Linux localhost.localdomain 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost jslagle]# rpm -q libvirt
libvirt-1.0.5.5-1.fc19.x86_64
[root@localhost jslagle]# lsmod | grep kvm
kvm_intel             138528  0 
kvm                   422809  1 kvm_intel


I've looked in dmesg output, /var/log/messages, and libvirt logs on both L0 and
L1 and can't see any errors.

Any ideas on how to debug this further?
Let me know if should file a bugzilla, I wasn't sure I had enough to go on to
go ahead and do that yet.

Thanks for any help.

--
-- James Slagle
--

[Cole Robinson]

On 08/19/2013 12:43 PM, James Slagle wrote:
>  Hi,
>  
>  When I try to use nested KVM with Fedora 19, I'm experiencing a hang of the L1
>  vm.  I eventually have to do a "Force Off" from virt-manager on L0 since the
> L1
>  vm is completely unresponsive.
>  
>  Both my L0 and L1 are F19, x86_64.  L2 doesn't really seem to matter as I can't
>  even get the guest to start before L1 completely locks up.  Even just
>  attempting to define a L2 guest that uses kvm virtualization locks up L1
>  completely.
>  
>  Here are some more details.
>  
>  On L0 (i've rebooted L0 after enabling the nested parameter):
>  [root@dublin ~]# uname -a
>  Linux dublin 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45 UTC 2013 x86_64
>  x86_64 x86_64 GNU/Linux
>  [root@dublin ~]# cat /sys/module/kvm_intel/parameters/nested 
>  Y
>  [root@dublin ~]# rpm -q libvirt
>  libvirt-1.0.5.5-1.fc19.x86_64
>  [root@dublin ~]# lsmod | grep kvm
>  kvm_intel             138528  3 
>  kvm                   422809  1 kvm_intel
>  
>  cpu section of libvirt xml for L1 vm:
>    <cpu mode='custom' match='exact'>
>      <model fallback='allow'>Nehalem</model>
>      <feature policy='require' name='vmx'/>
>    </cpu>
>  
>  
>  On L1:
>  [root@localhost jslagle]# uname -a
>  Linux localhost.localdomain 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45
>  UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>  [root@localhost jslagle]# rpm -q libvirt
>  libvirt-1.0.5.5-1.fc19.x86_64
>  [root@localhost jslagle]# lsmod | grep kvm
>  kvm_intel             138528  0 
>  kvm                   422809  1 kvm_intel
>  
>  
>  I've looked in dmesg output, /var/log/messages, and libvirt logs on both L0 and
>  L1 and can't see any errors.
>  
>  Any ideas on how to debug this further?
>  Let me know if should file a bugzilla, I wasn't sure I had enough to go on to
>  go ahead and do that yet.
>   
Unfortunately doesn't sound like there is much to go on, maybe the kvm
maintainers can give some hints. Please file a bug against the kernel and CC me.

Thanks,
Cole

[James Slagle]

On Mon, Aug 19, 2013 at 01:25:29PM -0400, Cole Robinson wrote:
>  On 08/19/2013 12:43 PM, James Slagle wrote:
>  > Hi,
>  > 
>  > When I try to use nested KVM with Fedora 19, I'm experiencing a hang of the L1
>  > vm.  I eventually have to do a "Force Off" from virt-manager on L0 since
> the L1
>  > vm is completely unresponsive.
>  > 
>  > Both my L0 and L1 are F19, x86_64.  L2 doesn't really seem to matter as I
> can't
>  > even get the guest to start before L1 completely locks up.  Even just
>  > attempting to define a L2 guest that uses kvm virtualization locks up L1
>  > completely.
>  > 
>  > Here are some more details.
>  > 
>  > On L0 (i've rebooted L0 after enabling the nested parameter):
>  > [root@dublin ~]# uname -a
>  > Linux dublin 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45 UTC 2013 x86_64
>  > x86_64 x86_64 GNU/Linux
>  > [root@dublin ~]# cat /sys/module/kvm_intel/parameters/nested 
>  > Y
>  > [root@dublin ~]# rpm -q libvirt
>  > libvirt-1.0.5.5-1.fc19.x86_64
>  > [root@dublin ~]# lsmod | grep kvm
>  > kvm_intel             138528  3 
>  > kvm                   422809  1 kvm_intel
>  > 
>  > cpu section of libvirt xml for L1 vm:
>  >   <cpu mode='custom' match='exact'>
>  >     <model fallback='allow'>Nehalem</model>
>  >     <feature policy='require' name='vmx'/>
>  >   </cpu>
>  > 
>  > 
>  > On L1:
>  > [root@localhost jslagle]# uname -a
>  > Linux localhost.localdomain 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45
>  > UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>  > [root@localhost jslagle]# rpm -q libvirt
>  > libvirt-1.0.5.5-1.fc19.x86_64
>  > [root@localhost jslagle]# lsmod | grep kvm
>  > kvm_intel             138528  0 
>  > kvm                   422809  1 kvm_intel
>  > 
>  > 
>  > I've looked in dmesg output, /var/log/messages, and libvirt logs on both L0 and
>  > L1 and can't see any errors.
>  > 
>  > Any ideas on how to debug this further?
>  > Let me know if should file a bugzilla, I wasn't sure I had enough to go on to
>  > go ahead and do that yet.
>  > 
>  
>  Unfortunately doesn't sound like there is much to go on, maybe the kvm
>  maintainers can give some hints. Please file a bug against the kernel and CC me. 
Filed: https://bugzilla.redhat.com/show_bug.cgi?id=998713

FWIW, I completely reinstalled L0, applied all updates, set things up in a
similar fashion, and still get the same issue.

--
-- James Slagle
--

[Ian Pilcher]

On 08/19/2013 11:43 AM, James Slagle wrote:
>  cpu section of libvirt xml for L1 vm:
>    <cpu mode='custom' match='exact'>
>      <model fallback='allow'>Nehalem</model>
>      <feature policy='require' name='vmx'/>
>    </cpu> 
Once upon a time I had RHEV running with nested KVM on F18.  I haven't
tried it on F19, though.  The CPU section of the XML for one of the
hypervisors looks like this:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Conroe</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vmx'/>
  </cpu>

So maybe try setting the CPU type back to Conroe?

Also, what type (generation) of processor does L0 have?  I'm not sure
how well pre-SandyBridge CPUs handle this stuff.

HTH

-- 
========================================================================
Ian Pilcher                                         arequipeno(a)gmail.com
Sometimes there's nothing left to do but crash and burn...or die trying.
========================================================================

[James Slagle]

On Mon, Aug 19, 2013 at 01:33:16PM -0500, Ian Pilcher wrote:
>  On 08/19/2013 11:43 AM, James Slagle wrote:
>  > cpu section of libvirt xml for L1 vm:
>  >   <cpu mode='custom' match='exact'>
>  >     <model fallback='allow'>Nehalem</model>
>  >     <feature policy='require' name='vmx'/>
>  >   </cpu>
>  
>  Once upon a time I had RHEV running with nested KVM on F18.  I haven't
>  tried it on F19, though.  The CPU section of the XML for one of the
>  hypervisors looks like this:
>  
>    <cpu mode='custom' match='exact'>
>      <model fallback='allow'>Conroe</model>
>      <vendor>Intel</vendor>
>      <feature policy='require' name='vmx'/>
>    </cpu>
>  
>  So maybe try setting the CPU type back to Conroe? 
I tried that...no luck.

I've actually tried a few different processor types there, but that doesn't
seem to have any effect.

>  Also, what type (generation) of processor does L0 have?  I'm not
> sure
>  how well pre-SandyBridge CPUs handle this stuff. 
It's a Nehalem, E5506.

--
-- James Slagle
--

[Kashyap Chamarthy]

On 08/19/2013 10:13 PM, James Slagle wrote:
>  Hi,
>  
>  When I try to use nested KVM with Fedora 19, I'm experiencing a hang of the L1
>  vm.  I eventually have to do a "Force Off" from virt-manager on L0 since the
> L1
>  vm is completely unresponsive.
>  
>  Both my L0 and L1 are F19, x86_64.  L2 doesn't really seem to matter as I can't
>  even get the guest to start before L1 completely locks up.  Even just
>  attempting to define a L2 guest that uses kvm virtualization locks up L1
>  completely. 
If you're using a full-blown X, maybe you cay try a minimal server setup across L0,
L1,
and L2?

>  
>  Here are some more details.
>  
>  On L0 (i've rebooted L0 after enabling the nested parameter):
>  [root@dublin ~]# uname -a
>  Linux dublin 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45 UTC 2013 x86_64
>  x86_64 x86_64 GNU/Linux
>  [root@dublin ~]# cat /sys/module/kvm_intel/parameters/nested 
>  Y
>  [root@dublin ~]# rpm -q libvirt
>  libvirt-1.0.5.5-1.fc19.x86_64
>  [root@dublin ~]# lsmod | grep kvm
>  kvm_intel             138528  3 
>  kvm                   422809  1 kvm_intel
>  
>  cpu section of libvirt xml for L1 vm:
>    <cpu mode='custom' match='exact'>
>      <model fallback='allow'>Nehalem</model>
>      <feature policy='require' name='vmx'/>
>    </cpu> 
You can also optionally try to do a host CPU pass-through instead of explicitly exposing
the VMX instruction (although that should suffice in most cases).

 <cpu mode='host-passthrough'/>

>  
>  
>  On L1:
>  [root@localhost jslagle]# uname -a
>  Linux localhost.localdomain 3.10.7-200.fc19.x86_64 #1 SMP Thu Aug 15 23:19:45
>  UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>  [root@localhost jslagle]# rpm -q libvirt
>  libvirt-1.0.5.5-1.fc19.x86_64
>  [root@localhost jslagle]# lsmod | grep kvm
>  kvm_intel             138528  0 
>  kvm                   422809  1 kvm_intel
>  
>  
>  I've looked in dmesg output, /var/log/messages, and libvirt logs on both L0 and
>  L1 and can't see any errors.
>  
>  Any ideas on how to debug this further? 
Hm, I've been running some tests on latest Intel Hardware (Haswell), and I have both
L1,
and L2 running consistently fine with rawhide kernels.


Here's my set-up details:

  https://github.com/kashyapc/nvmx-haswell/blob/master/SETUP-nVMX.rst

Libvirt XML files for L1 and L2:

  https://github.com/kashyapc/nvmx-haswell/tree/master/tests/libvirt-xmls-f...


Hope that helps a bit.


/kashyap

[Richard W.M. Jones]

On Tue, Aug 20, 2013 at 11:16:18AM +0530, Kashyap Chamarthy wrote:
>  Hm, I've been running some tests on latest Intel Hardware
> (Haswell), and I have both L1,
>  and L2 running consistently fine with rawhide kernels. 
Big difference between Haswell and Nehalem (which is, what, 5+ years old?)

Nested VMX is hard enough as it is.  I would suggest to the reporter
using Sandybridge or later ... or AMD :-)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Kashyap Chamarthy]

On 08/20/2013 01:35 PM, Richard W.M. Jones wrote:
>  On Tue, Aug 20, 2013 at 11:16:18AM +0530, Kashyap Chamarthy wrote:
> > Hm, I've been running some tests on latest Intel Hardware (Haswell), and I have
> both L1,
> > and L2 running consistently fine with rawhide kernels.
>  
>  Big difference between Haswell and Nehalem (which is, what, 5+ years old?) 
:-) That's true.

>  
>  Nested VMX is hard enough as it is.  I would suggest to the reporter
>  using Sandybridge or later ... or AMD :-) 
Right, AMD has been the easiest to get it working. But I don't see many further
patches/improvements for it upstream.

However, a lot of Intel patches are posted upstream (I'm sure you must be watching),
where
Paolo also reported nEPT is working decently well for him. I'm trying to test it by
building kvm/queue (it's not yet in mainline KVM), but hitting some problems.
Let's see if
I can make any progress there.


/kashyap

[Gerd Hoffmann]

  Hi,

>  > Nested VMX is hard enough as it is.  I would suggest to the
> reporter
>  > using Sandybridge or later ... or AMD :-)
>  
>  Right, AMD has been the easiest to get it working. But I don't see many further
>  patches/improvements for it upstream. 
All the goodies vmx is getting now are implemented for svm since
years ...

cheers,
  Gerd

[James Slagle]

On Tue, Aug 20, 2013 at 11:16:18AM +0530, Kashyap Chamarthy wrote:
>  If you're using a full-blown X, maybe you cay try a minimal
> server setup across L0, L1,
>  and L2? 
Unfortunately I can't test that b/c I need a desktop on this system.

>  You can also optionally try to do a host CPU pass-through instead of
> explicitly exposing
>  the VMX instruction (although that should suffice in most cases).
>  
>   <cpu mode='host-passthrough'/> 
I tried this, but got the same issue.

>  Hm, I've been running some tests on latest Intel Hardware
> (Haswell), and I have both L1,
>  and L2 running consistently fine with rawhide kernels.
>  
>  
>  Here's my set-up details:
>  
>    https://github.com/kashyapc/nvmx-haswell/blob/master/SETUP-nVMX.rst
>  
>  Libvirt XML files for L1 and L2:
>  
>    https://github.com/kashyapc/nvmx-haswell/tree/master/tests/libvirt-xmls-f...
>  
>  
>  Hope that helps a bit. 
Since it's working for others, I guess it may very well be my older hardware.
Thanks for the pointers!

--
-- James Slagle
--