Another weird KVM glitch

[Tom Horsley]

I've been looking at a few days worth of testbed results,
and rarely (but more than once) a fantastically weird failure
happens:

Some random process (bash, gcc, etc) will die with a
Trace/Breakpoint trap reported as the reason for the abort.

I am running tests of debugging on most of the virtual
machines, but I'm not debugging these apps, I'm debugging
other stuff on other virtual machines.

It is almost like the emulation code is looking at stale
register values when reporting a breakpoint or address
trap and directing it to the wrong process. So far, I've
never seen this on my Xeon based host, only on my Opteron
based host.

There are 8 CPUs on this thing. I wonder if there is
a missing lock somewhere in some infrequently executed
code path.

[Avi Kivity]

On 12/20/2009 05:10 PM, Tom Horsley wrote:
>  I've been looking at a few days worth of testbed results,
>  and rarely (but more than once) a fantastically weird failure
>  happens:
>
>  Some random process (bash, gcc, etc) will die with a
>  Trace/Breakpoint trap reported as the reason for the abort.
>
>  I am running tests of debugging on most of the virtual
>  machines, but I'm not debugging these apps, I'm debugging
>  other stuff on other virtual machines.
>
>  It is almost like the emulation code is looking at stale
>  register values when reporting a breakpoint or address
>  trap and directing it to the wrong process. So far, I've
>  never seen this on my Xeon based host, only on my Opteron
>  based host.
>
>  There are 8 CPUs on this thing. I wonder if there is
>  a missing lock somewhere in some infrequently executed
>  code path.
>      
kvm tries very hard to avoid switching cpu registers when it is 
expensive to do so; it's possible it tries too hard sometimes.

Can you give a reliable scenario for triggering this?

What host kernel are you running?


-- 
error compiling committee.c: too many arguments to function

[Tom Horsley]

>  > It is almost like the emulation code is looking at stale
>  > register values when reporting a breakpoint or address
>  > trap and directing it to the wrong process. So far, I've
>  > never seen this on my Xeon based host, only on my Opteron
>  > based host.
>  >
>  > There are 8 CPUs on this thing. I wonder if there is
>  > a missing lock somewhere in some infrequently executed
>  > code path.
>  
>  kvm tries very hard to avoid switching cpu registers when it is 
>  expensive to do so; it's possible it tries too hard sometimes.
>  
>  Can you give a reliable scenario for triggering this? 
It is pretty rare and pretty random. I have my testbed driver
randomly selecting VMs to test on and running regression tests
for my debugger on the randomly selected machines. It is booting
up and shutting down VMs as it does this, and the tests themselves
are doing all kinds of stuff besides just running debugger (like
compiling programs, and executing the shell scripts involved), so
it would be hard to come up with a reliable way to reproduce it :-).
I think I've only noticed it about 3 times in the (not quite) week
since I've had this running full blast.

>  What host kernel are you running? 
The host is a fully up to date fedora 12 x86_64 system (currently
running 2.6.31.6-166.fc12.x86_64).