9 p.m.
Hi,
I just migrated from Fedora 11 to Fedora 14 (via a fresh install) and so
far everything seems fine kvm seems to have major issues.
After starting one Centos 5 VM qemu-kvm keeps using 20% of the cpu
according to top even though the VM is completely idle. Does anyone have a
clue what might cause this because I wasn't seeing this behavior under
Fedora 11.
Regards,
Dennis
6:07 a.m.
On Mon, Nov 08, 2010 at 04:00:09AM +0100, Dennis Jacobfeuerborn wrote:
Hi,
I just migrated from Fedora 11 to Fedora 14 (via a fresh install) and so
far everything seems fine kvm seems to have major issues.
After starting one Centos 5 VM qemu-kvm keeps using 20% of the cpu
according to top even though the VM is completely idle. Does anyone have a
clue what might cause this because I wasn't seeing this behavior under
Fedora 11.
What command line arguments are we giving to qemu-kvm?
Interesting ones would be the '-M' option and any virtio and
pv-clock related ones, but it's useful to include the whole
command line here.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
6:17 a.m.
On Mon, Nov 08, 2010 at 12:07:33PM +0000, Richard W.M. Jones wrote:
What command line arguments are we giving to qemu-kvm?
Interesting ones would be the '-M' option and any virtio and
pv-clock related ones, but it's useful to include the whole
command line here.
This is a standard-cmdline of a centos5 vm including virtio net and disk
running on F14:
/usr/bin/qemu-kvm
-S -M pc-0.11
-enable-kvm -m 1024
-smp 2,sockets=2,cores=1,threads=1
-name centos -uuid 2daab0e5-e1ea-cee0-b1d5- d6cbac46d4ec -nodefconfig
-nodefaults -chardev socket,id=monitor,path=/-
var/lib/libvirt/qemu/centos.monitor,server,nowait
-mon chardev=monitor,mode=readline -rtc base=utc -boot c
-drive file=/var/lib/libvirt/images/centos.img,if=none,id=drive-virtio-
disk0,boot=on,format=raw
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio- disk0
-drive if=none,media=cdrom,id=drive-ide0-1- 0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1- 0
-device virtio-net- pci,vlan=0,id=net0,mac=52:54:00:12:00:e1,bus=pci.0,addr=0x5
-net tap,fd=45,vlan=0,name=hostnet0 -chardev pty,id=serial0
-device isa- serial,chardev=serial0 -usb -vnc 127.0.0.1:1 -k en-us
-vga cirrus - device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
--
sven === jabber/xmpp: sven(a)lankes.net
6:33 a.m.
On Mon, Nov 08, 2010 at 04:00:09AM +0100, Dennis Jacobfeuerborn wrote:
Hi,
I just migrated from Fedora 11 to Fedora 14 (via a fresh install) and so
far everything seems fine kvm seems to have major issues.
After starting one Centos 5 VM qemu-kvm keeps using 20% of the cpu
according to top even though the VM is completely idle. Does anyone have a
clue what might cause this because I wasn't seeing this behavior under
Fedora 11.
Another interesting thing for you to post would be precisely what
version of CentOS the guest is running. Guest kernel? Is the guest
using pv-clock[1]? Is it using virtio drivers?
Rich.
[1] cat /sys/devices/system/clocksource/clocksource0/current_clocksource
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw
6:38 a.m.
On Mon, Nov 08, 2010 at 04:00:09AM +0100, Dennis Jacobfeuerborn wrote:
Hi,
I just migrated from Fedora 11 to Fedora 14 (via a fresh install) and so
far everything seems fine kvm seems to have major issues.
After starting one Centos 5 VM qemu-kvm keeps using 20% of the cpu
according to top even though the VM is completely idle. Does anyone have a
clue what might cause this because I wasn't seeing this behavior under
Fedora 11.
Try adding divider=10 to the kernel boot command line for the guest.
This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
background idle load on the host. I doubt it accounts for 20%, but it is
always worth doing for RHEL5 kernels since they don't have tickless mode.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
7:22 a.m.
On Mon, Nov 08, 2010 at 12:38:22PM +0000, Daniel P. Berrange wrote:
Try adding divider=10 to the kernel boot command line for the
guest.
This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
background idle load on the host. I doubt it accounts for 20%, but it is
always worth doing for RHEL5 kernels since they don't have tickless mode.
I was seeing 10 - 15% cpu for the kvm process too. After adding
divider=10 this dropped to 1%.
--
sven === jabber/xmpp: sven(a)lankes.net
8:55 a.m.
On 11/08/2010 02:22 PM, Sven Lankes wrote:
On Mon, Nov 08, 2010 at 12:38:22PM +0000, Daniel P. Berrange wrote:
> Try adding divider=10 to the kernel boot command line for the guest.
> This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
> background idle load on the host. I doubt it accounts for 20%, but it is
> always worth doing for RHEL5 kernels since they don't have tickless mode.
I was seeing 10 - 15% cpu for the kvm process too. After adding
divider=10 this dropped to 1%.
The fact that the guest needs special kernel parameters to run properly
doesn't look right to me. Is there no way for KVM to handle this correctly?
Also I just imported this VM from my old Fedora 11 system on the new Fedora
14 install and there I didn't see this behavior.
BTW this might or might not be related to a general problem with the kernel
in Fedora 14:
https://bugzilla.redhat.com/show_bug.cgi?id=650934
https://bugzilla.redhat.com/show_bug.cgi?id=635813
Anyway, I'm off trying to boot my system with nohz=off and then add the
divider thing to the guest to see if that has any impact.
Regards,
Dennis
9:06 a.m.
On Mon, Nov 08, 2010 at 03:55:14PM +0100, Dennis Jacobfeuerborn wrote:
On 11/08/2010 02:22 PM, Sven Lankes wrote:
> On Mon, Nov 08, 2010 at 12:38:22PM +0000, Daniel P. Berrange wrote:
>
>> Try adding divider=10 to the kernel boot command line for the guest.
>> This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
>> background idle load on the host. I doubt it accounts for 20%, but it is
>> always worth doing for RHEL5 kernels since they don't have tickless mode.
>
> I was seeing 10 - 15% cpu for the kvm process too. After adding
> divider=10 this dropped to 1%.
The fact that the guest needs special kernel parameters to run properly
doesn't look right to me. Is there no way for KVM to handle this correctly?
Also I just imported this VM from my old Fedora 11 system on the new Fedora
14 install and there I didn't see this behavior.
KVM can't force the guest OS to use a particular timer rate. It is
simply unfortunate luck that some OS choose to use a 1000HZ timer
forcing KVM to spend alot of CPU time injecting interrupts. This
same problem also affects Xen HVM, VMWare and any other full machine
virtualization platform. Tickless guest kernels are the only solution
to this problem, and that's not something KVM can magically make
happen.
Here's what VMWare has to say on the matter, and this applies to
KVM too
"Note for uses on divider=10
For some Operating Systems, divider=10 is a supported kernel
configuration option, but might not be necessary for accurate
timekeeping. Using it reduces the frequency of timer interrupts
by 10x, which reduces the CPU overhead of processing timer
interrupts. This overhead is especially noticeable for idle
virtual machines. The only drawback of using divider=10 is
that the granularity of wakeups provided by the kernel changes
from 1ms to 10ms. The vast majority of applications are not
affected by this, but using divider=10 may not be the right
tradeoff for some time sensitive applications. For some Operating
Systems, specifically older versions, divider=10 greatly improves
timekeeping accuracy and is strongly recommended."
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&...
BTW this might or might not be related to a general problem with the
kernel
in Fedora 14:
https://bugzilla.redhat.com/show_bug.cgi?id=650934
https://bugzilla.redhat.com/show_bug.cgi?id=635813
Anyway, I'm off trying to boot my system with nohz=off and then add the
divider thing to the guest to see if that has any impact.
If your kernel supports 'nohz' then, the 'divider' flag will not exist,
and vica-verca. The 'divider=XX' arg is a special flag added only to
RHEL4/5 kernels which do not support tickless to allow their timer rate
to be decreased.
RHEL-3 had a 100HZ timer so wasn't impacted, and RHEL6 is fully tickless
so also isn't impacted. All currently supported Fedora are also tickless
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
9:30 a.m.
On 11/08/2010 04:06 PM, Daniel P. Berrange wrote:
On Mon, Nov 08, 2010 at 03:55:14PM +0100, Dennis Jacobfeuerborn
wrote:
> On 11/08/2010 02:22 PM, Sven Lankes wrote:
>> On Mon, Nov 08, 2010 at 12:38:22PM +0000, Daniel P. Berrange wrote:
>>
>>> Try adding divider=10 to the kernel boot command line for the guest.
>>> This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
>>> background idle load on the host. I doubt it accounts for 20%, but it is
>>> always worth doing for RHEL5 kernels since they don't have tickless
mode.
>>
>> I was seeing 10 - 15% cpu for the kvm process too. After adding
>> divider=10 this dropped to 1%.
>
> The fact that the guest needs special kernel parameters to run properly
> doesn't look right to me. Is there no way for KVM to handle this correctly?
> Also I just imported this VM from my old Fedora 11 system on the new Fedora
> 14 install and there I didn't see this behavior.
KVM can't force the guest OS to use a particular timer rate. It is
simply unfortunate luck that some OS choose to use a 1000HZ timer
forcing KVM to spend alot of CPU time injecting interrupts. This
same problem also affects Xen HVM, VMWare and any other full machine
virtualization platform. Tickless guest kernels are the only solution
to this problem, and that's not something KVM can magically make
happen.
I get that but why wasn't this affecting F11 as well? AFAIK Fedora 11 had a
tickless kernel so something else clearly changed between F11 and F14 that
causes this.
Here's what VMWare has to say on the matter, and this applies to
KVM too
"Note for uses on divider=10
For some Operating Systems, divider=10 is a supported kernel
configuration option, but might not be necessary for accurate
timekeeping. Using it reduces the frequency of timer interrupts
by 10x, which reduces the CPU overhead of processing timer
interrupts. This overhead is especially noticeable for idle
virtual machines. The only drawback of using divider=10 is
that the granularity of wakeups provided by the kernel changes
from 1ms to 10ms. The vast majority of applications are not
affected by this, but using divider=10 may not be the right
tradeoff for some time sensitive applications. For some Operating
Systems, specifically older versions, divider=10 greatly improves
timekeeping accuracy and is strongly recommended."
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&...
> BTW this might or might not be related to a general problem with the kernel
> in Fedora 14:
> https://bugzilla.redhat.com/show_bug.cgi?id=650934
> https://bugzilla.redhat.com/show_bug.cgi?id=635813
>
> Anyway, I'm off trying to boot my system with nohz=off and then add the
> divider thing to the guest to see if that has any impact.
If your kernel supports 'nohz' then, the 'divider' flag will not exist,
and vica-verca. The 'divider=XX' arg is a special flag added only to
RHEL4/5 kernels which do not support tickless to allow their timer rate
to be decreased.
RHEL-3 had a 100HZ timer so wasn't impacted, and RHEL6 is fully tickless
so also isn't impacted. All currently supported Fedora are also tickless
The "nohz" was for the host and "divider" for the guest. Here are the
results: After booting the host with "nohz=off" the qemu process takes
about 10% rather than 20% and once I add "divider=10" to the guest kernel
parameters that number goes down to 2%. So this clearly seems to help but
it doesn't explain why the same guest ran fine on F11 without any
additional kernel options whatsoever.
Regards,
Dennis
9:40 a.m.
On Mon, Nov 08, 2010 at 04:30:37PM +0100, Dennis Jacobfeuerborn wrote:
I get that but why wasn't this affecting F11 as well? AFAIK
Fedora 11 had a
tickless kernel so something else clearly changed between F11 and F14 that
causes this.
Dan is talking about the guest kernel, not the host kernel.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
10:14 a.m.
On 11/08/2010 04:40 PM, Richard W.M. Jones wrote:
On Mon, Nov 08, 2010 at 04:30:37PM +0100, Dennis Jacobfeuerborn
wrote:
> I get that but why wasn't this affecting F11 as well? AFAIK Fedora 11 had a
> tickless kernel so something else clearly changed between F11 and F14 that
> causes this.
Dan is talking about the guest kernel, not the host kernel.
Maybe there is a misunderstanding here.
What I'm saying is that the same guest is running on tickless kernels in
both cases (F11 and F14) yet on the F14 kernel it uses 20% cpu whereas it
uses just 1-2% on F11. So this leaves two questions:
What else besides tickless-ness in the host kernel is causing that the
identical guest uses much more cpu resources on the f14 kernel.
Why is the "divider=10" for the guest necessary on f14 but not on f11?
Regards,
Dennis
10:23 a.m.
On Mon, Nov 08, 2010 at 05:14:33PM +0100, Dennis Jacobfeuerborn wrote:
What I'm saying is that the same guest is running on tickless
kernels in both cases (F11 and F14) yet on the F14 kernel it uses
20% cpu whereas it uses just 1-2% on F11. So this leaves two
questions:
What else besides tickless-ness in the host kernel is causing that
the identical guest uses much more cpu resources on the f14 kernel.
Why is the "divider=10" for the guest necessary on f14 but not on f11?
The point is that it is the tickless kernel in the *guest* which
matters here. By having either a tickless guest kernel or using
divider=10 (for guest kernels which cannot support tickless), KVM has
to inject fewer timer interrupts into the guest.
A guest which is "idle" isn't really idle if it is demanding that the
hypervisor sends it a timer interrupt 1000 times a second. You can
either reduce that number (divider=10 => 100 times a second), or you
make it so you only need a timer interrupt when there is something you
need to schedule (tickless).
In any case the host can't control what the guest requests.
Why it is that Fedora 11 does so much better than Fedora 14 with these
idle guests is an entirely separate question. Please post the
qemu-kvm command line from both cases.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
6:48 p.m.
On 11/08/2010 05:23 PM, Richard W.M. Jones wrote:
On Mon, Nov 08, 2010 at 05:14:33PM +0100, Dennis Jacobfeuerborn
wrote:
> What I'm saying is that the same guest is running on tickless
> kernels in both cases (F11 and F14) yet on the F14 kernel it uses
> 20% cpu whereas it uses just 1-2% on F11. So this leaves two
> questions:
>
> What else besides tickless-ness in the host kernel is causing that
> the identical guest uses much more cpu resources on the f14 kernel.
>
> Why is the "divider=10" for the guest necessary on f14 but not on f11?
The point is that it is the tickless kernel in the *guest* which
matters here. By having either a tickless guest kernel or using
divider=10 (for guest kernels which cannot support tickless), KVM has
to inject fewer timer interrupts into the guest.
A guest which is "idle" isn't really idle if it is demanding that the
hypervisor sends it a timer interrupt 1000 times a second. You can
either reduce that number (divider=10 => 100 times a second), or you
make it so you only need a timer interrupt when there is something you
need to schedule (tickless).
In any case the host can't control what the guest requests.
Why it is that Fedora 11 does so much better than Fedora 14 with these
idle guests is an entirely separate question. Please post the
qemu-kvm command line from both cases.
Not that the attached measurements were done booting the VM with the
"single" option and without the "divider=10" option in the guest. On
the
Fedora 14 host I removed the "nohz=off" option (which wasn't present in the
F11 case in the first place). This should result in a comparison among equals.
(I edited the "top" header back in after it got remove by the grep for
readability)
Regards,
Dennis
4915
days inactive
4917
days old
12 comments
4 participants
participants (4)
-
Daniel P. Berrange -
Dennis Jacobfeuerborn -
Richard W.M. Jones -
Sven Lankes