[fedora-virt] Idle VM is hogging host cpu

[Dennis Jacobfeuerborn]

On 11/08/2010 04:06 PM, Daniel P. Berrange wrote:
> On Mon, Nov 08, 2010 at 03:55:14PM +0100, Dennis Jacobfeuerborn wrote:
>> On 11/08/2010 02:22 PM, Sven Lankes wrote:
>>> On Mon, Nov 08, 2010 at 12:38:22PM +0000, Daniel P. Berrange wrote:
>>>
>>>> Try adding   divider=10  to the kernel boot command line for the guest.
>>>> This will make it use a 100HZ tick rate, instead of 1000HZ, lowering the
>>>> background idle load on the host. I doubt it accounts for 20%, but it is
>>>> always worth doing for RHEL5 kernels since they don't have tickless mode.
>>>
>>> I was seeing 10 - 15% cpu for the kvm process too. After adding
>>> divider=10 this dropped to 1%.
>>
>> The fact that the guest needs special kernel parameters to run properly
>> doesn't look right to me. Is there no way for KVM to handle this correctly?
>> Also I just imported this VM from my old Fedora 11 system on the new Fedora
>> 14 install and there I didn't see this behavior.
>
> KVM can't force the guest OS to use a particular timer rate. It is
> simply unfortunate luck that some OS choose to use a 1000HZ timer
> forcing KVM to spend alot of CPU time injecting interrupts. This
> same problem also affects Xen HVM, VMWare and any other full machine
> virtualization platform. Tickless guest kernels are the only solution
> to this problem, and that's not something KVM can magically make
> happen.

I get that but why wasn't this affecting F11 as well? AFAIK Fedora 11 had a 
tickless kernel so something else clearly changed between F11 and F14 that 
causes this.

> Here's what VMWare has to say on the matter, and this applies to
> KVM too
>
>    "Note for uses on divider=10
>     For some Operating Systems, divider=10 is a supported kernel
>     configuration option, but might not be necessary for accurate
>     timekeeping. Using it reduces the frequency of timer interrupts
>     by 10x, which reduces the CPU overhead of processing timer
>     interrupts. This overhead is especially noticeable for idle
>     virtual machines. The only drawback of using divider=10 is
>     that the granularity of wakeups provided by the kernel changes
>     from 1ms to 10ms. The vast majority of applications are not
>     affected by this, but using divider=10 may not be the right
>     tradeoff for some time sensitive applications. For some Operating
>     Systems, specifically older versions, divider=10 greatly improves
>     timekeeping accuracy and is strongly recommended."
>
>    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
>
>
>> BTW this might or might not be related to a general problem with the kernel
>> in Fedora 14:
>> https://bugzilla.redhat.com/show_bug.cgi?id=650934
>> https://bugzilla.redhat.com/show_bug.cgi?id=635813
>>
>> Anyway, I'm off trying to boot my system with nohz=off and then add the
>> divider thing to the guest to see if that has any impact.
>
> If your kernel supports 'nohz' then, the 'divider' flag will not exist,
> and vica-verca. The 'divider=XX'  arg is a special flag added only to
> RHEL4/5 kernels which do not support tickless to allow their timer rate
> to be decreased.
>
> RHEL-3 had a 100HZ timer so wasn't impacted, and RHEL6 is fully tickless
> so also isn't impacted. All currently supported Fedora are also tickless

The "nohz" was for the host and "divider" for the guest. Here are the 
results: After booting the host with "nohz=off" the qemu process takes 
about 10% rather than 20% and once I add "divider=10" to the guest kernel 
parameters that number goes down to 2%. So this clearly seems to help but 
it doesn't explain why the same guest ran fine on F11 without any 
additional kernel options whatsoever.

Regards,
   Dennis