gnome-boxes, high cpu when guest is idle

[Eric Phetteplace]

Hello,

Using Boxes, I installed a Windows 10 guest.  I installed the spice drivers from
spice-space.org/download
(https://www.spice-space.org/download/windows/spice-guest-tools/spice-gues...)
and things look good.

However, when the guest is idle, cpu is 0-4% the host process is significantly higher
(28.7% for example below).  I'm looking for advice on how to figure out what the
problem is.  The only thing I did so far is switch the disk bus from ide to virtio, and
that didn't help.

I have an XPS 13 with plenty of ram and SSD disk space - i5 processor, running Fedora 28,
4.17.7-200.fc28.x86_64

Here's some output from top:

top - 07:13:02 up 12:49,  1 user,  load average: 1.10, 1.84, 1.76
Tasks: 272 total,   1 running, 204 sleeping,   0 stopped,   0 zombie
%Cpu0  :  4.7 us,  2.7 sy,  0.0 ni, 91.6 id,  0.0 wa,  0.7 hi,  0.3 si,  0.0 st
%Cpu1  :  3.0 us,  4.4 sy,  0.0 ni, 92.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  4.3 us,  4.6 sy,  0.0 ni, 90.4 id,  0.0 wa,  0.3 hi,  0.3 si,  0.0 st
%Cpu3  :  3.3 us,  5.0 sy,  0.0 ni, 91.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16144924 total,   272940 free,  5217992 used, 10653992 buff/cache
KiB Swap:  8224764 total,  8087004 free,   137760 used. 10409752 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND               
                                                       
27942 ericp     20   0 3886324 2.130g  26340 S  28.7 13.8  27:26.31 qemu-system-x86

Reading other items online I used perf to see what is going on:

$ sudo perf kvm --host stat live

Analyze events for all VMs, all VCPUs:

             VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg
time 

      IO_INSTRUCTION        493    56.93%    67.91%      2.91us  99913.26us   4255.31us (
+-  17.70% )
         APIC_ACCESS        237    27.37%     7.71%      0.56us  99775.52us   1004.83us (
+-  51.23% )
                 HLT        106    12.24%    23.89%      0.59us  99841.70us   6961.72us (
+-  22.82% )
  EXTERNAL_INTERRUPT         19     2.19%     0.00%      1.52us      5.56us      2.62us (
+-   9.42% )
       EPT_MISCONFIG          6     0.69%     0.49%     28.66us  14837.17us   2506.17us (
+-  98.41% )
   PENDING_INTERRUPT          3     0.35%     0.00%      0.74us      4.78us      2.91us (
+-  40.38% )
 TPR_BELOW_THRESHOLD          2     0.23%     0.00%      4.21us      4.28us      4.24us (
+-   0.84% )

Total Samples:866, Total events handled time:3089057.95us.

Lots of IO and APIC.  
I was hoping the virtio bus change would take care of the high IO_INSTRUCTIONs, but it
didn't.
I changed the windows powercfg to be "high performance," but that didn't
reduce the APIC calls.

Here's another perf:

$ sudo perf stat -e 'kvm:*' -a -- sleep 1 
 Performance counter stats for 'system wide':
             8,750      kvm:kvm_fpu                                                 
             6,823      kvm:kvm_exit                                                
             6,823      kvm:kvm_entry                                               
             4,375      kvm:kvm_userspace_exit                                      
             4,340      kvm:kvm_pio                                                 
             1,576      kvm:kvm_apic                                                
             1,386      kvm:kvm_set_irq                                             
             1,386      kvm:kvm_pic_set_irq                                         
             1,386      kvm:kvm_ioapic_set_irq                                      
               920      kvm:kvm_ple_window                                          
               866      kvm:kvm_mmio                                                
               847      kvm:kvm_emulate_insn                                        
               780      kvm:kvm_inj_virq                                            
               780      kvm:kvm_eoi                                                 
               780      kvm:kvm_apic_accept_irq                                     
               715      kvm:kvm_vcpu_wakeup                                         
               715      kvm:kvm_hv_timer_state                                      
               317      kvm:kvm_apic_ipi                                            
               177      kvm:kvm_halt_poll_ns                                        
               148      kvm:kvm_ack_irq                                             
                35      kvm:vcpu_match_mmio                                         

Lots of fpu ?!?!?  Lots of userspace exit.  Lots of userspace, and looking at the top, the
processors are doing an equal amount of usr and sys.  I figure something is constantly
doing system calls, causing lots of context switching.

I did "ps -Alf" to get the command line it creates.  Hopefully this helps.  I
can provide the boxes config file if that's easier.

/usr/bin/qemu-system-x86_64
 -machine accel=kvm
 -name guest=win10,debug-threads=on
 -S
 -object
secret,id=masterKey0,format=raw,file=/home/ericp/.config/libvirt/qemu/lib/domain-1-win10/master-key.aes
 -machine pc-i440fx-2.11,accel=kvm,usb=off,dump-guest-core=off
 -cpu Broadwell-noTSX-IBRS
 -m 2048
 -realtime mlock=off
 -smp 4,sockets=1,cores=2,threads=2
 -uuid c168275d-7cb1-4e98-8c66-6f51c3d0beb6
 -no-user-config
 -nodefaults
 -chardev
socket,id=charmonitor,path=/home/ericp/.config/libvirt/qemu/lib/domain-1-win10/monitor.sock,server,nowait
 -mon chardev=charmonitor,id=monitor,mode=control
 -rtc base=localtime,driftfix=slew
 -global kvm-pit.lost_tick_policy=delay
 -no-hpet
 -no-shutdown
 -global PIIX4_PM.disable_s3=1
 -global PIIX4_PM.disable_s4=1
 -boot strict=on
 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7
 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6
 -device usb-ccid,id=ccid0,bus=usb.0,port=2
 -device usb-hub,id=hub0,bus=usb.0,port=1
 -drive
file=/home/ericp/.local/share/gnome-boxes/images/win10,format=qcow2,if=none,id=drive-virtio-disk0,cache=writeback
 -device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
 -drive if=none,id=drive-ide0-1-0,readonly=on
 -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
 -netdev tap,fd=25,id=hostnet0
 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7f:55:17,bus=pci.0,addr=0x3
 -chardev spicevmc,id=charsmartcard0,name=smartcard
 -device ccid-card-passthru,chardev=charsmartcard0,id=smartcard0,bus=ccid0.0
 -chardev pty,id=charserial0
 -device isa-serial,chardev=charserial0,id=serial0
 -chardev spicevmc,id=charchannel0,name=vdagent
 -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
 -chardev spiceport,id=charchannel1,name=org.spice-space.webdav.0
 -device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.spice-space.webdav.0
 -device usb-tablet,id=input0,bus=usb.0,port=3
 -device usb-mouse,id=input1,bus=usb.0,port=4
 -device usb-kbd,id=input2,bus=usb.0,port=5
 -spice port=0,disable-ticketing,image-compression=off,seamless-migration=on
 -device virtio-vga,id=video0,max_outputs=1,bus=pci.0,addr=0x2
 -device intel-hda,id=sound0,bus=pci.0,addr=0x4
 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
 -chardev spicevmc,id=charredir0,name=usbredir
 -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=6
 -chardev spicevmc,id=charredir1,name=usbredir
 -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=1.1
 -chardev spicevmc,id=charredir2,name=usbredir
 -device usb-redir,chardev=charredir2,id=redir2,bus=usb.0,port=1.2
 -chardev spicevmc,id=charredir3,name=usbredir
 -device usb-redir,chardev=charredir3,id=redir3,bus=usb.0,port=1.3
 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8
 -msg timestamp=on

Any suggestions on what to do?

[Eric Phetteplace]

I've learned two more things:
1. I installed an ubuntu guest OS, and it has the same high host cpu when the guest is
idle.
2. using strace, I see this is 97% of the system calls:
ppoll([{fd=3, events=POLLIN}, {fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=8,
events=POLLIN}, {fd=13, events=POLLIN}, {fd=25, events=POLLIN}, {fd=26, events=POLLIN},
{fd=27, events=POLLIN}, {fd=30, events=POLLIN}, {fd=31, events=POLLIN}, {fd=32,
events=POLLIN}, {fd=33, events=POLLIN}, {fd=34, events=POLLIN}, {fd=35, events=POLLIN},
{fd=36, events=POLLIN}, {fd=37, events=POLLIN}, {fd=38, events=POLLIN}, {fd=39,
events=POLLIN}, {fd=40, events=POLLIN}, {fd=41, events=POLLIN}, {fd=42, events=POLLIN},
{fd=43, events=POLLIN}, {fd=44, events=POLLIN}, {fd=45, events=POLLIN}, {fd=46,
events=POLLIN}, {fd=47, events=POLLIN}, {fd=48, events=POLLIN}, {fd=49, events=POLLIN},
{fd=50, events=POLLIN}, {fd=51, events=POLLIN}, {fd=52, events=POLLIN}, {fd=53,
events=POLLIN}, ...], 84, {tv_sec=0, tv_nsec=0}, NULL, 8) = 0 (Timeout)

[Eric Phetteplace]

making progress.  If I start the VM using Virtual Machine Manager (virt-manager) the cpu
is noticeably lower.  ppoll seems to be lower, too.
Comparing the outputs from ps -lf I see some differences I can test...

[Eric Phetteplace]

Solved

I removed     
<smartcard mode='passthrough' type='spicevmc'>                      
       
      <address type='ccid' controller='0' slot='0'/>        
                   
</smartcard>                                                                

and the cpu usage dropped to match the guest.

FWIW, to track this down, I created a Fedora server using virt-manager and another with
Boxes and reproduced the issue.  I compared their ps -lf outputs and tried removing the
extra components, and removing smartcard made the difference.

[Cole Robinson]

On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
>  Solved
>  
>  I removed
>  <smartcard mode='passthrough' type='spicevmc'>
>         <address type='ccid' controller='0' slot='0'/>
>  </smartcard>
>  
>  and the cpu usage dropped to match the guest.
>  
>  FWIW, to track this down, I created a Fedora server using virt-manager and another with
> Boxes and reproduced the issue.  I compared their ps -lf outputs and tried removing the
> extra components, and removing smartcard made the difference. 
Thanks for tracking this down Eric. Can you file a bug against 
Product=Fedora component=qemu in  bugzilla.redhat.com so we don't lose 
these details?

Thanks,
Cole

[Eric Phetteplace]

Will do, tonight.


Eric
________________________________
From: Cole Robinson <crobinso(a)redhat.com&gt;
Sent: Tuesday, August 7, 2018 2:11:05 PM
To: Community support for Fedora users; Eric Phetteplace
Subject: Re: gnome-boxes, high cpu when guest is idle

On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
>  Solved
>
>  I removed
>  <smartcard mode='passthrough' type='spicevmc'>
>         <address type='ccid' controller='0' slot='0'/>
>  </smartcard>
>
>  and the cpu usage dropped to match the guest.
>
>  FWIW, to track this down, I created a Fedora server using virt-manager and another with
> Boxes and reproduced the issue.  I compared their ps -lf outputs and tried removing the
> extra components, and removing smartcard made the difference. 
Thanks for tracking this down Eric. Can you file a bug against
Product=Fedora component=qemu in  bugzilla.redhat.com so we don't lose
these details?

Thanks,
Cole

[Eric Phetteplace]

https://bugzilla.redhat.com/show_bug.cgi?id=1613990


Eric
________________________________
From: Cole Robinson <crobinso(a)redhat.com&gt;
Sent: Tuesday, August 7, 2018 2:11:05 PM
To: Community support for Fedora users; Eric Phetteplace
Subject: Re: gnome-boxes, high cpu when guest is idle

On 08/01/2018 08:56 AM, Eric Phetteplace wrote:
>  Solved
>
>  I removed
>  <smartcard mode='passthrough' type='spicevmc'>
>         <address type='ccid' controller='0' slot='0'/>
>  </smartcard>
>
>  and the cpu usage dropped to match the guest.
>
>  FWIW, to track this down, I created a Fedora server using virt-manager and another with
> Boxes and reproduced the issue.  I compared their ps -lf outputs and tried removing the
> extra components, and removing smartcard made the difference. 
Thanks for tracking this down Eric. Can you file a bug against
Product=Fedora component=qemu in  bugzilla.redhat.com so we don't lose
these details?

Thanks,
Cole