Cache=default gives appallingly awful disk performance

[Tom Horsley]

You might want to change the default virtio disk cache mode.

I was just doing some tests (after finally getting the latest
virtio scsi drivers installed correctly on my Windows XP virtual
machine).

The test was running the little "sdelete" tool which writes
a temp file full of zeroes to the disk in order to zero
out all the free space (very handy for converting the resulting
image to a maximally compressed qcow2 image).

It had just taken about 8 hours to do this on 20GB qcow2
image that has about 14GB unused and that seemed excessive,
so I thought I'd try different cache modes with updated
virtio drivers. My test was starting the sdelete run inside
the virtual machine and starting this on the host:

sleep 600 ; virsh destroy winxppro

That gives sdelete 10 minutes to grow the image file, then
I could ls -l the file to see how big it grew. With
the cache=default (which is apparently writethrough)
setting I get:

-rw-r--r-- 1 root root 423296512 Nov 13 16:27 winxppro.img

With cache=none I get:

-rw-r--r-- 1 root root 12813860864 Nov 13 16:44 winxppro.img

With cache=writeback I get:

-rw-r--r-- 1 root root 14574551040 Nov 13 16:58 winxppro.img

(which actually completed the whole sdelete zero fill process
in less than 10 minutes).

Needless to say, I have switched my KVM settings to writeback :-).

Default time to fill disk: 8 hours.
Writeback time to fill disk: less than 10 minutes.

That seems like just a bit of disparity. Since I was
using qcow2, I never expected really high performance, so
I didn't try different settings till now - I wish I had
tried it earlier :-).

[Avi Kivity]

On 11/14/2011 01:54 AM, Tom Horsley wrote:
>  You might want to change the default virtio disk cache mode.
>
>  I was just doing some tests (after finally getting the latest
>  virtio scsi drivers installed correctly on my Windows XP virtual
>  machine).
>
>  The test was running the little "sdelete" tool which writes
>  a temp file full of zeroes to the disk in order to zero
>  out all the free space (very handy for converting the resulting
>  image to a maximally compressed qcow2 image).
>  
This is a bad test.  Most likely the 'sdelete' utility insists on
overwriting the file multiple times (but strangely doesn't flush the
buffers); with cache=writeback these writes and overwrites hit the host
RAM and return immediately.  It's not the right workload to test
performance.

The recommendation is cache=none which gives good performance and cpu
utilization.  cache=writeback offers better performance, but it also
interferes with other workloads on the same host by filling up page
cache.  It's okay to use it, but understand that it isn't free.

-- 
error compiling committee.c: too many arguments to function

[Tom Horsley]

On Mon, 14 Nov 2011 19:41:38 +0200
Avi Kivity wrote:

>  Most likely the 'sdelete' utility insists on
>  overwriting the file multiple times 
Nope, I was only using it in the one pass
write lots of zeros mode (-c option).

In any case, even if it is a bad test, it
doesn't change the conclusion that the
disk performance with cache=default
is appallingly awful :-). Hours versus
minutes is insane.