virtio cache mode and windows XP bluescreens

[Tom Horsley]

I saw a thread a month or so ago on disk performance, and
just for grins, I thought I'd try different disk cache settings
for my Windows XP virtual machine that has a qcow2 image
file (I know qcow2 is never going to be really fast, but I
thought I'd see if I could get any improvement at all).

Every cache mode I tried (changing them in the advanced
options of the hardware viewer screen in virt-manager) resulted
in the XP machine getting into a bluescreen loop when I booted
it.

Finally set it back to "default" and the XP machine stopped
getting bluescreens.

Are other cache modes supposed to work OK? Should I
report a bug on some component? Should I try and capture
the bluescreen content? (It flickers in and out really fast).

The main reason I was trying this is because it sure seems
like the qcow2 image gets slower and slower every time I
upgrade fedora and get new and improved qemu and libvirt
code. I don't have solid benchmarks, but it sure feels
slower. Has optimization of cases that are supposed to be fast
slowed down qcow2 somehow?

I'm currently trying to build an updated base image by
doing the trick of writing a giant file of zeros to fill
up the disk so I can then convert the mostly zero image to
a new qcow2 file, and the rate it is going makes it seem like
it may well be tomorrow before it finishes writing the
10 GB or so of zeros it needs to finish filling up the
disk image.

None of my CPUs are showing over 1% utilization and gkrellm
is showing only 2.0M disk write rate, so it sure seems like
all the components involved are spending more time just
waiting than anything else.

[Dor Laor]

On 11/13/2011 12:29 AM, Tom Horsley wrote:
>  I saw a thread a month or so ago on disk performance, and
>  just for grins, I thought I'd try different disk cache settings
>  for my Windows XP virtual machine that has a qcow2 image
>  file (I know qcow2 is never going to be really fast, but I
>  thought I'd see if I could get any improvement at all).
>
>  Every cache mode I tried (changing them in the advanced
>  options of the hardware viewer screen in virt-manager) resulted
>  in the XP machine getting into a bluescreen loop when I booted
>  it.
>
>  Finally set it back to "default" and the XP machine stopped
>  getting bluescreens.
>
>  Are other cache modes supposed to work OK? Should I
>  report a bug on some component? Should I try and capture
>  the bluescreen content? (It flickers in and out really fast).
>
>  The main reason I was trying this is because it sure seems
>  like the qcow2 image gets slower and slower every time I
>  upgrade fedora and get new and improved qemu and libvirt
>  code. I don't have solid benchmarks, but it sure feels
>  slower. Has optimization of cases that are supposed to be fast
>  slowed down qcow2 somehow?
>
>  I'm currently trying to build an updated base image by
>  doing the trick of writing a giant file of zeros to fill
>  up the disk so I can then convert the mostly zero image to
>  a new qcow2 file, and the rate it is going makes it seem like
>  it may well be tomorrow before it finishes writing the
>  10 GB or so of zeros it needs to finish filling up the
>  disk image.
>
>  None of my CPUs are showing over 1% utilization and gkrellm
>  is showing only 2.0M disk write rate, so it sure seems like
>  all the components involved are spending more time just
>  waiting than anything else. 
We need more details - guest/host versions, whether you use virtio or 
ide, in case of virtio, what's the win pv version, etc.

cache=off and cache=writeothrough are safe and pretty much bugfree so I 
doubt the root cause is there. In addition, latest qcow2 code should 
perform relatively good, so please provide more details

>  _______________________________________________
>  virt mailing list
>  virt(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/virt

[Tom Horsley]

>  > None of my CPUs are showing over 1% utilization and gkrellm
>  > is showing only 2.0M disk write rate, so it sure seems like
>  > all the components involved are spending more time just
>  > waiting than anything else.
>  
>  We need more details - guest/host versions, whether you use virtio or 
>  ide, in case of virtio, what's the win pv version, etc. 
Here's the disk in the KVM xml definition:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/zooty/images/winxppro.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/>
    </disk>

So I'm using virtio. I tried mounting the virtio-win-0.1-15.iso in
the CDROM and asking XP to upgrade the disk driver, but it said the
on on the CD was not newer and did no upgrade. (But see below - maybe
I should have been upgrading the SCSI controller driver instead?)

But looking at the device manager in the XP machine gives me
pause. The Disk drives themselves say they are Redhat VirtIO SCSI disks
but are using a Microsoft provided driver 5.1.2535.0 dated 7/1/2001.
Should the disk drives themselves be on a different driver?

The SCSI controller inside the XP machine says Redhat VirtIO SCSI
and has a driver 5.3.0.17241 dated 10/10/2009.

That seems suspiciously old.

Also the disk image is probably the slowest possible image.
It is a qcow2 image, with a qcow2 backing file (so I can clone the
machine with different copies from the same backing file).

The write speed I calculated with "sdelete -c" running inside the
VM filling up the image file with zeroes was:

-rw-r--r-- 1 qemu qemu 3538087424 Nov 12 16:48 winxppro.img

-rw-r--r-- 1 qemu qemu 9181790208 Nov 12 19:49 winxppro.img

3 hours managed to write 5643702784 bytes
31,353,904.36 bytes per minute
522,565.07 bytes per second

It took about 8 hours to fill the unused 16 GB of the 20GB file
image with zeroes :-). I swear the last time I did it (probably around
6 months ago) it only took 5 hours (still pretty slow, mind you).

>  cache=off and cache=writeothrough are safe and pretty much bugfree so
> I 
>  doubt the root cause is there. In addition, latest qcow2 code should 
>  perform relatively good, so please provide more details 
It would not surprise me to find I have something misconfigured.

Also I note that the disk drive properties inside windows
has some performance settings available:

  ( ) Optimize for quick removal

  (*) Optimize for performance

      [ ] Enable write cacheing

Should any of those settings inside Windows be different?

[Tom Horsley]

On Sun, 13 Nov 2011 09:15:22 -0500
Tom Horsley wrote:

>  I tried mounting the virtio-win-0.1-15.iso in
>  the CDROM and asking XP to upgrade the disk driver, but it said the
>  on on the CD was not newer and did no upgrade. (But see below - maybe
>  I should have been upgrading the SCSI controller driver instead?) 
OK, I tried that CD image again, only this time updated the SCSI
driver instead of trying to update the disk driver.

Now the guest bluescreens at boot, so I guess it is time to
go back to my base image again.

I wonder if the code has a problem with a backing file? I
remember a while back I had problems due to bugs in the
backing file code, maybe they have cropped up again?

I'll make a copy of the base image and try the scsi driver
update again with no backing file in the image and see what
happens.

[Tom Horsley]

On Sun, 13 Nov 2011 10:03:06 -0500
Tom Horsley wrote:

>  I wonder if the code has a problem with a backing file? I
>  remember a while back I had problems due to bugs in the
>  backing file code, maybe they have cropped up again? 
Now, running with a plain qcow2 image, no backing file,
updateing to the latest version of the XP scsi virtio
driver makes my KVM bluescreen at boot time:

[Vadim Rozenfeld]

On Sun, 2011-11-13 at 10:26 -0500, Tom Horsley wrote:
>  On Sun, 13 Nov 2011 10:03:06 -0500
>  Tom Horsley wrote:
>  
>  > I wonder if the code has a problem with a backing file? I
>  > remember a while back I had problems due to bugs in the
>  > backing file code, maybe they have cropped up again?
>  
>  Now, running with a plain qcow2 image, no backing file,
>  updateing to the latest version of the XP scsi virtio
>  driver makes my KVM bluescreen at boot time: 
BugCheck code 0x7b means INACCESSIBLE_BOOT_DEVICE
Please post the QEMU command line.
>  _______________________________________________
>  virt mailing list
>  virt(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/virt

[Tom Horsley]

On Sun, 13 Nov 2011 21:35:21 +0200
Vadim Rozenfeld wrote:

>  BugCheck code 0x7b means INACCESSIBLE_BOOT_DEVICE
>  Please post the QEMU command line. 
Command line is coming up, but I think I solved the
problem. I believe it was installing the wrong driver.
I foolishly left the "search for driver" option checked
during the driver update process, and it must have found
the vista or windows 7 driver or something.

I went to reinstall to get the bluescreen again and
this time was very careful to take every opportunity
to tell XP not to search for drivers, but let it know
I wanted to point it to a specific one. That allowed me
to finally get the XP version installed 51.62.102.2000
dated 8/10/2011, and now it doesn't bluescreen when I
boot (pant, pant, wheeze... :-).

Anyway, here's the (line wrapped) qemu line from the log file with
the machine running correctly with the new driver
installed:

LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice
/usr/bin/qemu-kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp
2,sockets=2,cores=1,threads=1 -name winxppro -uuid
6d626c31-643c-ca9a-d36d-3c1612e39bbd -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/winxppro.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -drive
file=/zooty/images/winxppro.img,if=none,id=drive-virtio-disk0,format=qcow2
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/zooty/images/swap-winxppro.img,if=none,id=drive-virtio-disk1,format=raw
-device
virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/zooty/images/virtio-win-0.1-15.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
tap,fd=24,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:62:7b:5c,bus=pci.0,addr=0x5
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-usb -device usb-tablet,id=input0 -spice
port=5900,addr=127.0.0.1,disable-ticketing -vga qxl -global
qxl-vga.vram_size=67108864 -device AC97,id=sound0,bus=pci.0,addr=0x3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Now I'm off to try different cache modes again and see what happens...