Disk device performance in F11

[Jerry James]

I'm having a problem with a virtual machine created on an F-11 host
that I didn't have with an F-10 host, and I'm wondering if I've
misconfigured something or missed a trick somewhere.  I'm doing some
custom Linux kernel work for my employer, and I'm using a small
collection of virtual machines to test it.  In the short time I've had
F-11 on my host machine, I've seen that virtual disk performance is
noticeably worse than it was with F-10.  My virtual machines regularly
spew soft lockup messages to the system log, always with backtraces
that show the process is being starved of the disk.  The tests put a
lot of stress on the disk, because this custom kernel work is going to
help some overloaded servers deal with the load.

It's actually working pretty well on the real machines with real
disks, but with the upgrade to F-11, my test VMs have become almost
useless for testing further development work.  Before I take the
drastic step of rolling back to F-10, what factors affect the virtual
disk performance?

I created both the F-10 and F-11 machines with virt-manager, if that matters.

Thanks,

[Dor Laor]

On 06/29/2009 08:38 PM, Jerry James wrote:
> I'm having a problem with a virtual machine created on an F-11 host
> that I didn't have with an F-10 host, and I'm wondering if I've
> misconfigured something or missed a trick somewhere.  I'm doing some
> custom Linux kernel work for my employer, and I'm using a small
> collection of virtual machines to test it.  In the short time I've had
> F-11 on my host machine, I've seen that virtual disk performance is
> noticeably worse than it was with F-10.  My virtual machines regularly
> spew soft lockup messages to the system log, always with backtraces
> that show the process is being starved of the disk.  The tests put a
> lot of stress on the disk, because this custom kernel work is going to
> help some overloaded servers deal with the load.
>
> It's actually working pretty well on the real machines with real
> disks, but with the upgrade to F-11, my test VMs have become almost
> useless for testing further development work.  Before I take the
> drastic step of rolling back to F-10, what factors affect the virtual
> disk performance?
>
> I created both the F-10 and F-11 machines with virt-manager, if that matters.
>
> Thanks,

It might be due to different image caching defaults.
Can you grab the current command line of qemu and change -drive 
file=xxxx,..,cache=writeback and retest?

[Jerry James]

On Tue, Jun 30, 2009 at 12:31 AM, Dor Laor  wrote:
> It might be due to different image caching defaults.
> Can you grab the current command line of qemu and change -drive file=xxxx,..,cache=writeback and retest?

As far as I can tell, virt-manager doesn't use the -drive option, but
just passes the name of the hard disk image as the last command line
argument.  That -drive option has a lot of parameters.  I have no idea
which I should set, or to what, or how to do that in the context of
virt-manager.

The virt-manager concept seems nice, but I abandoned it after using it
for only a short time in F-10 because (a) it didn't give me an easy
way to see what options it ultimately passed to qemu-kvm, and (b) it
didn't give me a way to fiddle with nonstandard options.  It looks
like I'm probably going to abandon it again in F-11 after only using
it for a short time, for exactly the same reasons.
--
Jerry James
http://www.jamezone.org/

[Richard W.M. Jones]

On Tue, Jun 30, 2009 at 02:09:17PM -0600, Jerry James wrote:
> On Tue, Jun 30, 2009 at 12:31 AM, Dor Laor  wrote:
> > It might be due to different image caching defaults.
> > Can you grab the current command line of qemu and change -drive file=xxxx,..,cache=writeback and retest?
>
> As far as I can tell, virt-manager doesn't use the -drive option, but
> just passes the name of the hard disk image as the last command line
> argument.

I don't think this is correct.
> The virt-manager concept seems nice, but I abandoned it after using it
> for only a short time in F-10 because (a) it didn't give me an easy
> way to see what options it ultimately passed to qemu-kvm, and (b) it
> didn't give me a way to fiddle with nonstandard options.  It looks
> like I'm probably going to abandon it again in F-11 after only using
> it for a short time, for exactly the same reasons.

libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
different virtualization systems, giving you the benefit of
abstracting the details of the hypervisor from the management tools.
So for that reason we don't support tweaking the qemu command line
arbitrarily.  However if there are particular features of qemu or KVM
that you think we should support, please raise them.

Rich.

[Rich Mahn]

> libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
> different virtualization systems, giving you the benefit of
> abstracting the details of the hypervisor from the management tools.
> So for that reason we don't support tweaking the qemu command line
> arbitrarily.  However if there are particular features of qemu or KVM
> that you think we should support, please raise them.

I would like to see a feature equivalent to the -no-reboot in qemu.
When I manually shutdown a virtual machine I don't like it automatically
booting back up.

[Richard W.M. Jones]

On Wed, Jul 01, 2009 at 10:28:46AM -0400, Rich Mahn wrote:
> > libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
> > different virtualization systems, giving you the benefit of
> > abstracting the details of the hypervisor from the management tools.
> > So for that reason we don't support tweaking the qemu command line
> > arbitrarily.  However if there are particular features of qemu or KVM
> > that you think we should support, please raise them.
>
> I would like to see a feature equivalent to the -no-reboot in qemu.
> When I manually shutdown a virtual machine I don't like it automatically
> booting back up.

We support this already.  You have to set this:

<on_reboot>destroy</on_reboot>

See: http://libvirt.org/formatdomain.html#elementsLifecycle

To change the domain XML configuration use 'virsh edit <domname>'.

There might also be a way to change this setting in virt-manager.

Rich.

[Jerry James]

On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jones wrote:
> libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
> different virtualization systems, giving you the benefit of
> abstracting the details of the hypervisor from the management tools.
> So for that reason we don't support tweaking the qemu command line
> arbitrarily.  However if there are particular features of qemu or KVM
> that you think we should support, please raise them.

1) Higher resolution.  My boss gave me two big monitors that can do
1920x1200, and all I can get for my guest machines is tiny little
windows with hardly any room to work inside.

2) The ability to see what options qemu-kvm was invoked with.  Even if
I can't change them, at least I can understand what I am getting.
Right now, it's a black box to me.

Incidentally, the guest machine that I described in the first message
in this thread is using the virt-blk driver, so I wonder if I'm seeing
the same problem as described in the thread on the test slowdowns.  I
just checked, and the disk image is sparse, although there seems to be
some doubt that that has anything to do with the problem.

[Jerry James]

On Thu, Jul 2, 2009 at 8:40 AM, Jerry James wrote:
> Incidentally, the guest machine that I described in the first message
> in this thread is using the virt-blk driver, so I wonder if I'm seeing
> the same problem as described in the thread on the test slowdowns.  I
> just checked, and the disk image is sparse, although there seems to be
> some doubt that that has anything to do with the problem.

I just made a Rawhide virtual machine to see if it suffers from the
same problem.  Zillions of copies of this message are being spewed to
the console, although the machine itself seems to be running normally
(if sluggishly):

end_request: I/O error, dev vda, sector 0

Any idea what that's all about?

[Richard W.M. Jones]

On Thu, Jul 02, 2009 at 08:40:58AM -0600, Jerry James wrote:
> On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jones wrote:
> > libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
> > different virtualization systems, giving you the benefit of
> > abstracting the details of the hypervisor from the management tools.
> > So for that reason we don't support tweaking the qemu command line
> > arbitrarily.  However if there are particular features of qemu or KVM
> > that you think we should support, please raise them.
> Higher resolution.  My boss gave me two big monitors that can do
>
> 1920x1200, and all I can get for my guest machines is tiny little
> windows with hardly any room to work inside.

This is a limitation of the emulated Cirrus Logic graphics card.  In
the days or yore when those real graphics chipsets existed [I even had
one] 800x600 and 1024x768 in 16 bits was regarded as the cutting edge.

Eventually we'll have open source SPICE support (PV graphics driver)
which massively improves resolution and performance.  For example,
SPICE can stream videos by recompressing them on the fly between the
guest and host.
> The ability to see what options qemu-kvm was invoked with.  Even if
>
> I can't change them, at least I can understand what I am getting.
> Right now, it's a black box to me.

I believe this is being worked on in upstream libvirt.  Also the other
way around (turn a qemu command line into a libvirt configuration
file).

Rich.

[Jerry James]

On Thu, Jul 2, 2009 at 9:10 AM, Richard W.M. Jones wrote:
> This is a limitation of the emulated Cirrus Logic graphics card.  In
> the days or yore when those real graphics chipsets existed [I even had
> one] 800x600 and 1024x768 in 16 bits was regarded as the cutting edge.

Sure, but I can get a window NOW with a resolution I consider
reasonable by running qemu-kvm -vga std.  Even with the other benefits
of virt-manager, I'm tempted to drop it just for the bigger window.
I'm actually writing code inside the VMs (long story) and I need room.

I well remember CGA graphics.  EGA was great, VGA was greater, and
SVGA was just amazing!  Except, today, I am used to side-by-side
monitors that do 1920x1200.  Spoiled?  Perhaps.
> Eventually we'll have open source SPICE support (PV graphics driver)
> which massively improves resolution and performance.  For example,
> SPICE can stream videos by recompressing them on the fly between the
> guest and host.

I don't care about video.  I just want room for my accustomed programming tools.

I'm starting to whine.  I hate whiners.  Thanks for all the work on
making virtualization in Fedora better.  It really is appreciated, in
spite of the whining. :-)

[Tom Horsley]

On Thu, 2 Jul 2009 09:18:34 -0600
Jerry James wrote:
> I don't care about video.  I just want room for my accustomed programming tools.

Do what I do - don't use the console at all, run a vncserver in the virtual
machine and connect to it from the outside with vncviewer - get any resolution
you want :-).

[Jerry James]

On Thu, Jul 2, 2009 at 9:43 AM, Tom Horsley wrote:
> Do what I do - don't use the console at all, run a vncserver in the virtual
> machine and connect to it from the outside with vncviewer - get any resolution
> you want :-).

<*Sound of a light bulb going on in Jerry's brain*>

Where were you 5 months ago when I was getting started with virtual
machines? :-)

Thanks a ton.

[Daniel P. Berrange]

On Thu, Jul 02, 2009 at 08:40:58AM -0600, Jerry James wrote:
> On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jones wrote:
> > libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen
> > different virtualization systems, giving you the benefit of
> > abstracting the details of the hypervisor from the management tools.
> > So for that reason we don't support tweaking the qemu command line
> > arbitrarily.  However if there are particular features of qemu or KVM
> > that you think we should support, please raise them.
> Higher resolution.  My boss gave me two big monitors that can do
>
> 1920x1200, and all I can get for my guest machines is tiny little
> windows with hardly any room to work inside.

We'll have support for alternative graphics adapters in F12 libvirt
which will allow this, at least better than now
> The ability to see what options qemu-kvm was invoked with.  Even if
>
> I can't change them, at least I can understand what I am getting.
> Right now, it's a black box to me.

Any VM you launched has its args logged in

/var/log/libvirt/qemu/$NAME.log

Come F12 you can also explicitly do conversions without starting a VM

http://libvirt.org/drvqemu.html#imex

Daniel

[Mark McLoughlin]

Hi Jerry,

On Mon, 2009-06-29 at 11:38 -0600, Jerry James wrote:
> I'm having a problem with a virtual machine created on an F-11 host
> that I didn't have with an F-10 host, and I'm wondering if I've
> misconfigured something or missed a trick somewhere.  I'm doing some
> custom Linux kernel work for my employer, and I'm using a small
> collection of virtual machines to test it.  In the short time I've had
> F-11 on my host machine, I've seen that virtual disk performance is
> noticeably worse than it was with F-10.

I'm not sure if you noticed in the other thread, but from:

https://bugzilla.redhat.com/509383

Does running:

for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done

inside the guest help?

Cheers,
Mark.

[Jerry James]

On Fri, Jul 3, 2009 at 9:52 AM, Mark McLoughlin wrote:
> I'm not sure if you noticed in the other thread, but from:
>
> https://bugzilla.redhat.com/509383
>
> Does running:
>
> for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done
>
> inside the guest help?

I saw that.  However, the guest where I'm having the problem doesn't
have a file named "rotational" in the "queue" directory.  It's running
kernel version 2.6.28.9.  In what version did "rotational" appear?

I made a Rawhide guest a few days ago, but it is currently only
slightly usable (gdm appears to be having some problems).  I'll try
this once that guest is a little more usable.

Thanks,

[Mark McLoughlin]

On Fri, 2009-07-03 at 12:31 -0600, Jerry James wrote:
> On Fri, Jul 3, 2009 at 9:52 AM, Mark McLoughlin wrote:
> > I'm not sure if you noticed in the other thread, but from:
> >
> > https://bugzilla.redhat.com/509383
> >
> > Does running:
> >
> > for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done
> >
> > inside the guest help?
>
> I saw that.  However, the guest where I'm having the problem doesn't
> have a file named "rotational" in the "queue" directory.  It's running
> kernel version 2.6.28.9.  In what version did "rotational" appear?

Ah, okay - this file was only added in 2.6.29, and in 2.6.28 virtio-blk
used rotational mode anyway.

Cheers,
Mark.