I'm having a problem with a virtual machine created on an F-11 host that I didn't have with an F-10 host, and I'm wondering if I've misconfigured something or missed a trick somewhere. I'm doing some custom Linux kernel work for my employer, and I'm using a small collection of virtual machines to test it. In the short time I've had F-11 on my host machine, I've seen that virtual disk performance is noticeably worse than it was with F-10. My virtual machines regularly spew soft lockup messages to the system log, always with backtraces that show the process is being starved of the disk. The tests put a lot of stress on the disk, because this custom kernel work is going to help some overloaded servers deal with the load.
It's actually working pretty well on the real machines with real disks, but with the upgrade to F-11, my test VMs have become almost useless for testing further development work. Before I take the drastic step of rolling back to F-10, what factors affect the virtual disk performance?
I created both the F-10 and F-11 machines with virt-manager, if that matters.
Thanks,
On 06/29/2009 08:38 PM, Jerry James wrote:
I'm having a problem with a virtual machine created on an F-11 host that I didn't have with an F-10 host, and I'm wondering if I've misconfigured something or missed a trick somewhere. I'm doing some custom Linux kernel work for my employer, and I'm using a small collection of virtual machines to test it. In the short time I've had F-11 on my host machine, I've seen that virtual disk performance is noticeably worse than it was with F-10. My virtual machines regularly spew soft lockup messages to the system log, always with backtraces that show the process is being starved of the disk. The tests put a lot of stress on the disk, because this custom kernel work is going to help some overloaded servers deal with the load.
It's actually working pretty well on the real machines with real disks, but with the upgrade to F-11, my test VMs have become almost useless for testing further development work. Before I take the drastic step of rolling back to F-10, what factors affect the virtual disk performance?
I created both the F-10 and F-11 machines with virt-manager, if that matters.
Thanks,
It might be due to different image caching defaults. Can you grab the current command line of qemu and change -drive file=xxxx,..,cache=writeback and retest?
On Tue, Jun 30, 2009 at 12:31 AM, Dor Laor dlaor@redhat.com wrote:
It might be due to different image caching defaults. Can you grab the current command line of qemu and change -drive file=xxxx,..,cache=writeback and retest?
As far as I can tell, virt-manager doesn't use the -drive option, but just passes the name of the hard disk image as the last command line argument. That -drive option has a lot of parameters. I have no idea which I should set, or to what, or how to do that in the context of virt-manager.
The virt-manager concept seems nice, but I abandoned it after using it for only a short time in F-10 because (a) it didn't give me an easy way to see what options it ultimately passed to qemu-kvm, and (b) it didn't give me a way to fiddle with nonstandard options. It looks like I'm probably going to abandon it again in F-11 after only using it for a short time, for exactly the same reasons. -- Jerry James http://www.jamezone.org/
On Tue, Jun 30, 2009 at 02:09:17PM -0600, Jerry James wrote:
On Tue, Jun 30, 2009 at 12:31 AM, Dor Laor dlaor@redhat.com wrote:
It might be due to different image caching defaults. Can you grab the current command line of qemu and change -drive file=xxxx,..,cache=writeback and retest?
As far as I can tell, virt-manager doesn't use the -drive option, but just passes the name of the hard disk image as the last command line argument.
I don't think this is correct.
The virt-manager concept seems nice, but I abandoned it after using it for only a short time in F-10 because (a) it didn't give me an easy way to see what options it ultimately passed to qemu-kvm, and (b) it didn't give me a way to fiddle with nonstandard options. It looks like I'm probably going to abandon it again in F-11 after only using it for a short time, for exactly the same reasons.
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
Rich.
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
I would like to see a feature equivalent to the -no-reboot in qemu. When I manually shutdown a virtual machine I don't like it automatically booting back up.
On Wed, Jul 01, 2009 at 10:28:46AM -0400, Rich Mahn wrote:
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
I would like to see a feature equivalent to the -no-reboot in qemu. When I manually shutdown a virtual machine I don't like it automatically booting back up.
We support this already. You have to set this:
<on_reboot>destroy</on_reboot>
See: http://libvirt.org/formatdomain.html#elementsLifecycle
To change the domain XML configuration use 'virsh edit <domname>'.
There might also be a way to change this setting in virt-manager.
Rich.
On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jonesrjones@redhat.com wrote:
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
1) Higher resolution. My boss gave me two big monitors that can do 1920x1200, and all I can get for my guest machines is tiny little windows with hardly any room to work inside.
2) The ability to see what options qemu-kvm was invoked with. Even if I can't change them, at least I can understand what I am getting. Right now, it's a black box to me.
Incidentally, the guest machine that I described in the first message in this thread is using the virt-blk driver, so I wonder if I'm seeing the same problem as described in the thread on the test slowdowns. I just checked, and the disk image is sparse, although there seems to be some doubt that that has anything to do with the problem.
On Thu, Jul 2, 2009 at 8:40 AM, Jerry Jamesloganjerry@gmail.com wrote:
Incidentally, the guest machine that I described in the first message in this thread is using the virt-blk driver, so I wonder if I'm seeing the same problem as described in the thread on the test slowdowns. I just checked, and the disk image is sparse, although there seems to be some doubt that that has anything to do with the problem.
I just made a Rawhide virtual machine to see if it suffers from the same problem. Zillions of copies of this message are being spewed to the console, although the machine itself seems to be running normally (if sluggishly):
end_request: I/O error, dev vda, sector 0
Any idea what that's all about?
On Thu, Jul 02, 2009 at 08:40:58AM -0600, Jerry James wrote:
On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jonesrjones@redhat.com wrote:
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
- Higher resolution. My boss gave me two big monitors that can do
1920x1200, and all I can get for my guest machines is tiny little windows with hardly any room to work inside.
This is a limitation of the emulated Cirrus Logic graphics card. In the days or yore when those real graphics chipsets existed [I even had one] 800x600 and 1024x768 in 16 bits was regarded as the cutting edge.
Eventually we'll have open source SPICE support (PV graphics driver) which massively improves resolution and performance. For example, SPICE can stream videos by recompressing them on the fly between the guest and host.
- The ability to see what options qemu-kvm was invoked with. Even if
I can't change them, at least I can understand what I am getting. Right now, it's a black box to me.
I believe this is being worked on in upstream libvirt. Also the other way around (turn a qemu command line into a libvirt configuration file).
Rich.
On Thu, Jul 2, 2009 at 9:10 AM, Richard W.M. Jonesrjones@redhat.com wrote:
This is a limitation of the emulated Cirrus Logic graphics card. In the days or yore when those real graphics chipsets existed [I even had one] 800x600 and 1024x768 in 16 bits was regarded as the cutting edge.
Sure, but I can get a window NOW with a resolution I consider reasonable by running qemu-kvm -vga std. Even with the other benefits of virt-manager, I'm tempted to drop it just for the bigger window. I'm actually writing code inside the VMs (long story) and I need room.
I well remember CGA graphics. EGA was great, VGA was greater, and SVGA was just amazing! Except, today, I am used to side-by-side monitors that do 1920x1200. Spoiled? Perhaps.
Eventually we'll have open source SPICE support (PV graphics driver) which massively improves resolution and performance. For example, SPICE can stream videos by recompressing them on the fly between the guest and host.
I don't care about video. I just want room for my accustomed programming tools.
I'm starting to whine. I hate whiners. Thanks for all the work on making virtualization in Fedora better. It really is appreciated, in spite of the whining. :-)
On Thu, 2 Jul 2009 09:18:34 -0600 Jerry James wrote:
I don't care about video. I just want room for my accustomed programming tools.
Do what I do - don't use the console at all, run a vncserver in the virtual machine and connect to it from the outside with vncviewer - get any resolution you want :-).
On Thu, Jul 2, 2009 at 9:43 AM, Tom Horsleytom.horsley@att.net wrote:
Do what I do - don't use the console at all, run a vncserver in the virtual machine and connect to it from the outside with vncviewer - get any resolution you want :-).
<*Sound of a light bulb going on in Jerry's brain*>
Where were you 5 months ago when I was getting started with virtual machines? :-)
Thanks a ton.
On Thu, Jul 02, 2009 at 08:40:58AM -0600, Jerry James wrote:
On Wed, Jul 1, 2009 at 2:37 AM, Richard W.M. Jonesrjones@redhat.com wrote:
libvirt isn't just a wrapper around qemu-kvm, it drives half a dozen different virtualization systems, giving you the benefit of abstracting the details of the hypervisor from the management tools. So for that reason we don't support tweaking the qemu command line arbitrarily. However if there are particular features of qemu or KVM that you think we should support, please raise them.
- Higher resolution. My boss gave me two big monitors that can do
1920x1200, and all I can get for my guest machines is tiny little windows with hardly any room to work inside.
We'll have support for alternative graphics adapters in F12 libvirt which will allow this, at least better than now
- The ability to see what options qemu-kvm was invoked with. Even if
I can't change them, at least I can understand what I am getting. Right now, it's a black box to me.
Any VM you launched has its args logged in
/var/log/libvirt/qemu/$NAME.log
Come F12 you can also explicitly do conversions without starting a VM
http://libvirt.org/drvqemu.html#imex
Daniel
Hi Jerry,
On Mon, 2009-06-29 at 11:38 -0600, Jerry James wrote:
I'm having a problem with a virtual machine created on an F-11 host that I didn't have with an F-10 host, and I'm wondering if I've misconfigured something or missed a trick somewhere. I'm doing some custom Linux kernel work for my employer, and I'm using a small collection of virtual machines to test it. In the short time I've had F-11 on my host machine, I've seen that virtual disk performance is noticeably worse than it was with F-10.
I'm not sure if you noticed in the other thread, but from:
https://bugzilla.redhat.com/509383
Does running:
for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done
inside the guest help?
Cheers, Mark.
On Fri, Jul 3, 2009 at 9:52 AM, Mark McLoughlinmarkmc@redhat.com wrote:
I'm not sure if you noticed in the other thread, but from:
https://bugzilla.redhat.com/509383
Does running:
for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done
inside the guest help?
I saw that. However, the guest where I'm having the problem doesn't have a file named "rotational" in the "queue" directory. It's running kernel version 2.6.28.9. In what version did "rotational" appear?
I made a Rawhide guest a few days ago, but it is currently only slightly usable (gdm appears to be having some problems). I'll try this once that guest is a little more usable.
Thanks,
On Fri, 2009-07-03 at 12:31 -0600, Jerry James wrote:
On Fri, Jul 3, 2009 at 9:52 AM, Mark McLoughlinmarkmc@redhat.com wrote:
I'm not sure if you noticed in the other thread, but from:
https://bugzilla.redhat.com/509383
Does running:
for f in /sys/block/vd*/queue/rotational; do echo 1 > $f; done
inside the guest help?
I saw that. However, the guest where I'm having the problem doesn't have a file named "rotational" in the "queue" directory. It's running kernel version 2.6.28.9. In what version did "rotational" appear?
Ah, okay - this file was only added in 2.6.29, and in 2.6.28 virtio-blk used rotational mode anyway.
Cheers, Mark.