Hi, What is the best way to deal with I/O load when running several VMs on a physical machine with local or remote storage?
What I'm primarily worried about is the case when several VMs cause disk I/O at the same time. One example would be the "updatedb" cronjob of the mlocate package. If you have say 5 VMs running on a physical System with a local software raid-1 as storage and the all run updatedb at the same time that causes all of them to run really slowly because the starve each other fighting over the disk.
What is the best way to soften the impact of such a situation? Does it make sense to use a hardware raid instead? How would the raid type affect the performance in this case? Would the fact that the I/O load gets distributed across multiple spindles in, say, a 4 disk hardware raid-5 have a big impact on this?
I'm currently facing the problem where I fear that random disk I/O by too many VMs on a physical system could cripple their performance even though I have plenty of CPU cores/RAM left to run them.
Has anyone experience with this problem and maybe some data to shed some light on this potential bottleneck for virtualization?
Regards, Dennis
On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
Hi, What is the best way to deal with I/O load when running several VMs on a physical machine with local or remote storage?
What I'm primarily worried about is the case when several VMs cause disk I/O at the same time. One example would be the "updatedb" cronjob of the mlocate package. If you have say 5 VMs running on a physical System with a local software raid-1 as storage and the all run updatedb at the same time that causes all of them to run really slowly because the starve each other fighting over the disk.
What is the best way to soften the impact of such a situation? Does it make sense to use a hardware raid instead? How would the raid type affect the performance in this case? Would the fact that the I/O load gets distributed across multiple spindles in, say, a 4 disk hardware raid-5 have a big impact on this?
I'm currently facing the problem where I fear that random disk I/O by too many VMs on a physical system could cripple their performance even though I have plenty of CPU cores/RAM left to run them.
Has anyone experience with this problem and maybe some data to shed some light on this potential bottleneck for virtualization?
Well, there's only so much IOPS you can get from a single disk.. (or raid1).
IO generated from multiple virtual machines is pretty random, so you just need a lot of disks to make it faster.
Single 7200 rpm SATA disk will give you only 100-150 IOPS (IOs per second) max.. 10 disks will give you 1000-1500 IOPS max.. it's pretty easy maths.
Also good RAID controller with big battery-backed write-cache will help, in some cases a lot.
If the IOS are not split evenly between VMs, you could use some tool like dm-ioband to make sure IOs are split fairly between VMs.
-- Pasi
On Mon, Jul 27, 2009 at 07:22:27PM +0300, Pasi Kärkkäinen wrote:
On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
Hi, What is the best way to deal with I/O load when running several VMs on a physical machine with local or remote storage?
What I'm primarily worried about is the case when several VMs cause disk I/O at the same time. One example would be the "updatedb" cronjob of the mlocate package. If you have say 5 VMs running on a physical System with a local software raid-1 as storage and the all run updatedb at the same time that causes all of them to run really slowly because the starve each other fighting over the disk.
What is the best way to soften the impact of such a situation? Does it make sense to use a hardware raid instead? How would the raid type affect the performance in this case? Would the fact that the I/O load gets distributed across multiple spindles in, say, a 4 disk hardware raid-5 have a big impact on this?
I'm currently facing the problem where I fear that random disk I/O by too many VMs on a physical system could cripple their performance even though I have plenty of CPU cores/RAM left to run them.
Has anyone experience with this problem and maybe some data to shed some light on this potential bottleneck for virtualization?
Well, there's only so much IOPS you can get from a single disk.. (or raid1).
IO generated from multiple virtual machines is pretty random, so you just need a lot of disks to make it faster.
Single 7200 rpm SATA disk will give you only 100-150 IOPS (IOs per second) max.. 10 disks will give you 1000-1500 IOPS max.. it's pretty easy maths.
.. Or then get faster disks. Single 15k rpm SAS disk will give you up to 350 IOPS, so it's 2-3x faster than 7200 rpm SATA.
-- Pasi
On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
What is the best way to deal with I/O load when running several VMs on a physical machine with local or remote storage?
What I'm primarily worried about is the case when several VMs cause disk I/O at the same time. One example would be the "updatedb" cronjob of the mlocate package. If you have say 5 VMs running on a physical System with a local software raid-1 as storage and the all run updatedb at the same time that causes all of them to run really slowly because the starve each other fighting over the disk.
What is the best way to soften the impact of such a situation? Does it make sense to use a hardware raid instead? How would the raid type affect the performance in this case? Would the fact that the I/O load gets distributed across multiple spindles in, say, a 4 disk hardware raid-5 have a big impact on this?
I'm currently facing the problem where I fear that random disk I/O by too many VMs on a physical system could cripple their performance even though I have plenty of CPU cores/RAM left to run them.
Has anyone experience with this problem and maybe some data to shed some light on this potential bottleneck for virtualization?
For KVM, each VM isn't really any different from a host process, so you have to deal with them the same way that you'd deal with having lots of host processes doing I/O. So: lots of spindles, expensive I/O hardware, etc.
Rich.
-
Dennis Jacobfeuerborn -
Pasi Kärkkäinen -
Richard W.M. Jones