I/O load distribution

[Dennis Jacobfeuerborn]

Hi,
What is the best way to deal with I/O load when running several VMs on a 
physical machine with local or remote storage?

What I'm primarily worried about is the case when several VMs cause disk 
I/O at the same time. One example would be the "updatedb" cronjob of the 
mlocate package. If you have say 5 VMs running on a physical System with a 
local software raid-1 as storage and the all run updatedb at the same time 
that causes all of them to run really slowly because the starve each other 
fighting over the disk.

What is the best way to soften the impact of such a situation? Does it make 
sense to use a hardware raid instead? How would the raid type affect the 
performance in this case? Would the fact that the I/O load gets distributed 
across multiple spindles in, say, a 4 disk hardware raid-5 have a big 
impact on this?

I'm currently facing the problem where I fear that random disk I/O by too 
many VMs on a physical system could cripple their performance even though I 
have plenty of CPU cores/RAM left to run them.

Has anyone experience with this problem and maybe some data to shed some 
light on this potential bottleneck for virtualization?

Regards,
   Dennis

[Pasi Kärkkäinen]

On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
>  Hi,
>  What is the best way to deal with I/O load when running several VMs on a 
>  physical machine with local or remote storage?
>  
>  What I'm primarily worried about is the case when several VMs cause disk 
>  I/O at the same time. One example would be the "updatedb" cronjob of the 
>  mlocate package. If you have say 5 VMs running on a physical System with a 
>  local software raid-1 as storage and the all run updatedb at the same time 
>  that causes all of them to run really slowly because the starve each other 
>  fighting over the disk.
>  
>  What is the best way to soften the impact of such a situation? Does it make 
>  sense to use a hardware raid instead? How would the raid type affect the 
>  performance in this case? Would the fact that the I/O load gets distributed 
>  across multiple spindles in, say, a 4 disk hardware raid-5 have a big 
>  impact on this?
>  
>  I'm currently facing the problem where I fear that random disk I/O by too 
>  many VMs on a physical system could cripple their performance even though I 
>  have plenty of CPU cores/RAM left to run them.
>  
>  Has anyone experience with this problem and maybe some data to shed some 
>  light on this potential bottleneck for virtualization?
>   
Well, there's only so much IOPS you can get from a single disk.. (or raid1).

IO generated from multiple virtual machines is pretty random, so you just
need a lot of disks to make it faster.

Single 7200 rpm SATA disk will give you only 100-150 IOPS (IOs per second)
max.. 10 disks will give you 1000-1500 IOPS max.. it's pretty easy maths.

Also good RAID controller with big battery-backed write-cache will help, 
in some cases a lot.

If the IOS are not split evenly between VMs, you could use some tool like
dm-ioband to make sure IOs are split fairly between VMs.

-- Pasi

[Pasi Kärkkäinen]

On Mon, Jul 27, 2009 at 07:22:27PM +0300, Pasi Kärkkäinen wrote:
>  On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
>  > Hi,
>  > What is the best way to deal with I/O load when running several VMs on a 
>  > physical machine with local or remote storage?
>  > 
>  > What I'm primarily worried about is the case when several VMs cause disk 
>  > I/O at the same time. One example would be the "updatedb" cronjob of the 
>  > mlocate package. If you have say 5 VMs running on a physical System with a 
>  > local software raid-1 as storage and the all run updatedb at the same time 
>  > that causes all of them to run really slowly because the starve each other 
>  > fighting over the disk.
>  > 
>  > What is the best way to soften the impact of such a situation? Does it make 
>  > sense to use a hardware raid instead? How would the raid type affect the 
>  > performance in this case? Would the fact that the I/O load gets distributed 
>  > across multiple spindles in, say, a 4 disk hardware raid-5 have a big 
>  > impact on this?
>  > 
>  > I'm currently facing the problem where I fear that random disk I/O by too 
>  > many VMs on a physical system could cripple their performance even though I 
>  > have plenty of CPU cores/RAM left to run them.
>  > 
>  > Has anyone experience with this problem and maybe some data to shed some 
>  > light on this potential bottleneck for virtualization?
>  > 
>  
>  Well, there's only so much IOPS you can get from a single disk.. (or raid1).
>  
>  IO generated from multiple virtual machines is pretty random, so you just
>  need a lot of disks to make it faster.
>  
>  Single 7200 rpm SATA disk will give you only 100-150 IOPS (IOs per second)
>  max.. 10 disks will give you 1000-1500 IOPS max.. it's pretty easy maths.
>   
.. Or then get faster disks. Single 15k rpm SAS disk will give you up to 350
IOPS, so it's 2-3x faster than 7200 rpm SATA.

-- Pasi

[Richard W.M. Jones]

On Mon, Jul 27, 2009 at 03:15:05PM +0200, Dennis J. wrote:
>  What is the best way to deal with I/O load when running several VMs
> on a  
>  physical machine with local or remote storage?
>
>  What I'm primarily worried about is the case when several VMs cause disk  
>  I/O at the same time. One example would be the "updatedb" cronjob of the  
>  mlocate package. If you have say 5 VMs running on a physical System with 
>  a local software raid-1 as storage and the all run updatedb at the same 
>  time that causes all of them to run really slowly because the starve each 
>  other fighting over the disk.
>
>  What is the best way to soften the impact of such a situation? Does it 
>  make sense to use a hardware raid instead? How would the raid type affect 
>  the performance in this case? Would the fact that the I/O load gets 
>  distributed across multiple spindles in, say, a 4 disk hardware raid-5 
>  have a big impact on this?
>
>  I'm currently facing the problem where I fear that random disk I/O by too 
>  many VMs on a physical system could cripple their performance even though 
>  I have plenty of CPU cores/RAM left to run them.
>
>  Has anyone experience with this problem and maybe some data to shed some  
>  light on this potential bottleneck for virtualization? 
For KVM, each VM isn't really any different from a host process, so
you have to deal with them the same way that you'd deal with having
lots of host processes doing I/O.  So: lots of spindles, expensive I/O
hardware, etc.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
New in Fedora 11: Fedora Windows cross-compiler. Compile Windows
programs, test, and build Windows installers. Over 70 libraries supprt'd
http://fedoraproject.org/wiki/MinGW http://www.annexia.org/fedora_mingw