Building a new cluster of VM hosts

[Scott Baker]

We're in the process of building a new VM cluster that look similar to this:

-----     -----
|VM1|     |VM2|
-----     -----
    \ NFS /
   ---------
   |Backend|
   ---------

Are there any official recommendations on:

1) NFS export/mount options
2) File system choice
3) VM image format (qcow vs raw)
4) How to backup

[Cole Robinson]

On 04/01/2015 01:00 PM, Scott Baker wrote:
> We're in the process of building a new VM cluster that look similar to this:
>
> |VM1|     |VM2|
> \ NFS /
> |Backend|
>
> Are there any official recommendations on:
> NFS export/mount options
> File system choice
> VM image format (qcow vs raw)
> How to backup

If there's any specific document covering that usecase, I don't know about it.
Sorry.

Personally I haven't heard about any specific NFS mount options or FS
choice/options that are catered expressly to virt. Well, besides 'don't use
btrfs' :)

Use qcow2 if you care about any of the specific qcow2 features like internal
snapshots or overlay files. Otherwise use raw to maximize performance.

For backup, you can use live disk snapshots to get manual live backup:

https://fedoraproject.org/wiki/Features/Virt_Live_Snapshots

But if you don't want to worry about all these details, let a higher level
tool handle it for you and check out ovirt. Maybe it's overkill for your
needs, but they have all this stuff figured out already.

- Cole

[Richard W.M. Jones]

On Wed, Apr 01, 2015 at 10:00:43AM -0700, Scott Baker wrote:
> We're in the process of building a new VM cluster that look similar to this:
>
> |VM1|     |VM2|
> \ NFS /
> |Backend|
>
> Are there any official recommendations on:

Not "official" but ...
> NFS export/mount options

I used all defaults.

If you wish to migrate VMs, then you *must* use cache=none for the
VMs, as explained here: http://wiki.qemu.org/Migration/Storage
> File system choice

I used ext4, but any decent fs should work fine.  Avoid btrfs.
> VM image format (qcow vs raw)

It turned out that qcow2 has an amazing susceptibility to corruption,
especially when a node crashes.  So I converted my VMs to raw, because
at least if you get corruption it doesn't make the entire VM file
unreadable.
> How to backup

rsync -av /var/lib/libvirt/images /mnt/backups

- - -

I was using nfs on my cluster[1], but I'm no longer using it.  Why?
Because it's just way too slow over standard gigabit ethernet.

[1] https://rwmj.wordpress.com/2014/04/28/caseless-virtualization-cluster-part-5...

I also used iscsi for a while, backed with logical volumes.  The iscsi
management tools are absolutely hideous, but the performance and
reliability were far better (using the precise same hardware).

- - -

Anyway, I'm now using local disks attached to each node, and just
storing my VMs there.  Of course this means I can't easily migrate
VMs, but the performance is far superior.  My VMs are now fully
transitioned to cloud-like "cattle", using NFS mounts for shared data
(basically NFS-mounted /home and a few other mounts), so migration
isn't really much of an issue for me now.

My cloud management software in case anyone is interested:

http://git.annexia.org/?p=mclu.git;a=summary

Rich.