Dracut, dmsquash, and overlays

[Major Hayden]

On Jul 28, 2014, at 17:11, Will Woods <wwoods at redhat.com> wrote:

> You're not quite right about how the overlay works.
> 
> The default in-memory overlay is just 512MB. And the device-mapper docs
> note that "if it fills up the snapshot will become useless and be
> disabled, returning errors."[1].
> 
> You should also note that the overlay is a block-level snapshot - so any
> changes to existing files or filesystem metadata will cause data to be
> written to the overlay. Furthermore, the default chunk size is 4kb - so
> any change less than 4kb will take 4kb of space.

After hitting a wall several times today, I began to see what you're talking about here. ;)

> I assume you're using filesystem images 'cuz you don't have a reliable
> network connection (otherwise you'd probably be using NFS or iSCSI or
> something)?

The network connection is reliable, but I'm working with thousands of nodes that need a largely stateless system with only a few persistent items.

> Since your systems have lots of RAM, why not just use a regular ext4
> filesystem image as your root filesystem? Then you don't need to worry
> about blowing up the overlay at all.

Are you suggesting an ext4 r/w filesystem stored in RAM?  I haven't seen how to do that in dracut with the existing scripts.

> If you need compression to save RAM: why not use a squashfs image
> directly, and mount/bind a tmpfs to the places you'll be writing data?

I'd be interested in that for sure but the dmsquash module in dracut seems to require a real ext/btrfs/xfs filesystem for device mapper.  I couldn't find a way to boot a plain squashfs with a filesystem in it.

> Is there a particular reason you need to use dmsquash-live, or is this
> just a case of the hammer making all your problems look like nails?

My goal is to live boot our servers since the majority of our systems would be stateless.  Being able to reboot into a known good, tested state would be advantageous.  I've worked with Debian's Live Systems project[1] and their strategy is to mount a squashfs read only but then use aufs to provide a writeable filesystem overlay.  It's handy since you can fill up the overlay without causing the snapshot to overflow.  However, AUFS isn't in the upstream kernel and that makes things a bit challenging.

If there's a better strategy than using dmsquash-live in dracut, or if I need to do some work to build a new dracut module, I'm certainly up for that.  Finding documentation on the internals of dracut has been a bit challenging for me so far.

[1] http://live.debian.net/

--
Major Hayden