Dracut, dmsquash, and overlays

[Will Woods]

On Mon, 2014-07-28 at 09:37 -0500, Major Hayden wrote:
> Hello there,
> 
> I'm working with F20 and CentOS 7 to create some live booted images.
> I'm not looking to do live USB/CD media, but rather boot a server over
> the network with a kernel, initramfs, and squashfs.  It's working well
> so far, but I have a filesystem issue that I can't seem to fix.
> 
> My build scripts create a 10GB sparse file and I fill that with an
> ext4 filesystem.  I package that up into a squashfs as specified in
> the docs[1].  That boots just fine.
> 
> However, if I attempt to fill the filesystem, it fills and becomes
> corrupt much earlier than I'd expect.  I've put some log info into a
> gist[2].
> 
> My expectation is that if I create a 10GB live filesystem and and the
> system has > 10GB of RAM available, I'd expect that I could store
> somewhere around 10GB of data in the live filesystem before I run into
> a full filesystem.  Is that expectation incorrect?  Am I configuring
> something incorrectly?

You're not quite right about how the overlay works.

The default in-memory overlay is just 512MB. And the device-mapper docs
note that "if it fills up the snapshot will become useless and be
disabled, returning errors."[1].

You should also note that the overlay is a block-level snapshot - so any
changes to existing files or filesystem metadata will cause data to be
written to the overlay. Furthermore, the default chunk size is 4kb - so
any change less than 4kb will take 4kb of space.

Basically, the dmsquash-live + overlay thing is a gross hack. It was
designed to solve the problem of fitting an entire desktop OS on a 650MB
CD-ROM, to be used as a demonstration, or for other short-lived systems
(e.g. the installer). It was not designed for long-term use.

I assume you're using filesystem images 'cuz you don't have a reliable
network connection (otherwise you'd probably be using NFS or iSCSI or
something)?

Since your systems have lots of RAM, why not just use a regular ext4
filesystem image as your root filesystem? Then you don't need to worry
about blowing up the overlay at all.

If you need compression to save bandwidth/download time: you could just
xz-compress the filesystem image and uncompress it after download?

If you need compression to save RAM: why not use a squashfs image
directly, and mount/bind a tmpfs to the places you'll be writing data?

You might even consider using btrfs, mounted with the "compress" or
"compress-force" option[2], which would give you the benefit of
compression *and* a normal read-write filesystem.

Is there a particular reason you need to use dmsquash-live, or is this
just a case of the hammer making all your problems look like nails?

-w

[1] https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
[2] https://btrfs.wiki.kernel.org/index.php/Compression