Suggestion: bmap files and bmaptool

[Eric Sandeen]

On 8/13/13 8:58 AM, Artem Bityutskiy wrote:
> # Make the image to be sparse
> $ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw Fedora-x86_64-19-20130627-sda.raw.sparse
> 
> # Generate the bmap file
> $ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o Fedora-x86_64-19-20130627-sda.raw.sparse.bmap

So this is the part that interests me . . . 

There seem to be two issues here; how do we efficiently (compress and) transport sparse files while retaining sparseness, and how do we efficiently operate on files which are already sparse.

For the latter, you're using your bmap tool to map what is hopefully a static file (via fibmap or fiemap, I guess?).

I haven't looked at how you've done it, but you do need to be very careful that the file is stable & quiesced on disk.  Mapping it this way can be fraught with errors if the file is changing, or has delalloc blocks, etc.  And of course getting the mapping wrong means data corruption.  If the file is known to be sparse, then going forward, using SEEK_HOLE / SEEK_DATA is probably the best approach.

But then there's the issue of transporting these sparse files around.  We have had the same problem in the past with large e2image metadata image files, which may be terabytes in length, with only gigabytes or megabytes of real data.  e2image _itself_ creates a sparse file, but bzipping it or rsyncing it still processes terabytes of zeros, and loses all notion of sparseness.

xfs_metadump worked around this by creating its own compact format describing a sparse file's data & sparseness, which is "unpacked" into a normal sparse file by xfs_mdrestore.

More recently e2image gained something slightly similar, but used the existing qcow format to encode the sparseness.  qemu-image convert to "raw" type turns it back into a "normal" sparse file readable by e2fsprogs tools.

So I guess your solution requires 2 pieces of information; the existing file, and the mapping file.  Are there mechanisms to ensure that they are in sync?

Another approach which might (?) be more robust, is to somehow encode that sparseness in a single file format that can be transported/compressed/copied w/o losing the sparseness information, and another tool to operate efficiently on that format at the destination, either by unpacking it to a normal sparse file or piping it to some other process.

Just some thoughts...

-Eric