Suggestion: bmap files and bmaptool

[Artem Bityutskiy]

On Wed, 2013-08-14 at 21:16 -0700, Samuel Sieb wrote:
> On 08/14/2013 02:21 AM, Artem Bityutskiy wrote:
> > I think I covered this part in the documentation. But here is a short
> > description.
> >
> > 1. The bmap file should be created just after the image is generated.
> > 2. The blocks where zeroes were explicitly written will be mapped to
> > real sectors which will contain zeroes.
> > 3. The blocks which were not explicitely written to, will be unmapped.
> > 4. Creation of the bmap file is done using the FIEMAP ioctl
> > 5. Only unmapped blocks will be omited in the bmap files.
> >
> > While on this, I should note that this works best on ext4 file-system. I
> > did not test ext2/3, but they should work as well as ext4. Btrfs was
> > also tested, but it is a little bit worse than ext4, I can explain why
> > if someone is interested.
> 
> Have you looked at partimage?  It sounds like this except that it works 
> on many different filesystems and doesn't need the blocks to be unmapped 
> to compress it (i.e. it works on normal partitions as well as images).

No, never saw this project before. Yeah, it sounds like it uses similar
ideas to speed-up, but has different purposes and tries to know the
file-system internal format, and hence, does not support ext4/btrfs
simply because, as I guess, they are too complex and are developed too
quickly. It is just too difficult to maintain a parallel implementation
in user-space.

Bmaptool does not know anything about the internals of the file-system.
It does not care what is the FS underneath. bmaptool simply use the
FIEMAP ioctl and ask the FS about which blocks are mapped (used).

This would not work for partimage since it needs to know about all the
blocks (superblock, all the other meta-data blocks), not just blocks
belonging to a single file.

Now, why I said that ext4 is the best one to use on the server (most
probably ext2/3 are as good, but I did not verify). This is because ext4
is "perfect" in leaving the gaps. Even if you have one block gap, it
still will account it as unmapped. I have a test where I create random
mapped areas, and ext4 keeps all the gaps. But BTRFS sometimes maps
small 1-block gaps. This is related to its internal structure. So with
btrfs the bmap file becomes less ideal. 

Anyway, thanks for letting me know about partimage.

-- 
Best Regards,
Artem Bityutskiy