Suggestion: bmap files and bmaptool

[Artem Bityutskiy]

On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote:
> On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote:
> 
> > Other things like reading from remote sites, progress indicator,
> > protecting your mounted disks, uncompressing on-the-fly, checking sha1
> > of the data ond of the bmap file itself - are goodies, although
> > important ones.
> 
> Why sha1? If the check is there for security reasons, please use at
> least sha256.

Should not be difficult to implement if there is demand.

>  You can also encode the checksum as base64, base85 or
> base91 to reduce the size of the bmap file.

May be, this was not a problem for sha1, but for sha256 it would
probably make sense, thanks for suggestions.

> 
> > But the base principle is to utilize the inherent sparseness most raw
> > images have a lot of, record this in the bmap file before it is lost,
> > then publish the image in any form (compressed or not), and use the bmap
> > file for fetching the sparseness information and writing/copying only
> > the real data, and leaving out the zeroes.
> 
> This does not sound safe, because it does not ensure that all data that
> should be zero actually is a zero.

I think I covered this part in the documentation. But here is a short
description.

1. The bmap file should be created just after the image is generated.
2. The blocks where zeroes were explicitly written will be mapped to
real sectors which will contain zeroes.
3. The blocks which were not explicitely written to, will be unmapped.
4. Creation of the bmap file is done using the FIEMAP ioctl
5. Only unmapped blocks will be omited in the bmap files.

While on this, I should note that this works best on ext4 file-system. I
did not test ext2/3, but they should work as well as ext4. Btrfs was
also tested, but it is a little bit worse than ext4, I can explain why
if someone is interested.

>  It works well for unassigned file
> systems blocks, but if there is a file containing zeroes in the file
> system (that is not a sparse file) it might not contains zeroes
> afterwards as far as I understand bmap.

It will, those blocks will be explicitly specified in the bmap file. And
the zeroes will be copied.

And this is exactly why I said that 'cp --sparse=always' does not
generate the correct bmap file, I used it only for demosntration
purposes.

In bmap-tools project we have the 'BmapCreate' python module, which
implements bmap file creation, you can use it in the Fedora image build
tool for generating the correct bmap file just after the raw image is
created.

>  This does not sound like
> something that is safe to do.

It is safe, we are using it for a year already in Tizen IVI, so it is
somewhat tested as well. Of course, Tizen IVI community is a lot
smaller, but still, this is some indication of safeness already.

-- 
Best Regards,
Artem Bityutskiy