Suggestion: bmap files and bmaptool

Tue Aug 13 13:58:16 UTC 2013

Hi Fedora developers,

I would like suggest you to take a look at bmaptool, which you can use
for flashing Fedora ISO images to USB sticks (or other block devices).

Let me set the scene. In order to flash a Fedora ISO image, typically
first download the image and then use dd to copy it to the USB stick.
Bmaptool does this, but

a) a lot faster (several times faster if you have the bmap file)
b) simpler, because it can flash directly from an URL (decompress
   on-the fly too)
c) simpler, because it verifies data checksum automatically (only
   if bmap file is present)
d) safer, since it won't destroy your mounted partition if you specify
   wrong block device (say, for example, /dev/sda being the rootfs)

So what I am trying to "sell" here is a tool which flashes several times
faster than dd, which is safe, and can flash from http/ftp/https/ssh
URLs directly, and decompress gz/bz2/xz/gzip/tgz/tar.{gz/bz2/zx} files
on-the-fly.

We are using this in Tizen IVI, and I'd like to see other distros
using/supporting it. Or at least checking it.

The key element for bmaptool is the bmap file, which Fedora does not
produce for the ISO images. So bmaptool is not so useful, but the
community could find the tool useful and start providing the bmap
files.

Anyway, those who are interested in all the details may read the docs,
the links will be at the end. Let me give a couple of examples. I the
example I flash a Fedora image to my USB stick, which is /dev/sdd.

1. First of all, I can flash directly from the URL, I use the Finland
mirror here:

$ bmaptool copy --nobmap ftp://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/releases/19/Images/x86_64/Fedora-x86_64-19-20130627-sda.raw.xz /dev/sdd

Note, I use --nobmap because there is no bmap file. With bmap file
flashing would be _a lot_ faster.

2. Let's check how fast is dd. I downloaded and decompressed the image
   prior to running the tests.

$ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd
4194304+0 records in
4194304+0 records out
2147483648 bytes (2.1 GB) copied, 799.487 s, 2.7 MB/s

13.3 minutes.

3. With bmaptool, no bmap file.
$ bmaptool copy --nobmap Fedora-x86_64-19-20130627-sda.raw /dev/sdd
bmaptool: info: no bmap given, copy entire image to '/dev/sdd'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdd'
bmaptool: info: copying time: 7m 22.2s, copying speed 4.6 MiB/sec

Faster, but not too impressive. The speed is because bmaptool configures
the block device for optimal writing of large sequential files.

4. Now I've generated the bmap file myself. Note, the real, correct bmap
file can only be generated on the Fedora build server just after the
image is created, so this example is artificial, and not correct, do not
try to use it. And it will give better numbers than the real bmap file
would give, I can explain why if needed, or see the docs. Anyway, I've
made a sparse file out of the image, then generated the bmap file for
it, and flashed it:

# Make the image to be sparse
$ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw Fedora-x86_64-19-20130627-sda.raw.sparse

# Generate the bmap file
$ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o Fedora-x86_64-19-20130627-sda.raw.sparse.bmap

# Now flash with the bmap file
$ sudo ./bmaptool copy Fedora-x86_64-19-20130627-sda.raw.sparse /dev/sdd
bmaptool: info: discovered bmap file '/home/dedekind/tmp/Fedora-x86_64-19-20130627-sda.raw.sparse.bmap'
bmaptool: info: block map format version 1.3
bmaptool: info: 524288 blocks of size 4096 (2.0 GiB), mapped 176288 blocks (688.6 MiB or 33.6%)
bmaptool: info: copying image 'Fedora-x86_64-19-20130627-sda.raw.sparse' to block device '/dev/sdd' using bmap file 'Fedora-x86_64-19-20130627-sda.raw.sparse.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdd'
bmaptool: info: copying time: 2m 36.2s, copying speed 4.4 MiB/sec

Several times faster than dd.

And yes, bmaptool can flash the .raw.xz file as well, you do not have to
decompress.

Please, fine the bmaptool documentation here:
https://source.tizen.org/documentation/articles/bmaptool

It is a bit out-of-date, but not much, will be updated soon.

The tool is generic, not tizen-specific.

Please, see the information about the project here as well:
http://git.infradead.org/users/dedekind/bmap-tools.git/blob/refs/heads/devel:/docs/README

Note, the .xz support is only available in v2.6, which I did not release
yet, so you need to take the sources, not the packages.

The project is a python project, and it provides APIs so that other
python projects could use the functionality.

Here is an example of a bmap file can be found here:
https://source.tizen.org/documentation/articles/bmaptool

And yes, I suggest that Fedora could also generate and publish bmap
files, if this is found to be useful. In Tizen IVI people are very happy
to use bmaptool versus pain dd.

Anyway, please, check the docs, give feed-back, ask questions.

Thanks, and sorry for a messy e-mail  :-)

-- 
Best Regards,
Artem Bityutskiy