Two questions:
1) Deltaisos are capable of saving roughly half the download size in going from Fedora N to Fedora (N+1), but only work for installation images, not live images. Is there any form of delta compression for live images which is competitive with this?
2) (A little off topic) The installation images are still labeled i386, even though after the two package rebuilds, all packages on the disk will be i686. The live CDs are correctly labeled i686, presumably the live DVDs will be also. Are there any plans for F13 to relabel the installation images, and the repo directory names, to stop referring to i386?
Deltaisos are capable of saving roughly half the download size in going from Fedora N to Fedora (N+1), but only work for installation images, not live images. Is there any form of delta compression for live images which is competitive with this?
It hasn't been productized, but approximately: unsquashfs old-Live.img old-Live.tree # local ("slave") unsquashfs new-Live.img new-Live.tree # remote ("master") rsync remote:new-Live.tree local:old-Live.tree # "delta compression" happens here mksquashfs old-Live.tree new-Live.img # local
On 09/14/2009 12:05 PM, John Reiser wrote:
Deltaisos are capable of saving roughly half the download size in going from Fedora N to Fedora (N+1), but only work for installation images, not live images. Is there any form of delta compression for live images which is competitive with this?
It hasn't been productized, but approximately: unsquashfs old-Live.img old-Live.tree # local ("slave") unsquashfs new-Live.img new-Live.tree # remote ("master") rsync remote:new-Live.tree local:old-Live.tree # "delta compression" happens here mksquashfs old-Live.tree new-Live.img # local
Has anyone tried this on existing live images to see how much is saved (say going from a Fedora N to Fedora (N+1) Live CD)? I'm skeptical that rsync, which is completely general, would be as efficient as something specialized such as {make,apply}deltaiso. It may be necessary for someone to create a specialized tool for delta compression between live images, in order to be able to compress as well as deltaisos currently do for the install images.
On 09/14/2009 03:48 PM, Andre Robatino wrote:
On 09/14/2009 12:05 PM, John Reiser wrote:
Deltaisos are capable of saving roughly half the download size in going from Fedora N to Fedora (N+1), but only work for installation images, not live images. Is there any form of delta compression for live images which is competitive with this?
It hasn't been productized, but approximately: unsquashfs old-Live.img old-Live.tree # local ("slave") unsquashfs new-Live.img new-Live.tree # remote ("master") rsync remote:new-Live.tree local:old-Live.tree # "delta compression" happens here mksquashfs old-Live.tree new-Live.img # local
Has anyone tried this on existing live images to see how much is saved (say going from a Fedora N to Fedora (N+1) Live CD)? I'm skeptical that rsync, which is completely general, would be as efficient as something specialized such as {make,apply}deltaiso. It may be necessary for someone to create a specialized tool for delta compression between live images, in order to be able to compress as well as deltaisos currently do for the install images.
I tried doing this with the Live CDs for F10 and F11. Almost the entire content is in a single file LiveOS/squashfs.img. Attempting to use unsquashfs on this file gives
Parallel unsquashfs: Using 1 processor FATAL ERROR aborting: failed to read fragment table
What am I doing wrong? (I'm using the i686 live CDs. I originally tried it on an x86_64 host, then on an i686 host after reading somewhere that might be the problem, but it made no difference.)
I tried doing this with the Live CDs for F10 and F11:
Parallel unsquashfs: Using 1 processor FATAL ERROR aborting: failed to read fragment table
What am I doing wrong? (I'm using the i686 live CDs.
What versions are involved? [unsquashfs -v -ll foo.img]
unsquashfs version 4.0 works for me on F12-Snap2-i686-Live.iso/LiveOS/squashfs.img when run on either Fedora 11 or rawhide for Fedora 12.
On 09/15/2009 01:15 PM, John Reiser wrote:
I tried doing this with the Live CDs for F10 and F11:
Parallel unsquashfs: Using 1 processor FATAL ERROR aborting: failed to read fragment table
What am I doing wrong? (I'm using the i686 live CDs.
What versions are involved? [unsquashfs -v -ll foo.img]
unsquashfs version 4.0 works for me on F12-Snap2-i686-Live.iso/LiveOS/squashfs.img when run on either Fedora 11 or rawhide for Fedora 12.
I'm using the latest F11 version of squashfs-tools on a fully updated x86_64 F11 box. Just discovered that it works on Fedora-11-i686-Live.iso, but fails with F10-i686-Live.iso. So the new question is, why doesn't it work with the F10 image?
Also, after expanding squashfs.img for F11, it gives me another single huge (over 3 GB) file ext3fs.img. I know that rsync doesn't work particularly well between install images - going between the F11 Preview and Final DVDs required about half the full ISO size, while the deltaiso was more like 5%. It would be completely useless for the leap from Fedora N to (N+1). Unless there is a way to expand the Live image into a file tree where many of the files haven't changed, it looks like rsync won't be much help here.
On 09/15/2009 10:29 AM, Andre Robatino wrote:
I'm using the latest F11 version of squashfs-tools on a fully updated x86_64 F11 box. Just discovered that it works on Fedora-11-i686-Live.iso, but fails with F10-i686-Live.iso. So the new question is, why doesn't it work with the F10 image?
That's a bug, please file in bugzilla, with *specific* version info. Include the checksum of each *.iso, just to be sure.
Also, after expanding squashfs.img for F11, it gives me another single huge (over 3 GB) file ext3fs.img.
ext3fs.img is a complete, mountable filesystem. mount -o loop ext3fs.img /mnt/foo All the files are there with their actual names, actual contents, etc. So rsync will work on the tree /mnt/foo just as well as rsync works on any actual file system tree.
The downside is each rsync session requires processor cycles on both ends. The drawing card of the new zsync tool is that the rsync CPU time on the "master" side need be done only once; the results are cached as a companion file to each existing file. The companion file contains the checksums for chunks of the [new] file. The master side http server just serves the companion file like any other file. The zsync tool retrieves the whole companion file, does the rsync checksum computations for the local old file, then asks the master for the appropriate partial content (HTTP code 206) of the chunks that the local side does not have already. The gain is that the companion file is smaller. The risk is that anybody who can manufacture collisions for the checksum can pollute the result.
On 09/15/2009 02:01 PM, John Reiser wrote:
On 09/15/2009 10:29 AM, Andre Robatino wrote:
I'm using the latest F11 version of squashfs-tools on a fully updated x86_64 F11 box. Just discovered that it works on Fedora-11-i686-Live.iso, but fails with F10-i686-Live.iso. So the new question is, why doesn't it work with the F10 image?
That's a bug, please file in bugzilla, with *specific* version info. Include the checksum of each *.iso, just to be sure.
-
Andre Robatino -
John Reiser