12:30 p.m.
#4262: Provide deltaisos for Alpha, Beta, Final
----------------------+-----------------------------------------------------
Reporter: robatino | Owner: rel-eng(a)lists.fedoraproject.org
Type: task | Status: new
Milestone: | Component: koji
Keywords: |
----------------------+-----------------------------------------------------
For a while now I've been providing deltaisos between Alpha, Beta, and
Final (in addition to the ones for TCs/RCs) at
[http://thepiratebay.org/user/andre14965/].
* When N Alpha is released I create 2 torrents from (N-1) Final to N
Alpha (one for i386 and one for x86_64).
* When N Beta is released I create 2 torrents from (N-1) Final to N Beta,
and 2 torrents from N Alpha to N Beta.
* When N Final is released I create 2 torrents from (N-1) Final to N
Final, and 2 torrents from N Beta to N Final.
For each milestone, each corresponding torrent consists of
* The old and new signed checksum files for the full ISOs,
* A deltaiso from the old DVD to the new DVD, and
* Deltaisos from the new DVD to each of the new CDs and the new netinst
(since split media is gone for F15 and later, only the one for DVD ->
netinst will be needed in the future).
The size of the diso for DVD -> netinst is negligible (around 50K or
less). The size of the disos starting with (N-1) Final is roughly half the
size of the full ISO, and increases only slightly from N Alpha to N Beta
to N Final. The size of the disos for N Alpha -> N Beta and N Beta -> N
Final is around 10-20% of the full ISO size (you can see the exact sizes
at the above torrent link).
The same exact format could be used to provide these torrents at
[http://torrent.fedoraproject.org]. For direct download mirrors, just the
disos could be posted. It's unnecessary to create any new signed checksum
files. It may be desirable to provide a crude checksum for the disos just
to make sure the download is good before running applydeltaiso (which can
take a significant amount of time - around 40-45 minutes for the disos
starting with (N-1) Final, and around 10-20 minutes for the ones from N
Alpha -> N Beta and N Beta -> N Final - the time for DVD -> netinst is
negligible).
The main obstacle is that each RPM in the new ISO must be compressed using
the exact same compression. For example, the recent change in xz
compression means that Rawhide currently consists of RPMs built using both
the old and new compression, so it would be impossible to generate working
disos from 14 Final to either 15 Alpha, 15 Beta, or 15 Final. Deltaisos
for 15 Alpha -> 15 Beta and 15 Beta -> 15 Final should work (assuming that
the user has the new xz installed on the machine which runs
applydeltaiso). For the user to temporarily change xz version is a minor
nuisance - someone running anything from F11 to F14 can temporarily change
the xz-\* packages to the F15/Rawhide version, and someone running
F15/Rawhide can temporarily downgrade xz-\* to the F14 version. Of course
it's important to restore the original versions of xz-\* after running
applydeltaiso, otherwise yum-presto won't be able to apply drpms in
updates.
Some links regarding the xz compression change:
* https://bugzilla.redhat.com/show_bug.cgi?id=644046
* http://lists.fedoraproject.org/pipermail/devel/2010-October/144651.html
* http://lists.fedoraproject.org/pipermail/test/2010-October/094883.html
* https://fedorahosted.org/rel-eng/ticket/4224
I volunteer to do any necessary work in creating and checking the diso
content (though it's actually pretty trivial) or in writing user
documentation. In closing I'd like to point out that although refining
compression in order to reduce the size of full ISOs is important, at some
point there will be diminishing returns (I don't know if it's reasonable
to expect the large improvement of 10-15% in going from gzip to xz to
happen again). On the other hand, there's no reason to expect delta
compression, which reduces download size by 50% or more, to get any worse.
So in the long run, it doesn't make sense to keep ignoring delta
compression in the quest to make ISOs a few percent smaller.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project