#4262: Provide deltaisos for Alpha, Beta, Final

[Fedora Release Engineering]

#4262: Provide deltaisos for Alpha, Beta, Final
----------------------+-----------------------------------------------------
 Reporter:  robatino  |       Owner:  rel-eng at lists.fedoraproject.org
     Type:  task      |      Status:  new                            
Milestone:            |   Component:  koji                           
 Keywords:            |  
----------------------+-----------------------------------------------------
 For a while now I've been providing deltaisos between Alpha, Beta, and
 Final (in addition to the ones for TCs/RCs) at
 [http://thepiratebay.org/user/andre14965/].

  * When N Alpha is released I create 2 torrents from (N-1) Final to N
 Alpha (one for i386 and one for x86_64).
  * When N Beta is released I create 2 torrents from (N-1) Final to N Beta,
 and 2 torrents from N Alpha to N Beta.
  * When N Final is released I create 2 torrents from (N-1) Final to N
 Final, and 2 torrents from N Beta to N Final.

 For each milestone, each corresponding torrent consists of

  * The old and new signed checksum files for the full ISOs,
  * A deltaiso from the old DVD to the new DVD, and
  * Deltaisos from the new DVD to each of the new CDs and the new netinst
 (since split media is gone for F15 and later, only the one for DVD ->
 netinst will be needed in the future).

 The size of the diso for DVD -> netinst is negligible (around 50K or
 less). The size of the disos starting with (N-1) Final is roughly half the
 size of the full ISO, and increases only slightly from N Alpha to N Beta
 to N Final. The size of the disos for N Alpha -> N Beta and N Beta -> N
 Final is around 10-20% of the full ISO size (you can see the exact sizes
 at the above torrent link).

 The same exact format could be used to provide these torrents at
 [http://torrent.fedoraproject.org]. For direct download mirrors, just the
 disos could be posted. It's unnecessary to create any new signed checksum
 files. It may be desirable to provide a crude checksum for the disos just
 to make sure the download is good before running applydeltaiso (which can
 take a significant amount of time - around 40-45 minutes for the disos
 starting with (N-1) Final, and around 10-20 minutes for the ones from N
 Alpha -> N Beta and N Beta -> N Final - the time for DVD -> netinst is
 negligible).

 The main obstacle is that each RPM in the new ISO must be compressed using
 the exact same compression. For example, the recent change in xz
 compression means that Rawhide currently consists of RPMs built using both
 the old and new compression, so it would be impossible to generate working
 disos from 14 Final to either 15 Alpha, 15 Beta, or 15 Final. Deltaisos
 for 15 Alpha -> 15 Beta and 15 Beta -> 15 Final should work (assuming that
 the user has the new xz installed on the machine which runs
 applydeltaiso). For the user to temporarily change xz version is a minor
 nuisance - someone running anything from F11 to F14 can temporarily change
 the xz-\* packages to the F15/Rawhide version, and someone running
 F15/Rawhide can temporarily downgrade xz-\* to the F14 version. Of course
 it's important to restore the original versions of xz-\* after running
 applydeltaiso, otherwise yum-presto won't be able to apply drpms in
 updates.

 Some links regarding the xz compression change:
  * https://bugzilla.redhat.com/show_bug.cgi?id=644046
  * http://lists.fedoraproject.org/pipermail/devel/2010-October/144651.html
  * http://lists.fedoraproject.org/pipermail/test/2010-October/094883.html
  * https://fedorahosted.org/rel-eng/ticket/4224

 I volunteer to do any necessary work in creating and checking the diso
 content (though it's actually pretty trivial) or in writing user
 documentation. In closing I'd like to point out that although refining
 compression in order to reduce the size of full ISOs is important, at some
 point there will be diminishing returns (I don't know if it's reasonable
 to expect the large improvement of 10-15% in going from gzip to xz to
 happen again). On the other hand, there's no reason to expect delta
 compression, which reduces download size by 50% or more, to get any worse.
 So in the long run, it doesn't make sense to keep ignoring delta
 compression in the quest to make ISOs a few percent smaller.

-- 
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project