#4262: Provide deltaisos for Alpha, Beta, Final

[Fedora Release Engineering]

#4262: Provide deltaisos for Alpha, Beta, Final
----------------------+-----------------------------------------------------
 Reporter:  robatino  |       Owner:  rel-eng(a)lists.fedoraproject.org
     Type:  task      |      Status:  new                            
Milestone:            |   Component:  koji                           
 Keywords:            |  
----------------------+-----------------------------------------------------
 For a while now I've been providing deltaisos between Alpha, Beta, and
 Final (in addition to the ones for TCs/RCs) at
 [http://thepiratebay.org/user/andre14965/].

  * When N Alpha is released I create 2 torrents from (N-1) Final to N
 Alpha (one for i386 and one for x86_64).
  * When N Beta is released I create 2 torrents from (N-1) Final to N Beta,
 and 2 torrents from N Alpha to N Beta.
  * When N Final is released I create 2 torrents from (N-1) Final to N
 Final, and 2 torrents from N Beta to N Final.

 For each milestone, each corresponding torrent consists of

  * The old and new signed checksum files for the full ISOs,
  * A deltaiso from the old DVD to the new DVD, and
  * Deltaisos from the new DVD to each of the new CDs and the new netinst
 (since split media is gone for F15 and later, only the one for DVD ->
 netinst will be needed in the future).

 The size of the diso for DVD -> netinst is negligible (around 50K or
 less). The size of the disos starting with (N-1) Final is roughly half the
 size of the full ISO, and increases only slightly from N Alpha to N Beta
 to N Final. The size of the disos for N Alpha -> N Beta and N Beta -> N
 Final is around 10-20% of the full ISO size (you can see the exact sizes
 at the above torrent link).

 The same exact format could be used to provide these torrents at
 [http://torrent.fedoraproject.org]. For direct download mirrors, just the
 disos could be posted. It's unnecessary to create any new signed checksum
 files. It may be desirable to provide a crude checksum for the disos just
 to make sure the download is good before running applydeltaiso (which can
 take a significant amount of time - around 40-45 minutes for the disos
 starting with (N-1) Final, and around 10-20 minutes for the ones from N
 Alpha -> N Beta and N Beta -> N Final - the time for DVD -> netinst is
 negligible).

 The main obstacle is that each RPM in the new ISO must be compressed using
 the exact same compression. For example, the recent change in xz
 compression means that Rawhide currently consists of RPMs built using both
 the old and new compression, so it would be impossible to generate working
 disos from 14 Final to either 15 Alpha, 15 Beta, or 15 Final. Deltaisos
 for 15 Alpha -> 15 Beta and 15 Beta -> 15 Final should work (assuming that
 the user has the new xz installed on the machine which runs
 applydeltaiso). For the user to temporarily change xz version is a minor
 nuisance - someone running anything from F11 to F14 can temporarily change
 the xz-\* packages to the F15/Rawhide version, and someone running
 F15/Rawhide can temporarily downgrade xz-\* to the F14 version. Of course
 it's important to restore the original versions of xz-\* after running
 applydeltaiso, otherwise yum-presto won't be able to apply drpms in
 updates.

 Some links regarding the xz compression change:
  * https://bugzilla.redhat.com/show_bug.cgi?id=644046
  * http://lists.fedoraproject.org/pipermail/devel/2010-October/144651.html
  * http://lists.fedoraproject.org/pipermail/test/2010-October/094883.html
  * https://fedorahosted.org/rel-eng/ticket/4224

 I volunteer to do any necessary work in creating and checking the diso
 content (though it's actually pretty trivial) or in writing user
 documentation. In closing I'd like to point out that although refining
 compression in order to reduce the size of full ISOs is important, at some
 point there will be diminishing returns (I don't know if it's reasonable
 to expect the large improvement of 10-15% in going from gzip to xz to
 happen again). On the other hand, there's no reason to expect delta
 compression, which reduces download size by 50% or more, to get any worse.
 So in the long run, it doesn't make sense to keep ignoring delta
 compression in the quest to make ISOs a few percent smaller.

-- 
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project

[Fedora Release Engineering]

#4262: Provide deltaisos for Alpha, Beta, Final
-----------------------+----------------------------------------------------
  Reporter:  robatino  |       Owner:  rel-eng(a)lists.fedoraproject.org
      Type:  task      |      Status:  closed                         
 Milestone:            |   Component:  koji                           
Resolution:  wontfix   |    Keywords:                                 
-----------------------+----------------------------------------------------
Changes (by ausil):

  * status:  new => closed
  * resolution:  => wontfix

Comment:

 This is not something that really feasable for us to support. id rather
 see us support something like zsync of some tool that a user can run to
 easily sync from the isos. on a mirror.

 this requires releng to run a bunch of extra commands and qa to test a
 bunch of extra things.  making sure that it works as expected.

-- 
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262#comment:1>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project

[Fedora Release Engineering]

#4262: Provide deltaisos for Alpha, Beta, Final
-----------------------+----------------------------------------------------
  Reporter:  robatino  |       Owner:  rel-eng(a)lists.fedoraproject.org
      Type:  task      |      Status:  reopened                       
 Milestone:            |   Component:  koji                           
Resolution:            |    Keywords:                                 
-----------------------+----------------------------------------------------
Changes (by robatino):

  * status:  closed => reopened
  * resolution:  wontfix =>

Comment:

 For Alpha, Beta, and Final, which are available on the main mirrors, zsync
 is irrelevant since rsync is already available there as well. In any case,
 rsync/zsync only save space by not downloading unchanged RPMs, and between
 Alpha/Beta/Final, almost all RPMs change, so the savings are very small,
 which is why people don't bother to use it for converting between
 Alpha/Beta/Final. AFAIK there is no tool that works as you suggest that
 can provide large bandwidth savings under these conditions. If there was,
 it could replace deltarpms for updating, so that entire infrastructure
 would be unnecessary. And there are only a few disos which need to be
 created for each milestone, which can easily be done manually (and I've
 already offered to do it).

 I'm looking into getting enough disk space in either my fedorapeople
 account or alt.fp.o to host these, so if that's possible I could do this
 entirely by myself and releng wouldn't need to be involved at all.
 Something on the order of 20 GB (about 10 times the standard quota of 2000
 MiB) would be enough.

-- 
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262#comment:2>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project

[Fedora Release Engineering]

#4262: Provide deltaisos for Alpha, Beta, Final
-----------------------+----------------------------------------------------
  Reporter:  robatino  |       Owner:  rel-eng(a)lists.fedoraproject.org
      Type:  task      |      Status:  closed                         
 Milestone:            |   Component:  koji                           
Resolution:  wontfix   |    Keywords:                                 
-----------------------+----------------------------------------------------
Changes (by ausil):

  * status:  reopened => closed
  * resolution:  => wontfix

Comment:

 this is really something releng is not going to do. it really will not be
 fixed closing again

-- 
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/4262#comment:3>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project