Updating RPMs using binary deltas (demo)

[Nigel Metheringham]

On Wed, 2004-01-28 at 13:55, Leonard den Ottolander wrote:
> > rsync currently puts a load on the server because it reads the file and
> > builds a set of hash data for each request.
> > 
> > It would appear quite possible to produce either:-
> >       * a prebuilt table of file hashes using some external tool
> >       * a cache of hashes held by the rsync server
> 
> Please have a look at the scripts that came with the post that started
> this thread. Do you have any reason to assume that these "rsync hashes"
> can be more efficient than the deltas generated by Mike's script?

This getting way off topic, *but* it would be very useful to have an
rsync server which used/maintained a saved file hash set because it
would be of general use on the mirrors, saving them significant CPU/IO
usage (or saving them turning off the rsync crc code due to lack of
CPU/IO).

Whether this would be particularly useful for the specific case of
downloading update rpms when you have the original rpm is very debatable
- probably not since the major rpm payload is compressed and probably
not amenable to rsync.

However, even if this rpm delta stuff works well it requires the mirrors
to carry another pile of files of the order of 50% of the size of
current updates *as* *well* *as* the current updates themselves.  As a
mirror operator I have to say that the case for giving a distribution
yet another huge chunk of disk for what appears to be a corner case
seems pretty weak.

But this is still rattling on on a distribution list.  A basic proof of
concept has been posted (serious kudos to the guy who did it - it took
me a good few minutes of puzzling to work out what the hell he had
done), but this needs to be taken to the rpm list or some new rpm delta
development forum and made into something that really could be used,
including a file format (shipping pairs of delta files will not work), a
naming, metadata and maybe signing system, and then think about
integrating it.  Present stuff is about as appropriate as discussing how
we are going to handle the 2.8 (yes thats the right number) kernel in
Fedora - and exercise in vapourware planning.

	Nigel.

-- 
[ Nigel Metheringham           Nigel.Metheringham at InTechnology.co.uk ]
[ - Comments in this message are my own and not ITO opinion/policy - ]