On Wed, 2004-01-28 at 13:55, Leonard den Ottolander wrote:
rsync currently puts a load on the server because it reads the file and builds a set of hash data for each request.
It would appear quite possible to produce either:- * a prebuilt table of file hashes using some external tool * a cache of hashes held by the rsync server
Please have a look at the scripts that came with the post that started this thread. Do you have any reason to assume that these "rsync hashes" can be more efficient than the deltas generated by Mike's script?
This getting way off topic, *but* it would be very useful to have an rsync server which used/maintained a saved file hash set because it would be of general use on the mirrors, saving them significant CPU/IO usage (or saving them turning off the rsync crc code due to lack of CPU/IO).
Whether this would be particularly useful for the specific case of downloading update rpms when you have the original rpm is very debatable - probably not since the major rpm payload is compressed and probably not amenable to rsync.
However, even if this rpm delta stuff works well it requires the mirrors to carry another pile of files of the order of 50% of the size of current updates *as* *well* *as* the current updates themselves. As a mirror operator I have to say that the case for giving a distribution yet another huge chunk of disk for what appears to be a corner case seems pretty weak.
But this is still rattling on on a distribution list. A basic proof of concept has been posted (serious kudos to the guy who did it - it took me a good few minutes of puzzling to work out what the hell he had done), but this needs to be taken to the rpm list or some new rpm delta development forum and made into something that really could be used, including a file format (shipping pairs of delta files will not work), a naming, metadata and maybe signing system, and then think about integrating it. Present stuff is about as appropriate as discussing how we are going to handle the 2.8 (yes thats the right number) kernel in Fedora - and exercise in vapourware planning.
Nigel.