On Mon, 2018-03-12 at 15:42 +0100, Michal Domonkos wrote:
To me, the zchunk idea looks good.
Incidentally, for the last couple of months, I have been trying to
rethink the way we cache metadata on the clients, as part of the
libdnf (re)design efforts. My goal was to de-duplicate the data
between similar repos in the cache as well as decrease the size that
needs to be downloaded every time (inevitably leading to this topic).
I came up with two different strategies:
That made me think that either using git (libgit2) directly or doing
small, lightweight implementation of the core concepts might be the
way to go. I even played with the latter a bit (I didn't get to
breaking down primary.xml, though):
In the context of this thread, this is basically what you do with
zchunk (just much better) :)
Yes, I guess. ;) The git concept sounds interesting, but it will be a
lot of work and will require some huge changes in how we deal with
metadata. For the moment, I think I'll focus on more evolutionary
Later, during this year's devconf, I had a few "brainstorming"
sessions with Florian Festi who pointed out that the differences in
metadata updates might often be on the sub-package level (e.g. NEVRA
in the version tag) so chunking on the package boundaries might not
give us the best results possible. Instead perhaps, we could generate
deltas on the binary level.
My current tests were chunked on srpm boundaries, not package
boundaries (not sure if we're using the same terminology here). The
problem with chunking on the package boundary in the xml is that it
creates many more chunks than grouping by srpm, and the smaller chunks
hurt compression (even with a dictionary) and the larger number
increase the size of the index. In Fedora, I think it's impossible to
only change the metadata for one sub-package belonging to an srpm, and,
even if it *is* possible, it's very rare.
An alternative would be to pre-generate (compressed) binary deltas
the last N versions and let clients download an index file that will
tell them what deltas they're missing and should download. This is
basically what debian's pdiff format does. One downside to this
approach is that it doesn't give us the de-duplication on clients
consuming multiple repos with similar content (probably quite common
with RHEL subscriptions at least).
I'm not a huge fan of pre-generated binary deltas. They might give
smaller deltas than a chunking solution, but at the cost of generating
and maintaining the deltas. It's been enough of a pain for rpms that
(at least on an individual basis) change infrequently. For metadata
that changes every day, I think it would be too much.
The beauty of the zchunk format (or zsync, or any other chunked format)
is that we don't have to download different files based on what we
have, but rather, we download either fewer or more parts of the same
file based on what we have. From the server side, we don't have to
worry about the deltas, and the clients just get what they need.
Having said that, the current zchunk proposal doesn't really address
deduplication from multiple repos with similar content. I suppose some
way of extending zchunk to allow you to specify multiple local sources
would be a way fixing that, but I'm still working on getting it to
build from *one* local source (getting very close, but not quite
Then I stumbled upon casync which combines the benefits of both
strategies; it chunks based on the shape of the data (arguably giving
better results than chunking on the package boundaries), and it
doesn't require a smart protocol. However, it involves a lot of HTTP
requests as you already mentioned.
Just to be clear, zchunk is casync with smart boundaries (or, better
put, application-specified boundaries) and a single file backend so
requests can be sent as http range requests rather than hundreds of
individual http requests.
Despite that, I'm still leaning towards chunking as being the
solution of the two. The question is, how much granularity we want.
You made a good point: the repodata format is fixed (be it xml or
solv), so we might as well take advantage of it to detect boundaries
for chunking, rather than using a rolling hash (but I have no data to
back it up). I'm not sure how to approach the many-GET-requests (or
the lack of range support) problem, though.
If you've been following the conversation between myself and Michael
Schroeder on rpm-ecosystem (starting with http://lists.rpm.org/pipermai
l/rpm-ecosystem/2018-March/000553.html), I did some comparisons between
zsync (which uses a rolling hash) and zchunk. There's also the fact
that zsync uses gzip compression, while zchunk uses zstd by default,
but quite a bit of the difference is due to the larger window size a
rolling hash provides.
With zchunk, if you get a server with no range support, you just
download the full zchunk file. I did run a check on the Fedora
mirrors, and I think there were two or three that didn't support range
requests at all.
As part of my efforts, I created this "small" git repo that
metadata snapshots since ~February which can be useful to see how
typical metadata updates look like. Feel free to use it (e.g. for
testing out zchunk):
This looks great! I've got some random days in January and February,
so this looks far more consistent.
Thanks much for your insight,