F21 downloads repository metadata in 3 places!

[Richard Hughes]

On 13 December 2014 at 21:10, Hedayat Vatankhah <hedayat.fwd at gmail.com> wrote:
> Surprisingly, PackageKit uses its own separate cache.

Not surprising at all, when you're familiar with how PackageKit works.
PackageKit has to accept transactions from clients and return results
very quickly. Just something as simple as SHA'ing a metadata file
destroys our latency, which is one of the biggest reasons nobody liked
the command-not-found functionality when it was introduced: it was
SLOW. This interactive command had to return results in ~100ms, not
tens of seconds.

By having 100% complete control of a copy of the cache we can keep
certain files locked in memory, and we can be aggressive about caching
pools of packages. This allows us to achieve the low-latency design
required by gnome-software, which is firing off tons of transactions
in parallel at startup with expected latency guarantees. Another thing
it allows us to do is atomically update the cache, so if we're
updating the cache in the background and we get interrupted or the
transaction is cancelled to make room for a user-requester
"interactive" transaction, we can just continue to use the old cache,
and then atomically rename the new location to the proper location and
update pools when done. You just can't do this when there are three
things fiddling with files behind your back without any co-ordination.

Now, if we had a metadata format that was just one file (e.g. primary,
other, filelists, etc) smushed into one file (possibly also joined
fedora+updates, as the packages expect to be depsolved together) then
we could share it with dnf and yum if the promise was to do atomic
renames. What can't happen if for yum to just update repomd.xml and
primary.xml, leaving the other files in a half-dead and invalid state
(the files have indexes to each other internally) and then for the
user to launch gnome-software and there be a ~5 minute delay while all
the other required files are downloaded and indexes created.

Note, if yum or DNF wanted to use the PK cache, it's guaranteed to be
valid, complete and up to date, although I'm not sure a dependency
from the package manager CLI to PK would be acceptable for their
maintainers.

Richard.