[RFC] Optionally using git repositories instead of the lookaside cache
Hi folks,
The Fedora kernel is moving to maintaining the package in a source
(sometimes people refer to it as an "exploded") tree. Basically just a
fork of upstream. This makes a lot of packager tasks easier, but has
introduced a minor issue with respect to the lookaside cache.
Right now, it's configured to create a tarball from the git tree and
upload it to the lookaside cache for each build. We build the rawhide
kernel every weekday (give or take) and the xz compressed source
tarball is ~110MB. This works out to about 28GB per year for Rawhide
alone (if this is a drop in the bucket and no one cares please let me
know and we'll just do this). The old approach uploaded a release
tarball and then incremental tarballs on top of that.
If, however, Fedora allowed packagers to optionally generate tarballs
from a git repository we could just push the linux git repository. The
entire repository with history going back 15 years is under 4GB total,
which is pretty good when compared to ~419GB which is the space
required for the equivalent time using the lookaside cache.
What would need to change:
* Fedora offers a git repository to push source trees to.
* A new file in the dist-git repository could be added if the packager
wishes called "source-repos". In it, it contains a git url and commit
identifier. For example, an entry might look like:
"
https://src.fedoraproject.org/sources/kernel.git v5.6"
where v5.6 is a tag in the repository. We can restrict it so the git
repository must be hosted by Fedora so we keep all the sources
forever.
* fedpkg and fedpkg-minimal would need to be updated to pull the
source tree if the "source-repos" file is found and run
"git archive". Fortunately this work is actually already done since
Red Hat's version of fedpkg already supports this.
I'm happy do to all the work for fedpkg/fedpkg-minimal to make this
possible because the other option is to add a bunch of hacks to the
kernel tooling to spit out a bunch of incremental tarballs to reduce
what we have to upload.
I assume this is something that will need to go through the packaging
SIG, but from an infra side of things are there any thoughts/concerns?
Regards,
Jeremy