Koji patches for no longer creating hardlinks for package repodata

[Jesse Keating]

We've talked about ways of making new repo tasks faster, and one of the
big speedups is no longer making a tree of hardlinks to run createrepo
over.

Recently createrepo gained[1] the ability to read from an include list.
Paired with the option to specify a baseurl to make everything relative
from, we can now run createrepo directly on the packages/ dir, using
the include file (generated from the list of packages we /would/ have
hardlinked), use an http url as the baseurl (so that static-repos
continue to work) and skip the hardlink step all together.  An added
bonus is that when a static-repo expires, the metadata you may be
working with will continue to reference valid package URLS so your
transaction will continue without failure.

While I was working with this code, I also noticed that we can stop
creating a buildsys-build rpm and instead rely on the groupdata in the
repodata and just use 'groupinstall buildsys-build' as are chroot init
command.  Add to that we were unnecessarily linking comps.xml files
into each arch directory of a repo directory, when instead we can just
reference the one we create in groups/ when creating repodata.

Since we no longer need to make the buildgroups rpm, the PrepRepo task
in kojid is no longer needed.  I took the one part that it was still
usefully doing (asking the hub to prep the repo dirs) and move it up to
the NewRepo task.  This saves a task creation and pickup/handoff.

I introduced a config item in kojid.conf, which is pkgurl.  This is
used as the baseurl component to the repodata and should be the url
which leads to the packages/ directory.

Finally, since the repodata itself will continue to live in the same
expected locations, builders can continue to consume the repodata via
nfs.  But since the repodata itself references http, yum will use http
to download the packages to populate the buildroot.  We just have to
ensure that the url used in pkgurl is not only valid in the outside
world (so that static-repos continue to work) but valid within the
buildsystem structure as well.

I've committed my changes to a git tree (cloned from tip today) here:

http://jkeating.fedorapeople.org/git/koji/

Feedback would be welcome.

We're going to get the createrepo change into RHEL5.2 (along with
--update support) with the idea that when we refresh the Fedora
buildsystem machines in early December, we'll go with RHEL5 and perhaps
a newer createrepo package.  Ideally we'd get the changes I've made
into koji beforehand so that we can release a new build of koji during
the refresh.

[1]
https://lists.dulug.duke.edu/pipermail/rpm-metadata/2007-November/000812....

-- 
Jesse Keating
Fedora -- All my bits are free, are yours?

[Jesse Keating]

On Wed, 14 Nov 2007 20:35:08 -0500
Jesse Keating <jkeating(a)redhat.com&gt; wrote:

>  I've committed my changes to a git tree (cloned from tip today)
> here:
>  
>  http://jkeating.fedorapeople.org/git/koji/
>  
>  Feedback would be welcome. 
I've rebased this repo against current git.  Attached is a diff of the
result of applying all the commits from the above repo.



-- 
Jesse Keating
Fedora -- All my bits are free, are yours?