Thanks for the comments. Responses below.
-----Original Message-----
From: fedora-buildsys-list-bounces(a)redhat.com
[mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
Clark Williams
Sent: Wednesday, May 10, 2006 3:59 PM
To: Discussion of Fedora build system
Subject: Re: [RFC, PATCH] Build multiple srpms
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael_E_Brown(a)Dell.com wrote:
> [RFC, PATCH] Build multiple srpms
>
> Here is a patch for review. I have made the "obvious"
changes in order
> to be able to do the following:
> mock -r CFG srpm1 srpm2 srpm3 srpm4
>
I like the patch. It doesn't change the currently expected
behavior with one srpm, just adds the ability to build
multiple srpms from one chroot.
Exactly my intent.
Have you tried it with plague? If not, I'll put your version
on my local plague server and see if it causes any grief
(don't expect it to).
I did not see an *easy* way to set up a plague server, so we do not have
one, yet.
>
> Limitations:
> - There are no checks for dependencies between the srpms, so they
> should be independent. (no re-ordering of SRPMS)
> - Building one SRPM can leave the buildroot in an
inconsistent state
> for the next SRPM. (no clean of chroot between builds)
>
I don't have a problem with this. I'm not sure I buy the
argument that we need to do a clean of the chroot every time.
Partially that's because I do a lot of cross-tools stuff
which requires that I keep a chroot around for multiple
builds. But even discounting that, I don't see what building
an srpm in a chroot can do that will corrupt the chroot so
that a subsequent build will fail or be incorrect. Mostly
you're in there because you want a particular set of binaries
(programs and libraries). Once those are installed, who cares
if the rpm database gets trashed or the passwd file has some
crufty entries in it?
My thoughts as well. :)
> - Failure to build one SRPM stops the whole process.
>
I'm not sure that I would consider the "failure stops
everything" a limitation, since it saves you having to dig
through tons of log file entries to find where the failure
occurred (I never liked that make option anyway :)). You
could probably get away with removing the
sys.exit() in the for loop, but then you'd have to remember
the exit status, etc.
Same here.
> - Resulting RPMs are not installed into the build
environment for use
> by subsequent SRPMS.
>
> Reasoning:
> One problem I have been having with mock is speed. We are
> using mock to build for 16 distinct configurations. Doing
prep on each
> configuration was taking a minute and a half to two minutes. As an
> optimization, our mock wrapper script was doing a single prep per
> chroot and then using --no-clean, but this complicates our wrapper
> script. I am building a set of related RPMS, so I don have much
> concern about cross-pollution. This simple patch simplifies
my wrapper
> script considerably.
>
What sort of speed improvement are you seeing now?
None, yet. Right now, this patch really just functions to make the rest
of my scripting simpler. I believe that the speed boost will probably
happen in stage 3. This patch is stage 1, and the patch I just sent next
is stage 2. In stage 3, I would like to overlap prep() of one build root
with building in another.
> Future:
> One future direction I'd like to take this is a
parallel mock
> that can prep/build multiple configurations at the same
time to try to
> amortize the cost of prep stage by running prep of one
environment in
> parallel with build of a different configuration.
> Currently, it is taking me an hour to build 16 sets of RPMS
(5 SRPMS
> per set), and I am hoping to get this down. We have already
> implemented squid and some other measures to try to speed things up.
>
You could probably achieve the same result by using plague
and invoking multiple plague-client builds from your wrapper
script. You wouldn't be able to "pipeline" the builds (i.e.
wait for build1 to reach build state, then start build2), but
this will be simpler.
Certainly simpler than threading mock.
Plague looked a bit complicated to set up when I last looked at it. I
will probably look at it again soon.
--
Michael