RE: mock goals and non-goals
by Michael E Brown
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> seth vidal
> Sent: Thursday, May 11, 2006 9:15 AM
> To: Discussion of Fedora build system
> Subject: mock goals and non-goals
Thanks for the valuable and constructive feedback. I was just testing
the limits, as I have a mildly complex build environment that I am
trying to set up and manage. Looks like I will be investigating plague a
bit more.
> So, with regard to the patches we've seen so far here's what
> makes sense to me:
> - global/shared config to reduce the replication in the mock configs
> - specifying multiple srpms on the command line for a SINGLE
> config - provided that the buildroot is refreshed/cleaned
> each time - unless, of course, the --no-clean option is passed in
I will refresh my patch from yesterday to do this and resend. Doing it
this way is a one line change.
> - caching mechanism to create a cached pristine chroot that
> will allow for quickly recreating the chroot into that state
> simply by putting that cached copy back.
I will dig out the patch from the archives and test/resend (though I am
not the original author). I have 5 RPMS I build across 16 configs. With
clean, this takes about 5 hours. With single clean/init to set up the
environment and --no-clean for each individual RPM, that reduces the
time to 45-mins to an hour.
>
> Patches that don't make sense to me:
> - multiconfig - this is what shells and/or plague is for, use
> one of them.
Secret message received and understood. :)
> - parallel builds - plague is your friend.
Ditto.
Thanks,
Michael
16 years, 10 months
RE: [RFC, PATCH] Build multiple srpms
by Michael E Brown
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> Clark Williams
>
> I'm going to set up another plague system on my home server.
> I'll document those steps and send them to you; maybe we can
> either make a Plague for the Clueless document (I'm
> *eminently* qualified) or we can automate it so that you
> could install an RPM and voila! a plague server.
That would be much appreciated. An RPM that installs a functioning
plague server would be goodness. One nice thing about mock is that you
install the RPM and have a working (but slow) configuration. Make a
couple of obvious config changes to use a local mirror and everything is
good.
An equivalent situation for plague would be goodness.
--
Michael
16 years, 10 months
Plague 0.5 update: builder-to-server communication
by Dan Williams
Hi,
I checked in the last bits for builder-initiated communication earlier
this week to cvs HEAD. This means that builder instances no longer have
to run _any_ server (XML-RPC, HTTP, or otherwise), and that builders can
be behind a firewall without problems. I call these "Active" builders,
as they initiate all communication to to the build server.
It appears to work with light use, but there are still bugs and
edge-cases to work out.
Next steps include beating down bugs introduced by this refactor,
finishing up the server-to-builder ("Passive") communication code,
writing documentation, and getting more testing. I'd also like to make
sure the plague directory structure is FHS-clean
Somebody is sending me an SVN patch too.
Dan
16 years, 10 months
RE: [RFC, PATCH] Build multiple srpms
by Michael E Brown
Thanks for the comments. Responses below.
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> Clark Williams
> Sent: Wednesday, May 10, 2006 3:59 PM
> To: Discussion of Fedora build system
> Subject: Re: [RFC, PATCH] Build multiple srpms
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Michael_E_Brown(a)Dell.com wrote:
> > [RFC, PATCH] Build multiple srpms
> >
> > Here is a patch for review. I have made the "obvious"
> changes in order
> > to be able to do the following:
> > mock -r CFG srpm1 srpm2 srpm3 srpm4
> >
>
> I like the patch. It doesn't change the currently expected
> behavior with one srpm, just adds the ability to build
> multiple srpms from one chroot.
Exactly my intent.
>
> Have you tried it with plague? If not, I'll put your version
> on my local plague server and see if it causes any grief
> (don't expect it to).
I did not see an *easy* way to set up a plague server, so we do not have
one, yet.
> >
> > Limitations:
> > - There are no checks for dependencies between the srpms, so they
> > should be independent. (no re-ordering of SRPMS)
> > - Building one SRPM can leave the buildroot in an
> inconsistent state
> > for the next SRPM. (no clean of chroot between builds)
> >
> I don't have a problem with this. I'm not sure I buy the
> argument that we need to do a clean of the chroot every time.
> Partially that's because I do a lot of cross-tools stuff
> which requires that I keep a chroot around for multiple
> builds. But even discounting that, I don't see what building
> an srpm in a chroot can do that will corrupt the chroot so
> that a subsequent build will fail or be incorrect. Mostly
> you're in there because you want a particular set of binaries
> (programs and libraries). Once those are installed, who cares
> if the rpm database gets trashed or the passwd file has some
> crufty entries in it?
My thoughts as well. :)
>
> > - Failure to build one SRPM stops the whole process.
> >
>
> I'm not sure that I would consider the "failure stops
> everything" a limitation, since it saves you having to dig
> through tons of log file entries to find where the failure
> occurred (I never liked that make option anyway :)). You
> could probably get away with removing the
> sys.exit() in the for loop, but then you'd have to remember
> the exit status, etc.
Same here.
>
> > - Resulting RPMs are not installed into the build
> environment for use
> > by subsequent SRPMS.
> >
> > Reasoning:
> > One problem I have been having with mock is speed. We are
> > using mock to build for 16 distinct configurations. Doing
> prep on each
> > configuration was taking a minute and a half to two minutes. As an
> > optimization, our mock wrapper script was doing a single prep per
> > chroot and then using --no-clean, but this complicates our wrapper
> > script. I am building a set of related RPMS, so I don have much
> > concern about cross-pollution. This simple patch simplifies
> my wrapper
> > script considerably.
> >
>
> What sort of speed improvement are you seeing now?
None, yet. Right now, this patch really just functions to make the rest
of my scripting simpler. I believe that the speed boost will probably
happen in stage 3. This patch is stage 1, and the patch I just sent next
is stage 2. In stage 3, I would like to overlap prep() of one build root
with building in another.
>
> > Future:
> > One future direction I'd like to take this is a
> parallel mock
> > that can prep/build multiple configurations at the same
> time to try to
> > amortize the cost of prep stage by running prep of one
> environment in
> > parallel with build of a different configuration.
> > Currently, it is taking me an hour to build 16 sets of RPMS
> (5 SRPMS
> > per set), and I am hoping to get this down. We have already
> > implemented squid and some other measures to try to speed things up.
> >
> You could probably achieve the same result by using plague
> and invoking multiple plague-client builds from your wrapper
> script. You wouldn't be able to "pipeline" the builds (i.e.
> wait for build1 to reach build state, then start build2), but
> this will be simpler.
> Certainly simpler than threading mock.
Plague looked a bit complicated to set up when I last looked at it. I
will probably look at it again soon.
--
Michael
16 years, 10 months
[RFC PATCH] Mock multi-config build patch -- implement multi-config
by Michael E Brown
And here is the multi-srpm, multi-config patch, built on top of the
previous two. The changes are relatively minor, just re-indenting the
last part of main(), no significant code changes except changing
options.chroot to an array and adding a loop.
Please let me know what you think.
--
Michael
________________________________
From: fedora-buildsys-list-bounces(a)redhat.com
[mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
Michael_E_Brown(a)Dell.com
Sent: Wednesday, May 10, 2006 4:21 PM
To: fedora-buildsys-list(a)redhat.com
Subject: [RFC PATCH] Mock multi-config build patch -- prepare
All,
Here is stage one of a patch series to implement
multi-config builds with mock. I believe this patch stands on its own,
so I am sending it separately. It is applied on top of the earlier patch
today to do multi-srpm builds.
The patch simply re-arranges main() by taking and
putting several sections into their own function. For example, the clean
and prep sections were very similar and have been factored into their
own function. The setting of defaults for config_opts is moved into its
own function.
The result is no functional change to the operation of
mock, but it makes the next patch in my series much easier, which is the
addition of multiple config builds, like this:
$ mock -r CFG1 -r CFG2 -r CFG3 -r CFG4 ... srpm1 srpm2
srpm3 ...
The next patch in the series will be a straightforward
patch to implement the above on top of this patch. The last patch I
would like to send would be to be able to run the prep() sections for
the different configs in parallel (simple, via fork()) to reduce setup
time.
Comments welcome.
--
Michael
<<mock-multicfg.patch>>
16 years, 10 months
RE: Patches for shared configurations items and specifying additional dependencies
by Michael E Brown
I cannot really speak for the mock guys, but to me, these patches look
very good.
-- self-contained
-- limited scope
-- elegant
They add required functionality without making changes across the
codebase. In particular, your solution for the additional dependencies
looks elegant and cool.
--
Michael
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> Andreas Thienemann
> Sent: Wednesday, May 10, 2006 3:27 PM
> To: Discussion of Fedora build system
> Subject: Patches for shared configurations items and
> specifying additional dependencies
>
> Hello List,
>
> as it seems to be patch submission time, I'm pushing two
> patches to the list, I had lying around for some time now.
>
> The first patch allows mock to parse a generic side-wide
> config file, where general configuration-items can be stored
> and later overridden by specific chroot-definitions.
>
>
> The second patch, is a bit different:
> While pushing the Fedora Core SRPMS through the mock
> buildsystem we noticed many SRPMS which do not correctly
> define their buildrequirements.
> Until the fedora core developer responsible for the package
> has updated the spec in cvs and pushed an updated srpm into
> rawhide, there is no way to make the package in question
> build in mock without having to change the .spec.
> Thus, in order to allow even these broken packages being
> built, we needed a way to work around these broken
> dependencies to see if anything else was missing.
> This is done by the second patch, which adds a more_buildreqs
> config option. If set the given dependencies are installed
> _in addition_ to the ones from the .spec file.
>
> While the second patch should not be used during normal use
> and is definetly not for the fedora-extras buildservers, it
> still is of interest to other parties, using mock.
>
>
>
> regards,
> andreas
>
16 years, 10 months
[RFC PATCH] Mock multi-config build patch -- prepare
by Michael E Brown
All,
Here is stage one of a patch series to implement multi-config
builds with mock. I believe this patch stands on its own, so I am
sending it separately. It is applied on top of the earlier patch today
to do multi-srpm builds.
The patch simply re-arranges main() by taking and putting
several sections into their own function. For example, the clean and
prep sections were very similar and have been factored into their own
function. The setting of defaults for config_opts is moved into its own
function.
The result is no functional change to the operation of mock, but
it makes the next patch in my series much easier, which is the addition
of multiple config builds, like this:
$ mock -r CFG1 -r CFG2 -r CFG3 -r CFG4 ... srpm1 srpm2 srpm3
...
The next patch in the series will be a straightforward patch to
implement the above on top of this patch. The last patch I would like to
send would be to be able to run the prep() sections for the different
configs in parallel (simple, via fork()) to reduce setup time.
Comments welcome.
--
Michael
<<mock-multicfg.patch>>
16 years, 10 months
SLES9 and SLES10 enablement patches committed
by Matt Domsch
I've committed the two patches below to mock in CVS, which enables
mock to be used to create build environments for Novell/SuSE SLES 9
and SLES10. In their environments, some files already exist so mock
shouldn't create them; useradd needs the -m flag to create the users's
home dir; we moved several iterations of chmod/chroot invocations out of the loop, and
instead do a 'chown -R' once (saves calling chroot as often); and
finally make sure files in /etc/ that should be writeable are.
Patches are by Michael E. Brown <Michael_E_Brown(a)dell.com> and myself.
Thanks,
Matt
--
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com
--- /usr/bin/mock 2006-04-19 14:31:47.000000000 -0500
+++ ./mock 2006-04-19 14:32:16.000000000 -0500
@@ -566,8 +566,9 @@
for item in [os.path.join(self.rootdir, 'etc', 'mtab'),
os.path.join(self.rootdir, 'etc', 'fstab'),
os.path.join(self.rootdir, 'var', 'log', 'yum.log')]:
- fo = open(item, 'w')
- fo.close()
+ if not os.path.exists(item):
+ fo = open(item, 'w')
+ fo.close()
# write in yum.conf into chroot
yumconf = os.path.join(self.rootdir, 'etc', 'yum.conf')
@@ -589,7 +590,7 @@
if not os.path.exists(self.rootdir + self.homedir):
if not os.path.exists(os.path.join(self.rootdir, 'usr/sbin/useradd')):
raise RootError, "Could not find useradd in chroot, maybe the install failed?"
- cmd = '/usr/sbin/useradd -u %s -d %s %s' % (self.config['chrootuid'],
+ cmd = '/usr/sbin/useradd -m -u %s -d %s %s' % (self.config['chrootuid'],
self.homedir, self.config['chrootuser'])
self.do_chroot(cmd, fatal = True)
@@ -604,9 +605,11 @@
for subdir in ('RPMS', 'SRPMS', 'SOURCES', 'SPECS', 'BUILD', 'originals'):
cmd = "mkdir -p %s/%s" % (self.builddir, subdir)
self.do_chroot(cmd, fatal = True)
- cmd = "chown %s.%s %s/%s" % (self.config['chrootuser'],
- self.config['chrootgroup'], self.builddir, subdir)
- self.do_chroot(cmd, fatal = True)
+
+ # change ownership so we can write to build home dir
+ cmd = "chown -R %s.%s %s" % (self.config['chrootuser'],
+ self.config['chrootgroup'], self.homedir)
+ self.do_chroot(cmd, fatal = True)
# rpmmacros default
macrofile_out = '%s%s/.rpmmacros' % (self.rootdir, self.homedir)
--- mock 2006-04-24 10:04:14.000000000 -0500
+++ mock.new 2006-05-05 16:16:09.000000000 -0500
@@ -579,6 +579,13 @@
# files in /etc that need doing
filedict = self.config['files']
for key in filedict:
+ # ensure permisssions
+ if os.path.exists( "%s%s" % (self.rootdir, key) ):
+ cmd = "chown %s.%s %s" % (self.config['chrootuser'],
+ self.config['chrootgroup'], "%s" % key)
+ self.do_chroot(cmd, fatal = True)
+
+ # write file
fn = '%s%s' % (self.rootdir, key)
fo = open(fn, 'w')
fo.write(filedict[key])
16 years, 10 months
Re: [Fedora-sysadmin-list] rpm on build machines
by seth vidal
On Thu, 2006-05-04 at 21:29 -0400, Jesse Keating wrote:
> Due to the bug we found in rpm, tickled by the glibc package in rawhide,
> we'll need to get an updated rpm onto the build machines.
>
> We have one package for RHEL4/CentOS4 that skips over the bug, but Paul
> Nasrat is working on a real fix. Until we get something on the build
> machines no builds on development can happen.
>
> I can provide the rpms that skip over the bug for now, your call.
What bug is this? Does it have a bug #?
-sv
16 years, 11 months
