[PATCH] slim main()
by Michael E Brown
The attached patch contains no functional changes. It simply introduces
a few helper functions and slims down main().
def setup_default_config_opts(config_opts):
move the setup of default config_opts to a helper function
def set_config_opts_per_cmdline(config_opts, options):
All of the logic that sets config_opts stuff based on cmdline options
is here.
def do_clean(config_opts, init=0):
Consolidate clean/init handling into one function
def do_run_cmd(config_opts, cmd, raw_chroot=0):
Consolidate chroot/shell handling into one function
def ensure_filetype_srpm(srpms):
Check all cmdline srpm args to ensure they exist and are srpms
def do_rebuild(config_opts, srpms):
rebuild logic.
Please apply.
16 years, 10 months
[TRIVIAL PATCH] - Tab cleanup
by Michael E Brown
Attached is a patch to clean up a tab-error in my patch from last week.
I accidentally let a tab creep into one of the comments.
Please apply.
16 years, 10 months
Patches for shared configurations items and specifying additional dependencies
by Andreas Thienemann
Hello List,
as it seems to be patch submission time, I'm pushing two patches to the
list, I had lying around for some time now.
The first patch allows mock to parse a generic side-wide config file,
where general configuration-items can be stored and later overridden by
specific chroot-definitions.
The second patch, is a bit different:
While pushing the Fedora Core SRPMS through the mock buildsystem we
noticed many SRPMS which do not correctly define their buildrequirements.
Until the fedora core developer responsible for the package has updated
the spec in cvs and pushed an updated srpm into rawhide, there is no way
to make the package in question build in mock without having to change the
.spec.
Thus, in order to allow even these broken packages being built, we needed
a way to work around these broken dependencies to see if anything else was
missing.
This is done by the second patch, which adds a more_buildreqs config
option. If set the given dependencies are installed _in addition_ to the
ones from the .spec file.
While the second patch should not be used during normal use and is
definetly not for the fedora-extras buildservers, it still is of interest
to other parties, using mock.
regards,
andreas
Provide generic config defaults.
* Support a generic /etc/mock/defaults.cfg config file. The values
defined in this file are overwritten by any definitions in the
specific /etc/mock/CHROOT.cfg files.
This part just introduces mock-0.4/etc/defaults.cfg and
adds the last chunk in the patch to mock-0.4/mock.py.
This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in
2006 after an idea by Andreas Thienemann. Testing and final clean-up was
one by Andreas Thienemann.
The source code this patch is based on has been downloaded from
http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz
diff -Nurp mock-0.4.orig/etc/defaults.cfg mock-0.4.bawue/etc/defaults.cfg
--- mock-0.4.orig/etc/defaults.cfg 1970-01-01 01:00:00.000000000 +0100
+++ mock-0.4.bawue/etc/defaults.cfg 2006-04-05 21:59:22.000000000 +0200
@@ -0,0 +1,8 @@
+# mock defaults
+#
+# Define default values here.
+# These values are overwritten in the /etc/mock/CHROOT.cfg files.
+#
+# Example:
+#
+# config_opts['foo'] = bar
diff -Nurp mock-0.4.orig/mock.py mock-0.4.bawue/mock.py
--- mock-0.4.orig/mock.py 2005-07-26 22:33:52.000000000 +0200
+++ mock-0.4.bawue/mock.py 2006-04-06 06:59:51.000000000 +0200
@@ -683,6 +701,14 @@ def main():
if len(args) < 1:
error("No srpm or command specified - nothing to do")
sys.exit(50)
+
+ # Read in the default values which can be overwritten
+ # with the more specific config being loaded below.
+ cfg = os.path.join(config_path, 'defaults.cfg')
+ if os.path.exists(cfg):
+ execfile(cfg)
+ else:
+ pass # not finding the defaults.cfg file is no error
# read in the config file by chroot name
if options.chroot.endswith('.cfg'):
Provide extra build requirements for packages.
* Introduce a new config variable 'more_buildreqs' which allows
defining additional build dependencies for specific packages.
This is useful for buggy SRPMs or SRPMs which compile some
parts conditionally, depending on the software installed.
This mechanism should not be used in normal operation.
The extra build requirements can be configured like in the following
example, in which we use the example of a fictitional SRPM file
called 'ser-0.9.6-3'.
# matches only the 'ser-0.9.6-3' package
config_opts['more_buildreqs']['ser-0.9.6-3'] = [ 'smtp-devel >= 1.2',
'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ]
# matches all 'ser-0.9.6-*' packages
config_opts['more_buildreqs']['ser-0.9.6'] = [ 'smtp-devel >= 1.2',
'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ]
# matches all 'ser-*-*' packages
config_opts['more_buildreqs']['ser'] = [ 'smtp-devel >= 1.2',
'mysql-devel >= 5.0', 'png-devel < 0.1', 'tcsh' ]
The most specific match is used, i.e. if requirements for
'ser-0.9.6-3' are defined, the definitions for 'ser-0.9.6' and 'ser'
are ignored. Each time, /etc/mock/CHROOT.cfg is checked first, and
only if there is no dependency definition here, the definition from
/etc/mock/defaults.cfg is checked.
This patch was written in 2006 for bawue.net by Hans Ulrich Niedermann in
2006 after an idea by Andreas Thienemann. Testing and final clean-up was
done by Andreas Thienemann.
The source code this patch is based on has been downloaded from
http://fedoraproject.org/projects/mock/releases/mock-0.4.tar.gz
diff -Nurp mock-0.4.orig/mock.py mock-0.4.bawue/mock.py
--- mock-0.4.orig/mock.py 2005-07-26 22:33:52.000000000 +0200
+++ mock-0.4.bawue/mock.py 2006-04-06 06:59:51.000000000 +0200
@@ -278,7 +278,7 @@ class Root:
hdr = rpmUtils.miscutils.hdrFromPackage(ts, srpm)
# get text buildreqs
- buildreqs = self._text_requires_from_hdr(hdr)
+ buildreqs = self._text_requires_from_hdr(hdr, srpm)
arg_string = ""
for item in buildreqs:
@@ -469,7 +469,7 @@ class Root:
return (ret, output)
- def _text_requires_from_hdr(self, hdr):
+ def _text_requires_from_hdr(self, hdr, srpm):
"""take a header and hand back a unique'd list of the requires as
strings"""
@@ -487,6 +487,23 @@ class Root:
req = rpmUtils.miscutils.formatRequire(n, v, f)
reqlist.append(req)
+ # Extract SRPM name components - still not nice, shouldn't this
+ # be somewhere in the "hdr" parameter?
+ fname = os.path.split(str(srpm))[1]
+ name, ver, rel, epoch, arch = rpmUtils.miscutils.splitFilename(fname)
+
+ # Add the 'more_buildreqs' for this SRPM (if defined)
+ for this_srpm in ['-'.join([name,ver,rel]),
+ '-'.join([name,ver]),
+ '-'.join([name]),]:
+ if self.config['more_buildreqs'].has_key(this_srpm):
+ more_reqs = self.config['more_buildreqs'][this_srpm]
+ if type(more_reqs) in (type(u''), type(''),):
+ more_reqs = [more_reqs] # be nice if we get a string
+ for req in more_reqs:
+ reqlist.append(req)
+ break
+
return rpmUtils.miscutils.unique(reqlist)
def _prep_install(self):
@@ -674,6 +691,7 @@ def main():
""" % config_opts['chroothome']
+ config_opts['more_buildreqs'] = {}
config_opts['files']['/etc/resolv.conf'] = "nameserver 192.168.1.1\n"
config_opts['files']['/etc/hosts'] = "127.0.0.1 localhost localhost.localdomain\n"
16 years, 10 months
Avoiding install of x86_64 packages for i386 builds
by Jeff Sheltren
I'm having a problem when building packages which have a dependency
sitting in the plague 'repo_dir'. Since plague puts all arches of a
package in the same sub directory under 'repo_dir', if a package has
both i386 and x86_64 binaries, later builds which pull in those
packages as a dependency will install the x86_64 binary rather than
the i386 one. Yes, I have brought this problem on myself - I want to
have stuff in repo_dir be usable for builds even before those
packages are pushed to the main repository. In order to do this, I
have to point mock to the same repodir for both i386 and x86_64.
So, I have a couple of questions: is there a way to have plague sort
packages in the repo_dir differently than it does by default? How
does Extras avoid this problem?
Here's an example of the directory structure used:
---- Snip from /etc/mock/fedora-3-i386-core.cfg ---
[updates-plague]
name=updates-plague
baseurl=file:///build/plague/repodir/fedora-3-core/
---- End Snip ----
$ pwd
/build/plague/repodir/fedora-3-core
$ ls
mozilla repodata
$ ls mozilla/1.7.13-1.3.1.legacy/
i386 SRPM x86_64
Thanks,
Jeff
16 years, 10 months
mock goals and non-goals
by seth vidal
Hi folks,
Given the patchfest from yesterday I felt it would be best to have some
discussion of goals/non-goals for mock in/around fedora's build
environment.
This is my opinion on the subject. It is not final nor terminal, it just
is where I am:
goals:
- make clean and consistent build environments to be used by fedora's
buildsystem and packagers
- simplify building and testing process for packagers
- be able to canonically identify the 'build environment' for fedora
systems to discourage confusion on what should be where
- simple interface with good defaults.
non-goals:
- this is not a tool to manage many, many builds with queuing, etc -
that is plague.
So, with regard to the patches we've seen so far here's what makes sense
to me:
- global/shared config to reduce the replication in the mock configs
- specifying multiple srpms on the command line for a SINGLE config -
provided that the buildroot is refreshed/cleaned each time - unless, of
course, the --no-clean option is passed in
- caching mechanism to create a cached pristine chroot that will allow
for quickly recreating the chroot into that state simply by putting that
cached copy back.
Patches that don't make sense to me:
- multiconfig - this is what shells and/or plague is for, use one of
them.
- parallel builds - plague is your friend.
Patches which I am ambivalent about:
- the fake buildreqs/sneaky buildreqs patch for rebuilding items with
BROKEN buildrequires:
- I hate the idea of enabling broken shit
- I realize that there are lots of broken things that people need to
work around and I'm sympathetic to that, to some extent. Doesn't mean I
can't hate it.
so, there, my flames are all away, feel free to bombard me with your
own.
-sv
16 years, 10 months
[PATCH] resubmit: multisrpm build
by Michael E Brown
This is a resubmit of the patch from yesterday to build multiple
srpms in one command line:
$ mock -r CFG srpm1 srpm2 srpm3
It exits immediately on the first failure.
It cleans and preps the build environment for each srpm unless
--no-clean is passed.
--
Michael
<<multisrpm.patch>>
16 years, 10 months
interactive shell in chroot patch
by Clark Williams
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
While we're in patch-a-thon mode, thought I'd submit this for comment.
While attempting to track down a bizarre bug with elinks, I hacked
mock so that I could get an interactive shell in the chroot
environment. Not useful for everyone, but handy if you want to look at
the environment in which something is failing. Oh and I converted some
prints in my chroot command to use the debug method.
Index: mock.py
===================================================================
RCS file: /cvs/fedora/mock/mock.py,v
retrieving revision 1.43
diff -u -r1.43 mock.py
- --- mock.py 8 May 2006 16:39:53 -0000 1.43
+++ mock.py 11 May 2006 17:04:34 -0000
@@ -800,13 +800,28 @@
print 'Finished initializing root'
elif args[0] == 'chroot':
+ # catch-all for executing arbitrary commands in the chroot
config_opts['clean'] = config_opts['quiet'] = False
- - my= Root(config_opts)
+ my = Root(config_opts)
+ cmd = ' '.join(args[1:])
+ my.debug("executing: %s" % cmd)
my._mount()
- - my.do_chroot(' '.join(args[1:]), True)
+ my.do_chroot(cmd, True)
my.close()
- - print 'Finished chroot command'
+ my.debug('finished chroot command')
+ elif args[0] == 'shell':
+ # debugging tool for interactive poking around in the chroot
+ config_opts['clean'] = config_opts['quiet'] = False
+ my = Root(config_opts)
+ cmd = "PS1='mock-chroot> ' %s %s /bin/bash" %
(config_opts['chroot'],
+ my.rootdir)
+ my.debug("executing: %s" % cmd)
+ my._mount()
+ ret = os.system(cmd)
+ my.close()
+ my.debug("finished shell with retval %d" % ret)
+
else:
if args[0] == 'rebuild':
if len(args) > 1:
Index: src/mock-helper.c
===================================================================
RCS file: /cvs/fedora/mock/src/mock-helper.c,v
retrieving revision 1.7
diff -u -r1.7 mock-helper.c
- --- src/mock-helper.c 14 Jul 2005 18:00:26 -0000 1.7
+++ src/mock-helper.c 11 May 2006 17:04:34 -0000
@@ -25,7 +25,7 @@
static char const * const ALLOWED_ENV[] =
{
"dist",
- - "ftp_proxy", "http_proxy", "https_proxy", "no_proxy"
+ "ftp_proxy", "http_proxy", "https_proxy", "no_proxy", "PS1",
};
#define ALLOWED_ENV_SIZE (sizeof (ALLOWED_ENV) / sizeof (ALLOWED_ENV[0]))
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFEY3BpHyuj/+TTEp0RAvnBAJ9tqOtxB102szDxsv/QZGbHkF5SFgCg1mCn
QzgvjxfcsjBrfcKXOAg3Efs=
=iMVS
-----END PGP SIGNATURE-----
16 years, 10 months
RE: Patches for shared configurations items and specifying additional dependencies
by Michael E Brown
Don't know if anybody else brought this up, but could the
shared-configuration be implemented by putting an
"execfile('/etc/mock/default.cfg')" at the beginning of the config file?
No patches necessary.
--
Michael
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> Andreas Thienemann
> Sent: Wednesday, May 10, 2006 3:27 PM
> To: Discussion of Fedora build system
> Subject: Patches for shared configurations items and
> specifying additional dependencies
>
> Hello List,
>
> as it seems to be patch submission time, I'm pushing two
> patches to the list, I had lying around for some time now.
>
> The first patch allows mock to parse a generic side-wide
> config file, where general configuration-items can be stored
> and later overridden by specific chroot-definitions.
>
>
> The second patch, is a bit different:
> While pushing the Fedora Core SRPMS through the mock
> buildsystem we noticed many SRPMS which do not correctly
> define their buildrequirements.
> Until the fedora core developer responsible for the package
> has updated the spec in cvs and pushed an updated srpm into
> rawhide, there is no way to make the package in question
> build in mock without having to change the .spec.
> Thus, in order to allow even these broken packages being
> built, we needed a way to work around these broken
> dependencies to see if anything else was missing.
> This is done by the second patch, which adds a more_buildreqs
> config option. If set the given dependencies are installed
> _in addition_ to the ones from the .spec file.
>
> While the second patch should not be used during normal use
> and is definetly not for the fedora-extras buildservers, it
> still is of interest to other parties, using mock.
>
>
>
> regards,
> andreas
>
16 years, 10 months
RE: mock goals and non-goals
by Michael E Brown
> -----Original Message-----
> From: fedora-buildsys-list-bounces(a)redhat.com
> [mailto:fedora-buildsys-list-bounces@redhat.com] On Behalf Of
> Andreas Thienemann
> Sent: Thursday, May 11, 2006 2:10 PM
> To: Discussion of Fedora build system
> Subject: Re: mock goals and non-goals
>
> Hello,
>
> On Thu, 11 May 2006, seth vidal wrote:
>
> > goals:
> > - make clean and consistent build environments to be used
> by fedora's
> > buildsystem and packagers
> > - simplify building and testing process for packagers
> > - be able to canonically identify the 'build environment'
> for fedora
> > systems to discourage confusion on what should be where
> > - simple interface with good defaults.
> Why not a bit more generic and see mock as a tool to build
> rpms in a clean and consistent buildroot. After all, we heard
> that SuSE RPMs can be built with it too. And that's a feat s
> the last time I looked SuSE RPMs were pretty bad dependency-wise.
SuSE has taken a little bit of work to get working with mock. My
position, though, is that I cannot afford, time-wise, the effort it
would take to set up hundreds of different build environments for every
distro out there. It is much more efficient for me to just add the extra
bits to support SUSE in mock. I really would like to see one build
environment to rule them all(TM). Mock seems simple, flexible, and
extensible enough to handle lots of different distros with a modicum of
cooperation from the distros. Yes, there are a few problems, and no, you
cannot do an entire distro build of SUSE out of the box. But it works
for my stuff, which is all I need.
>From my perspective, every major rpm-based distro coming out has native
yum support, so mock (built on yum) looks like the winner.
> > - specifying multiple srpms on the command line for a
> SINGLE config -
> > provided that the buildroot is refreshed/cleaned each time
> - unless,
> > of
> This time, I do not really see the difference between calling
> mock with several srpms and calling mock several times on the shell.
It is just a matter of complexity. The added complexity in my script to
handle this is a pain, but inside mock itself, the changes are minimal.
Or, stated another way, the total system complexity is smaller doing
this inside mock.
>
> > course, the --no-clean option is passed in
> > - caching mechanism to create a cached pristine chroot that
> will allow
> > for quickly recreating the chroot into that state simply by putting
> > that cached copy back.
> This would be simply great. Right now, the cycle of cearting
> the chroot, installing the base-packages, doing the full
> depsolve-cycle, building the rpm and then removing the
> buildroot again takes ages.
> Something faster there might be great.
Seconded.
--
Michael
16 years, 10 months
[RFC, PATCH] Build multiple srpms
by Michael E Brown
Here is a patch for review. I have made the "obvious" changes in order
to be able to do the following:
mock -r CFG srpm1 srpm2 srpm3 srpm4
Limitations:
- There are no checks for dependencies between the srpms, so they should
be independent. (no re-ordering of SRPMS)
- Building one SRPM can leave the buildroot in an inconsistent state for
the next SRPM. (no clean of chroot between builds)
- Failure to build one SRPM stops the whole process.
- Resulting RPMs are not installed into the build environment for use by
subsequent SRPMS.
Reasoning:
One problem I have been having with mock is speed. We are using
mock to build for 16 distinct configurations. Doing prep on each
configuration was taking a minute and a half to two minutes. As an
optimization, our mock wrapper script was doing a single prep per chroot
and then using --no-clean, but this complicates our wrapper script. I am
building a set of related RPMS, so I don have much concern about
cross-pollution. This simple patch simplifies my wrapper script
considerably.
Future:
One future direction I'd like to take this is a parallel mock
that can prep/build multiple configurations at the same time to try to
amortize the cost of prep stage by running prep of one environment in
parallel with build of a different configuration. Currently, it is
taking me an hour to build 16 sets of RPMS (5 SRPMS per set), and I am
hoping to get this down. We have already implemented squid and some
other measures to try to speed things up.
--
Michael
<<mock-multi.patch>>
16 years, 10 months