Pungi gathering - too many kernels
by Peter Åstrand
I have a KS file that looks like:
%packages --nobase
@core
@base-x
kernel
@hardware-support
-fedora-logos
# Compose Needs
anaconda-runtime
#iscsi-initiator-utils
memtest86+
#vnc-server
%end
Despite this, *all* kernels gets shipped:
Pungi.Gather.INFO: Checking deps of xorg-x11-drv-nouveau.i386
yum.verbose.YumBase.DEBUG: Matched xorg-x11-server-Xorg - 1.3.0.0-33.fc8.i386 to require for xorg-x11-server-Xorg
Pungi.Gather.INFO: Added xorg-x11-server-Xorg.i386 for xorg-x11-drv-nouveau.i386
yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-42.fc8.i586 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-PAE - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-PAE-debug - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-debug - 2.6.23.1-42.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-debug - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-49.fc8.i586 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-PAE-debug - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel-PAE - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau
yum.verbose.YumBase.DEBUG: Matched kernel - 2.6.23.1-49.fc8.i686 to require for kernel-drm-nouveau
Pungi.Gather.INFO: Added kernel-PAE.i686 for xorg-x11-drv-nouveau.i386
Pungi.Gather.INFO: Added kernel-debug.i686 for xorg-x11-drv-nouveau.i386
Pungi.Gather.INFO: Added kernel-PAE-debug.i686 for xorg-x11-drv-nouveau.i386
Pungi.Gather.INFO: Added kernel.i586 for xorg-x11-drv-nouveau.i386
Pungi.Gather.INFO: Added kernel.i686 for xorg-x11-drv-nouveau.i386
Is this really correct? The normal "kernel" package provides
kernel-drm-nouveau, so why is it necessary to pull in all other kernel
packages? To me, it looks like the dependency resolver is not satisfied
with one package fulfilling the dependencies, but rather includes all.
Regards,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00
16 years, 5 months
Koji patches for no longer creating hardlinks for package repodata
by Jesse Keating
We've talked about ways of making new repo tasks faster, and one of the
big speedups is no longer making a tree of hardlinks to run createrepo
over.
Recently createrepo gained[1] the ability to read from an include list.
Paired with the option to specify a baseurl to make everything relative
from, we can now run createrepo directly on the packages/ dir, using
the include file (generated from the list of packages we /would/ have
hardlinked), use an http url as the baseurl (so that static-repos
continue to work) and skip the hardlink step all together. An added
bonus is that when a static-repo expires, the metadata you may be
working with will continue to reference valid package URLS so your
transaction will continue without failure.
While I was working with this code, I also noticed that we can stop
creating a buildsys-build rpm and instead rely on the groupdata in the
repodata and just use 'groupinstall buildsys-build' as are chroot init
command. Add to that we were unnecessarily linking comps.xml files
into each arch directory of a repo directory, when instead we can just
reference the one we create in groups/ when creating repodata.
Since we no longer need to make the buildgroups rpm, the PrepRepo task
in kojid is no longer needed. I took the one part that it was still
usefully doing (asking the hub to prep the repo dirs) and move it up to
the NewRepo task. This saves a task creation and pickup/handoff.
I introduced a config item in kojid.conf, which is pkgurl. This is
used as the baseurl component to the repodata and should be the url
which leads to the packages/ directory.
Finally, since the repodata itself will continue to live in the same
expected locations, builders can continue to consume the repodata via
nfs. But since the repodata itself references http, yum will use http
to download the packages to populate the buildroot. We just have to
ensure that the url used in pkgurl is not only valid in the outside
world (so that static-repos continue to work) but valid within the
buildsystem structure as well.
I've committed my changes to a git tree (cloned from tip today) here:
http://jkeating.fedorapeople.org/git/koji/
Feedback would be welcome.
We're going to get the createrepo change into RHEL5.2 (along with
--update support) with the idea that when we refresh the Fedora
buildsystem machines in early December, we'll go with RHEL5 and perhaps
a newer createrepo package. Ideally we'd get the changes I've made
into koji beforehand so that we can release a new build of koji during
the refresh.
[1]
https://lists.dulug.duke.edu/pipermail/rpm-metadata/2007-November/000812....
--
Jesse Keating
Fedora -- All my bits are free, are yours?
16 years, 5 months
adding debug files to tree with pungi
by Doug Chapman
I have been working on building Fedora 8 ia64 trees with pungi. I have
it working for the normal binary and source rpms but I want to be able
to use it to add the debuginfo rpms to the tree.
The closest I have been able to get this to work was to add *debuginfo
to the %packages section of my pungi .ks file. This however just adds
the debuginfo packages in 8/ia64/os/Packages/ instead of
8/ia64/debuginfo/ where I find them on other Fedora distros.
Can this be done with pungi?
btw, I am running the F8 version of pungi: pungi-1.1.9-1.fc8
thanks,
- Doug
16 years, 5 months
RE: potential patch
by Michael E Brown
Better patch:
@@ -264,7 +264,7 @@ def main(retParams):
log_cfg = ConfigParser.ConfigParser()
logging.config.fileConfig(log_ini)
log_cfg.read(log_ini)
- except (IOError, OSError), e:
+ except (IOError, OSError, ConfigParser.NoSectionError), e:
log.error("Could not find required logging config file: %s" % log_ini)
sys.exit(50)
About branching:
If they ever upgrade fedoraproject.org to a non-ancient version of git, that would be fine. With git >= 1.5, you can delete remote branches. With older git, there is no way to remove a remote branch.
Are there any other changes you are needing for cross-building?
--
Michael
-----Original Message-----
From: Clark Williams [mailto:williams@redhat.com]
Sent: Fri 11/16/2007 9:03 AM
To: Brown, Michael E
Subject: potential patch
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Michael,
I'm porting our cross-build environment to use the new mock (0.8.8+) and ran into a
small issues. Evidently the ConfigParser stuff doesn't barf if the input configfile
doesn't exist, so you get a very strange error about "no formatters section exists"
or something similar.
What do you think of the change below?
I'm still struggling with how I should push these sorts of changes back to the main
git repo. I wonder if we shouldn't have michael,clark, and jessie branches up there
so we could just push to that for review then merge to master? Dunno...
Clark
- From 5add76128fcc10555264392307d5da998a05401b Mon Sep 17 00:00:00 2001
From: Clark Williams <williams(a)redhat.com>
Date: Thu, 15 Nov 2007 09:20:01 -0600
Subject: [PATCH] fix for no logging.ini situation
- ---
src/mock.py | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/src/mock.py b/src/mock.py
index 08832dd..9954dcf 100755
- --- a/src/mock.py
+++ b/src/mock.py
@@ -260,12 +260,15 @@ def main(retParams):
# reconfigure logging in case config file was overridden
log_ini = os.path.join(config_path, config_opts["log_config_file"])
+ if not os.path.exists(log_ini):
+ log.error("Could not find required logging config file: %s" % log_ini)
+ sys.exit(50)
try:
log_cfg = ConfigParser.ConfigParser()
logging.config.fileConfig(log_ini)
log_cfg.read(log_ini)
except (IOError, OSError), e:
- - log.error("Could not find required logging config file: %s" % log_ini)
+ log.error("Error configuring logging: %s" % e)
sys.exit(50)
# set up logging format strings
- --
1.5.3.4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHPbFJHyuj/+TTEp0RAga1AJ0ddE0ReUVdJpqGwSIHuFQqmhimEACgpej2
WX/yAWX3mo+J7GoS20t9JLg=
=OQg0
-----END PGP SIGNATURE-----
16 years, 5 months
What replaced plague
by John Villalovos
I get the impression that another program has replaced plague for
doing the distributed build stuff.
Can someone tell me what it is and a link to a site with more info?
Thanks!
John
16 years, 5 months
Mock 0.8.x available
by Michael E Brown
Mock users,
I have posted a new version of mock for review and comments.
RPM download: http://linux.dell.com/libsmbios/download/mock/mock-0.8.1/
Git repository: http://linux.dell.com/git/mock-v2.git
Compatibility:
-- Mock 0.8.x+ has been written on Fedora 7 and is intended for use
on Fedora 7 and higher. A limited amount of testing has been done on
FC6, so please report any FC6-specific bugs and we will make
best-effort attempts to fix them.
-- I have tried to maintain a high degree of compatibility with mock
0.7 in cmdline args and
New stuff:
-- plugin system to better modularize things
-- new plugins:
-- Yum cache
-- root cache
-- ccache
-- bind mount
-- root cache (formerly called autocache) is now significantly
smaller in size
-- speed increases: mock 0.8 is now in almost every case minutes
faster than 0.7, especially for multiple builds.
-- uses python logging.py module for increased logging flexibility
-- new command: 'mock install PACKAGE' to run a 'yum install
PACKAGE' inside the buildroot
-- expanded command: 'mock installdeps RPM_FILE' can now install
deps for a local binary RPM. Formerlly, only SRPMS were
supported.
-- new option: "--cleanup-after" that can be used with
"--resultdir". This will do a cleanup of the buildroot after the
build. This is enabled by default such that any '--resultdir'
builds will be automatically cleaned up
-- All of the config files delivered with mock have been tested and
should work. (The test case compiles mock using each of the
config files delivered with mock.)
Changed:
-- cmdline requires "rebuild" argument when rebuilding srpms.
Previously you could just pass srpm name.
-- output has slightly changed. Mock is now slightly more verbose.
-- formerly /dev from the host was bind-mounted into the chroot.
This is now not enabled by default, but can be configured easily
per-chroot using the bind plugin. See the 'defaults.cfg' files
for config details.
-- '-r CONFIG' option can no longer accept full config filenames
("config.cfg"). Leave off the '.cfg' for mock 0.8+. Formerly,
config ('-r' option) could be specified using either
"config.cfg" or
"config" and mock would look for /etc/mock/config.cfg,
automatically
adding the '.cfg', if necessary.
-- logs are not overwritten or truncated for --no-clean or
--resultdir builds.
-- cmdline options removed:
--autocache, --rebuild-cache: use "--enable-plugin" or
"--disable-plugin" instead
--verbose, --debug, --quiet: the log files contain all the
information. --verbose could possibly return.
--statedir: statedir didnt have much point in life and has gone
away
Config files:
-- Old config files will, by and large, still work. There are
several options in the old config files that are no longer
applicable now that 'mock-helper' has went away. The next
release of mock should have warnings enabled if it sees that
your config file has obsolete options.
Internal Changes:
-- now modularized
-- mock-helper is gone. Instead there is a setuid-wrapper that calls
mock.py. This vastly simplifies development.
Plugin Notes:
-- the plugin system is somewhat rudimentary at this time. If this
becomes something that is truly useful for out-of-tree plugins or
third-parties, we can formalize the interface some more. As of now,
the plugin interface should have most of the hooks to do useful
stuff, but some other things like the 'conduit' interface that yum
uses for plugins is not present. This would formalize the interface
a bit more and make sure the plugins dont grope too badly into
internals they shouldnt.
Future plans:
-- Upstream mock git will be updated to this version next week after
we sort out branching for the 0.7 release.
-- This version of mock will be checked into Fedora 9 next week.
-- yum integration: plan to try to use the yum API rather than the
cmdline. this will enable several optimizations and speed ups
due to not redownloading the yum metadata multiple times.
-- config file format change: at some future point, we probably will
switch to an .ini format config file.
SPECIAL SECURITY WARNING:
1) default /usr/bin/mock owner is root:mock with permissions 4550
(-rwsrwx---). Thus, it may only be run by the 'root' user or members of
the 'mock' group. DO NOT PUT UNTRUSTED USERS IN THE 'MOCK' GROUP! The
current code (as well as all previous versions) has many easily-used
mechanisms that an untrusted user could use to elevate their local
privileges. IT IS NOT THE GOAL OF THE MOCK DEVELOPERS TO MAKE MOCK A
TOOL THAT CAN BE USED BY UNTRUSTED LOCAL USERS.
2) All 'rpmbuild' commands, as well as the 'rpm' command to install and
unpack the to-be-built SRPM are run as the user running 'mock' with no
elevated privileges. These 'rpm' and 'rpmbuild' commands are run in the
chroot environment with these lower privileges. Thus, it should be
relatively safe to use mock to compile code from untrusted sources. That
being said, MOCK HAS NOT BEEN THOROUGHLY AUDITED TO GUARANTEE THE SAFETY
OF THIS.
--
Michael
16 years, 5 months
RHEL 5.1
by Rob Myers
I know RHEL is a bit off topic for this list, but bear with me. I
import all the RHEL5 bits into koji and then use koji to build rpms for
RHEL5. The release of RHEL 5.1 gave me a few kinks to work out:
The RHEL5 supplementary channel includes some noarch packages which do
not come with source rpms. This breaks the assumption in mash that
excludearch and exclusive arch can safely be initialized on SRPMS only.
I know that mash is not designed for this use case, but I thought it
might be worth pointing out. The patch I used is below.
diff --git a/mash/__init__.py b/mash/__init__.py
index b02d6c8..6e7a496 100644
--- a/mash/__init__.py
+++ b/mash/__init__.py
@@ -209,6 +209,18 @@ class Mash:
# now deal with noarch
for pkg in noarch:
for target_arch in self.config.arches:
+
+ # if excludearch is not set this build likely has no src.rpm
+ # so set excludearch and exclusivearch from the binary
+ if pkg['build_id'] fg
not in excludearch:
+ path = os.path.join(koji.pathinfo.build(builds_hash[pkg['build_id']]), koji.pathinfo.rpm(pkg))
+ fn = open(path, 'r')
+ hdr = koji.get_rpm_header(fn)
+ excludearch[pkg['build_id']] = hdr['EXCLUDEARCH']
+ exclusivearch[pkg['build_id']] = hdr['EXCLUSIVEARCH']
+ fn.close()
+ continue
+
if (excludearch[pkg['build_id']] and has_any(masharch.compat[target_arch], excludearch[pkg['build_id']])) or \
(exclusivearch[pkg['build_id']] and not has_any(masharch.compat[target_arch], exclusivearch[pkg['build_id']])):
print "Excluding %s.%s from %s due to EXCLUDEARCH/EXCLUSIVEARCH" % (pkg['name'], pkg['arch'], target_arch)
Another problem was that I had already imported RHEL 5.1 beta, which has
the same rpms in RHEL 5.1 signed with a different key- one that I do not
have access to. Since I wanted the rpms written out with the release
key, I added a function take the 5.1 release sighdrs and add them to the
previously imported RHEL 5.1 beta rpms. Would anyone else ever need this
functionality? The patch I used is below.
diff --git a/hub/kojihub.py b/hub/kojihub.py
index f9ba39a..c5a1261 100644
--- a/hub/kojihub.py
+++ b/hub/kojihub.py
@@ -4706,6 +4706,15 @@ class RootExports(object):
context.session.assertPerm('sign')
return add_rpm_sig(an_rpm, base64.decodestring(data))
+ def addRPMSigByPath(self, an_rpm, path):
+ """Store a signature header for an rpm
+
+ an_rpm: N-V-R.a (for example)
+ path: the path to the signed rpm
+ """
+ context.session.assertPerm('sign')
+ return add_rpm_sig(an_rpm, koji.rip_rpm_sighdr(path))
+
findBuildID = staticmethod(find_build_id)
getTagID = staticmethod(get_tag_id)
getTag = staticmethod(get_tag)
Comments?
rob.
16 years, 5 months
[Patch] for pungi-1.1.8-1.fc8 build on fedora 7
by Sergio Monteiro Basto
Hi,
To fix this error:
Pungi.Gather:INFO: Finished downloading packages.
Traceback (most recent call last):
File "/usr/bin/pungi", line 178, in <module>
main()
File "/usr/bin/pungi", line 68, in main
mygather.makeCompsFile()
File "/usr/lib/python2.5/site-packages/pypungi/gather.py", line 397,
in makeCompsFile
ourcomps.writelines(compslines[start:end])
UnboundLocalError: local variable 'start' referenced before assignment
16 years, 5 months
RFC - Patch - koji SCM generalization
by Rob Myers
attached is a patch that attempts to generalize checking out the
components used to build an SRPM. this patch supports CVS, GIT, and
SVN, but only CVS and SVN have been tested. the idea is to provide the
infrastructure for different SCM systems to be configured at run-time so
that users can choose their favorite system.
is there a better approach? did i miss something obvious? general
comments?
thanks.
rob.
16 years, 5 months
