[BUG] Koji and Lustre
by Thomas Guthmann
Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object
return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data
return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod
raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas
12 years
Strange mock build failure due to typo
by Paul Howarth
Hi,
today I have been preparing an update to perl-Math-Pari and came across
a very strange build failure whilst doing a local mock build on a Fedora
13 x86_64 host. My package built successfully on x86_64 but when I tried
to build for i386, the build failed during %setup but without any
diagnostics. The SRPM was installed but no attempt to install its build
requirements was made. The root.log showed an exit status of 0 for all
commands that had been run.
After much experimentation bisecting the changes I had made, I
discovered that a typo in the changelog entry was the culprit: I had set
the year to 2100 instead of 2010. So it would appear that somewhere in
the mock/yum/rpm stack there may be a year 2038 problem waiting to bite
us (though I suspect there may not be too many 32-bit builds happening
by then).
Seriously though, it would be nice to have better diagnostics for this
and perhaps an rpmlint check for changelog entries in the future?
Paul.
12 years, 8 months
Speeding up koji-generated mock builds
by Alan Franzoni
Hello,
I'm trying to get our koji instance faster; since we build a lot of
packages, we'd like to reduce the latency.
There're some things that usually make mock faster: the options I
tinkered with and seem to improve performance are root_cache,
yum_cache and tmpfs.
All of those are disabled by default when generating mock config
through koji, and there seems no way to override them:
https://fedorahosted.org/koji/browser/koji/__init__.py#L1276
So here come the questions:
1) Am i missing anything?
2) are all those options intentionally locked out?
3) would a patch which let them be configured be accepted into your git repo?
4) any other way to speedup mock?
Thanks!
--
Alan Franzoni
--
contact me at public(a)[mysurname].eu
12 years, 9 months
make iso failed
by Vijay N. Majagaonkar
Hi all,
I have updated my system to f14 and now i am not able to build iso out of
koji,
mock : mock-1.1.6-1.fc14.noarch
python : python-2.7-8.fc14.1.x86_64
koji : koji-1.4.0-4.fc14.noarch
[Error]
Building Installation Images: ########################################
100.0%
DEBUG util.py:267: Going to replace isolinux/isolinux.cfg with
/etc/revisor/conf.d/revisor-proxyone-isolinux.cfg
DEBUG util.py:267: Deleted the old isolinux.cfg
DEBUG util.py:267: Inserted the new isolinux.cfg
DEBUG util.py:267: Removing files and directories that do not need to be on
the media:
DEBUG util.py:267: Traceback (most recent call last):
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/__init__.py", line 441, in run
DEBUG util.py:267: self.base.run()
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/base.py", line 106, in run
DEBUG util.py:267: self.cli.run()
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/cli.py", line 44, in run
DEBUG util.py:267: self.base.lift_off()
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/base.py", line 1172, in lift_off
DEBUG util.py:267: self.buildInstallationMedia()
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/base.py", line 1487, in
buildInstallationMedia
DEBUG util.py:267: self.plugins.exec_hook('post_exec_buildinstall')
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/plugins.py", line 191, in
exec_hook
DEBUG util.py:267: exec("self.%s.%s()" % (plugin,hook))
DEBUG util.py:267: File "<string>", line 1, in ?
DEBUG util.py:267: File
"/usr/lib/python2.4/site-packages/revisor/modremove/__init__.py", line 61,
in post_exec_buildinstall
DEBUG util.py:267: for file in self.cfg.rm:
DEBUG util.py:267: TypeError: iteration over non-sequence
[/Error]
Is this something to do with python version, if I downgrade the version to
previous version that is f12 it works correct, do i need to patch these
packages make it work under f14 ?
Thanks for your help
V!jay
12 years, 9 months
[PATCH 1/3] Install build deps with yum-builddep.
by Ville Skyttä
No longer need to screen-scrape resolvedep and feed that to yum
install, and we have a chance to get BuildConflicts handing "for free"
(when RHBZ #614191 is done in yum(-builddep)).
---
mock.spec.in | 2 +-
py/mock/backend.py | 33 ++++++++++++++++++++++-----------
2 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/mock.spec.in b/mock.spec.in
index e2580ca..0949975 100644
--- a/mock.spec.in
+++ b/mock.spec.in
@@ -18,7 +18,7 @@ Source: https://fedorahosted.org/mock/attachment/wiki/MockTarballs/%{name}-%{ver
URL: http://fedoraproject.org/wiki/Projects/Mock
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArch: noarch
-Requires: python >= 2.4, yum >= 2.4, tar, pigz, python-ctypes, python-decoratortools, usermode
+Requires: python >= 2.4, yum >= 2.4, yum-utils >= 1.1.9, tar, pigz, python-ctypes, python-decoratortools, usermode
Requires: createrepo
Requires(pre): shadow-utils
BuildRequires: python-devel
diff --git a/py/mock/backend.py b/py/mock/backend.py
index ba0ad14..81a98d0 100644
--- a/py/mock/backend.py
+++ b/py/mock/backend.py
@@ -69,6 +69,7 @@ class Root(object):
self.chroot_file_contents = config['files']
self.chroot_setup_cmd = config['chroot_setup_cmd']
self.yum_path = '/usr/bin/yum'
+ self.yum_builddep_path = '/usr/bin/yum-builddep'
self.macros = config['macros']
self.more_buildreqs = config['more_buildreqs']
self.cache_topdir = config['cache_topdir']
@@ -444,23 +445,28 @@ class Root(object):
"""figure out deps from srpm. call yum to install them"""
try:
self.uidManager.becomeUser(0, 0)
+
+ def _yum_and_check(cmd):
+ output = self._yum(cmd, returnOutput=1)
+ for line in output.split('\n'):
+ if line.lower().find('No Package found for'.lower()) != -1:
+ raise mock.exception.BuildError, "Bad build req: %s. Exiting." % line
+
+ # first, install pre-existing deps and configured additional ones
arg_string = self.preExistingDeps
for hdr in mock.util.yieldSrpmHeaders(srpms, plainRpmOk=1):
# get text buildreqs
- a = mock.util.requiresTextFromHdr(hdr)
- b = mock.util.getAddtlReqs(hdr, self.more_buildreqs)
- for item in mock.util.uniqReqs(a, b):
+ for item in mock.util.getAddtlReqs(hdr, self.more_buildreqs):
arg_string = arg_string + " '%s'" % item
-
- # everything exists, okay, install them all.
- # pass build reqs (as strings) to installer
if arg_string != "":
- output = self._yum('resolvedep %s' % arg_string, returnOutput=1)
- for line in output.split('\n'):
- if line.lower().find('No Package found for'.lower()) != -1:
- raise mock.exception.BuildError, "Bad build req: %s. Exiting." % line
+ # everything exists, okay, install them all.
+ # pass build reqs (as strings) to installer
+ _yum_and_check('resolvedep %s' % arg_string)
# nothing made us exit, so we continue
self._yum('install %s' % arg_string, returnOutput=1)
+
+ # install actual build dependencies
+ _yum_and_check("builddep '%s'" % "' '".join(srpms))
finally:
self.uidManager.restorePrivs()
@@ -676,7 +682,12 @@ class Root(object):
if not self.online:
cmdOpts = "-C"
- cmd = '%s --installroot %s %s %s' % (self.yum_path, self.makeChrootPath(), cmdOpts, cmd)
+ # invoke yum-builddep instead of yum if cmd is builddep
+ exepath = self.yum_path
+ if cmd.startswith("builddep "):
+ exepath = self.yum_builddep_path
+ cmd = cmd[len("builddep "):]
+ cmd = '%s --installroot %s %s %s' % (exepath, self.makeChrootPath(), cmdOpts, cmd)
self.root_log.debug(cmd)
output = ""
try:
--
1.7.2.3
12 years, 9 months
How do folks use an SCM?
by Jon Stanley
So I have an interesting situation. I have an SCM (CVS, don't laugh)
that requires kerberized gserver authentication. How do I use Koji
with this? I don't mind embedding a password for a user that has
read-only access to the repo somewhere, but I really don't want to if
I can avoid it.
Also, with the interesting requirement of a Makefile with target srpm,
how do folks generate that for externally developed packages? Frankly,
most of the packages that we're going to build are rebuilds of RHEL
content with minor changes (sometimes a patch, sometimes just pathname
changes, etc), so generating an SRPM and feeding it directly to koji
is easier than maintaining some SCM layout that's foreign to us and a
lookaside cache. Note that the reason we want to use koji is build
reproducibility, but we'll be saving the SRPM's used in some location.
12 years, 9 months
for help
by Jinze Xue
I want to move all data from a old koji build system to a new os, what's
the best way to do?
and who have documents about configs and steps to build cvs system for a
koji build system ? send me ,thanks very much !!!!!!!
12 years, 9 months
A question on Fedora Koji infrastructure
by Alan Franzoni
Hello everybody,
lately we're experiencing some strange and random issues with koji on
fc14 64 bit; we're still tracking down some errors and we'll later
submit tickets, but my question is: which OS uses
koji.fedoraproject.org main hub and builders? Is the host machine
running fedora, centos, rhel or what?
--
Alan Franzoni
--
contact me at public(a)[mysurname].eu
12 years, 10 months
Koji build slaves configuration
by Alan Franzoni
Hello,
after successfully using koji on a single machine we've hit a
performance bottleneck, hence we're extending our build infrastracture
by adding new builders.
There's a step which is not clear to me: /mnt/koji on the main machine
should be remotely mounted read-only to the builders, right? is there
any preferred mount method (nfs/sshfs/...) which is known not to be a
source of issues?
--
Alan Franzoni
--
contact me at public(a)[mysurname].eu
12 years, 10 months
growing /mnt/koji/work/tasks
by Alan Franzoni
Hello, after upgrading to fc14 and koji 1.4 I'm still partially
experiencing an issue with the "tasks" directory; it keeps growing and
contains very old builds.
In the old thread it was suggested that this shouldn't happen if
kojira is running:
http://www.mail-archive.com/buildsys@lists.fedoraproject.org/msg00472.html
In fact now that kojira is running and everything is updated the
cli-build dir is perfectly clean, but the tasks dir content seems to
stay there (forever?) alo. I can't see any error in kojira.log, and
all files are owned by apache.
koji-gc is scheduled and runs every night.
Where should I look at to track my problems?
--
Alan Franzoni
--
contact me at public(a)[mysurname].eu
12 years, 10 months
