[BUG] Koji and Lustre
by Thomas Guthmann
Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object
return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data
return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod
raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas
12 years, 7 months
Kojid unable to regen-repo in Fedora 15
by Anthony Joseph Messina
I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15. My
koji hub is still running Fedora 14.
When running a regen-repo, I now get this error:
2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
(most recent call last):
File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
runTask
response = (handler.run(),)
File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
return self.handler(*self.params,**self.opts)
File "/usr/sbin/kojid", line 2552, in handler
self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
File "/usr/sbin/kojid", line 2606, in create_local_repo
% parseStatus(status, ' '.join(cmd))
GenericError: failed to create repo: /usr/bin/createrepo -vd -o
/tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
-i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
/mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
/mnt/koji/packages/ exited with status 1
Any ideas on where I should go? Thanks in advance. -A
--
Anthony - http://messinet.com - http://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
12 years, 8 months
[PATCH] Add support for custom checkout command
by Jan Pazdziora
Hello
please consider included patch which adds support for custom command
to get the sources from SCM.
The reason why we needed something like this is the fact that
in the Spacewalk project, we have multiple packages in one git repo,
in different directories. To get .spec and .tar.gz for a particular
package, a slightly different workflow is needed than koji generally
supports.
In our case, I wanted to be able to just use git format to get the
.tar.gz from the remote repo (no local checkout needed (even if we
might have one eventually to speed things up)), but before that a bit
more work needs to be done to find out which subdirectory the package
is in, similar to what the tito tool does (even if we do not use tito
directly).
Seeing that the allowed_scms already has fourth option to specify the
alternative to make sources, and since we don't need to run the
checkout at all, I've just added fifth option to specify custom
script, which then does all the heavy lifting -- it checks if it
already has the .spec and .tar.gz cached (as we build Spacewalk for
four OSes from one source), it retrieves them from
git.fedorahosted.org if it does not, and generally populates the
sourcedir properly, short-cutting any checkout and make sources.
I'd appreciate your comments about the patch, or inclusion of the
patch to koji master as is.
The patch has been in use at koji.spacewalkproject.org (albeit on top
of koji 1.4, so in the /usr/sbin/kojid) for a week or so and it seems
to achieve the goal properly.
Thank you.
>From 8e4072de758926a58c93226a44b08eda8b7ded87 Mon Sep 17 00:00:00 2001
From: Jan Pazdziora <jpazdziora(a)redhat.com>
Date: Wed, 8 Jun 2011 10:48:08 +0200
Subject: [PATCH] Support custom checkout command to populate sourcedir from
SCM.
The configuration is via the fifth part of the allowed_scms
configuration value:
allowed_scms=git.fedorahosted.org:/git/spacewalk.git:0::/usr/local/bin/get-tgz-from-git
The custom script is passed the sourcedir, host, repository, and
revision as parameters.
---
koji/daemon.py | 9 ++++++++-
1 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/koji/daemon.py b/koji/daemon.py
index b6f775a..07e1be7 100644
--- a/koji/daemon.py
+++ b/koji/daemon.py
@@ -192,6 +192,7 @@ class SCM(object):
self.revision = fragment
self.use_common = True
self.source_cmd = ['make', 'sources']
+ self.checkout_cmd = None
for scmtype, schemes in SCM.types.items():
if self.scheme in schemes:
@@ -275,6 +276,9 @@ class SCM(object):
else:
# there was nothing after the trailing :, so they don't want to run a source_cmd at all
self.source_cmd = None
+ if len(scm_tuple) >= 5:
+ if scm_tuple[4]:
+ self.checkout_cmd = scm_tuple[4].split(',')
break
else:
self.logger.warn('Ignoring incorrectly formatted SCM host:repository: %s' % allowed_scm)
@@ -311,7 +315,10 @@ class SCM(object):
raise koji.BuildError, 'Error running %s command "%s", see %s for details' % \
(self.scmtype, ' '.join(cmd), os.path.basename(logfile))
- if self.scmtype == 'CVS':
+ if self.checkout_cmd:
+ module_checkout_cmd = self.checkout_cmd + [sourcedir, self.host, self.repository, self.revision]
+
+ elif self.scmtype == 'CVS':
pserver = ':pserver:%s@%s:%s' % ((self.user or 'anonymous'), self.host, self.repository)
module_checkout_cmd = ['cvs', '-d', pserver, 'checkout', '-r', self.revision, self.module]
common_checkout_cmd = ['cvs', '-d', pserver, 'checkout', 'common']
--
1.7.4.4
--
Jan Pazdziora
Principal Software Engineer, Satellite Engineering, Red Hat
12 years, 9 months
--install issue with mock-1.1.10-1.fc14
by Paul B Schroeder
I have a F14 system with all of the latest updates, including the latest
mock.
I --init my chroot with:
$ mock -r fedora-14-x86_64 --init
And then try to install additional packages in my chroot with:
mock -r fedora-14-x86_64 --verbose --install "revisor-cli revisor-comps
revisor-isolinux"
And get:
ERROR: Command failed:
# ['/usr/bin/yum', '--installroot',
'/var/lib/mock/fedora-14-x86_64/root/', 'install', 'revisor-cli
revisor-comps revisor-isolinux', '--setopt=tsflags=nocontexts']
Error: Nothing to do
When I do a 'yum downgrade' to mock-1.1.5-1.fc14, I am able to install
the additional packages without issue. The error seems to be coming
from yum, but only causes an issue with the latest mock. Has anybody
seen this issue or have any idea as to what is going on?
Thanks...Paul...
--
---
Paul B Schroeder <paul.schroeder "at" vbridges "dot" com>
Virtual Bridges, Inc.
12 years, 10 months
What is the difference among those tags in koji?
by yanchuan
Hi,
What is the difference among thos tags in the buildsystem web of Fedora http://koji.fedoraproject.org/koji?
There are 13 tags for Fedora 15,and I'm confused with the relationship among those tags.I know dist-f15-updates
is inherited from dist-f15,and dist-f15-override is
inherited from dist-f15-updates.I what to know which package should
goes to dist-f15-updates? In what situation, a package should goes to dist-f15-override but not dist-f15-updates?
Where can I find some relevant docs?I'm new here.
Thanks.
Yanchuan
12 years, 10 months
Kojihub fails after F14 -> f15 upgrade
by Anthony Joseph Messina
After upgrading from Fedora 14 to Fedora 15, I'm having the following
issue when trying to access my koji web install. I use Kerberos for my
private Koji instance.
I've see this:
http://www.redhat.com/archives/fedora-buildsys-list/2008-July/msg00022.html
I've checked my pg_hba.conf (it's the same as it was a few hours ago
with a working Fedora 14 Koji setup). Of course, now I'm using
PostgreSQL 9.
I run the PostgreSQL service on the same box and am able to access other
PostgreSQL stuff without issue, using Kerberos.
Any help would be appreciated. Thanks in advance. -Anthony
MOD_PYTHON ERROR
ProcessId: 26326
Interpreter: 'messinet.com'
ServerName: 'messinet.com'
DocumentRoot: '/var/www/egroupware/sitemgr/sitemgr-site'
URI: '/koji/'
Location: None
Directory: '/usr/share/koji-web/scripts/'
Filename: '/usr/share/koji-web/scripts/index.py'
PathInfo: ''
Phase: 'PythonHandler'
Handler: 'kojiweb.publisher'
Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line
1537, in HandlerDispatch
default=default_handler, arg=req, silent=hlist.silent)
File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line
1229, in _process_target
result = _execute_target(config, req, object, arg)
File "/usr/lib64/python2.7/site-packages/mod_python/importer.py", line
1128, in _execute_target
result = object(arg)
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 39, in handler
return mod_python.publisher.handler(req)
File "/usr/lib64/python2.7/site-packages/mod_python/publisher.py",
line 213, in handler
published = publish_object(req, object)
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 34, in
publish_object
return old_publish_object(req, _genHTML(req, 'error.chtml'))
File "/usr/share/koji-web/lib/kojiweb/util.py", line 69, in _genHTML
req._values['mavenEnabled'] = req._session.mavenEnabled()
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1510,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1740,
in _callMethod
return proxy.__getattr__(name)(*args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1570, in __request
verbose=self.__verbose
File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib64/python2.7/xmlrpclib.py", line 1312, in single_request
response.msg,
ProtocolError: <ProtocolError for messinet.com/kojihub: 500 Internal
Server Error>
MODULE CACHE DETAILS
Accessed: Wed Jun 8 19:04:04 2011
Generation: 0
_mp_0dec3ca8c086f5baed01b0d5504fa2b0 {
FileName: '/usr/share/koji-web/scripts/index.py'
Instance: 1 [IMPORT]
Generation: 1
Modified: Thu Dec 16 15:13:17 2010
Imported: Wed Jun 8 19:04:04 2011
}
--
Anthony - http://messinet.com - http://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
12 years, 10 months
Koji host selection/load balancing
by Anthony Towns
Hi,
The host selection algorithm in koji/daemon.py (TaskManager.checkRelAvail) is designed to choose a host that's in the top half of hosts by most available capacity. I think it doesn't quite work as might be expected though -- in the case where you have five hosts all with a capacity of 3.0, you could have the following behaviour:
A takes job, capacities: [3.0, 3.0, 3.0, 3.0, 2.0] (median = 3.0)
B takes job, capacities: [3.0, 3.0, 3.0, 2.0, 2.0] (median = 3.0)
C takes job, capacities: [3.0, 3.0, 2.0, 2.0, 2.0] (median = 2.0)
A takes job, capacities: [3.0, 3.0, 2.0, 2.0, 1.0] (median = 2.0)
B takes job, capacities: [3.0, 3.0, 2.0, 1.0, 1.0] (median = 2.0)
C takes job, capacities: [3.0, 3.0, 1.0, 1.0, 1.0] (median = 1.0)
A takes job, capacities: [3.0, 3.0, 1.0, 1.0] (median = 3.0)
D takes job, capacities: [3.0, 2.0, 1.0, 1.0] (median = 2.0)
D takes job, capacities: [3.0, 1.0, 1.0, 1.0] (median = 1.0)
B takes job, capacities: [3.0, 1.0, 1.0] (median = 1.0)
C takes job, capacities: [3.0, 1.0] (median = 3.0)
So eleven jobs get divided up as:
3 on A
3 on B
3 on C
2 on D
0 on E
This is because the >=median test doesn't actually ensure this host is in the top half, just that there's a host in the top half that's no better than this one. I think a better test would be ==best or >median, in which case you'd get:
A takes job, capacities: [3.0, 3.0, 3.0, 3.0, 2.0]
B takes job, capacities: [3.0, 3.0, 3.0, 2.0, 2.0]
C takes job, capacities: [3.0, 3.0, 2.0, 2.0, 2.0]
D takes job, capacities: [3.0, 2.0, 2.0, 2.0, 2.0]
E takes job, capacities: [2.0, 2.0, 2.0, 2.0, 2.0]
Patch attached for consideration.
Cheers,
aj
--
Anthony Towns <atowns(a)redhat.com>
12 years, 10 months
koji troubles building
by Clint Savage
Hi all,
I know a few of you have helped me with my buildout of koji over the
past couple weeks. Thank you all for your help! I've been making good
progress on the buildout until last night, when I got stuck with a
very interesting problem. At this point, I've searched high and low,
and done everything I can think of to address this problem. I'm still
stuck here, 24-hours later, with no solution.
After a good 6 hours of work, I finally got all of the dependencies
working for a build of a tool called 'pydf'. A colorized version of
df, written in python. I recently added it to Fedora's koji.
When I attempt to build this rpm from a src.rpm, I get through all of
the deps. It seems about the time that mock is to write data to the
chroot is when things start to break. The machine I am running is
Fedora 15, x86_64, fully updated as of tonight. Koji is setup to only
build x86_64 rpms. Everything is on just one machine for now.
$ koji -c ~/.koji/config-goose build dist-g6 rpmbuild/SRPMS/pydf-9-3.el6.src.rpm
..snip..
865 buildArch (pydf-9-3.el6.src.rpm, noarch): open (kojibuilder1) ->
FAILED: BuildError: error building package (arch noarch), mock exited
with status 1; see build.log for more information
0 free 1 open 0 done 1 failed
864 build (dist-g6, pydf-9-3.el6.src.rpm): open (kojibuilder1) ->
FAILED: BuildError: error building package (arch noarch), mock exited
with status 1; see build.log for more information
0 free 0 open 0 done 2 failed
When I look at the logs, the most interesting is the 'mock_output.log'
State Changed: setup
ERROR: Exception(/mnt/koji/work/cli-build/1306989306.986684.cIQpcXny/pydf-9-3.el6.src.rpm)
Config(dist-g6-build-116-205) 0 minutes 1 seconds
INFO: Results and/or logs in: /var/lib/mock/dist-g6-build-116-205/result
Traceback (most recent call last):
File "/usr/sbin/mock", line 491, in <module>
def do_buildsrpm(config_opts, chroot, options, args):
File "/usr/sbin/mock", line 794, in main
do_rebuild(config_opts, chroot, args)
File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at
0x01F9EB90>", line 3, in do_rebuild
File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py",
line 70, in trace
result = func(*args, **kw)
File "/usr/sbin/mock", line 463, in do_rebuild
chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'])
File "<peak.util.decorators.rewrap wrapping mock.backend.build at
0x01F8EB90>", line 3, in build
File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py",
line 70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.7/site-packages/mock/backend.py", line 530, in build
gid=self.chrootgid,
File "/usr/lib/python2.7/site-packages/mock/backend.py", line 449, in doChroot
shell=shell, *args, **kargs )
File "<peak.util.decorators.rewrap wrapping mock.util.do at
0x01F895F0>", line 3, in do
File "/usr/lib/python2.7/site-packages/mock/trace_decorator.py",
line 70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.7/site-packages/mock/util.py", line 292, in do
preexec_fn = preexec,
File "/usr/lib64/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Everything above the 'Status Changed: setup' appears to be normal.
Some things that were suggested to me in #koji.
- SELinux permissions. Currently in permissive mode.
- The user is not in the mock group - all users allowed to use koji
are in the mock group (clints,kojibuilder to be exact)
I cannot see any reason this is failing, so I am hoping to find some
insight here. Also, I should note that I did run across some abrt
dumps which looked interesting and will be happy to post, if desired.
Thank you all for your continued help. I sure hope this issue is
easier to solve than it currently looks.
Cheers,
Clint
12 years, 11 months
