[BUG] Koji and Lustre
by Thomas Guthmann
Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object
return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data
return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod
raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas
11 years, 8 months
Kojid unable to regen-repo in Fedora 15
by Anthony Joseph Messina
I'm running a Kojid instance, freshly upgraded from Fedora 14 -> 15. My
koji hub is still running Fedora 14.
When running a regen-repo, I now get this error:
2011-05-25 20:20:41,405 [WARNING] koji.TaskManager: TRACEBACK: Traceback
(most recent call last):
File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 1114, in
runTask
response = (handler.run(),)
File "/usr/lib/python2.7/site-packages/koji/tasks.py", line 146, in run
return self.handler(*self.params,**self.opts)
File "/usr/sbin/kojid", line 2552, in handler
self.create_local_repo(rinfo, arch, pkglist, groupdata, oldrepo)
File "/usr/sbin/kojid", line 2606, in create_local_repo
% parseStatus(status, ' '.join(cmd))
GenericError: failed to create repo: /usr/bin/createrepo -vd -o
/tmp/koji/tasks/1796/1796/repo -u http://messinet.com/koji-mnt/packages
-i /mnt/koji/repos/dist-f14-build/221/x86_64/pkglist -g
/mnt/koji/repos/dist-f14-build/221/groups/comps.xml --update --skip-stat
/mnt/koji/packages/ exited with status 1
Any ideas on where I should go? Thanks in advance. -A
--
Anthony - http://messinet.com - http://messinet.com/~amessina/gallery
8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
11 years, 10 months
Koji instances?
by Mike McLean
I'm writing an article about Koji and I would like to mention some of
the folks that are using it. If you're running a Koji instance, I'd love
to hear about it.
11 years, 10 months
for help sparc
by Jinze Xue
hello!
I want to build some packages of arch sparcv9 and sparc64 with koji build
system.
As you known , kernel is sparc64, no sparcv9
so , when i build packages of sparcv9,
how to solve the problems like this:
ERROR with rpm chek_debug vs depsolve:
lermel >= 2.6.29 is needed by libdrm...........
thanks anyway~~
11 years, 10 months
Question about Koji hub
by Kiu Leung
Hi all,
My name is Ken Leung, one of the Eclipse team new interns from the Red Hat Toronto office working on a Mylyn Builds Koji connector project. So I will be relying heavily on Apache's Java XmlRpc API as well as the Koji hub web service.
Right now, I am running into a problem with how to query Koji builds by a user_id or task_id. According to the signature of the web method:
listBuilds(packageID=None, userID=None, taskID=None, prefix=None, state=None, createdBefore=None, createdAfter=None, completeBefore=None, completeAfter=None, type=None, typeInfo=None, queryOpts=None)
The web method takes multiple parameter lists and can have default parameter values - which are not native to the Java language.
I tried the following to query builds of user ID 2, who has about 360 builds pushed to the target :
ArrayList<Object> params = new ArrayList<Object>();
params.add(new ArrayList<Integer>().add(null));
params.add(2);
xmlrpcClient.execute("listBuilds", params);//the Xml Rpc client was referenced by xmlrpcClient
While I expected a list of 360 hash maps (360 builds), I only received 1 hash map with a package_id of 1. I also tried to pass other user_ids which are known to have builds pushed to the target Koji instance, but the web service returned nothing.
I also tried to query Koji by task_id # 8 with the following code:
ArrayList<Object> params = new ArrayList<Object>();
params.add(new ArrayList<Integer>().add(null));
params.add(new ArrayList<Integer>().add(null));
params.add(8);
xmlrpcClient.execute("listBuilds", params);//the Xml Rpc client was referenced by xmlrpcClient
But unfortunately, the web service also returned nothing while there are more than 3000 tasks pushed onto the server.
So I would be most grateful for your advice on how to form the parameter list properly with Apache XmlRpc. Thank you very much!
Best regards,
Ken Leung
11 years, 10 months
Question about channel
by Kiu Leung
Dear all,
My name is Ken Leung. I am trying to learn the terminology used by the koji build system. I run into the term 'channel' and I don't really understand what a channel is. So I should be grateful if somebody can give me a brief explanation.
Thank you very much and have a great weekend!
Ken Leung
11 years, 10 months
Allow mash to export arbitrary packages in multilib repositories
by joaopfv＠br.ibm.com
Hello,
In one of our project, we tried to use the 'multilib_method = file' in the
configuration_file.mash; however it seems like that the current version of
mash doesn't correctly supports it.
The constructor of FileMultilibMethod, implemented on multilib.py, demands
a valid path for a file containing a list o packages to be exported.
However the method detectMultilibMethod implemented on __init__.py doesn't
provide it; thus the program crashes.
I would like to know if I may suggest a modification that apparently fix
this issue. This would allow the user provide the file through the option
'multilib_file = file_list.multilib', where file_list.multilib must be on
the same directory of the configuration_file.mash.
Thank you for all your attention.
--
João Paulo Fernandes Ventura
Linux Technology Center Brazil Software Engineer
IBM Systems & Technology Group
joaopfv(a)br.ibm.com
11 years, 10 months
[PATCH] Add support for custom checkout command
by Jan Pazdziora
Hello
please consider included patch which adds support for custom command
to get the sources from SCM.
The reason why we needed something like this is the fact that
in the Spacewalk project, we have multiple packages in one git repo,
in different directories. To get .spec and .tar.gz for a particular
package, a slightly different workflow is needed than koji generally
supports.
In our case, I wanted to be able to just use git format to get the
.tar.gz from the remote repo (no local checkout needed (even if we
might have one eventually to speed things up)), but before that a bit
more work needs to be done to find out which subdirectory the package
is in, similar to what the tito tool does (even if we do not use tito
directly).
Seeing that the allowed_scms already has fourth option to specify the
alternative to make sources, and since we don't need to run the
checkout at all, I've just added fifth option to specify custom
script, which then does all the heavy lifting -- it checks if it
already has the .spec and .tar.gz cached (as we build Spacewalk for
four OSes from one source), it retrieves them from
git.fedorahosted.org if it does not, and generally populates the
sourcedir properly, short-cutting any checkout and make sources.
I'd appreciate your comments about the patch, or inclusion of the
patch to koji master as is.
The patch has been in use at koji.spacewalkproject.org (albeit on top
of koji 1.4, so in the /usr/sbin/kojid) for a week or so and it seems
to achieve the goal properly.
Thank you.
>From 8e4072de758926a58c93226a44b08eda8b7ded87 Mon Sep 17 00:00:00 2001
From: Jan Pazdziora <jpazdziora(a)redhat.com>
Date: Wed, 8 Jun 2011 10:48:08 +0200
Subject: [PATCH] Support custom checkout command to populate sourcedir from
SCM.
The configuration is via the fifth part of the allowed_scms
configuration value:
allowed_scms=git.fedorahosted.org:/git/spacewalk.git:0::/usr/local/bin/get-tgz-from-git
The custom script is passed the sourcedir, host, repository, and
revision as parameters.
---
koji/daemon.py | 9 ++++++++-
1 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/koji/daemon.py b/koji/daemon.py
index b6f775a..07e1be7 100644
--- a/koji/daemon.py
+++ b/koji/daemon.py
@@ -192,6 +192,7 @@ class SCM(object):
self.revision = fragment
self.use_common = True
self.source_cmd = ['make', 'sources']
+ self.checkout_cmd = None
for scmtype, schemes in SCM.types.items():
if self.scheme in schemes:
@@ -275,6 +276,9 @@ class SCM(object):
else:
# there was nothing after the trailing :, so they don't want to run a source_cmd at all
self.source_cmd = None
+ if len(scm_tuple) >= 5:
+ if scm_tuple[4]:
+ self.checkout_cmd = scm_tuple[4].split(',')
break
else:
self.logger.warn('Ignoring incorrectly formatted SCM host:repository: %s' % allowed_scm)
@@ -311,7 +315,10 @@ class SCM(object):
raise koji.BuildError, 'Error running %s command "%s", see %s for details' % \
(self.scmtype, ' '.join(cmd), os.path.basename(logfile))
- if self.scmtype == 'CVS':
+ if self.checkout_cmd:
+ module_checkout_cmd = self.checkout_cmd + [sourcedir, self.host, self.repository, self.revision]
+
+ elif self.scmtype == 'CVS':
pserver = ':pserver:%s@%s:%s' % ((self.user or 'anonymous'), self.host, self.repository)
module_checkout_cmd = ['cvs', '-d', pserver, 'checkout', '-r', self.revision, self.module]
common_checkout_cmd = ['cvs', '-d', pserver, 'checkout', 'common']
--
1.7.4.4
--
Jan Pazdziora
Principal Software Engineer, Satellite Engineering, Red Hat
11 years, 10 months
newRepo error (merging external repo fails)
by Severin Gehwolf
Hi,
I've added an external repo to my local koji instance like so:
$ koji add-external-repo -t dist-rawhide-build \
dist-rawhide-external-repo \
http://mirrors.kernel.org/fedora/development/rawhide/\$arch/os
After that I'm getting the following errors for newRepo tasks:
GenericError: failed to merge repos: /usr/libexec/kojid/mergerepos -a
x86_64 -b /mnt/koji/repos/dist-rawhide-build/881/x86_64/blocklist
-o /tmp/koji/tasks/2427/2427/repo
-g /mnt/koji/repos/dist-rawhide-build/881/groups/comps.xml -r
file:///tmp/koji/tasks/2427/2427/repo_881_premerge/ -r
http://mirrors.kernel.org/fedora/development/rawhide/x86_64/os/ exited
with status 1
createrepo.log:
------------------------------------------------------------
$ /usr/bin/createrepo -vd -o /tmp/koji/tasks/2427/2427/repo -u
http://koji-toronto.usersys.redhat.com/packages
-g /mnt/koji/repos/dist-rawhide-build/881/groups/comps.xml /tmp/koji/tasks/2427/2427/repo
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Starting other db creation: Fri Jul 15 09:37:41 2011
Ending other db creation: Fri Jul 15 09:37:41 2011
Starting filelists db creation: Fri Jul 15 09:37:41 2011
Ending filelists db creation: Fri Jul 15 09:37:41 2011
Starting primary db creation: Fri Jul 15 09:37:41 2011
Ending primary db creation: Fri Jul 15 09:37:41 2011
Sqlite DBs complete
mergerepos.log:
------------------------------------------
$ /usr/libexec/kojid/mergerepos -a x86_64
-b /mnt/koji/repos/dist-rawhide-build/881/x86_64/blocklist
-o /tmp/koji/tasks/2427/2427/repo
-g /mnt/koji/repos/dist-rawhide-build/881/groups/comps.xml -r
file:///tmp/koji/tasks/2427/2427/repo_881_premerge/ -r
http://mirrors.kernel.org/fedora/development/rawhide/x86_64/os/
Adding repo: file:///tmp/koji/tasks/2427/2427/repo_881_premerge/
Adding repo:
http://mirrors.kernel.org/fedora/development/rawhide/x86_64/os/
Traceback (most recent call last):
File "/usr/libexec/kojid/mergerepos", line 247, in <module>
main(sys.argv[1:])
File "/usr/libexec/kojid/mergerepos", line 242, in main
merge.write_metadata()
File "/usr/libexec/kojid/mergerepos", line 222, in write_metadata
mdgen.doPkgMetadata()
File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line
395, in doPkgMetadata
self.writeMetadataDocs(packages)
File "/usr/lib/python2.7/site-packages/createrepo/__init__.py", line
587, in writeMetadataDocs
self.read_pkgs.append(po.localpath)
File "/usr/lib/python2.7/site-packages/yum/sqlitesack.py", line 270,
in __getattr__
raise KeyError, str(e)
KeyError: 'no such column: localpath'
I'm using koji* packages from F15 repos.
$ rpm -qa | grep koji
koji-hub-1.6.0-2.fc15.noarch
koji-hub-plugins-1.6.0-2.fc15.noarch
koji-1.6.0-2.fc15.noarch
koji-web-1.6.0-2.fc15.noarch
koji-utils-1.6.0-2.fc15.noarch
koji-builder-1.6.0-2.fc15.noarch
Does anybody have thoughts on this? Thanks!
--Severin
11 years, 10 months
Local koji setup: Canceled build. Yum exits with return code 1
by Severin Gehwolf
Hi,
I have a local koji setup and I got to the point where yum seems to be
able to run for some time and install base packages. However, it still
seems to fail eventually. I'm trying to build ed and used:
$ koji build --scratch dist-f15 path/to/srpm
(srpm generated from Fedora Git with fedpkg srpm on the f15 branch)
Attached is Koji Web's watchlogs output of the task in question. Any
ideas as to what might be going wrong? BTW: Koji Web shows the build
with state "canceled".
Thanks for any pointers!
--Severin
11 years, 10 months
