[BUG] Koji and Lustre
by Thomas Guthmann
Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object
return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data
return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod
raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas
12 years, 6 months
Koji 1.6 - Mock SRPM Build failing - NFS Path - no such file or directory
by Ciaran Bryan
Hi,
I have setup a koji server and client builder successfully and can submit koji client commands and issue build tasks successfully.
When an SRPM build task is taken by my builder server (same server as koji-hub/web) the mock buildroot gets setup ok and yum installed the correct build group binary packages etc within the buildroot.
Then as mock goes to build the source RPM it fails as the location of the source RPM is the /mnt/work/cli-build..... NFS folder, but that NFS folder is not mounted to/available within the buildroot.
How do I make my source rpm available to the mock buildroot environment without using NFS or is my setup wrong ?
Thanks,
C.
My koji build task was submitted as follows:
[kojiadmin@nse-build ~]$ koji build newbay-nse /usr/src/redhat/SRPMS/zookeeper-3.2.2-1.src.rpm
Mock output:
INFO: mock.py version 1.0.7 starting...
State Changed: init plugins
State Changed: start
State Changed: lock buildroot
State Changed: clean
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.7
INFO: Mock Version: 1.0.7
State Changed: running yum
INFO: mock.py version 1.0.7 starting...
State Changed: init plugins
State Changed: start
INFO: Start(/mnt/koji/work/cli-build/1298901262.7331891.WlovUVjV/zookeeper-3.2.2-1.src.rpm) Config(nse-build-41-31)
State Changed: init
State Changed: lock buildroot
Mock Version: 1.0.7
INFO: Mock Version: 1.0.7
State Changed: running yum
State Changed: setup
ERROR: Exception(/mnt/koji/work/cli-build/1298901262.7331891.WlovUVjV/zookeeper-3.2.2-1.src.rpm) Config(nse-build-41-31) 0 minutes 0 seconds
INFO: Results and/or logs in: /var/lib/mock/newbay-nse-build-41-31/result
Traceback (most recent call last):
File "/usr/sbin/mock", line 693, in ?
main(retParams)
File "/usr/sbin/mock", line 635, in main
do_rebuild(config_opts, chroot, args)
File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x1BC285F0>", line 3, in do_rebuild
File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace
result = func(*args, **kw)
File "/usr/sbin/mock", line 372, in do_rebuild
chroot.build(srpm, timeout=config_opts['rpmbuild_timeout'])
File "<peak.util.decorators.rewrap wrapping mock.backend.build at 0x1BBE6B90>", line 3, in build
File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.4/site-packages/mock/backend.py", line 439, in build
gid=self.chrootgid,
File "/usr/lib/python2.4/site-packages/mock/backend.py", line 366, in doChroot
shell=shell, *args, **kargs )
File "<peak.util.decorators.rewrap wrapping mock.util.do at 0x1BBEC500>", line 3, in do
File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace
result = func(*args, **kw)
File "/usr/lib/python2.4/site-packages/mock/util.py", line 299, in do
preexec_fn = preexec,
File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 996, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
________________________________
STRICTLY PRIVATE, CONFIDENTIAL AND PRIVILEGED COMMUNICATION.
This message (including attachments) may contain information that is
privileged, confidential or protected from disclosure. They are intended
solely for the use of the intended recipient. If you are not the
intended recipient, you are hereby notified that dissemination,
disclosure, copying, distribution, printing, transmission or use of this
message or any information contained in it is strictly prohibited. If
you have received this message from NewBay Software in error, please
immediately notify the sender by reply email and delete this message
from your computer. The content of this e-mail, and any files
transmitted with it, may have been changed or altered without the
consent of the author. Although we have taken steps to ensure that this
email and attachments are free from any virus, we advise that in keeping
with good computing practice the recipient should ensure they are
actually virus free.
13 years
need 32bit rpms in a mock setup for a 64 bit build
by Chris Edsall
Hi,
I have a x86_64 builder and I am trying to make an x86_64 build. This particular build needs a compiler that is only available as a binary tarball. I have made an RPM out of the tarball, however it has i386 dependancies. When I come to use it in koji (by adding it as a BuildRequires) mock complains because the yum configuration can't satisfy the dependencies:
DEBUG backend.py:603: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ resolvedep 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel'
DEBUG util.py:291: Executing command: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ resolvedep 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel'
DEBUG util.py:256: 0:zlib-devel-1.2.3-3.x86_64
DEBUG util.py:256: 1:valgrind-3.5.0-1.el5.x86_64
DEBUG util.py:256: 0:gawk-3.1.5-14.el5.x86_64
DEBUG util.py:256: 0:pgi-10.3-7.x86_64
DEBUG util.py:256: 0:hdf5-devel-1.8.5.patch1-5.x86_64
DEBUG util.py:256: 0:curl-devel-7.15.5-9.el5.x86_64
DEBUG util.py:330: Child returncode was: 0
DEBUG backend.py:603: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ install 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel'
DEBUG util.py:291: Executing command: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ install 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel'
DEBUG util.py:256: Package gawk-3.1.5-14.el5.x86_64 already installed and latest version
DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems
DEBUG util.py:256: --> Missing Dependency: libnsl.so.1 is needed by package pgi-10.3-7.x86_64 (build)
DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems
DEBUG util.py:256: --> Missing Dependency: libXext.so.6 is needed by package pgi-10.3-7.x86_64 (build)
DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems
DEBUG util.py:256: --> Missing Dependency: libc.so.6(GLIBC_2.1.2) is needed by package pgi-10.3-7.x86_64 (build)
However the same compiler package installs fine on the command line with the same yum reops configured
[root@cli-dev ~]# yum -y install pgi
[ ... ]
Installed:
pgi.x86_64 0:10.3-7
Complete!
It needed these non x86_64 packages:
[root@cli-dev ~]# yum deplist pgi | egrep 'i[3456]86' | sort -u
provider: alsa-lib.i386 1.0.17-1.el5
provider: glibc.i686 2.5-49
provider: IBMJava2-142-ia32-JRE.i386 1.4.2-13.6
provider: libgcj.i386 4.1.2-48.el5
provider: libX11.i386 1.0.3-11.el5
provider: libXext.i386 1.0.1-2.1
provider: libXi.i386 1.0.1-4.el5_4
provider: libXtst.i386 1.0.1-3.1
The difference, of course, is that the repo mock is pointing at has been created by mergerepos:
$ /usr/libexec/kojid/mergerepos -a x86_64 -b /mnt/koji/repos/ec5-build/251/x86_64/blocklist -o /tmp/koji/tasks/1382/1382/repo -g /mnt/koji/repos/ec5-build/251/groups/comps.xml -r file:///tmp/koji/tasks/1382/1382/repo_251_premerge/ -r http://10.64.1.239/install/rhels5.5/x86_64/ -r http://mirror.aarnet.edu.au/pub/epel/5/x86_64/ -r http://10.64.1.239/install/HPC/
Adding repo: file:///tmp/koji/tasks/1382/1382/repo_251_premerge/
Adding repo: http://10.64.1.239/install/rhels5.5/x86_64/
Adding repo: http://mirror.aarnet.edu.au/pub/epel/5/x86_64/
Adding repo: http://10.64.1.239/install/HPC/
where the -a x86_64 flag appears to be filtering out all the 32 bit packages that are in http://10.64.1.239/install/rhels5.5/x86_64/
I get the feeling I'm going about this the wrong way...
Any suggestions?
Cheers,
Chris Edsall
Please consider the environment before printing this email.
NIWA is the trading name of the National Institute of Water & Atmospheric Research Ltd.
13 years
ProxyDNs
by Jamie Couture
Hi there,
I've recently been setting up a koji environment. I've been relying heavily
on the docs http://fedoraproject.org/wiki/Koji/ServerHowTo. The docs have
been a great help.
However, I ran into a small issue when modifying the ProxyDNs option in
/etc/koji-hub/hub.conf
DNUsernameComponent = CN
ProxyDNs = "/C=US/ST=Massachusetts/O=Example Org/OU=Example
User/CN=example/emailAddress=example(a)example.com"
I had problems with the server responding with: "(user) is not authorized to
login other users". I did change the value to my cert's settings.
Turns out that if I removed the quotes then everything is happy.
I'm not sure if it is intended to allow a quoted string for this option or
not, considering the current behaviour is to split multiple ProxyDNs' via |
Attached is a patch that will help avoid users seeing the above message when
authenticating with SSL. Of course, removing the quotes from your config
will correct the issue; the patch merely tries to avoid the problem by
removing quotes from the string when the client_dn is sought in the *if
<string> in <list>* statement.
13 years, 1 month
How do I remove a package from koji?
by steve.webb@beatport.com
How do I remove or clean out a build repo/tag from koji to start over with
a new build tag?
- Steve
--
Steve Webb | System Administrator
Beatport | Play With Music
------------------------------------------
2399 Blake Street, Suite 170
Denver, Colorado USA 80205
tel: +1.720.932.9103
fax: +1.720.932.9104
noc: +1.303.565.2710
mobile: +1.303.564.4269
13 years, 1 month
Re: koji sign plugin
by Pierre Guillet
Hello,
Thank you for your Koji callback.
I perform some modifications:
+ to handle DEFAULT section (if section for Tag name is not defined)
+ to support empty GPG pass-phrase
+ to work on localized OS
+ to log GPG messages if rpm --resign fails
Note: your GPG directory (gpg_path in .conf file) must be readable and
writeable by apache (the user which runs Koji hub)
Regards,
Pierre
# Koji callback for GPG signing RPMs before import
#
# Author:
# Paul B Schroeder <paulbsch "at" vbridges "dot" com>
from koji.plugin import register_callback
import logging
# Configuration file in /etc like for other plugins
CONFIG_FILE = '/etc/koji-hub/plugins/sign.conf'
def sign(cbtype, *args, **kws):
if kws['type'] != 'build':
return
# Get the tag name from the buildroot map
import sys
sys.path.insert(0, '/usr/share/koji-hub')
from kojihub import get_buildroot
br_id = kws['brmap'].values()[0]
br = get_buildroot(br_id)
tag_name = br['tag_name']
# Get GPG info using the config for the tag name
import ConfigParser
config = ConfigParser.ConfigParser()
config.read(CONFIG_FILE)
try:
rpm = config.get(tag_name, 'rpm')
except ConfigParser.NoSectionError:
rpm = config.get(ConfigParser.DEFAULTSECT, 'rpm')
try:
gpgbin = config.get(tag_name, 'gpgbin')
except ConfigParser.NoSectionError:
gpgbin = config.get(ConfigParser.DEFAULTSECT, 'gpgbin')
try:
gpg_path = config.get(tag_name, 'gpg_path')
except ConfigParser.NoSectionError:
gpg_path = config.get(ConfigParser.DEFAULTSECT, 'gpg_path')
try:
gpg_name = config.get(tag_name, 'gpg_name')
except ConfigParser.NoSectionError:
gpg_name = config.get(ConfigParser.DEFAULTSECT, 'gpg_name')
try:
gpg_pass = config.get(tag_name, 'gpg_pass')
except ConfigParser.NoSectionError:
gpg_pass = config.get(ConfigParser.DEFAULTSECT, 'gpg_pass')
# Get the package paths set up
from koji import pathinfo
uploadpath = pathinfo.work()
rpms = ''
for relpath in [kws['srpm']] + kws['rpms']:
rpms += '%s/%s ' % (uploadpath, relpath)
# Get the packages signed
import pexpect
import os
os.environ['LC_ALL'] = 'C'
logging.getLogger('koji.plugin.sign').info('Attempting to sign packages'
' (%s) with key "%s"' % (rpms, gpg_name))
rpm_cmd = "%s --resign --define '_signature gpg'" % rpm
rpm_cmd += " --define '_gpgbin %s'" % gpgbin
rpm_cmd += " --define '_gpg_path %s'" % gpg_path
rpm_cmd += " --define '_gpg_name %s' %s" % (gpg_name, rpms)
pex = pexpect.spawn(rpm_cmd, timeout=1000)
# Add rpm output to a temporary file
fout = os.tmpfile()
pex.logfile = fout
pex.expect('(E|e)nter (P|p)ass (P|p)hrase:', timeout=1000)
if not gpg_pass:
pex.sendline('\r')
else:
pex.sendline(gpg_pass)
i = pex.expect(['good', 'failed', 'skipping', pexpect.TIMEOUT])
pex.expect(pexpect.EOF)
if i == 0:
logging.getLogger('koji.plugin.sign').info('Package sign
successful!')
elif i == 1:
logging.getLogger('koji.plugin.sign').error('Pass phrase check
failed!')
elif i == 2:
logging.getLogger('koji.plugin.sign').error('Package sign skipped!')
elif i == 3:
logging.getLogger('koji.plugin.sign').error('Package sign timed
out!')
else:
logging.getLogger('koji.plugin.sign').error('Unexpected sign
result!')
if i != 0:
# Rewind in rpm output
fout.seek(0)
# Add GPG errors to log
for line in fout.readlines():
if 'gpg:' in line:
logging.getLogger('koji.plugin.sign').error(line.rstrip('\n'))
fout.close()
raise Exception, 'Package sign failed!'
else:
fout.close()
register_callback('preImport', sign)
Regards,
Pierre
13 years, 1 month