buildsys February 2011

buildsys@lists.fedoraproject.org

16 participants
8 discussions

[BUG] Koji and Lustre
by Thomas Guthmann 10 Sep '11

10 Sep '11

Hey guys, We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py) when used with a Lustre filesystem. This function parses all attributes of every file of a build and is used when you want to display build.log/root.log through the web interface. Everything returned by listTaskOutput is returned through XML-RPC and as a result we had this : An error has occurred while processing your request. Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> Traceback (most recent call last): File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in publish_object return old_publish_object(req, object) File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py", line 412, in publish_object return publish_object(req,util.apply_fs_data(object, req.form, req=req)) File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line 439, in apply_fs_data return object(**args) File "/usr/share/koji-web/scripts/index.py", line 649, in getfile output = server.listTaskOutput(taskID, stat=True) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718, in _callMethod raise err Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC limits'> The issue comes from the st_dev value gathered by getStat (stat). In Lustre this value can be very high and that's why it complains. To fix that, we used the same condition than st_size, we cast the value as a string. See attached patch. Example (see 'Device:' the value after the '/'): # stat build.log File: `build.log' Size: 106021 Blocks: 208 IO Block: 2097152 regular file Device: e04ae70eh/3763005198d Inode: 721664 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache) Access: 2010-07-13 10:44:40.000000000 +1000 Modify: 2010-07-12 12:54:15.000000000 +1000 Change: 2010-07-12 12:54:52.000000000 +1000 # That's what is read by listTaskOutput relfilename=build.log ATTR=st_atime GETATTR=1278981616 relfilename=build.log ATTR=st_blksize GETATTR=2097152 relfilename=build.log ATTR=st_blocks GETATTR=208 relfilename=build.log ATTR=st_ctime GETATTR=1278903292 relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\ relfilename=build.log ATTR=st_gid GETATTR=48 relfilename=build.log ATTR=st_ino GETATTR=721664 relfilename=build.log ATTR=st_mode GETATTR=33188 relfilename=build.log ATTR=st_mtime GETATTR=1278903255 relfilename=build.log ATTR=st_nlink GETATTR=1 relfilename=build.log ATTR=st_rdev GETATTR=0 relfilename=build.log ATTR=st_size GETATTR=106021 relfilename=build.log ATTR=st_uid GETATTR=48 Hope it helps, lost a good amount of time on that one :) Cheers, Thomas

2 3

Koji 1.6 - Mock SRPM Build failing - NFS Path - no such file or directory
by Ciaran Bryan 01 Mar '11

01 Mar '11

Hi, I have setup a koji server and client builder successfully and can submit koji client commands and issue build tasks successfully. When an SRPM build task is taken by my builder server (same server as koji-hub/web) the mock buildroot gets setup ok and yum installed the correct build group binary packages etc within the buildroot. Then as mock goes to build the source RPM it fails as the location of the source RPM is the /mnt/work/cli-build..... NFS folder, but that NFS folder is not mounted to/available within the buildroot. How do I make my source rpm available to the mock buildroot environment without using NFS or is my setup wrong ? Thanks, C. My koji build task was submitted as follows: [kojiadmin@nse-build ~]$ koji build newbay-nse /usr/src/redhat/SRPMS/zookeeper-3.2.2-1.src.rpm Mock output: INFO: mock.py version 1.0.7 starting... State Changed: init plugins State Changed: start State Changed: lock buildroot State Changed: clean State Changed: init State Changed: lock buildroot Mock Version: 1.0.7 INFO: Mock Version: 1.0.7 State Changed: running yum INFO: mock.py version 1.0.7 starting... State Changed: init plugins State Changed: start INFO: Start(/mnt/koji/work/cli-build/1298901262.7331891.WlovUVjV/zookeeper-3.2.2-1.src.rpm) Config(nse-build-41-31) State Changed: init State Changed: lock buildroot Mock Version: 1.0.7 INFO: Mock Version: 1.0.7 State Changed: running yum State Changed: setup ERROR: Exception(/mnt/koji/work/cli-build/1298901262.7331891.WlovUVjV/zookeeper-3.2.2-1.src.rpm) Config(nse-build-41-31) 0 minutes 0 seconds INFO: Results and/or logs in: /var/lib/mock/newbay-nse-build-41-31/result Traceback (most recent call last): File "/usr/sbin/mock", line 693, in ? main(retParams) File "/usr/sbin/mock", line 635, in main do_rebuild(config_opts, chroot, args) File "<peak.util.decorators.rewrap wrapping __main__.do_rebuild at 0x1BC285F0>", line 3, in do_rebuild File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/sbin/mock", line 372, in do_rebuild chroot.build(srpm, timeout=config_opts['rpmbuild_timeout']) File "<peak.util.decorators.rewrap wrapping mock.backend.build at 0x1BBE6B90>", line 3, in build File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.4/site-packages/mock/backend.py", line 439, in build gid=self.chrootgid, File "/usr/lib/python2.4/site-packages/mock/backend.py", line 366, in doChroot shell=shell, *args, **kargs ) File "<peak.util.decorators.rewrap wrapping mock.util.do at 0x1BBEC500>", line 3, in do File "/usr/lib/python2.4/site-packages/mock/trace_decorator.py", line 70, in trace result = func(*args, **kw) File "/usr/lib/python2.4/site-packages/mock/util.py", line 299, in do preexec_fn = preexec, File "/usr/lib64/python2.4/subprocess.py", line 550, in __init__ errread, errwrite) File "/usr/lib64/python2.4/subprocess.py", line 996, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory ________________________________ STRICTLY PRIVATE, CONFIDENTIAL AND PRIVILEGED COMMUNICATION. This message (including attachments) may contain information that is privileged, confidential or protected from disclosure. They are intended solely for the use of the intended recipient. If you are not the intended recipient, you are hereby notified that dissemination, disclosure, copying, distribution, printing, transmission or use of this message or any information contained in it is strictly prohibited. If you have received this message from NewBay Software in error, please immediately notify the sender by reply email and delete this message from your computer. The content of this e-mail, and any files transmitted with it, may have been changed or altered without the consent of the author. Although we have taken steps to ensure that this email and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.

2 2

need 32bit rpms in a mock setup for a 64 bit build
by Chris Edsall 22 Feb '11

22 Feb '11

Hi, I have a x86_64 builder and I am trying to make an x86_64 build. This particular build needs a compiler that is only available as a binary tarball. I have made an RPM out of the tarball, however it has i386 dependancies. When I come to use it in koji (by adding it as a BuildRequires) mock complains because the yum configuration can't satisfy the dependencies: DEBUG backend.py:603: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ resolvedep 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel' DEBUG util.py:291: Executing command: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ resolvedep 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel' DEBUG util.py:256: 0:zlib-devel-1.2.3-3.x86_64 DEBUG util.py:256: 1:valgrind-3.5.0-1.el5.x86_64 DEBUG util.py:256: 0:gawk-3.1.5-14.el5.x86_64 DEBUG util.py:256: 0:pgi-10.3-7.x86_64 DEBUG util.py:256: 0:hdf5-devel-1.8.5.patch1-5.x86_64 DEBUG util.py:256: 0:curl-devel-7.15.5-9.el5.x86_64 DEBUG util.py:330: Child returncode was: 0 DEBUG backend.py:603: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ install 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel' DEBUG util.py:291: Executing command: /usr/bin/yum --installroot /var/lib/mock/ec5-build-402-251/root/ install 'zlib-devel' 'valgrind' 'gawk' 'pgi' 'hdf5-devel >= 1.8.4' 'curl-devel' DEBUG util.py:256: Package gawk-3.1.5-14.el5.x86_64 already installed and latest version DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems DEBUG util.py:256: --> Missing Dependency: libnsl.so.1 is needed by package pgi-10.3-7.x86_64 (build) DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems DEBUG util.py:256: --> Missing Dependency: libXext.so.6 is needed by package pgi-10.3-7.x86_64 (build) DEBUG util.py:256: pgi-10.3-7.x86_64 from build has depsolving problems DEBUG util.py:256: --> Missing Dependency: libc.so.6(GLIBC_2.1.2) is needed by package pgi-10.3-7.x86_64 (build) However the same compiler package installs fine on the command line with the same yum reops configured [root@cli-dev ~]# yum -y install pgi [ ... ] Installed: pgi.x86_64 0:10.3-7 Complete! It needed these non x86_64 packages: [root@cli-dev ~]# yum deplist pgi | egrep 'i[3456]86' | sort -u provider: alsa-lib.i386 1.0.17-1.el5 provider: glibc.i686 2.5-49 provider: IBMJava2-142-ia32-JRE.i386 1.4.2-13.6 provider: libgcj.i386 4.1.2-48.el5 provider: libX11.i386 1.0.3-11.el5 provider: libXext.i386 1.0.1-2.1 provider: libXi.i386 1.0.1-4.el5_4 provider: libXtst.i386 1.0.1-3.1 The difference, of course, is that the repo mock is pointing at has been created by mergerepos: $ /usr/libexec/kojid/mergerepos -a x86_64 -b /mnt/koji/repos/ec5-build/251/x86_64/blocklist -o /tmp/koji/tasks/1382/1382/repo -g /mnt/koji/repos/ec5-build/251/groups/comps.xml -r file:///tmp/koji/tasks/1382/1382/repo_251_premerge/ -r http://10.64.1.239/install/rhels5.5/x86_64/ -r http://mirror.aarnet.edu.au/pub/epel/5/x86_64/ -r http://10.64.1.239/install/HPC/ Adding repo: file:///tmp/koji/tasks/1382/1382/repo_251_premerge/ Adding repo: http://10.64.1.239/install/rhels5.5/x86_64/ Adding repo: http://mirror.aarnet.edu.au/pub/epel/5/x86_64/ Adding repo: http://10.64.1.239/install/HPC/ where the -a x86_64 flag appears to be filtering out all the 32 bit packages that are in http://10.64.1.239/install/rhels5.5/x86_64/ I get the feeling I'm going about this the wrong way... Any suggestions? Cheers, Chris Edsall Please consider the environment before printing this email. NIWA is the trading name of the National Institute of Water & Atmospheric Research Ltd.

3 3

[PUNGI PATCH] Add an option for only pulling the best provider
by Bill Nottingham 15 Feb '11

15 Feb '11

This is so it doesn't pull 200 fonts for "font(:lang=en)", or all the MTAs. Can easily change option name, etc. if necessary. Bill

3 2

Cleaning scratch & work directories
by Anthony Joseph Messina 07 Feb '11

07 Feb '11

I'm continuing work a on a private Koji instance (Kerberos) and am now working on automating cleanup, etc. It is mentioned here http://lists.fedoraproject.org/pipermail/buildsys/2010-December/003415.html that "Koji leaves it to the administrator to clean out the scratch build directory." Is there anything special about cleaning up scratch builds or the work/{cli-build,cli-import,tasks} directories for that matter? Do I need to interact with the Koji database in any way to let koji know these files are gone? What I mean is, can I simply remove the files and directories without any Koji integration? Thanks. -A -- Anthony - http://messinet.com - http://messinet.com/~amessina/gallery 8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E

4 6

ProxyDNs
by Jamie Couture 04 Feb '11

04 Feb '11

Hi there, I've recently been setting up a koji environment. I've been relying heavily on the docs http://fedoraproject.org/wiki/Koji/ServerHowTo. The docs have been a great help. However, I ran into a small issue when modifying the ProxyDNs option in /etc/koji-hub/hub.conf DNUsernameComponent = CN ProxyDNs = "/C=US/ST=Massachusetts/O=Example Org/OU=Example User/CN=example/emailAddress=example(a)example.com" I had problems with the server responding with: "(user) is not authorized to login other users". I did change the value to my cert's settings. Turns out that if I removed the quotes then everything is happy. I'm not sure if it is intended to allow a quoted string for this option or not, considering the current behaviour is to split multiple ProxyDNs' via | Attached is a patch that will help avoid users seeing the above message when authenticating with SSL. Of course, removing the quotes from your config will correct the issue; the patch merely tries to avoid the problem by removing quotes from the string when the client_dn is sought in the *if <string> in <list>* statement.

1 0

How do I remove a package from koji?
by steve.webb＠beatport.com 04 Feb '11

04 Feb '11

How do I remove or clean out a build repo/tag from koji to start over with a new build tag? - Steve -- Steve Webb | System Administrator Beatport | Play With Music ------------------------------------------ 2399 Blake Street, Suite 170 Denver, Colorado USA 80205 tel: +1.720.932.9103 fax: +1.720.932.9104 noc: +1.303.565.2710 mobile: +1.303.564.4269

6 10

Re: koji sign plugin
by Pierre Guillet 04 Feb '11

04 Feb '11

Hello, Thank you for your Koji callback. I perform some modifications: + to handle DEFAULT section (if section for Tag name is not defined) + to support empty GPG pass-phrase + to work on localized OS + to log GPG messages if rpm --resign fails Note: your GPG directory (gpg_path in .conf file) must be readable and writeable by apache (the user which runs Koji hub) Regards, Pierre # Koji callback for GPG signing RPMs before import # # Author: # Paul B Schroeder <paulbsch "at" vbridges "dot" com> from koji.plugin import register_callback import logging # Configuration file in /etc like for other plugins CONFIG_FILE = '/etc/koji-hub/plugins/sign.conf' def sign(cbtype, *args, **kws): if kws['type'] != 'build': return # Get the tag name from the buildroot map import sys sys.path.insert(0, '/usr/share/koji-hub') from kojihub import get_buildroot br_id = kws['brmap'].values()[0] br = get_buildroot(br_id) tag_name = br['tag_name'] # Get GPG info using the config for the tag name import ConfigParser config = ConfigParser.ConfigParser() config.read(CONFIG_FILE) try: rpm = config.get(tag_name, 'rpm') except ConfigParser.NoSectionError: rpm = config.get(ConfigParser.DEFAULTSECT, 'rpm') try: gpgbin = config.get(tag_name, 'gpgbin') except ConfigParser.NoSectionError: gpgbin = config.get(ConfigParser.DEFAULTSECT, 'gpgbin') try: gpg_path = config.get(tag_name, 'gpg_path') except ConfigParser.NoSectionError: gpg_path = config.get(ConfigParser.DEFAULTSECT, 'gpg_path') try: gpg_name = config.get(tag_name, 'gpg_name') except ConfigParser.NoSectionError: gpg_name = config.get(ConfigParser.DEFAULTSECT, 'gpg_name') try: gpg_pass = config.get(tag_name, 'gpg_pass') except ConfigParser.NoSectionError: gpg_pass = config.get(ConfigParser.DEFAULTSECT, 'gpg_pass') # Get the package paths set up from koji import pathinfo uploadpath = pathinfo.work() rpms = '' for relpath in [kws['srpm']] + kws['rpms']: rpms += '%s/%s ' % (uploadpath, relpath) # Get the packages signed import pexpect import os os.environ['LC_ALL'] = 'C' logging.getLogger('koji.plugin.sign').info('Attempting to sign packages' ' (%s) with key "%s"' % (rpms, gpg_name)) rpm_cmd = "%s --resign --define '_signature gpg'" % rpm rpm_cmd += " --define '_gpgbin %s'" % gpgbin rpm_cmd += " --define '_gpg_path %s'" % gpg_path rpm_cmd += " --define '_gpg_name %s' %s" % (gpg_name, rpms) pex = pexpect.spawn(rpm_cmd, timeout=1000) # Add rpm output to a temporary file fout = os.tmpfile() pex.logfile = fout pex.expect('(E|e)nter (P|p)ass (P|p)hrase:', timeout=1000) if not gpg_pass: pex.sendline('\r') else: pex.sendline(gpg_pass) i = pex.expect(['good', 'failed', 'skipping', pexpect.TIMEOUT]) pex.expect(pexpect.EOF) if i == 0: logging.getLogger('koji.plugin.sign').info('Package sign successful!') elif i == 1: logging.getLogger('koji.plugin.sign').error('Pass phrase check failed!') elif i == 2: logging.getLogger('koji.plugin.sign').error('Package sign skipped!') elif i == 3: logging.getLogger('koji.plugin.sign').error('Package sign timed out!') else: logging.getLogger('koji.plugin.sign').error('Unexpected sign result!') if i != 0: # Rewind in rpm output fout.seek(0) # Add GPG errors to log for line in fout.readlines(): if 'gpg:' in line: logging.getLogger('koji.plugin.sign').error(line.rstrip('\n')) fout.close() raise Exception, 'Package sign failed!' else: fout.close() register_callback('preImport', sign) Regards, Pierre

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys February 2011