[BUG] Koji and Lustre
by Thomas Guthmann
Hey guys,
We found a bug(?) in listTaskOutput (/usr/share/koji-hub/kojihub.py)
when used with a Lustre filesystem. This function parses all attributes
of every file of a build and is used when you want to display
build.log/root.log through the web interface. Everything returned by
listTaskOutput is returned through XML-RPC and as a result we had this :
An error has occurred while processing your request.
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
Traceback (most recent call last):
File "/usr/share/koji-web/lib/kojiweb/publisher.py", line 16, in
publish_object
return old_publish_object(req, object)
File "/usr/lib64/python2.4/site-packages/mod_python/publisher.py",
line 412, in publish_object
return publish_object(req,util.apply_fs_data(object, req.form,
req=req))
File "/usr/lib64/python2.4/site-packages/mod_python/util.py", line
439, in apply_fs_data
return object(**args)
File "/usr/share/koji-web/scripts/index.py", line 649, in getfile
output = server.listTaskOutput(taskID, stat=True)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1468,
in __call__
return self.__func(self.__name,args,opts)
File "/usr/lib/python2.4/site-packages/koji/__init__.py", line 1718,
in _callMethod
raise err
Fault: <Fault 1: 'exceptions.OverflowError: long int exceeds XML-RPC
limits'>
The issue comes from the st_dev value gathered by getStat (stat). In
Lustre this value can be very high and that's why it complains. To fix
that, we used the same condition than st_size, we cast the value as a
string. See attached patch.
Example (see 'Device:' the value after the '/'):
# stat build.log
File: `build.log'
Size: 106021 Blocks: 208 IO Block: 2097152 regular file
Device: e04ae70eh/3763005198d Inode: 721664 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 48/ apache) Gid: ( 48/ apache)
Access: 2010-07-13 10:44:40.000000000 +1000
Modify: 2010-07-12 12:54:15.000000000 +1000
Change: 2010-07-12 12:54:52.000000000 +1000
# That's what is read by listTaskOutput
relfilename=build.log ATTR=st_atime GETATTR=1278981616
relfilename=build.log ATTR=st_blksize GETATTR=2097152
relfilename=build.log ATTR=st_blocks GETATTR=208
relfilename=build.log ATTR=st_ctime GETATTR=1278903292
relfilename=build.log ATTR=st_dev GETATTR=3763005198 <----- /!\
relfilename=build.log ATTR=st_gid GETATTR=48
relfilename=build.log ATTR=st_ino GETATTR=721664
relfilename=build.log ATTR=st_mode GETATTR=33188
relfilename=build.log ATTR=st_mtime GETATTR=1278903255
relfilename=build.log ATTR=st_nlink GETATTR=1
relfilename=build.log ATTR=st_rdev GETATTR=0
relfilename=build.log ATTR=st_size GETATTR=106021
relfilename=build.log ATTR=st_uid GETATTR=48
Hope it helps, lost a good amount of time on that one :)
Cheers,
Thomas
12 years, 8 months
[PATCH] fedpkg: call sources before running mockbuild
by Josh Boyer
Currently a 'fedpkg mockbuild' in a fresh clone of a package will
result in something like:
[jwboyer@r2d2 kernel]$ fedpkg mockbuild
error: File /home/jwboyer/src/fedora/kernel/linux-2.6.34.tar.bz2: No
such file or directory
Could not run mockbuild: Command 'rpmbuild --define '_sourcedir
/home/jwboyer/src/fedora/kernel' --define '_specdir
/home/jwboyer/src/fedora/kernel' --define '_builddir
/home/jwboyer/src/fedora/kernel' --define '_srcrpmdir
/home/jwboyer/src/fedora/kernel' --define '_rpmdir
/home/jwboyer/src/fedora/kernel' --define 'dist .fc13' --define
'fedora 13' --define 'fc13 1' --nodeps -bs
/home/jwboyer/src/fedora/kernel/kernel.spec' returned non-zero exit
status 1
[jwboyer@r2d2 kernel]$
because the sources have not been downloaded yet. Under the old
dist-cvs system, a mockbuild had a dependency on srpm, which had a
dependency on sources.
The following patch accomplishes the a similar dependency in fedpkg by
making it fetch the sources prior to doing the mock setup. Tested
locally on F14.
josh
---
diff --git a/src/fedpkg.py b/src/fedpkg.py
index dbae1a8..31c18dd 100755
--- a/src/fedpkg.py
+++ b/src/fedpkg.py
@@ -576,6 +576,12 @@ def local(args):
sys.exit(1)
def mockbuild(args):
+ try:
+ pyfedpkg.sources(args.path)
+ except pyfedpkg.FedpkgError, e:
+ log.error('Could not download sources: %s' % e)
+ sys.exit(1)
+
# Pick up any mockargs from the env
mockargs = []
try:
13 years, 4 months
Strange mock build failure due to typo
by Paul Howarth
Hi,
today I have been preparing an update to perl-Math-Pari and came across
a very strange build failure whilst doing a local mock build on a Fedora
13 x86_64 host. My package built successfully on x86_64 but when I tried
to build for i386, the build failed during %setup but without any
diagnostics. The SRPM was installed but no attempt to install its build
requirements was made. The root.log showed an exit status of 0 for all
commands that had been run.
After much experimentation bisecting the changes I had made, I
discovered that a typo in the changelog entry was the culprit: I had set
the year to 2100 instead of 2010. So it would appear that somewhere in
the mock/yum/rpm stack there may be a year 2038 problem waiting to bite
us (though I suspect there may not be too many 32-bit builds happening
by then).
Seriously though, it would be nice to have better diagnostics for this
and perhaps an rpmlint check for changelog entries in the future?
Paul.
13 years, 4 months
Building EL6 with koji/mock on a EL5.5 host
by Brian_Kosick@McAfee.com
Hi All,
I'm trying to build el6 on a koji build host that is el5.5.
I'm running into issues in the mock environment with rpm and what version of BDB it is using. Basically, I was originally having issues with yum failing on deps of rpmlib(PayLoadIsXZ), so after a little research I found this link from a list: http://infrastructure.fedoraproject.org/builder-rpms/x86_64/. These rpms (basically a intermediate version of rpm that supports XZ compression) got me around the missing rpm libs and on to mock exiting complaining about not being able to read the rpm __db files. I believe that I have correctly diagnosed the issue here; that the versions of __db files that the rpm 4.6 (from the link) produces is not liked by the rpm (4.8) of the el6 mock root. From what I've gleaned about mock is that there is no easy way to get around this rpm db incompatibility. And there is no such mock option that will let you do the equivalent of "rpm /var/lib/rpm/__db* && rpm --rebuilddb". Has anyone run into this before? And if so how did you address/handle it?
I believe that what I may want to do is create a el6 build host and use channels to control which host builds the el6 dist tag. However, I cannot seem to find information on how to create and use custom channels with koji and only vague references to the concept of channels in koji. For instance my questions would be:
How to create a new channel?
How to associate a tag/target/dist with a channel?
I'm assuming that the koji add-host-to-channel command would add the new el6 host correctly, but how to then remove the the new host from the "default" channel so that it doesn't build el5 and which,. I suspect, would have the same rpm db issues in reverse.
Thanks,
Brian
13 years, 4 months
koji using krb - having problems
by steve.webb@beatport.com
Hello there.
I'm having some problems running koji using kerberos authentication.
I've verified that I have standard kerberos authentication from the
machine. I've followed the
http://fedoraproject.org/wiki/Koji/ServerHowTo howto document and I'm up
to the point where I'm trying to use the koji cli to create a user and I'm
getting:
The end of my koji-hub/hub.conf file looks like this:
AuthPrincipal host/bpbuild001.co0.nar.beatportcorp.net
AuthKeytab /etc/koji.keytab
ProxyPrincipals koji/bpbuild001.co0.nar.beatportcorp.net
HostPrincipalFormat compile/bpbuild001.co0.nar.beatportcorp.net
[root@bpbuild001 etc]# koji --authtype=kerberos add-user kojira
Kerberos authentication failed: No credentials cache found (-1765328189)
[root@bpbuild001 etc]# koji add-user kojira
Unable to log in, no authentication methods available
The document doesn't have any methods to verify/debug that I've gotten the
krb configs correct.. Is there a way to debug that I've done the krb
configs properly?
- Steve Webb
--
Steve Webb | System Administrator
Beatport | Music for DJ's
------------------------------------------
2399 Blake Street, Suite 170
Denver, Colorado USA 80205
tel: +1.720.932.9103
fax: +1.720.932.9104
noc: +1.303.565.2710
mobile: +1.303.564.4269
13 years, 4 months
Building rpm-4.8.1 fails with "sh: no job control"
by Alex
Hi,
I have a FC13 x86_64 devel box that I've been working with for some
time, and I recently started to have a problem building RPMs. It
always fails with some kind of "bad file descriptor" or "no job
control" message. I'm not certain the problem is limited to just rpm,
but there doesn't appear to be any other problems with the server.
Building rpm-4.8.1 fails on test 43. I've included the rpmtests.log
file output below. Please note the "sh: no job control" and "sort:
stat failed: -: Bad file descriptor" messages.
How does rpm pass data to check-files? Could this be a more general
interprocess communications problem and not specific to RPM?
# -*- compilation -*-
43. rpmbuild.at:23: testing ...
./rpmbuild.at:25:
rm -rf ${TOPDIR}
as_dir=${TOPDIR}/SOURCES; as_fn_mkdir_p
cp "${abs_srcdir}"/data/SOURCES/hello-1.0.tar.gz ${TOPDIR}/SOURCES
run rpmbuild \
-ba "${abs_srcdir}"/data/SPECS/hello.spec
stderr:
sh: line 0: fg: no job control
+ umask 022
+ cd /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILD
+ cd /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILD
+ rm -rf hello-1.0
+ /usr/bin/gzip -dc
/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/SOURCES/hello-1.0.tar.gz
+ /bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd hello-1.0
+ exit 0
+ umask 022
+ cd /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILD
+ cd hello-1.0
+ make
hello.c: In function 'main':
hello.c:4: warning: incompatible implicit declaration of built-in
function 'printf'
+ exit 0
+ umask 022
+ cd /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILD
+ cd hello-1.0
+ rm -rf /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64
+ mkdir -p /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/local/bin
+ make DESTDIR=/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64
install
+ /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing//usr/lib/rpm/brp-compress
+ /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing//usr/lib/rpm/brp-strip
+ /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing//usr/lib/rpm/brp-strip-static-archive
+ /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing//usr/lib/rpm/brp-strip-comment-note
+ umask 022
+ cd /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILD
+ cd hello-1.0
+ DOCDIR=/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/share/doc/hello-1.0
+ export DOCDIR
+ rm -rf /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/share/doc/hello-1.0
+ /bin/mkdir -p
/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/share/doc/hello-1.0
+ cp -pr FAQ /home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/share/doc/hello-1.0
+ exit 0
sort: stat failed: -: Bad file descriptor
error: Installed (but unpackaged) file(s) found:
/usr/local/bin/hello
/usr/share/doc/hello-1.0/FAQ
Installed (but unpackaged) file(s) found:
/usr/local/bin/hello
/usr/share/doc/hello-1.0/FAQ
stdout:
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.UdWceT
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.ep7X3B
make[4]: Entering directory
`/usr/src/engarde/community-3.0.24/engarde/BUILD/rpm-4.8.1/tests/testing/build/BUILD/hello-1.0'
cc hello.c -o hello
make[4]: Leaving directory
`/usr/src/engarde/community-3.0.24/engarde/BUILD/rpm-4.8.1/tests/testing/build/BUILD/hello-1.0'
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.Us7xcl
make[4]: Entering directory
`/usr/src/engarde/community-3.0.24/engarde/BUILD/rpm-4.8.1/tests/testing/build/BUILD/hello-1.0'
install -m 0755 hello
/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64/usr/local/bin
make[4]: Leaving directory
`/usr/src/engarde/community-3.0.24/engarde/BUILD/rpm-4.8.1/tests/testing/build/BUILD/hello-1.0'
Processing files: hello-1.0-1.x86_64
Executing(%doc): /bin/sh -e /var/tmp/rpm-tmp.wEYIY4
Provides: hi
Requires(interp): /bin/sh /bin/sh /bin/sh /bin/sh
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(pre): /bin/sh
Requires(post): /bin/sh
Requires(preun): /bin/sh
Requires(postun): /bin/sh
Requires: libc.so.6()(64bit) libc.so.6(GLIBC_2.2.5)(64bit) rtld(GNU_HASH)
Conflicts: goodbye
Obsoletes: howdy
Checking for unpackaged file(s):
/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing//usr/lib/rpm/check-files
/home/packman/rpmbuild/BUILD/rpm-4.8.1/tests/testing/build/BUILDROOT/hello-1.0-1.x86_64
RPM build errors:
./rpmbuild.at:25: exit code was 1, expected 0
43. rpmbuild.at:23: 43. rpmbuild -ba *.spec (rpmbuild.at:23): FAILED
(rpmbuild.at:25)
What could be the cause of this? How can I troubleshoot it?
Thanks,
Alex
13 years, 5 months
Signing built RPMs or how to create signed RPMs.
by Allen Hewes
Hi,
I have my koji setup building RPMs from my subversion repo. I have mash sucking the RPMs out of koji and making yum repos. Right now, the RPMs in my yum repo aren't signed.
I have read some old posts from Jesse w.r.t adding signatures into/with koji. I found the commands for koji cli, but I am not clear as to what to do or how to do it.
My end goal is have the RPMs signed with my key when a client machine runs yum install/update from my repo. What I use to do before I started with koji was rpm --resign *.rpm (enter pass phrase) and then do createrepo -d.
I know I can build & sign stuff with rpmbuild --sign but where(how) would someone change this inside mock running inside kojid?
So my question is where does this happen with koji? Or does it happen outside of koji?
Thanks,
/allen
13 years, 5 months
Re: koji using krb - having problems
by Oliver Falk
He can get a valid ticket for steve(a)EXAMPLE.COM even if he logs in as root...
Steve, do you want us to take a look at your configs?
And can you provide a step-by-step example of what you have done?
-of
Anthony Messina <amessina(a)messinet.com> schrieb:
>On 12/16/2010 06:14 PM, steve.webb(a)beatport.com wrote:
>> [root@bpbuild001 etc]# koji add-user kojira
>> Unable to log in, no authentication methods available
>>
>> The document doesn't have any methods to verify/debug that I've gotten the
>> krb configs correct.. Is there a way to debug that I've done the krb
>> configs properly?
>
>You are doing this under the root account. I'm guessing that your root
>user might not be the koji administrative user you added during setup
>and that you don't have kerberos credentials as that administrative user.
>
>If the koji admin user you created had a username of 'steve' and
>kerberos principal of steve(a)EXAMPLE.COM, then if you are logged in as
>'steve' and have done a kinit steve(a)EXAMPLE.COM, you should then be able
>to perform the tasks.
>
>-A
>
>--
>Anthony - http://messinet.com - http://messinet.com/~amessina/gallery
>8F89 5E72 8DF0 BCF0 10BE 9967 92DC 35DC B001 4A4E
>
>
>--
>buildsys mailing list
>buildsys(a)lists.fedoraproject.org
>https://admin.fedoraproject.org/mailman/listinfo/buildsys
13 years, 5 months
for help
by Jinze Xue
hello!
I'm building some rpms of x86_64,and some of them need glibc-devel of i686
to build. but I can't add it into buildrequires. so it failed with error:
No Package Found For /usr/include/gnu/stubs-32.h
How could I solve this problem.Waiting for help,thanks all!!!
13 years, 5 months
Re: koji sign plugin
by Oliver Falk
The gpg key must only be available on the hub host, right!?
-of
Paul B Schroeder <paulbsch(a)vbridges.com> schrieb:
>As a follow up to the recent thread on singing RPMs in koji...and the many times this question pops
>up on the list. I've written some code that uses the koji plugin framework for signing packages.
>I'm betting this may be useful to many folks that don't want/need sigul. It might also be useful
>to get this into the koji-hub-plugins package?
>
>At any rate, here is the code and an example config file.. sign.py goes into your PluginPath.
>The config file needs to be readable by the apache user and should probably be chmoded 600. Also,
>make sure you add sign to the Plugins option in hub.conf. Oh, you'll want to install pexpect too.
>
>
>sign.py:
># Koji callback for GPG signing RPMs before import
>#
># Author:
># Paul B Schroeder <paulbsch "at" vbridges "dot" com>
>
>from koji.plugin import register_callback
>import logging
>
>config_file = '/usr/lib/koji-hub-plugins/sign.conf'
>
>def sign(cbtype, *args, **kws):
> if kws['type'] != 'build':
> return
>
> # Get the tag name from the buildroot map
> import sys
> sys.path.insert(0, '/usr/share/koji-hub')
> from kojihub import get_buildroot
> br_id = kws['brmap'].values()[0]
> br = get_buildroot(br_id)
> tag_name = br['tag_name']
>
> # Get GPG info using the config for the tag name
> from ConfigParser import ConfigParser
> config = ConfigParser()
> config.read(config_file)
> rpm = config.get(tag_name, 'rpm')
> gpgbin = config.get(tag_name, 'gpgbin')
> gpg_path = config.get(tag_name, 'gpg_path')
> gpg_name = config.get(tag_name, 'gpg_name')
> gpg_pass = config.get(tag_name, 'gpg_pass')
>
> # Get the package paths set up
> from koji import pathinfo
> uploadpath = pathinfo.work()
> rpms = ''
> for relpath in [kws['srpm']] + kws['rpms']:
> rpms += '%s/%s ' % (uploadpath, relpath)
>
> # Get the packages signed
> import pexpect
> logging.getLogger('koji.plugin.sign').info('Attempting to sign packages'
> ' (%s) with key "%s"' % (rpms, gpg_name))
> rpm_cmd = "%s --resign --define '_signature gpg'" % rpm
> rpm_cmd += " --define '_gpgbin %s'" % gpgbin
> rpm_cmd += " --define '_gpg_path %s'" % gpg_path
> rpm_cmd += " --define '_gpg_name %s' %s" % (gpg_name, rpms)
> pex = pexpect.spawn(rpm_cmd, timeout=1000)
> pex.expect('(E|e)nter (P|p)ass (P|p)hrase:', timeout=1000)
> pex.sendline(gpg_pass)
> i = pex.expect(['good', 'failed', 'skipping', pexpect.TIMEOUT])
> if i == 0:
> logging.getLogger('koji.plugin.sign').info('Package sign successful!')
> elif i == 1:
> logging.getLogger('koji.plugin.sign').error('Pass phrase check failed!')
> elif i == 2:
> logging.getLogger('koji.plugin.sign').error('Package sign skipped!')
> elif i == 3:
> logging.getLogger('koji.plugin.sign').error('Package sign timed out!')
> else:
> logging.getLogger('koji.plugin.sign').error('Unexpected sign result!')
> if i != 0:
> raise Exception, 'Package sign failed!'
> pex.expect(pexpect.EOF)
>
>register_callback('preImport', sign)
>
>
>sign.conf:
>[DEFAULT]
>rpm = /bin/rpm
>gpgbin = /usr/bin/gpg
>gpg_path = /usr/lib/koji-hub-plugins/sign_gnupg
>gpg_name = My Company, Inc. <support(a)mycompany.com>
>gpg_pass = my_passphrase
>
># Defaults can be overridden on a per-tag basis
>[dist-foo-build]
>gpg_name = My Other Company, Inc. <support(a)myothercompany.com>
>gpg_pass = my_other_passphrase
>
>
>
>
>Cheers...Paul...
>
>
>--
>---
>Paul B Schroeder
><paulbsch "at" vbridges "dot" com>
>--
>buildsys mailing list
>buildsys(a)lists.fedoraproject.org
>https://admin.fedoraproject.org/mailman/listinfo/buildsys
13 years, 5 months