buildsys July 2015

buildsys@lists.fedoraproject.org

18 participants
15 discussions

[PATCH] use createrepo_c instead of createrepo to make repos if it is available
by Dennis Gilmore 30 Oct '15

30 Oct '15

Signed-off-by: Dennis Gilmore <dennis(a)ausil.us> --- builder/kojid | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/builder/kojid b/builder/kojid index 3a92ec3..4e53fb5 100755 --- a/builder/kojid +++ b/builder/kojid @@ -4252,7 +4252,11 @@ class CreaterepoTask(BaseTaskHandler): def create_local_repo(self, rinfo, arch, pkglist, groupdata, oldrepo): koji.ensuredir(self.outdir) - cmd = ['/usr/bin/createrepo', '-vd', '-o', self.outdir] + if os.path.isfile('/usr/bin/createrepo_c'): + cmd = ['/usr/bin/createrepo_c'] + else: + cmd = ['/usr/bin/createrepo'] + cmd.extend(['-vd', '-o', self.outdir]) if pkglist is not None: cmd.extend(['-i', pkglist]) if os.path.isfile(groupdata): -- 2.4.0

2 3

koji and mergerepo_c
by Thomas 08 Sep '15

08 Sep '15

Hi Folks, Quick question, would you consider to use mergerepo_c with option --all [1] in koji ? I am happy to work on a patch if it would be accepted. Maybe, we can keep default behavior and enable with an option on the tag. My use case is to be able to ship different release built against fix external repo package that may not be latest version (Buildrequires:pkg = 1.0.0 while external repo has already 1.1.0) [1]: --all Include all packages with the same name and arch if version or release is different. If used --method argument is ignored! -- Thomas

2 1

[PATCH] Drop tables and functions only if exist
by Pavol Babincak 20 Aug '15

20 Aug '15

At the time of first initialization of Koji database tables and functions don't exist. This isn't really a problem as SQL commands later in the script create them. For non-existent tables or functions psql prints NOTICE instead of an ERROR. Therefore one can check if database was initialized correctly with ON_ERROR_STOP set (see psql(1)): psql returns 3 if an error occurred in a script and the variable ON_ERROR_STOP was set. --- docs/schema.sql | 70 ++++++++++++++++++++++++++++----------------------------- 1 file changed, 35 insertions(+), 35 deletions(-) diff --git a/docs/schema.sql b/docs/schema.sql index 56418c9..7a46f7f 100644 --- a/docs/schema.sql +++ b/docs/schema.sql @@ -1,56 +1,56 @@ -- vim:noet:sw=8 -- still needs work -DROP TABLE build_notifications; +DROP TABLE IF EXISTS build_notifications; -DROP TABLE log_messages; +DROP TABLE IF EXISTS log_messages; -DROP TABLE buildroot_listing; -DROP TABLE image_listing; +DROP TABLE IF EXISTS buildroot_listing; +DROP TABLE IF EXISTS image_listing; -DROP TABLE rpminfo; -DROP TABLE image_builds; -DROP TABLE image_archives; +DROP TABLE IF EXISTS rpminfo; +DROP TABLE IF EXISTS image_builds; +DROP TABLE IF EXISTS image_archives; -DROP TABLE group_package_listing; -DROP TABLE group_req_listing; -DROP TABLE group_config; -DROP TABLE groups; +DROP TABLE IF EXISTS group_package_listing; +DROP TABLE IF EXISTS group_req_listing; +DROP TABLE IF EXISTS group_config; +DROP TABLE IF EXISTS groups; -DROP TABLE tag_listing; -DROP TABLE tag_packages; +DROP TABLE IF EXISTS tag_listing; +DROP TABLE IF EXISTS tag_packages; -DROP TABLE buildroot; -DROP TABLE repo; +DROP TABLE IF EXISTS buildroot; +DROP TABLE IF EXISTS repo; -DROP TABLE build_target_config; -DROP TABLE build_target; +DROP TABLE IF EXISTS build_target_config; +DROP TABLE IF EXISTS build_target; -DROP TABLE tag_config; -DROP TABLE tag_inheritance; -DROP TABLE tag; +DROP TABLE IF EXISTS tag_config; +DROP TABLE IF EXISTS tag_inheritance; +DROP TABLE IF EXISTS tag; -DROP TABLE build; +DROP TABLE IF EXISTS build; -DROP TABLE task; +DROP TABLE IF EXISTS task; -DROP TABLE host_channels; -DROP TABLE host; +DROP TABLE IF EXISTS host_channels; +DROP TABLE IF EXISTS host; -DROP TABLE channels; -DROP TABLE package; +DROP TABLE IF EXISTS channels; +DROP TABLE IF EXISTS package; -DROP TABLE user_groups; -DROP TABLE user_perms; -DROP TABLE permissions; +DROP TABLE IF EXISTS user_groups; +DROP TABLE IF EXISTS user_perms; +DROP TABLE IF EXISTS permissions; -DROP TABLE sessions; -DROP TABLE users; +DROP TABLE IF EXISTS sessions; +DROP TABLE IF EXISTS users; -DROP TABLE event_labels; -DROP TABLE events; -DROP FUNCTION get_event(); -DROP FUNCTION get_event_time(INTEGER); +DROP TABLE IF EXISTS event_labels; +DROP TABLE IF EXISTS events; +DROP FUNCTION IF EXISTS get_event(); +DROP FUNCTION IF EXISTS get_event_time(INTEGER); BEGIN WORK; -- 1.9.3

2 2

RFC: dynamic generation of mock config from template?
by Clark Williams 03 Aug '15

03 Aug '15

The other day someone suggested that mock should dynamically create configs from a set of templates, so that we wouldn't have to keep delivering and deleting a series of config files each time a new Fedora release goes out. Not sure who it was since I can't find the discussion in my IRC logs, but might have been dgilmore. Nirik? Don't know... Anyway, I started thinking about it and it seems doable. Right now configs are named with three fields: <distro>-<release>-<arch>.cfg We could come up with templates for distro-arch that would be used to generate a config. The idea is someone invokes mock with this command line: $ mock -r fedora-73-x86_64 --init We go look in /etc/mock and find no fedora-73-x86_64.cfg, so we go grab /etc/mock/template/fedora-x86_64 and substitute in '73' for the release number, then write /etc/mock/fedora-73-x86_64.cfg. Then we continue on our way, at least until the build fails due to non-existant repositories (presuming that we're not talking about the year 2035 here). Obviously you could fat-finger the release number and generate a bogus config file. Worse you might want to be using F22 but mis-type '21' and use a wrong, but existent config. Also, I don't really like the idea of dynamically creating files in /etc, so we might need to move the created configs off to a /var/mock/configs directory or something like that. The upside though is that we could stop having to add and delete configs when new Fedora releases come out. What do you guys think? Clark

6 8

[PATCH] backend: Ensure output files are owned by unpriv user with nspawn
by Colin Walters 29 Jul '15

29 Jul '15

In the systemd-nspawn case, while we need to be root to call the build, we still need to chown the output files. This regressed rpmdistro-gitoverlay which was expecting to be able to `link()` the SRPMs, and it couldn't because they were root owned. --- py/mockbuild/backend.py | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/py/mockbuild/backend.py b/py/mockbuild/backend.py index c342573..414475c 100644 --- a/py/mockbuild/backend.py +++ b/py/mockbuild/backend.py @@ -193,14 +193,14 @@ class Commands(object): buildstate = "build phase for %s" % baserpm self.state.start(buildstate) + # remove rpm db files to prevent version mismatch problems + # note: moved to do this before the user change below! + self.buildroot._nuke_rpm_db() + dropped_privs = False try: - # remove rpm db files to prevent version mismatch problems - # note: moved to do this before the user change below! - self.buildroot._nuke_rpm_db() - if not util.USE_NSPAWN: - # drop privs and become mock user self.uid_manager.becomeUser(self.buildroot.chrootuid, self.buildroot.chrootgid) + dropped_privs = True buildsetup = "build setup for %s" % baserpm self.state.start(buildsetup) @@ -222,6 +222,12 @@ class Commands(object): self.plugins.call_hooks('prebuild') results = self.rebuild_package(spec_path, timeout, check) + # In the nspawn case, we retained root until here, but we + # need to ensure our output files are owned by the caller's uid. + # So drop them now. + if not dropped_privs: + self.uid_manager.becomeUser(self.buildroot.chrootuid, self.buildroot.chrootgid) + dropped_privs = True if results: self.build_results.extend(self.copy_build_results(results)) elif self.config.get('short_circuit'): @@ -233,7 +239,7 @@ class Commands(object): self.state.finish(rpmbuildstate) finally: - if not util.USE_NSPAWN: + if dropped_privs: self.uid_manager.restorePrivs() if self.state.result != 'success': self.state.result = 'fail' -- 1.8.3.1

2 1

[PATCH] Add "rpmbuild_networking" key (False by default) for nspawn backend
by Colin Walters 29 Jul '15

29 Jul '15

>From 1726c9b75abad3361e6207d2ea4a8e8cb78c85c4 Mon Sep 17 00:00:00 2001 From: Colin Walters <walters(a)verbum.org> Date: Mon, 27 Jul 2015 21:35:20 -0400 Subject: [PATCH] Add "rpmbuild_networking" key (False by default) for nspawn backend Build systems like Koji and many others want to support one aspect of reproducible builds by ensuring all input artifacts are cached within the organization. This is the "If Github goes down, you can still build" aspect. Traditionally, Koji installations implement this via firewalling rules on the server. However, now with the systemd nspawn backend, we can rely on much more robust container features. In particular, systemd has a `--private-network` option which does exactly what we want. Teach mock how to use it, and enable this by default if the nspawn backend is in use. This has the potential to break mock users who were relying on networking - but it's easy to turn back on. But I assert that the majority of people using mock are exactly the kind of user who want to do full packaging, and it's best if they find out about any network issues before they do a Koji/whatever build. --- etc/mock/site-defaults.cfg | 3 +++ py/mockbuild/backend.py | 2 ++ py/mockbuild/util.py | 13 +++++++++---- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/etc/mock/site-defaults.cfg b/etc/mock/site-defaults.cfg index 4ce0392..355870c 100644 --- a/etc/mock/site-defaults.cfg +++ b/etc/mock/site-defaults.cfg @@ -54,6 +54,9 @@ # By default Mock use simple chroot(1). When you set this to True # it will use systemd-nspawn(1) # config_opts['use_nspawn'] = False +# If you're using nspawn, then by default networking will be turned off +# for rpmbuild. This helps ensure more reproducible builds. +# config_opts['rpmbuild_networking'] = False # The default package manager is Yum # config_opts['package_manager'] = 'yum' diff --git a/py/mockbuild/backend.py b/py/mockbuild/backend.py index c342573..8a5e644 100644 --- a/py/mockbuild/backend.py +++ b/py/mockbuild/backend.py @@ -428,10 +428,12 @@ class Commands(object): command = [rpmbuild_cmd] if not util.USE_NSPAWN: command = ["bash", "--login", "-c"] + command + print "network: %s" % (not self.config['rpmbuild_networking'],) self.buildroot.doChroot(command, shell=False, logger=self.buildroot.build_log, timeout=timeout, uid=self.buildroot.chrootuid, gid=self.buildroot.chrootgid, user=self.buildroot.chrootuser, + private_network=not self.config['rpmbuild_networking'], printOutput=self.config['print_main_output']) bd_out = self.make_chroot_path(self.buildroot.builddir) results = glob.glob(bd_out + '/RPMS/*.rpm') diff --git a/py/mockbuild/util.py b/py/mockbuild/util.py index 58b4f90..36b2188 100644 --- a/py/mockbuild/util.py +++ b/py/mockbuild/util.py @@ -420,7 +420,8 @@ def selinuxEnabled(): @traceLog() def do(command, shell=False, chrootPath=None, cwd=None, timeout=0, raiseExc=True, returnOutput=0, uid=None, gid=None, user=None, personality=None, - printOutput=False, env=None, pty=False, *args, **kargs): + printOutput=False, env=None, pty=False, private_network=False, + *args, **kargs): logger = kargs.get("logger", getLog()) output = "" @@ -437,7 +438,7 @@ def do(command, shell=False, chrootPath=None, cwd=None, timeout=0, raiseExc=True command = ['/bin/sh', '-c'] + command shell = False if chrootPath and USE_NSPAWN: - command = _prepare_nspawn_command(chrootPath, user, command) + command = _prepare_nspawn_command(chrootPath, user, command, private_network=private_network) logger.debug("Executing command: {0} with env {1} and shell {2}".format(command, env, shell)) child = subprocess.Popen( command, @@ -531,7 +532,7 @@ def is_in_dir(path, directory): return os.path.commonprefix([path, directory]) == directory -def _prepare_nspawn_command(chrootPath, user, cmd): +def _prepare_nspawn_command(chrootPath, user, cmd, private_network=False): cmd_is_list = isinstance(cmd, list) if user: # needs to be /bin because of el5 and el6 targets @@ -541,7 +542,10 @@ def _prepare_nspawn_command(chrootPath, user, cmd): cmd = ['/bin/su', '-l', user, '-c', '"{0}"'.format(cmd)] elif not cmd_is_list: cmd = [ cmd, ] - cmd = ['/usr/bin/systemd-nspawn', '-M' , uuid.uuid4().hex, '-D', chrootPath] + cmd + nspawn_argv = ['/usr/bin/systemd-nspawn', '-M' , uuid.uuid4().hex, '-D', chrootPath] + if private_network: + nspawn_argv.append('--private-network') + cmd = nspawn_argv + cmd if cmd_is_list: return cmd else: @@ -629,6 +633,7 @@ def setup_default_config_opts(unprivUid, version, pkgpythondir): config_opts['state_log_fmt_name'] = "state" config_opts['online'] = True config_opts['use_nspawn'] = False + config_opts['rpmbuild_networking'] = False config_opts['internal_dev_setup'] = True config_opts['internal_setarch'] = True -- 1.8.3.1

2 3

Changes in mock development
by Miroslav Suchý 29 Jul '15

29 Jul '15

Hi, there is small change in mock development. All future development will happen in origin/devel branch of mock.git. Branch origin/master will be used purely for releasing. If you want to submit you patch, then please base it origin/devel. I already updated related wiki/docs. -- Miroslav Suchy, RHCA Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

1 0

[PATCH] Don't retry on SSL failures
by Mathieu Bridon 25 Jul '15

25 Jul '15

From: Mathieu Bridon <bochecha(a)daitauha.fr> With the current code, trying to SSL-login with a bad certificate will just make it look like the client code is hanging. That's because it tries and tries again, silently, until it reaches it's maximum retry limit. But in the case of an SSL error, such as an expired client cert, there's really no point in retrying. --- koji/__init__.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/koji/__init__.py b/koji/__init__.py index 58971da..9ebe83a 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -57,7 +57,7 @@ import xmlrpclib import xml.sax import xml.sax.handler from xmlrpclib import loads, dumps, Fault -#import OpenSSL.SSL +import OpenSSL import zipfile def _(args): @@ -1938,6 +1938,9 @@ class ClientSession(object): except (SystemExit, KeyboardInterrupt): #(depending on the python version, these may or may not be subclasses of Exception) raise + except OpenSSL.SSL.Error as e: + # There's no point in retrying this + raise except Exception, e: self._close_connection() if not self.logged_in: -- 2.4.3

8 11

[PATCH] Check that opts is not None before accessing.
by Ralph Bean 24 Jul '15

24 Jul '15

This should fix the periodic 500 error we get from koji-web in Fedora Infra's staging environment. Signed-off-by: Ralph Bean <rbean(a)redhat.com> --- www/lib/kojiweb/util.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/www/lib/kojiweb/util.py b/www/lib/kojiweb/util.py index 89a5ddb..6610beb 100644 --- a/www/lib/kojiweb/util.py +++ b/www/lib/kojiweb/util.py @@ -466,7 +466,7 @@ def taskScratchClass(task_object): request = task_object['request'] if len(request) >= 3: opts = request[2] - if opts.get('scratch'): + if opts and opts.get('scratch'): return "scratch" return "" -- 2.1.0

4 6

koji client authentication from python code
by Josh Boyer 22 Jul '15

22 Jul '15

Hi All, Would it be possible to put the client authentication code found in the koji CLI into a python module that can be used by other code? Right now I've found no other way to interact with koji via python other than copying a large chunk of the code from koji. It seems even in the koji codebase, koji-gc and koji-shadow have also done the same. Specifically, the activate_session, ensure_connection, and koji config file parsing functions have to be copied. josh

1 0

Koji 1.10
by Mike McLean 15 Jul '15

15 Jul '15

Let's call this an RC for the moment. Grab the tarball and have at it. We'll call it final when I get back from vacation next week. Also, expect a 1.11 release before long. You can get the tarball here: https://fedorahosted.org/released/koji/ See the changelog for full details. E.g. git log koji-1.9.0..b8b03099c85f76ca8292a8ed99296c7d850ab22b == Highlights == support extra options for tags ability to set mock package_manager via tag extra options optionally handle extra footer as a template runroot plugin support conf.d directories for hub and web Add new {pre,post}RPMSign plugin callbacks numerous kojira optimizations list-history --watch image builds using the Indirection plugin Use iterators for potentially large queries to reduce memory use list-tasks cli improvements new "koji chainmaven" command move default kojid workdir from /tmp/koji/ to /var/tmp/koji/

3 2

[PATCH] Improved support for koji profiles.
by Daniel Mach 14 Jul '15

14 Jul '15

--- cli/koji | 137 ++++++++++++------------------------------------------- docs/profiles | 57 +++++++++++++++++++++++ koji/__init__.py | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 215 insertions(+), 109 deletions(-) create mode 100644 docs/profiles diff --git a/cli/koji b/cli/koji index f521d6d..29f26bb 100755 --- a/cli/koji +++ b/cli/koji @@ -186,88 +186,9 @@ def get_options(): list_commands() parser.error('Unknown command: %s' % args[0]) assert False - # load local config - defaults = { - 'server' : 'http://localhost/kojihub', - 'weburl' : 'http://localhost/koji', - 'topurl' : None, - 'pkgurl' : None, - 'topdir' : '/mnt/koji', - 'max_retries' : None, - 'retry_interval': None, - 'anon_retry' : None, - 'offline_retry' : None, - 'offline_retry_interval' : None, - 'keepalive' : True, - 'timeout' : None, - 'use_fast_upload': False, - 'poll_interval': 5, - 'krbservice': 'host', - 'cert': '~/.koji/client.crt', - 'ca': '~/.koji/clientca.crt', - 'serverca': '~/.koji/serverca.crt', - 'authtype': None - } - #note: later config files override earlier ones - configs = koji.config_directory_contents('/etc/koji.conf.d') - if os.access('/etc/koji.conf', os.F_OK): - configs.append('/etc/koji.conf') - if options.configFile: - fn = os.path.expanduser(options.configFile) - if os.path.isdir(fn): - contents = koji.config_directory_contents(fn) - if not contents: - parser.error("No config files found in directory: %s" % fn) - configs.extend(contents) - else: - if not os.access(fn, os.F_OK): - parser.error("No such file: %s" % fn) - configs.append(fn) - else: - user_config_dir = os.path.expanduser("~/.koji/config.d") - configs.extend(koji.config_directory_contents(user_config_dir)) - fn = os.path.expanduser("~/.koji/config") - if os.access(fn, os.F_OK): - configs.append(fn) - got_conf = False - for configFile in configs: - f = open(configFile) - config = ConfigParser.ConfigParser() - config.readfp(f) - f.close() - if config.has_section(options.profile): - got_conf = True - for name, value in config.items(options.profile): - #note the defaults dictionary also serves to indicate which - #options *can* be set via the config file. Such options should - #not have a default value set in the option parser. - if defaults.has_key(name): - if name in ('anon_retry', 'offline_retry', 'keepalive', 'use_fast_upload'): - defaults[name] = config.getboolean(options.profile, name) - elif name in ('max_retries', 'retry_interval', - 'offline_retry_interval', 'poll_interval', 'timeout'): - try: - defaults[name] = int(value) - except ValueError: - parser.error("value for %s config option must be a valid integer" % name) - assert False - else: - defaults[name] = value - if configs and not got_conf: - warn("Warning: no configuration for profile name: %s" % options.profile) - for name, value in defaults.iteritems(): - if getattr(options, name, None) is None: - setattr(options, name, value) - dir_opts = ('topdir', 'cert', 'ca', 'serverca') - for name in dir_opts: - # expand paths here, so we don't have to worry about it later - value = os.path.expanduser(getattr(options, name)) - setattr(options, name, value) - - #honor topdir - if options.topdir: - koji.BASEDIR = options.topdir - koji.pathinfo.topdir = options.topdir + + defaults = koji.read_config(options.profile, user_config=options.configFile) + options._update_loose(defaults.__dict__) #pkgurl is obsolete if options.pkgurl: @@ -1503,11 +1424,11 @@ def linked_upload(localfile, path, name=None): try: if name is None: name = os.path.basename(localfile) - dest_dir = os.path.join(koji.pathinfo.work(), path) + dest_dir = os.path.join(koji_profile_module.pathinfo.work(), path) dst = os.path.join(dest_dir, name) koji.ensuredir(dest_dir) # fix uid/gid to keep httpd happy - st = os.stat(koji.pathinfo.work()) + st = os.stat(koji_profile_module.pathinfo.work()) os.chown(dest_dir, st.st_uid, st.st_gid) print "Linking rpm to: %s" % dst os.link(localfile, dst) @@ -2018,7 +1939,7 @@ def handle_prune_signed_copies(options, session, args): for sig in sigs: sigkey = sig['sigkey'] by_sig.setdefault(sigkey, []).append(rpminfo) - builddir = koji.pathinfo.build(binfo) + builddir = koji_profile_module.pathinfo.build(binfo) build_files = 0 build_space = 0 if not by_sig and options.debug: @@ -2028,7 +1949,7 @@ def handle_prune_signed_copies(options, session, args): archdirs = {} sigdirs = {} for rpminfo in rpms: - signedpath = "%s/%s" % (builddir, koji.pathinfo.signed(rpminfo, sigkey)) + signedpath = "%s/%s" % (builddir, koji_profile_module.pathinfo.signed(rpminfo, sigkey)) try: st = os.lstat(signedpath) except OSError: @@ -2285,8 +2206,8 @@ def handle_list_signed(options, session, args): binfo = session.getBuild(rinfo['build_id']) build_idx[binfo['id']] = binfo binfo['name'] = binfo['package_name'] - builddir = koji.pathinfo.build(binfo) - signedpath = "%s/%s" % (builddir, koji.pathinfo.signed(rinfo, sigkey)) + builddir = koji_profile_module.pathinfo.build(binfo) + signedpath = "%s/%s" % (builddir, koji_profile_module.pathinfo.signed(rinfo, sigkey)) if not os.path.exists(signedpath): if options.debug: print "No copy: %s" % signedpath @@ -2492,7 +2413,6 @@ def anon_handle_latest_build(options, session, args): if len(args) < 2: parser.error(_("A tag name and package name must be specified")) assert False - pathinfo = koji.PathInfo() for pkg in args[1:]: if options.arch: @@ -2503,7 +2423,7 @@ def anon_handle_latest_build(options, session, args): for x in data: z = x.copy() x['name'] = builds_hash[x['build_id']]['package_name'] - x['path'] = os.path.join(pathinfo.build(x), pathinfo.rpm(z)) + x['path'] = os.path.join(koji_profile_module.pathinfo.build(x), koji_profile_module.pathinfo.rpm(z)) fmt = "%(path)s" else: fmt = "%(name)s-%(version)s-%(release)s.%(arch)s" @@ -2515,11 +2435,11 @@ def anon_handle_latest_build(options, session, args): if options.paths: if options.type == 'maven': for x in data: - x['path'] = pathinfo.mavenbuild(x) + x['path'] = koji_profile_module.pathinfo.mavenbuild(x) fmt = "%(path)-40s %(tag_name)-20s %(maven_group_id)-20s %(maven_artifact_id)-20s %(owner_name)s" else: for x in data: - x['path'] = pathinfo.build(x) + x['path'] = koji_profile_module.pathinfo.build(x) fmt = "%(path)-40s %(tag_name)-20s %(owner_name)s" else: if options.type == 'maven': @@ -2597,7 +2517,6 @@ def anon_handle_list_tagged(options, session, args): parser.error(_("Only one package name may be specified")) assert False activate_session(session) - pathinfo = koji.PathInfo() package = None if len(args) > 1: package = args[1] @@ -2630,14 +2549,14 @@ def anon_handle_list_tagged(options, session, args): build_idx = dict([(b['id'],b) for b in builds]) for rinfo in data: build = build_idx[rinfo['build_id']] - builddir = pathinfo.build(build) + builddir = koji_profile_module.pathinfo.build(build) if options.sigs: sigkey = rinfo['sigkey'] - signedpath = os.path.join(builddir, pathinfo.signed(rinfo, sigkey)) + signedpath = os.path.join(builddir, koji_profile_module.pathinfo.signed(rinfo, sigkey)) if os.path.exists(signedpath): rinfo['path'] = signedpath else: - rinfo['path'] = os.path.join(builddir, pathinfo.rpm(rinfo)) + rinfo['path'] = os.path.join(builddir, koji_profile_module.pathinfo.rpm(rinfo)) fmt = "%(path)s" data = [x for x in data if x.has_key('path')] else: @@ -2649,11 +2568,11 @@ def anon_handle_list_tagged(options, session, args): if options.paths: if options.type == 'maven': for x in data: - x['path'] = pathinfo.mavenbuild(x) + x['path'] = koji_profile_module.pathinfo.mavenbuild(x) fmt = "%(path)-40s %(tag_name)-20s %(maven_group_id)-20s %(maven_artifact_id)-20s %(owner_name)s" else: for x in data: - x['path'] = pathinfo.build(x) + x['path'] = koji_profile_module.pathinfo.build(x) fmt = "%(path)-40s %(tag_name)-20s %(owner_name)s" else: if options.type == 'maven': @@ -2721,7 +2640,6 @@ def anon_handle_list_untagged(options, session, args): opts = {} if package: opts['name'] = package - pathinfo = koji.PathInfo() data = session.untaggedBuilds(**opts) if options.show_references: @@ -2741,7 +2659,7 @@ def anon_handle_list_untagged(options, session, args): #data = [x for x in data if not refs.has_key(x['id'])] if options.paths: for x in data: - x['path'] = pathinfo.build(x) + x['path'] = koji_profile_module.pathinfo.build(x) fmt = "%(path)s" else: fmt = "%(name)s-%(version)s-%(release)s" @@ -3053,9 +2971,9 @@ def anon_handle_rpminfo(options, session, args): print "External Repository: %(name)s [%(id)i]" % repo print "External Repository url: %(url)s" % repo else: - print "RPM Path: %s" % os.path.join(koji.pathinfo.build(buildinfo), koji.pathinfo.rpm(info)) + print "RPM Path: %s" % os.path.join(koji_profile_module.pathinfo.build(buildinfo), koji_profile_module.pathinfo.rpm(info)) print "SRPM: %(epoch)s%(name)s-%(version)s-%(release)s [%(id)d]" % buildinfo - print "SRPM Path: %s" % os.path.join(koji.pathinfo.build(buildinfo), koji.pathinfo.rpm(buildinfo)) + print "SRPM Path: %s" % os.path.join(koji_profile_module.pathinfo.build(buildinfo), koji_profile_module.pathinfo.rpm(buildinfo)) print "Built: %s" % time.strftime('%a, %d %b %Y %H:%M:%S %Z', time.localtime(info['buildtime'])) print "Payload: %(payloadhash)s" %info print "Size: %(size)s" %info @@ -3127,23 +3045,23 @@ def anon_handle_buildinfo(options, session, args): if maven_archives: print "Maven archives:" for archive in maven_archives: - print os.path.join(koji.pathinfo.mavenbuild(info), koji.pathinfo.mavenfile(archive)) + print os.path.join(koji_profile_module.pathinfo.mavenbuild(info), koji_profile_module.pathinfo.mavenfile(archive)) win_archives = session.listArchives(buildID=info['id'], type='win') if win_archives: print "Windows archives:" for archive in win_archives: - print os.path.join(koji.pathinfo.winbuild(info), koji.pathinfo.winfile(archive)) + print os.path.join(koji_profile_module.pathinfo.winbuild(info), koji_profile_module.pathinfo.winfile(archive)) rpms = session.listRPMs(buildID=info['id']) image_info = session.getImageBuild(info['id']) img_archives = session.listArchives(buildID=info['id'], type='image') if img_archives: print 'Image archives:' for archive in img_archives: - print os.path.join(koji.pathinfo.imagebuild(info), archive['filename']) + print os.path.join(koji_profile_module.pathinfo.imagebuild(info), archive['filename']) if rpms: print "RPMs:" for rpm in rpms: - print os.path.join(koji.pathinfo.build(info), koji.pathinfo.rpm(rpm)) + print os.path.join(koji_profile_module.pathinfo.build(info), koji_profile_module.pathinfo.rpm(rpm)) if options.changelog: changelog = session.getChangelogEntries(info['id']) if changelog: @@ -4261,7 +4179,7 @@ def _printTaskInfo(session, task_id, level=0, recurse=True, verbose=True): files = session.listTaskOutput(task_id) logs = [filename for filename in files if filename.endswith('.log')] output = [filename for filename in files if not filename.endswith('.log')] - files_dir = '%s/%s' % (koji.pathinfo.work(), koji.pathinfo.taskrelpath(task_id)) + files_dir = '%s/%s' % (koji_profile_module.pathinfo.work(), koji_profile_module.pathinfo.taskrelpath(task_id)) owner = session.getUser(info['owner'])['name'] @@ -6124,9 +6042,8 @@ def anon_handle_download_build(options, session, args): if not suboptions.topurl: print "You must specify --topurl to download images" return 1 - pi = koji.PathInfo(topdir=suboptions.topurl) for archive in archives: - url = '%s/%s' % (pi.imagebuild(info), archive['filename']) + url = '%s/%s' % (pathinfo.imagebuild(info), archive['filename']) urls.append((url, archive['filename'])) else: # can't happen @@ -6628,7 +6545,9 @@ def activate_session(session): if __name__ == "__main__": global options + global koji_profile_module options, command, args = get_options() + koji_profile_module = koji.get_profile_module(options.profile) logger = logging.getLogger("koji") handler = logging.StreamHandler(sys.stderr) diff --git a/docs/profiles b/docs/profiles new file mode 100644 index 0000000..2f21723 --- /dev/null +++ b/docs/profiles @@ -0,0 +1,57 @@ +============= +Koji Profiles +============= +This document describes how to work with koji profiles. + + +Command Line Interface +====================== +Koji client allows connecting to multiple koji instances from CLI +by using profiles. The default profile is given by executable file name, +which is 'koji'. + +To change koji profile, you can: +* run koji with --profile=$profile_name argument +* change executable file name by symlinking $profile_name -> koji + + +Configuration Files +=================== +Configuration files are located in following locations: +* /etc/koji.conf +* /etc/koji.conf.d/*.conf +* ~/.koji/config.d/*.conf +* user-specified config +Koji reads them all, looking for [$profile_name] sections. + + +Using Koji Profiles in Python +============================= +Instead of using koji module directly, +get profile specific module by calling: +>>> mod = koji.get_profile_module($profile_name) + +This module is clone of koji module with additional +profile specific tweaks. + +Profile configuration is available via: +>>> mod.config + + +Example +------- +import koji + +fedora_koji = koji.get_profile_module("koji") +ppc_koji = koji.get_profile_module("ppc-koji") + +for i in (fedora_koji, ppc_koji): + print "PROFILE: %s" % i.config.profile + for key, value in sorted(i.config.__dict__.items()): + print " %s = %s" % (key, value) + print + + +TODO +==== +* consider using pyxdg for user config locations diff --git a/koji/__init__.py b/koji/__init__.py index c0d5a30..dae52fc 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -28,13 +28,16 @@ except ImportError: sys.stderr.write("Warning: Could not install krbV module. Kerberos support will be disabled.\n") sys.stderr.flush() import base64 +import ConfigParser import datetime import errno from fnmatch import fnmatch import httplib +import imp import logging import logging.handlers from koji.util import md5_constructor +import optparse import os import os.path import pwd @@ -1457,6 +1460,133 @@ def config_directory_contents(dir_name): return configs +CONFIG_DEFAULTS = { + 'server': 'http://localhost/kojihub', + 'weburl': 'http://localhost/koji', + 'topurl': None, + 'pkgurl': None, + 'topdir': '/mnt/koji', + 'max_retries': None, + 'retry_interval': None, + 'anon_retry': None, + 'offline_retry': None, + 'offline_retry_interval': None, + 'keepalive': True, + 'timeout': None, + 'use_fast_upload': False, + 'poll_interval': 5, + 'krbservice': 'host', + 'cert': '~/.koji/client.crt', + 'ca': '~/.koji/clientca.crt', + 'serverca': '~/.koji/serverca.crt', + 'authtype': None +} + + +INT_OPTIONS = ['max_retries', 'retry_interval', 'offline_retry_interval', 'poll_interval', 'timeout'] +BOOL_OPTIONS = ['anon_retry', 'offline_retry', 'keepalive', 'use_fast_upload'] +PATH_OPTIONS = ['topdir', 'cert', 'ca', 'serverca'] + + +def config_directory_contents(dir_name): + configs = [] + try: + conf_dir_contents = os.listdir(dir_name) + except OSError, exception: + if exception.errno != errno.ENOENT: + raise + else: + for name in sorted(conf_dir_contents): + if not name.endswith('.conf'): + continue + config_full_name = os.path.join(dir_name, name) + configs.append(config_full_name) + return configs + + +def read_config(profile_name, user_config=None): + result = CONFIG_DEFAULTS.copy() + for option in CONFIG_DEFAULTS: + if option in PATH_OPTIONS: + result[option] = os.path.expanduser(result[option]) + + configs = [] + + # main config + configs.append("/etc/koji.conf") + + # conf.d + configs.extend(config_directory_contents("/etc/koji.conf.d")) + + # user config + configs.append(os.path.expanduser("~/.koji/config")) + + # user conf.d + configs.extend(config_directory_contents(os.path.expanduser("~/.koji/conf.d"))) + + # TODO: read configs via xdg.BaseDirectory.load_config_path("koji") + + # user config specified in runtime + if user_config is not None: + configs.append(user_config) + + # read configs in particular order, use the last value found + for config_path in configs: + if not os.access(config_path, os.F_OK): + continue + config = ConfigParser.SafeConfigParser() + config.readfp(open(config_path, "r")) + + if profile_name not in config.sections(): + continue + + # check for invalid options + invalid_options = [] + for option in config.options(profile_name): + if option not in result: + invalid_options.append(option) + + if invalid_options: + raise ValueError("Invalid options: %s" % ", ".join(invalid_options)) + + for option in config.options(profile_name): + if option in BOOL_OPTIONS: + result[option] = config.getboolean(profile_name, option) + elif option in INT_OPTIONS: + result[option] = config.getint(profile_name, option) + else: + result[option] = config.get(profile_name, option) + if option in PATH_OPTIONS: + result[option] = os.path.expanduser(result[option]) + + result["profile"] = profile_name + + # convert dict to optparse Values + options = optparse.Values(result) + return options + + +config = read_config("koji") + + +def get_profile_module(profile_name, config=None): + """ + Create module for a koji instance. + Override profile specific module attributes: + * BASEDIR + * config + * pathinfo + """ + if config is None: + config = read_config(profile_name) + mod = imp.new_module("_koji__%s" % profile_name) + mod.__dict__.update(globals()) + mod.config = config + mod.BASEDIR = config.topdir + mod.pathinfo = PathInfo(topdir=config.topdir) + return mod + + class PathInfo(object): # ASCII numbers and upper- and lower-case letter for use in tmpdir() ASCII_CHARS = [chr(i) for i in range(48, 58) + range(65, 91) + range(97, 123)] -- 2.4.3

3 2

[PATCH] docs: Document how to write a plugin
by Mathieu Bridon 11 Jul '15

11 Jul '15

--- Koji supports different types of plugins. This commit documents the 3 types that I know of, but I have absolutely no idea if there are others. docs/Writing_a_plugin.md | 153 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 docs/Writing_a_plugin.md diff --git a/docs/Writing_a_plugin.md b/docs/Writing_a_plugin.md new file mode 100644 index 0000000..36cddd9 --- /dev/null +++ b/docs/Writing_a_plugin.md @@ -0,0 +1,153 @@ +# Writing Koji plugins + +Depending on what you are trying to do, there are different ways to write a +Koji plugin. + +Each is described in this file, by use case. + +## Adding new task types + +Koji can do several things, for example build RPMs, or live CDs. Those are +types of tasks which Koji knows about. + +If you need to do something which Koji does not know yet how to do, you could +create a Koji Builder plugin. + +Such a plugin would minimally look like this: + + from koji.tasks import BaseTaskHandler + + class MyTask(BaseTaskHandler): + Methods = ['mytask'] + _taskWeight = 2.0 + + def handler(self, arg1, arg2, kwarg1=None): + self.logger.debug("Running my task...") + + # Here is where you actually do something + +A few explanations on what goes on here: + +* Your task needs to inherit from `koji.tasks.BaseTaskHandler` +* Your task must have a `Methods` attribute, which is a list of the method + names your task can handle. +* You can specify the weight of your task with the `_taskWeight` attribute. + The more intensive (CPU, IO, ...) your task is, the higher this number + should be. +* The task object has a `logger` attribute, which is a Python logger with the + usual `debug`, `info`, `warning` and `error` methods. The messages you send + with it will end up in the Koji Builder logs (`kojid.log`) +* Your task must have a `handler()` method. That is the method Koji will call + to run your task. It is the method that should actually do what you need. It + can have as many positional and named arguments as you want. + +Save your plugin as e.g `mytask.py`, then install it in the Koji Builder +plugins folder: `/usr/lib/koji-builder-plugins/` + +Finally, edit the Koji Builder config file, `/etc/kojid/kojid.conf`: + + # A space-separated list of plugins to enable + plugins = mytask + +Restart the Koji Builder service, and your plugin will be enabled. + +You can try running a task from your new task type with the command-line: + + $ koji make-task mytask arg1 arg2 kwarg1 + +## Exporting new API methods over XMLRPC + +Koji clients talk to the Koji Hub via an XMLRPC API. + +It is sometimes desirable to add to that API, so that clients can request +things Koji does not expose right now. + +Such a plugin would minimally look like this: + + def mymethod(arg1, arg2, kwarg1=None): + # Here is where you actually do something + + mymethod.exported = True + +A few explanations on what goes on here: + +* Your plugin is just a method, with whatever positional and/or named + arguments you need. +* You must export your method by setting its `exported` attribute to `True` +* The `context.session.assertPerm()` is how you ensure that the + +Save your plugin as e.g `mymethod.py`, then install it in the Koji Hub plugins +folder: `/usr/lib/koji-hub-plugins/` + +Finally, edit the Koji Hub config file, `/etc/koji-hub/hub.conf`: + + # A space-separated list of plugins to enable + Plugins = mymethod + +Restart the Koji Hub service, and your plugin will be enabled. + +You can try calling the new XMLRPC API with the Python client library: + + >>> import koji + >>> session = koji.ClientSession("http://koji/example.org/kojihub") + >>> session.mymethod(arg1, arg2, kwarg1='some value') + +### Ensuring the user has the required permissions + +If you want your new XMLRPC API to require specific permissions from the user, +all you need to do is add the following to your method: + + from koji.context import context + + def mymethod(arg1, arg2, kwarg1=None): + context.session.assertPerm("admin") + + # Here is where you actually do something + + mymethod.exported = True + +In the example above, Koji will ensure that the user is an administrator. You +could of course create your own permission, and check for that. + +## Running code automatically triggered on events + +You might want to run something automatically when something else happens in +Koji. + +A typical example is to automatically sign a package right after a build +finished. Another would be to send a notification to a message bus after any +kind of event. + +This can be achieved with a plugin, which would look minimally as follows: + + from koji.plugin import callback + + @callback('preTag', 'postTag') + def mycallback(cbtype, tag, build, user, force=False): + # Here is where you actually do something + +A few explanations on what goes on here: + +* The `@callback` decorator allows you to declare which events should trigger + your function. You can pass as many as you want. For a list of supported + events, see `koji/plugins.py`. +* The arguments of the function depend on the event you subscribed to. As a + result, you need to know how it will be called by Koji. You probably should + use `*kwargs` to be safe. You can see how callbacks are called in the + `hub/kojihub.py` file, search for calls of the `run_callbacks` function. + +Save your plugin as e.g `mycallback.py`, then install it in the Koji Hub +plugins folder: `/usr/lib/koji-hub-plugins` + +Finally, edit the Koji Hub config file, `/etc/koji-hub/hub.conf`: + + # A space-separated list of plugins to enable + Plugins = mycallback + +Restart the Koji Hub service, and your plugin will be enabled. + +You can try triggering your callback plugin with the command-line. For +example, if you registered a callback for the `postTag` event, try tagging a +build: + + $ koji tag-build mytag mypkg-1.0-1 -- 2.4.3

3 2

Koji 2.0 planning
by Mike McLean 02 Jul '15

02 Jul '15

It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive. Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13. Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x. The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development. https://lists.fedorahosted.org/mailman/listinfo/koji-devel Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation. = High level goals = • better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process = Highlights/Major changes = • python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc = Yet more changes = • store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)

8 8

Documenting mock-scm plugin
by Miroslav Suchý 02 Jul '15

02 Jul '15

FIY: I wrote documentation for mock-scm https://fedoraproject.org/wiki/Projects/Mock/Plugin/Scm I find some bugs during writing this doc and it will be fixed in next version of mock. If you want to enhance this doc, feel free to do it. -- Miroslav Suchy, RHCA Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

buildsys July 2015