With mock switching to dnf, we needed to conditionally adjust mock's package_manager config on a per-tag basis. I didn't want to add yet another specialized field in the tag_config table, so I extended the schema to support more flexible extra options for tags.
This set of patches adds the extra options functionality and uses it to control set mock's package_manager option.
Each tag can now be assigned arbitrary extra options. These are stored as json in the database, so they can be of any type that json supports. The getBuildConfig call follows inheritance when determining these.
To tell mock to use dnf in a tag, you would run a command like:
# koji edit-tag TAG -x mock.package_manager=dnf
This setting is inherited.
tag_extra table in schema report tag extras in taginfo and buildconfig --- docs/schema.sql | 16 ++++++++++++++++ hub/kojihub.py | 59 ++++++++++++++++++++++++++++++++++++++------------------- 2 files changed, 56 insertions(+), 19 deletions(-)
diff --git a/docs/schema.sql b/docs/schema.sql index aad24a9..d5ffa87 100644 --- a/docs/schema.sql +++ b/docs/schema.sql @@ -371,6 +371,22 @@ CREATE TABLE tag_config ( UNIQUE (tag_id,active) ) WITHOUT OIDS;
+CREATE TABLE tag_extra ( + tag_id INTEGER NOT NULL REFERENCES tag(id), + key TEXT NOT NULL, + value TEXT NOT NULL, -- TODO - move this to jsonb when we can +-- versioned - see desc above + create_event INTEGER NOT NULL REFERENCES events(id) DEFAULT get_event(), + revoke_event INTEGER REFERENCES events(id), + creator_id INTEGER NOT NULL REFERENCES users(id), + revoker_id INTEGER REFERENCES users(id), + active BOOLEAN DEFAULT 'true' CHECK (active), + CONSTRAINT active_revoke_sane CHECK ( + (active IS NULL AND revoke_event IS NOT NULL AND revoker_id IS NOT NULL) + OR (active IS NOT NULL AND revoke_event IS NULL AND revoker_id IS NULL)), + PRIMARY KEY (create_event, tag_id, key), + UNIQUE (tag_id, key, active) +) WITHOUT OIDS;
-- the tag_updates table provides a mechanism to indicate changes relevant to tag -- that are not reflected in a versioned table. For example: builds changing volumes, diff --git a/hub/kojihub.py b/hub/kojihub.py index 9cc8ff2..d9dec97 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -2761,14 +2761,15 @@ def get_tag(tagInfo, strict=False, event=None): a string (the tag name) or an int (the tag ID). Returns a map containing the following keys:
- - id - - name - - perm_id (may be null) - - perm (name, may be null) - - arches (may be null) - - locked - - maven_support - - maven_include_all + - id : unique id for the tag + - name : name of the tag + - perm_id : permission id (may be null) + - perm : permission name (may be null) + - arches : tag arches (string, may be null) + - locked : lock setting (boolean) + - maven_support : maven support flag (boolean) + - maven_include_all : maven include all flag (boolean) + - extra : extra tag parameters (dictionary)
If there is no tag matching the given tagInfo, and strict is False, return None. If strict is True, raise a GenericError. @@ -2807,8 +2808,28 @@ def get_tag(tagInfo, strict=False, event=None): if strict: raise koji.GenericError, "Invalid tagInfo: %r" % tagInfo return None + result['extra'] = get_tag_extra(result) + return result + + +def get_tag_extra(tagInfo, event=None): + """ Get tag extra info (no inheritance) """ + tables = ['tag_extra'] + fields = ['key', 'value'] + clauses = [eventCondition(event, table='tag_extra'), "tag_id = %(id)i"] + query = QueryProcessor(columns=fields, tables=tables, clauses=clauses, values=tagInfo, + opts={'asList': True}) + result = {} + for key, value in query.execute(): + try: + value = simplejson.loads(value) + except Exception: + # this should not happen + raise koji.GenericError("Invalid tag extra data: %s : %r", key, value) + result[key] = value return result
+ def edit_tag(tagInfo, **kwargs): """Edit information for an existing tag.
@@ -8755,17 +8776,17 @@ class RootExports(object): def getBuildConfig(self,tag,event=None): """Return build configuration associated with a tag""" taginfo = get_tag(tag,strict=True,event=event) - arches = taginfo['arches'] - if arches is None: - #follow inheritance for arches - order = readFullInheritance(taginfo['id'],event=event) - for link in order: - if link['noconfig']: - continue - arches = get_tag(link['parent_id'],strict=True,event=event)['arches'] - if arches is not None: - taginfo['arches'] = arches - break + order = readFullInheritance(taginfo['id'], event=event) + #follow inheritance for arches and extra + for link in order: + if link['noconfig']: + continue + ancestor = get_tag(link['parent_id'], strict=True, event=event) + if taginfo['arches'] is None and ancestor['arches'] is not None: + taginfo['arches'] = ancestor['arches'] + for key in ancestor['extra']: + if key not in taginfo['extra']: + taginfo['extra'][key] = ancestor['extra'][key] return taginfo
def getRepo(self,tag,state=None,event=None):
--- builder/kojid | 2 ++ koji/__init__.py | 2 ++ 2 files changed, 4 insertions(+)
diff --git a/builder/kojid b/builder/kojid index b6b79c1..5b9d66b 100755 --- a/builder/kojid +++ b/builder/kojid @@ -239,6 +239,8 @@ class BuildRoot(object): opts['maven_envs'] = self.maven_envs opts['bind_opts'] = self.bind_opts opts['target_arch'] = self.target_arch + if 'mock.package_manager' in self.config['extra']: + opts['package_manager'] = self.config['extra']['mock.package_manager'] output = koji.genMockConfig(self.name, self.br_arch, managed=True, **opts)
#write config diff --git a/koji/__init__.py b/koji/__init__.py index 58971da..c0d5a30 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -1286,6 +1286,8 @@ def genMockConfig(name, arch, managed=False, repoid=None, tag_name=None, **opts) # Don't let a build last more than 24 hours 'rpmbuild_timeout': opts.get('rpmbuild_timeout', 86400) } + if opts.get('package_manager'): + config_opts['package_manager'] = opts['package_manager']
# bind_opts are used to mount parts (or all of) /dev if needed. # See kojid::LiveCDTask for a look at this option in action.
--- cli/koji | 15 +++++++++++++++ hub/kojihub.py | 41 ++++++++++++++++++++++++++++++----------- 2 files changed, 45 insertions(+), 11 deletions(-)
diff --git a/cli/koji b/cli/koji index 908d829..7968460 100755 --- a/cli/koji +++ b/cli/koji @@ -4353,6 +4353,12 @@ def anon_handle_taginfo(options, session, args): if session.mavenEnabled(): print "Maven support?: %s" % (info['maven_support'] and 'yes' or 'no') print "Include all Maven archives?: %s" % (info['maven_include_all'] and 'yes' or 'no') + if 'extra' in info: + print "Tag options:" + keys = info['extra'].keys() + keys.sort() + for key in keys: + print " %s : %s" % (key, pprint.pformat(info['extra'][key])) dest_targets = session.getBuildTargets(destTagID=info['id'], **event_opts) build_targets = session.getBuildTargets(buildTagID=info['id'], **event_opts) repos = {} @@ -4446,6 +4452,8 @@ def handle_edit_tag(options, session, args): parser.add_option("--no-maven-support", action="store_true", help=_("Disable creation of Maven repos for this tag")) parser.add_option("--include-all", action="store_true", help=_("Include all packages in this tag when generating Maven repos")) parser.add_option("--no-include-all", action="store_true", help=_("Do not include all packages in this tag when generating Maven repos")) + parser.add_option("-x", "--extra", action="append", default=[], metavar="key=value", + help=_("Set tag extra option")) (options, args) = parser.parse_args(args) if len(args) != 1: parser.error(_("Please specify a name for the tag")) @@ -4473,6 +4481,13 @@ def handle_edit_tag(options, session, args): opts['maven_include_all'] = True if options.no_include_all: opts['maven_include_all'] = False + if options.extra: + extra = {} + for xopt in options.extra: + key, value = xopt.split('=') + value = arg_filter(value) + extra[key] = value + opts['extra'] = extra #XXX change callname session.editTag2(tag,**opts)
diff --git a/hub/kojihub.py b/hub/kojihub.py index d9dec97..d41e6b8 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -44,6 +44,7 @@ import os import re import rpm import shutil +import simplejson as json import stat import subprocess import sys @@ -2822,7 +2823,7 @@ def get_tag_extra(tagInfo, event=None): result = {} for key, value in query.execute(): try: - value = simplejson.loads(value) + value = json.loads(value) except Exception: # this should not happen raise koji.GenericError("Invalid tag extra data: %s : %r", key, value) @@ -2885,18 +2886,36 @@ def edit_tag(tagInfo, **kwargs): if kwargs.has_key(key) and data[key] != kwargs[key]: changed = True data[key] = kwargs[key] - if not changed: - return + if changed: + update = UpdateProcessor('tag_config', values=data, clauses=['tag_id = %(id)i']) + update.make_revoke() + update.execute()
- update = UpdateProcessor('tag_config', values=data, clauses=['tag_id = %(id)i']) - update.make_revoke() - update.execute() + insert = InsertProcessor('tag_config', data=dslice(data, ('arches', 'perm_id', 'locked'))) + insert.set(tag_id=data['id']) + insert.set(**dslice(data, ('maven_support', 'maven_include_all'))) + insert.make_create() + insert.execute() + + # handle extra data + if 'extra' in kwargs: + for key in kwargs['extra']: + value = kwargs['extra'][key] + if key not in tag['extra'] or tag['extra'] != value: + data = { + 'tag_id' : tag['id'], + 'key' : key, + 'value' : json.dumps(kwargs['extra'][key]), + } + # revoke old entry, if any + update = UpdateProcessor('tag_extra', values=data, clauses=['tag_id = %(tag_id)i', 'key=%(key)s']) + update.make_revoke() + update.execute() + # add new entry + insert = InsertProcessor('tag_extra', data=data) + insert.make_create() + insert.execute()
- insert = InsertProcessor('tag_config', data=dslice(data, ('arches', 'perm_id', 'locked'))) - insert.set(tag_id=data['id']) - insert.set(**dslice(data, ('maven_support', 'maven_include_all'))) - insert.make_create() - insert.execute()
def old_edit_tag(tagInfo, name, arches, locked, permissionID): """Edit information for an existing tag."""
--- cli/koji | 8 ++++++++ hub/kojihub.py | 1 + 2 files changed, 9 insertions(+)
diff --git a/cli/koji b/cli/koji index 7968460..fd6d409 100755 --- a/cli/koji +++ b/cli/koji @@ -3853,6 +3853,13 @@ def _print_histline(entry, **kwargs): fmt = "new tag: %(tag.name)s" else: fmt = "tag deleted: %(tag.name)s" + elif table == 'tag_extra': + if edit: + fmt = "tag option %(key)s for tag %(tag.name)s altered" + elif create: + fmt = "added tag option %(key)s for tag %(tag.name)s" + else: + fmt = "tag option %(key)s removed for %(tag.name)s" elif table == 'build_target_config': if edit: fmt = "build target configuration for %(build_target.name)s updated" @@ -3962,6 +3969,7 @@ _table_keys = { 'user_groups' : ['user_id', 'group_id'], 'tag_inheritance' : ['tag_id', 'parent_id'], 'tag_config' : ['tag_id'], + 'tag_extra' : ['tag_id', 'key'], 'build_target_config' : ['build_target_id'], 'external_repo_config' : ['external_repo_id'], 'tag_external_repos' : ['tag_id', 'external_repo_id'], diff --git a/hub/kojihub.py b/hub/kojihub.py index d41e6b8..ca6985a 100644 --- a/hub/kojihub.py +++ b/hub/kojihub.py @@ -5500,6 +5500,7 @@ def query_history(tables=None, **kwargs): 'user_groups' : ['user_id', 'group_id'], 'tag_inheritance' : ['tag_id', 'parent_id', 'priority', 'maxdepth', 'intransitive', 'noconfig', 'pkg_filter'], 'tag_config' : ['tag_id', 'arches', 'perm_id', 'locked', 'maven_support', 'maven_include_all'], + 'tag_extra' : ['tag_id', 'key', 'value'], 'build_target_config' : ['build_target_id', 'build_tag', 'dest_tag'], 'external_repo_config' : ['external_repo_id', 'url'], 'tag_external_repos' : ['tag_id', 'external_repo_id', 'priority'],
--- www/kojiweb/taginfo.chtml | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff --git a/www/kojiweb/taginfo.chtml b/www/kojiweb/taginfo.chtml index 8a0701d..1298996 100644 --- a/www/kojiweb/taginfo.chtml +++ b/www/kojiweb/taginfo.chtml @@ -1,4 +1,5 @@ #from kojiweb import util +#import pprint
#include "includes/header.chtml"
@@ -153,6 +154,17 @@ <td colspan="2"><a href="tagdelete?tagID=$tag.id$util.authToken($self)">Delete tag</a></td> </tr> #end if + #if $tag.get('extra') + <tr> + <th>Extra options:</th> + </tr> + #for $key in $tag['extra'] + <tr> + <th>$key</th> + <td>$pprint.pformat($tag['extra'][$key])</td> + </tr> + #end for + #end if </table>
#include "includes/footer.chtml"
On 06/17/2015 03:33 PM, Mike McLean wrote:
With mock switching to dnf, we needed to conditionally adjust mock's package_manager config on a per-tag basis. I didn't want to add yet another specialized field in the tag_config table, so I extended the schema to support more flexible extra options for tags.
This set of patches adds the extra options functionality and uses it to control set mock's package_manager option.
Each tag can now be assigned arbitrary extra options. These are stored as json in the database, so they can be of any type that json supports. The getBuildConfig call follows inheritance when determining these.
To tell mock to use dnf in a tag, you would run a command like:
# koji edit-tag TAG -x mock.package_manager=dnf
This setting is inherited.
Also, I should add that I expect to find more use for these extra options in the future.
Dne 17.6.2015 v 21:33 Mike McLean napsal(a):
With mock switching to dnf, we needed to conditionally adjust mock's package_manager config on a per-tag basis. I didn't want to add yet another specialized field in the tag_config table, so I extended the schema to support more flexible extra options for tags.
This set of patches adds the extra options functionality and uses it to control set mock's package_manager option.
Each tag can now be assigned arbitrary extra options. These are stored as json in the database, so they can be of any type that json supports. The getBuildConfig call follows inheritance when determining these.
To tell mock to use dnf in a tag, you would run a command like:
# koji edit-tag TAG -x mock.package_manager=dnf
This setting is inherited.
Nice.
However when builder(s) are upgraded to F22 you need to add: config_opts['yum_command'] = '/usr/bin/yum-deprecated' too. Can this extra option can be utilized for this too? How you will know the version of builder OS? I assume that not all builders will be migrated to F22 at once.
I'm just raising issues, which caused me problems in Copr.
Mirek
On 06/17/2015 05:01 PM, Miroslav Suchy wrote:
Dne 17.6.2015 v 21:33 Mike McLean napsal(a):
With mock switching to dnf, we needed to conditionally adjust mock's package_manager config on a per-tag basis. I didn't want to add yet another specialized field in the tag_config table, so I extended the schema to support more flexible extra options for tags.
This set of patches adds the extra options functionality and uses it to control set mock's package_manager option.
Each tag can now be assigned arbitrary extra options. These are stored as json in the database, so they can be of any type that json supports. The getBuildConfig call follows inheritance when determining these.
To tell mock to use dnf in a tag, you would run a command like:
# koji edit-tag TAG -x mock.package_manager=dnf
This setting is inherited.
Nice.
However when builder(s) are upgraded to F22 you need to add: config_opts['yum_command'] = '/usr/bin/yum-deprecated' too. Can this extra option can be utilized for this too? How you will know the version of builder OS? I assume that not all builders will be migrated to F22 at once.
We could use this mechanism for other build parameters, but it sounds like this one isn't quite so much per-tag as per-builder.
Is it safe to set config_opts['yum_command'] all the time (on F22+ biulders), or will it cause problems when package_manager is set to dnf?
The options I see are: - set config_opts['yum_command'] in site-defaults on F22+ builders - add some per-builder config in kojid to control this - have kojid auto-detect whether to set this - have mock auto-detect instead
I'm just raising issues, which caused me problems in Copr.
Mirek _______________________________________________ koji-devel mailing list koji-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/koji-devel
On Wednesday, June 17, 2015 06:10:47 PM Mike McLean wrote:
On 06/17/2015 05:01 PM, Miroslav Suchy wrote:
Dne 17.6.2015 v 21:33 Mike McLean napsal(a):
With mock switching to dnf, we needed to conditionally adjust mock's package_manager config on a per-tag basis. I didn't want to add yet another specialized field in the tag_config table, so I extended the schema to support more flexible extra options for tags.
This set of patches adds the extra options functionality and uses it to control set mock's package_manager option.
Each tag can now be assigned arbitrary extra options. These are stored as json in the database, so they can be of any type that json supports. The getBuildConfig call follows inheritance when determining these.
To tell mock to use dnf in a tag, you would run a command like:
# koji edit-tag TAG -x mock.package_manager=dnf
This setting is inherited.
Nice.
However when builder(s) are upgraded to F22 you need to add: config_opts['yum_command'] = '/usr/bin/yum-deprecated'
too. Can this extra option can be utilized for this too? How you will know the version of builder OS? I assume that not all builders will be migrated to F22 at once.
We could use this mechanism for other build parameters, but it sounds like this one isn't quite so much per-tag as per-builder.
Is it safe to set config_opts['yum_command'] all the time (on F22+ biulders), or will it cause problems when package_manager is set to dnf?
The options I see are:
- set config_opts['yum_command'] in site-defaults on F22+ builders
- add some per-builder config in kojid to control this
- have kojid auto-detect whether to set this
- have mock auto-detect instead
having mock auto-detect it has been what I have been asking for.
Dennis
I'm just raising issues, which caused me problems in Copr.
Mirek _______________________________________________ koji-devel mailing list koji-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/koji-devel
koji-devel mailing list koji-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/koji-devel
koji-devel@lists.fedorahosted.org
-
Dennis Gilmore -
Mike McLean -
Miroslav Suchý