koji-devel June 2015

koji-devel@lists.fedorahosted.org

10 participants
16 discussions

setting up a koji stage env
by Mike McLean 05 Jun '15

05 Jun '15

There are a number of ways one might use to set up a stage instance of koji that is based on a production instance. This is one that I have used on my koji systems. The two major bits of data that koji contains are: 1) the koji db 2) /mnt/koji With the db it is pretty easy to make a duplicate db based on a snapshot. The filesystem can be trickier. If you're lucky, your filesystem may support writable snapshots, in which case, awesome. Otherwise, you might like this approach. Koji has supported the notion of split storage since 1.7.0. There's a brief summary of how that works here: https://lists.fedoraproject.org/pipermail/buildsys/2012-May/003892.html In this staging approach, we mount the production volume READ-ONLY on the staging systems, and make /mnt/koji/vol/prod on the _stage_ fs point to that RO mount. When we restore the database from the production snapshot, we update the builds to point to the prod volume (seen the attached example script). The stage koji instance will then know where to find them. Because the builds have changed locations, the old repodata becomes invalid, so all the repos need to be regenerated. The attached script just marks them expired (which should trigger the kojira running in stage to regen them). The attached example sql script is based on one that I have used before. It will need to be adapted to your particular case.

1 0

Any chance for koji 1.10?
by Pat Riehecky 04 Jun '15

04 Jun '15

I realize we are all excited about koji 2.0, but there are a few minor fixes in the current koji tree I'd love to see on my build servers. Any chance for a koji 1.10 release? Pat -- Pat Riehecky Scientific Linux developer Fermi National Accelerator Laboratory www.fnal.gov www.scientificlinux.org

2 2

[PATCH] fix more #pylint issues
by Mike McLean 04 Jun '15

04 Jun '15

--- hub/kojixmlrpc.py | 2 +- koji/__init__.py | 4 ++-- koji/daemon.py | 2 +- koji/server.py | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hub/kojixmlrpc.py b/hub/kojixmlrpc.py index 0e51012..efb99a6 100644 --- a/hub/kojixmlrpc.py +++ b/hub/kojixmlrpc.py @@ -192,7 +192,7 @@ class HandlerAccess(object): return self.__reg.get(__name)(*args, **kwargs) def get(self, name): - return self.__Reg.get(name) + return self.__reg.get(name) class ModXMLRPCRequestHandler(object): diff --git a/koji/__init__.py b/koji/__init__.py index c3a0990..e38a208 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -2037,10 +2037,10 @@ class ClientSession(object): chk_opts['verify'] = 'adler32' result = self._callMethod('checkUpload', (path, name), chk_opts) if int(result['size']) != ofs: - raise koji.GenericError, "Uploaded file is wrong length: %s/%s, %s != %s" \ + raise GenericError, "Uploaded file is wrong length: %s/%s, %s != %s" \ % (path, name, result['sumlength'], ofs) if problems and result['hexdigest'] != full_chksum.hexdigest(): - raise koji.GenericError, "Uploaded file has wrong checksum: %s/%s, %s != %s" \ + raise GenericError, "Uploaded file has wrong checksum: %s/%s, %s != %s" \ % (path, name, result['hexdigest'], full_chksum.hexdigest()) self.logger.debug("Fast upload: %s complete. %i bytes in %.1f seconds", localfile, size, t2) diff --git a/koji/daemon.py b/koji/daemon.py index d76355b..0570faa 100644 --- a/koji/daemon.py +++ b/koji/daemon.py @@ -266,7 +266,7 @@ class SCM(object): # check for validity: params should be empty, query may be empty, everything else should be populated if params : - raise koji.GenericError, 'Unable to parse SCM URL: %s . Param element %s should be empty.' % (self.url,param) + raise koji.GenericError, 'Unable to parse SCM URL: %s . Params element %s should be empty.' % (self.url, params) if not scheme : raise koji.GenericError, 'Unable to parse SCM URL: %s . Could not find the scheme element.' % self.url if not netloc : diff --git a/koji/server.py b/koji/server.py index 7cc8be8..52f13f5 100644 --- a/koji/server.py +++ b/koji/server.py @@ -158,7 +158,7 @@ class WSGIWrapper(object): for chunk in result: if chunk and not self.set_headers: raise RuntimeError, "write() called before start_response()" - write(data) + write(chunk) if not req.bytes_sent: #application sent nothing back req.set_content_length(0) -- 1.9.3

1 0

[PATCH] handle symlinks in BuildMavenTask._zip_dir()
by Mike McLean 04 Jun '15

04 Jun '15

--- builder/kojid | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/builder/kojid b/builder/kojid index dcd0840..ad2cf1f 100755 --- a/builder/kojid +++ b/builder/kojid @@ -1281,7 +1281,17 @@ class BuildMavenTask(BaseBuildTask): dirnames.remove(skip) for filename in filenames: filepath = os.path.join(dirpath, filename) - zfo.write(filepath, filepath[roottrim:]) + if os.path.islink(filepath): + content = os.readlink(filepath) + st = os.lstat(filepath) + mtime = time.localtime(st.st_mtime) + info = zipfile.ZipInfo(filepath[roottrim:]) + info.external_attr |= 0120000 << 16L # symlink file type + info.compress_type = zipfile.ZIP_STORED + info.date_time = mtime[:6] + zfo.writestr(info, content) + else: + zfo.write(filepath, filepath[roottrim:]) zfo.close() def checkHost(self, hostdata): -- 1.9.3

1 0

[PATCH] Avoid errors logging anonymous retries (ticket 317)
by Mike McLean 04 Jun '15

04 Jun '15

see: https://fedorahosted.org/koji/ticket/317 --- koji/__init__.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/koji/__init__.py b/koji/__init__.py index 4ca9c08..36772bb 100644 --- a/koji/__init__.py +++ b/koji/__init__.py @@ -1946,7 +1946,7 @@ class ClientSession(object): if self.logger.isEnabledFor(logging.DEBUG): tb_str = ''.join(traceback.format_exception(*sys.exc_info())) self.logger.debug(tb_str) - self.logger.info("Try #%d for call %s (%s) failed: %s", tries, self.callnum, name, e) + self.logger.info("Try #%s for call %s (%s) failed: %s", tries, self.callnum, name, e) if tries > 1: # first retry is immediate, after that we honor retry_interval time.sleep(interval) -- 1.9.3

1 0

Koji 2.0 planning
by Mike McLean 02 Jun '15

02 Jun '15

It's been eight years since koji 1.0. In that time Koji has grown a lot, but always incrementally and with great care to avoid breaking compatibility. Over the years, we've found numerous things that we wanted to add or change, but that we dismissed as too big, too complicated, or too invasive. Bumping the major release number gives use the freedom to shake things up a bit. Koji 2.0 is about making major changes, otherwise it would just be koji 1.13. Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is sufficiently stable. Some features (e.g. content generators) will also appear in 1.x. The list below is probably not complete, it is ambitious, and it is certainly open to discussion. If you are a Koji user, or otherwise invested in Koji, then you are encouraged to join in. I expect there will be a lot to say, so I've created a new mailing list devoted specifically to Koji development. https://lists.fedorahosted.org/mailman/listinfo/koji-devel Also, apologies for the tense summary listing. I can certain expound on any of these as necessary. I hope this suffices to start some conversation. = High level goals = • better documentation • more community involvement • refactor/modernize code • more modular design ∘ content generators ∘ broader, better plugin framework • better support for different types of build processes • better support for for different types build output • make hard-wired restrictions more configurable • easier to deploy • better qa process • better release process = Highlights/Major changes = • python3 support ∘ the bulk of the code will target python 2.6 + python-six ∘ we'll create a basic client lib for older systems (e.g rhel5 clients/builders) • drop xmlrpc in favor a json based rpc • build namespaces ∘ allow for use cases that require multiple builds of the same NVR (or NVRA) • refactor task scheduling • extend content generator support ∘ content generators will land in 1.x fairly soon, but in 2.0 they will be more integral ∘ refactor kojid to use content generator calls ∘ (possibly) tighter integration in the db • unify handling of rpms with other build types in the db ∘ e.g. unify rpminfo and archiveinfo tables • support different ways of building (possibly via content generators) • utilize jsonb fields in postgres • modular auth ∘ make it easier to add new auth mechanisms ∘ support openid auth • improve plugins ∘ make the plugin framework cleaner and more pythonic ∘ support plugins in cli and web ui • improve/update web ui ∘ drop cheetah templates in favor of python-jinja2 ∘ more parity with cli ∘ history browsing ∘ landing page search or history ∘ support plugins • change how tag pkglists (and blocking) work • refactor package ownership • refactor uploads • more flexible gc = Yet more changes = • store all task requests by named args ∘ (for ease of inspection) • get rid of tagBuild tasks • drop odd event refererences in favor of timestamps • streamlined cli options • marker files for many things on disk • more history data • drop modpython support • policy code ∘ more robust syntax ‣ test negation ‣ OR ‣ parentheses ‣ quoted strings ∘ multiple result policies (non-terminal actions) ∘ all-matches policy (needed for scheduler?) ∘ break action (breaks out of nesting) ∘ stop action (halts processing of an all-matches policy)

6 9

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

koji-devel June 2015