dist-git build problem on epel-6-x86_64 chroot
by Jean-Marc Liger
Hy,
I'm facing the problem below on epel-6-x86_64 chroot for both from URL
and upload SRPM :
[2016-01-19 15:16:26,790][ INFO][PID:3241] Setting up builder: 172.25.94.167
[2016-01-19 15:16:27,991][ INFO][PID:3241] marking build dir with build_id,
[2016-01-19 15:16:27,992][ INFO][PID:3241] Start build: BuildJob<id: 154509, owner: jmliger, project: virt6-upstream, git branch: el6, git_hash: acef5efd6153aa8f9ea760f468333e65c9b46a4e, status: 3 >
[2016-01-19 15:16:27,993][ INFO][PID:3241] putting into minimal buildroot of epel-6-x86_64
[2016-01-19 15:16:28,902][ INFO][PID:3241] Cloning Dist Git repo jmliger/virt6-upstream/libvirt, branch acef5efd6153aa8f9ea760f468333e65c9b46a4e, hash el6
[2016-01-19 15:16:30,906][ ERROR][PID:3241] Failed to obtain srpm from dist-git
Traceback (most recent call last):
File "/usr/share/copr/backend/mockremote/builder.py", line 209, in download_job_pkg_to_builder
self.remote_pkg_path = list(results["contacted"].values())[0][u"stdout"].split("Wrote: ")[1]
IndexError: list index out of range
[2016-01-19 15:16:30,911][ ERROR][PID:3241] builder.build error building pkg `libvirt`: BuildError: Failed to obtain srpm from dist-git: ansible results {'dark': {}, 'contacted': {'172.25.94.167': {'cmd': 'rm -rf /tmp/build_package_repo && mkdir /tmp/build_package_repo && cd /tmp/build_package_repo && git clone http://copr-dist-git.fedorainfracloud.org/git/jmliger/virt6-upstream/libv... && cd libvirt && git checkout acef5efd6153aa8f9ea760f468333e65c9b46a4e && fedpkg-copr --dist el6 srpm', 'end': '2016-01-19 15:16:30.308282', 'stdout': 'Downloading libvirt-1.3.1.tar.gz\n\r 0.0%\r 100.0%\r######################################################################## 100.0%', 'changed': True, 'start': '2016-01-19 15:16:28.758321', 'delta': '0:00:01.549961', 'stderr': "Cloning into 'libvirt'...\nNote: checking out 'acef5efd6153aa8f9ea760f468333e65c9b46a4e'.\n\nYou are in 'detached HEAD' state. You can look around, make experimental\nchanges and commit them, and you can discard any commits you make in this\nstate without impacting any branches by performing another checkout.\n\nIf you want to create a new branch to retain commits you create, you may\ndo so (now or later) by using -b with the checkout command again. Example:\n\n git checkout -b <new-branch-name>\n\nHEAD is now at acef5ef... import_srpm\nerror: line 1800: Trigger fired by the same package is already defined in spec file: %post daemon-config-network\n\nerror: query of specfile /tmp/build_package_repo/libvirt/libvirt.spec failed, can't parse\n\nCould not execute srpm: Could not get n-v-r-e from '\\n\\n'", 'rc': 1, 'invocation': {'module_name': 'shell', 'module_complex_args': {}, 'module_args': u'rm -rf /tmp/build_package_repo && mkdir /tmp/build_package_repo && cd /tmp/build_package_repo && git clone http://copr-dist-git.fedorainfracloud.org/git/jmliger/virt6-upstream/libv... && cd libvirt && git checkout acef5efd6153aa8f9ea760f468333e65c9b46a4e && fedpkg-copr --dist el6 srpm'}, 'warnings': ['Consider using file module with state=absent rather than running rm']}}}
[2016-01-19 15:16:31,759][ INFO][PID:3241] End Build: BuildJob<id: 154509, owner: jmliger, project: virt6-upstream, git branch: el6, git_hash: acef5efd6153aa8f9ea760f468333e65c9b46a4e, status: 3 >
https://copr.fedoraproject.org/coprs/jmliger/virt6-upstream/builds/
Regards,
Jean-Marc LIGER
8 years
ppc64le strange build errors, needs 1947 inodes on the / filesystem
!?
by Sérgio Basto
Hi,
I tried 4 days ago and after I decide to wait for "Planned Outage: Copr
upgrade", today still have the same errors.
https://copr-be.cloud.fedoraproject.org/results/sergiomb/kde4for23/fedo
ra-22-ppc64le/00154683-kde-baseapps/root.log.gz
DEBUG util.py:377: Transaction check error:
DEBUG util.py:377: installing package kdelibs-devel-6:4.14.14-1.fc22.ppc64le needs 1947 inodes on the / filesystem
DEBUG util.py:377: installing package xapian-core-libs-1.2.21-1.fc22.ppc64le needs 1956 inodes on the / filesystem
DEBUG util.py:377: installing package libical-1.0.1-1.fc22.ppc64le needs 1968 inodes on the / filesystem
DEBUG util.py:377: installing package kdepimlibs-4.14.10-3.fc22.ppc64le needs 2236 inodes on the / filesystem
DEBUG util.py:377: installing package kdepimlibs-akonadi-4.14.10-3.fc22.ppc64le needs 2306 inodes on the / filesystem
DEBUG util.py:377: installing package baloo-libs-4.14.3-1.fc22.ppc64le needs 2335 inodes on the / filesystem
DEBUG util.py:377: installing package baloo-widgets-4.14.3-1.fc22.ppc64le needs 2340 inodes on the / filesystem
DEBUG util.py:377: installing package baloo-devel-4.14.3-1.fc22.ppc64le needs 2379 inodes on the / filesystem
DEBUG util.py:377: installing package baloo-widgets-devel-4.14.3-1.fc22.ppc64le needs 2391 inodes on the / filesystem
DEBUG util.py:377: installing package kactivities-devel-4.13.3-9.fc22.ppc64le needs 2416 inodes on the / filesystem
DEBUG util.py:377: installing package kfilemetadata-devel-4.14.3-1.fc22.ppc64le needs 2432 inodes on the / filesystem
DEBUG util.py:377: installing package libappstream-glib-0.4.1-1.fc22.ppc64le needs 2448 inodes on the / filesystem
DEBUG util.py:377: installing package glib2-devel-2.44.1-2.fc22.ppc64le needs 2811 inodes on the / filesystem
DEBUG util.py:377: installing package libtidy-devel-0.99.0-33.20091203.fc22.ppc64le needs 2925 inodes on the / filesystem
DEBUG util.py:377: Error Summary
Thanks,
--
Sérgio M. B.
8 years, 1 month
Unable to connect with copr-cli
by Pavel Raiskup
FYI: Seems like there are some issues on frontend side ATM:
09:32:04 ~/rh/projects/distgen/rpm$ copr --debug list
[09:32:12] {/usr/lib/python2.7/site-packages/copr_cli/main.py:416} DEBUG - # Debug log enabled #
[09:32:12] {/usr/lib/python2.7/site-packages/copr/client/client.py:172} DEBUG - Fetching url: https://copr.fedoraproject.org/api/coprs/praiskup/, for login: dmeretcesbfrquybdyxb
[09:32:12] {/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:758} INFO - Starting new HTTPS connection (1): copr.fedoraproject.org
[..hangs here..]
Pavel
8 years, 1 month
Removal of F21 chroots
by Miroslav Suchý
Fedora 21 is EOLed for one month. I plan to remove from Copr in 14 days too.
All builds will be preserved. You will not be able to submit new builds for F21.
Please update your scripts accordingly.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years, 1 month
copr-keygen upgrade post-mortem
by Miroslav Suchý
Hi,
I have planned outage for copr-keygen on Tuesday. It did not went well, so here is post-mortem for those interested in.
Copr-keygen machine was Fedora 21 so we wanted to upgrade it to some supported version of Fedora.
I did the upgrade of dev machine and it went well so I moved to production machine.
Well, after upgrade of production machine to Fedora 23 it did not worked. I was getting error:
files-are-digests doesn't work with v4 sigs
I compared the dev and producton machines, but they were identical.
I even tried downgrade to Fedora 22 (which is still supported), but it did not worked too. So I had to dive into source
code. Code of copr-keygen, obs-signd and finally gnupg2 where I find that option --force-v3-sigs is (since gnupg 2.1)
silently ignored.
I took gnupg2 from Fedora21 (latest with 2.0.x version of gnupg), rebuilt it for Fedora 22 and tried. Fortunately it worked.
I put gnupg2 to protected packages on copr-keygen so the situation is stabilized for now.
I notified obs guys about this situation, but I'm afraid that they still use gnupg 2.0.x so we are first who hit this
problem.
In the mean time if you are running your own instance of Copr (I'm looking at you Pavel) be careful when upgrading
keygen machine.
We need to solve it somehow in the near future. The options are backport --force-v3-sigs into gnupg2 (unlikely) or add
support for v4 into obs-sign.
BTW why it worked on dev machine? I'm still not 100% sure, but I suspect the data. Dev machine is always completly
wiped, including old keys. While old keys are preserved on production machine.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years, 1 month
[PATCH 1/2] [cli] add --config option
by Pavel Raiskup
---
cli/copr | 25 +++++++++++++++++++++++++
cli/copr_cli/main.py | 9 ++++++---
2 files changed, 31 insertions(+), 3 deletions(-)
create mode 100755 cli/copr
diff --git a/cli/copr b/cli/copr
new file mode 100755
index 0000000..a119b23
--- /dev/null
+++ b/cli/copr
@@ -0,0 +1,25 @@
+#! /bin/sh
+
+# Run copr-cli script directly from git.
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+absdir="$(dirname "$(readlink -f "$0")")"
+cd "$absdir"
+
+export PYTHONPATH="$absdir/../python"
+
+python -m copr_cli.main "$@"
diff --git a/cli/copr_cli/main.py b/cli/copr_cli/main.py
index 8d55466..6dc8b9f 100644
--- a/cli/copr_cli/main.py
+++ b/cli/copr_cli/main.py
@@ -40,10 +40,10 @@ no_config_warning = """
class Commands(object):
- def __init__(self):
+ def __init__(self, config):
try:
- self.client = CoprClient.create_from_file_config()
+ self.client = CoprClient.create_from_file_config(config)
except (copr_exceptions.CoprNoConfException,
copr_exceptions.CoprConfigException):
print(no_config_warning)
@@ -295,6 +295,9 @@ def setup_parser():
parser.add_argument("--debug", dest="debug", action="store_true",
help="Enable debug output")
+ parser.add_argument("--config", dest="config",
+ help="Path to an alternative configuration file")
+
subparsers = parser.add_subparsers(title="actions")
# create the parser for the "list" command
@@ -422,7 +425,7 @@ def main(argv=sys.argv[1:]):
if arg.debug:
enable_debug()
- commands = Commands()
+ commands = Commands(arg.config)
getattr(commands, arg.func)(arg)
except KeyboardInterrupt:
--
2.5.0
8 years, 2 months
[PATCH] [backend] abstraction above [BE <-> JG <-> Builders] channels
by Pavel Raiskup
Previously, if backend started a bit faster than job-grabber, it
was fine, but the other way around (because job-grabber is
prerequisite for backend in service file):
1. Job Grabber started
2. JobGrabber took the list of tasks
3. JobGrabber filled the redis task queue && and filled its
internal representation of queue
4. Backend started, which resulted in redis task queue cleanup
5. JobGrabber was confused because the filled redis queue did
not match its internal queue representation state
This resulted in unprocessed queue.
Now backend precisely tells job_grabber via "different" (Queue)
channel about backend re-start (so backend does not have to touch
the Queue itself and the inconsistency does not happen).
---
backend/backend/daemons/backend.py | 32 ++----------
backend/backend/daemons/dispatcher.py | 17 +++---
backend/backend/daemons/job_grab.py | 97 ++++++++++++++++++++++-------------
backend/backend/helpers.py | 13 +++++
backend/backend/jobgrabcontrol.py | 80 +++++++++++++++++++++++++++++
5 files changed, 166 insertions(+), 73 deletions(-)
create mode 100644 backend/backend/jobgrabcontrol.py
diff --git a/backend/backend/daemons/backend.py b/backend/backend/daemons/backend.py
index 934a566..8b54bf1 100644
--- a/backend/backend/daemons/backend.py
+++ b/backend/backend/daemons/backend.py
@@ -15,13 +15,13 @@ from collections import defaultdict
import lockfile
from daemon import DaemonContext
from requests import RequestException
-from retask.queue import Queue
from retask import ConnectionError
from backend.frontend import FrontendClient
from ..exceptions import CoprBackendError
from ..helpers import BackendConfigReader, get_redis_logger
from .dispatcher import Worker
+from .. import jobgrabcontrol
class CoprBackend(object):
@@ -42,7 +42,7 @@ class CoprBackend(object):
raise CoprBackendError("Must specify config_file")
self.config_file = config_file
- self.ext_opts = ext_opts # to stow our cli options for read_conf()
+ self.ext_opts = ext_opts # to show our cli options for read_conf()
self.workers_by_group_id = defaultdict(list)
self.max_worker_num_by_group_id = defaultdict(int)
@@ -55,35 +55,14 @@ class CoprBackend(object):
self.log = get_redis_logger(self.opts, "backend.main", "backend")
self.frontend_client = FrontendClient(self.opts, self.log)
+ self.jg_control = jobgrabcontrol.Channel(self.opts, self.log)
self.is_running = False
- def clean_task_queues(self):
- """
- Make sure there is nothing in our task queues
- """
- try:
- for queue in self.task_queues.values():
- while queue.length:
- queue.dequeue()
- except ConnectionError:
- raise CoprBackendError(
- "Could not connect to a task queue. Is Redis running?")
-
def init_task_queues(self):
"""
- Connect to the retask.Queue for each group_id. Remove old tasks from queues.
+ Remove old tasks from queues.
"""
- try:
- for group in self.opts.build_groups:
- group_id = group["id"]
- queue = Queue("copr-be-{0}".format(group_id))
- queue.connect()
- self.task_queues[group_id] = queue
- except ConnectionError:
- raise CoprBackendError(
- "Could not connect to a task queue. Is Redis running?")
-
- self.clean_task_queues()
+ self.jg_control.backend_start()
def update_conf(self):
"""
@@ -158,7 +137,6 @@ class CoprBackend(object):
for w in self.workers_by_group_id[group_id][:]:
self.workers_by_group_id[group_id].remove(w)
w.terminate_instance()
- self.clean_task_queues()
try:
self.log.info("Rescheduling unfinished builds before stop")
diff --git a/backend/backend/daemons/dispatcher.py b/backend/backend/daemons/dispatcher.py
index 10375ba..ae3f460 100644
--- a/backend/backend/daemons/dispatcher.py
+++ b/backend/backend/daemons/dispatcher.py
@@ -7,8 +7,6 @@ import shutil
import multiprocessing
from setproctitle import setproctitle
-from retask.queue import Queue
-
from ..vm_manage.manager import VmManager
from ..exceptions import MockRemoteError, CoprWorkerError, VmError, NoVmAvailable
from ..job import BuildJob
@@ -16,6 +14,7 @@ from ..mockremote import MockRemote
from ..constants import BuildStatus, JOB_GRAB_TASK_END_PUBSUB, build_log_format
from ..helpers import register_build_result, get_redis_connection, get_redis_logger, \
local_file_logger
+from .. import jobgrabcontrol
# ansible_playbook = "ansible-playbook"
@@ -32,8 +31,6 @@ class Worker(multiprocessing.Process):
Worker process dispatches building tasks. Backend spin-up multiple workers, each
worker associated to one group_id and process one task at the each moment.
- Worker listens for the new tasks from :py:class:`retask.Queue` associated with its group_id
-
:param Munch opts: backend config
:param int worker_num: worker number
:param int group_id: group_id from the set of groups defined in config
@@ -51,9 +48,7 @@ class Worker(multiprocessing.Process):
self.log = get_redis_logger(self.opts, self.logger_name, "worker")
- # job management stuff
- self.task_queue = Queue("copr-be-{0}".format(str(group_id)))
- self.task_queue.connect()
+ self.jg = jobgrabcontrol.Channel(self.opts, self.log)
# event queue for communicating back to dispatcher
self.kill_received = False
@@ -238,14 +233,16 @@ class Worker(multiprocessing.Process):
# this sometimes caused TypeError in random worker
# when another one picekd up a task to build
# why?
+ # praiskup: not reproduced
try:
- task = self.task_queue.dequeue()
- except TypeError:
+ task = self.jg.get_build(self.group_id)
+ except TypeError as err:
+ self.log.warning(err)
return
if not task:
return
- job = BuildJob(task.data, self.opts)
+ job = BuildJob(task, self.opts)
self.update_process_title(suffix="Task: {} chroot: {}, obtained at {}"
.format(job.build_id, job.chroot, str(datetime.now())))
diff --git a/backend/backend/daemons/job_grab.py b/backend/backend/daemons/job_grab.py
index 21871ae..7bca75b 100644
--- a/backend/backend/daemons/job_grab.py
+++ b/backend/backend/daemons/job_grab.py
@@ -10,8 +10,6 @@ import time
from setproctitle import setproctitle
from requests import get, RequestException
-from retask.task import Task
-from retask.queue import Queue
from backend.frontend import FrontendClient
@@ -19,11 +17,22 @@ from ..actions import Action
from ..constants import JOB_GRAB_TASK_END_PUBSUB
from ..helpers import get_redis_connection, get_redis_logger
from ..exceptions import CoprJobGrabError
-
+from .. import jobgrabcontrol
# TODO: Replace entire model with asynchronous queue, so that frontend push task,
# and workers listen for them
-
+# praiskup: Please don't. I doubt this would help too much, and I really don't
+# think it is worth another rewrite. Reasons (imho):
+# a. there still needs to be "one" organizator, aka jobgrabber on the backend
+# VM side -- we do not want allow Workers to contact frontend directly
+# because of (1) security and (2) process synchronization.
+# b. in frontend, we _never_ want to block UI differently than on database,
+# so the push to BE can't be done instantly -- and thus there would have
+# to be something like buffered "JobPusher" (and that would be most
+# probably implemented as poll anyway). Maybe we could use some "pipe"
+# approach through infinite (http?) connection, or opened database
+# connection, .. but I don't think it does matter too much who will
+# control the "pipe".
class CoprJobGrab(object):
"""
@@ -36,41 +45,33 @@ class CoprJobGrab(object):
:param Munch opts: backend config
:param lock: :py:class:`multiprocessing.Lock` global backend lock
+ TODO: Not yet fully ready for config reload.
"""
def __init__(self, opts):
""" base class initialization """
self.opts = opts
+
+ # Maps e.g. x86_64 && i386 => PC (.
self.arch_to_group_id_map = dict()
- for group in self.opts.build_groups:
- for arch in group["archs"]:
- self.arch_to_group_id_map[arch] = group["id"]
-
- self.task_queues_by_arch = {}
- self.task_queues_by_group = {}
-
- self.added_jobs_dict = dict() # task_id -> task dict
-
+ # PC => max N builders per user
+ self.group_to_usermax = dict()
+ # task_id -> task dict
+ self.added_jobs_dict = dict()
self.rc = None
self.channel = None
self.ps_thread = None
self.log = get_redis_logger(self.opts, "backend.job_grab", "job_grab")
+ self.jg_control = jobgrabcontrol.Channel(self.opts, self.log)
self.frontend_client = FrontendClient(self.opts, self.log)
- def connect_queues(self):
- """
- Connects to the retask queues. One queue per builders group.
- """
- for group in self.opts.build_groups:
- queue = Queue("copr-be-{0}".format(group["id"]))
- queue.connect()
- self.task_queues_by_group[group["name"]] = queue
- for arch in group["archs"]:
- self.task_queues_by_arch[arch] = queue
+ def group(self, arch):
+ return self.arch_to_group_id_map[arch]
+
def listen_to_pubsub(self):
"""
@@ -84,6 +85,7 @@ class CoprJobGrab(object):
self.log.info("Subscribed to {} channel".format(JOB_GRAB_TASK_END_PUBSUB))
+
def route_build_task(self, task):
"""
Route build task to the appropriate queue.
@@ -101,24 +103,23 @@ class CoprJobGrab(object):
if "task_id" in task:
if task["task_id"] not in self.added_jobs_dict:
arch = task["chroot"].split("-")[2]
- if arch not in self.task_queues_by_arch:
- raise CoprJobGrabError("No builder group for architecture: {}, task: {}"
- .format(arch, task))
+ group = self.group(arch)
username = task["project_owner"]
- group_id = int(self.arch_to_group_id_map[arch])
active_jobs_count = len([t for t_id, t in self.added_jobs_dict.items()
if t["project_owner"] == username])
- if active_jobs_count > self.opts.build_groups[group_id]["max_vm_per_user"]:
- self.log.debug("User can not acquire more VM (active builds #{}), "
+ if active_jobs_count > self.group_to_usermax[group]:
+ self.log.debug("User can not acquire more VM (active builds #{0}), "
"don't schedule more tasks".format(active_jobs_count))
return 0
+ msg = "enqueue task for user {0}: id={1}, arch={2}, group={3}, active={4}"
+ self.log.debug(msg.format(username, task["task_id"], arch, group, active_jobs_count))
+
+ # Add both to local list and control channel queue.
self.added_jobs_dict[task["task_id"]] = task
-
- task_obj = Task(task)
- self.task_queues_by_arch[arch].enqueue(task_obj)
+ self.jg_control.add_build(group, task)
count += 1
else:
@@ -226,22 +227,46 @@ class CoprJobGrab(object):
self.log.debug("Added jobs after remove and load: {}".format(self.added_jobs_dict))
self.log.debug("# of executed jobs: {}".format(len(self.added_jobs_dict)))
- for group, queue in self.task_queues_by_group.items():
- if queue.length > 0:
- self.log.debug("# of pending jobs for `{}`: {}".format(group, queue.length))
+
+ def init_internal_structures(self):
+ self.arch_to_group_id_map = dict()
+ self.group_to_usermax = dict()
+ for group in self.opts.build_groups:
+ group_id = group["id"]
+ for arch in group["archs"]:
+ self.arch_to_group_id_map[arch] = group_id
+ self.log.debug("mapping {0} to {1} group".format(arch, group_id))
+
+ self.log.debug("user might use only {0}VMs for {1} group".format(group["max_vm_per_user"], group_id))
+ self.group_to_usermax[group_id] = group["max_vm_per_user"]
+
+ self.added_jobs_dict = dict()
+
+
+ def handle_control_channel(self):
+ if not self.jg_control.backend_started():
+ return
+ self.log.info("backend gave us signal to start")
+ self.init_internal_structures()
+ self.jg_control.remove_all_builds()
+ self.jg_control.job_graber_initialized()
def run(self):
"""
Starts job grabber process
"""
setproctitle("CoprJobGrab")
- self.connect_queues()
self.listen_to_pubsub()
self.log.info("JobGrub started.")
+
+ self.init_internal_structures()
try:
while True:
try:
+ # This effectively delays job_grabbing until backend
+ # gives as signal to start.
+ self.handle_control_channel()
self.load_tasks()
self.log_queue_info()
time.sleep(self.opts.sleeptime)
diff --git a/backend/backend/helpers.py b/backend/backend/helpers.py
index 3e9c028..fa1a3e5 100644
--- a/backend/backend/helpers.py
+++ b/backend/backend/helpers.py
@@ -11,6 +11,7 @@ import ConfigParser
import os
import sys
import errno
+import time
from contextlib import contextmanager
import traceback
@@ -28,6 +29,18 @@ from backend.constants import DEF_BUILD_USER, DEF_BUILD_TIMEOUT, DEF_CONSECUTIVE
CONSECUTIVE_FAILURE_REDIS_KEY, default_log_format
from backend.exceptions import CoprBackendError
+
+def wait_log(log, reason="I don't know why.", timeout=5):
+ """
+ We need to wait a while, this should happen only when copr converges to
+ boot-up/restart/..
+ """
+ if not log:
+ return
+ log.warning("I'm waiting {0}s because: {1}".format(timeout, reason))
+ time.sleep(timeout)
+
+
class SortedOptParser(optparse.OptionParser):
"""Optparser which sorts the options by opt before outputting --help"""
diff --git a/backend/backend/jobgrabcontrol.py b/backend/backend/jobgrabcontrol.py
new file mode 100644
index 0000000..c8d7414
--- /dev/null
+++ b/backend/backend/jobgrabcontrol.py
@@ -0,0 +1,80 @@
+from retask.queue import Queue
+from retask.task import Task
+
+from .helpers import wait_log
+
+class Channel(object):
+ """
+ Abstraction above retask (the set of "channels" between backend(s),
+ jobgrabber and workers). We could use multiple backends and/or diffferent
+ "atomic" medium (other implemntation than Queue) in future. But
+ make sure nobody needs to touch the "medium" directly.
+ """
+
+ def __init__(self, opts, log=None):
+ self.log = log
+ self.opts = opts
+ # channel for Backend <--> JobGrabber communication
+ self.jg_start = Queue("jg_control_start")
+ # channel for JobGrabber <--> [[Builders]] communication
+ self.build_queues = dict()
+ while not self.jg_start.connect():
+ wait_log("waiting for redis", 5)
+
+ def _get_queue(self, bgroup):
+ if not bgroup in self.build_queues:
+ q_id = "copr-be-{0}".format(bgroup)
+ q = Queue(q_id)
+ if not q.connect():
+ # As we already connected to jg_control_message, this should
+ # be also OK.
+ raise Exception("can't connect to redis, should never happen!")
+ return q
+
+ return self.build_queues[bgroup]
+
+ def add_build(self, bgroup, build):
+ """ this should be used by job_grab only for now """
+ q = self._get_queue(bgroup)
+ try:
+ q.enqueue(Task(build))
+ except Exception as err:
+ # I've seen isses Task() was not able to jsonify urllib exceptions
+ if not self.log:
+ return False
+ self.log.error("can't enqueue build {0}, reason:\n{1}".format(
+ build, err
+ ))
+
+ return True
+
+ # Builder's API
+ def get_build(self, bgroup):
+ """
+ Return task from queue or return 0
+ """
+ q = self._get_queue(bgroup)
+ t = q.dequeue()
+ return t.data if t else None
+
+ # JobGrab's API
+ def backend_started(self):
+ return self.jg_start.length
+
+ def job_graber_initialized(self):
+ while self.jg_start.dequeue():
+ pass
+
+ def remove_all_builds(self):
+ for bgroup in self.build_queues:
+ q = self._get_queue(bgroup)
+ while q.dequeue():
+ pass
+ self.build_queues = dict()
+
+ # Backend's API
+ def backend_start(self):
+ """ Notify jobgrab about service start. """
+ self.jg_start.enqueue("start")
+ while self.jg_start.length:
+ wait_log(self.log, "waiting until jobgrabber initializes queue")
--
2.5.0
8 years, 2 months