Beaker-devel January 2013

beaker-devel@lists.fedorahosted.org

5 participants
4 discussions

Defining a harness API
by Dan Callaghan 07 Feb '13

07 Feb '13

An important first step towards supporting alternative harnesses and/or the mythical Beaker Simple Harness is coming up with a stable, documented API for the harness to interact with Beaker. So I'm starting this thread now to get the ball rolling. My first thoughts: * It should have the smallest possible surface area -- just enough to expose all of Beaker's functionality and nothing more. * It should be defined from the point of view of lab controller <-> test system, not scheduler <-> test system. Corollary: we might need to start treating the lab controller as a first-class citizen instead of just a dumb proxy to the scheduler. * Just because we use XML-RPC now doesn't mean it's the best choice, and doesn't mean we need to keep using it. * In particular, the HTTP protocol probably supports everything we need to build the API (in other words, it can be a "RESTful API" even though I hate that phrase). For example logs could be uploaded using HTTP PUT (with Content-Range), which means if we wanted we could potentially use Apache mod_dav_fs to efficiently write these directly to the filesystem without any intervening Beaker code. There are four areas the API needs to cover (that I can think of). The harness needs to be able to: * find out what to run * report results and upload logs * extend the watchdog time * synchronize with other recipes in the recipe set This last area is a new one, it's currently handled entirely in beah and there is no API on the Beaker side for it. But the dynamic FQDNs in Beaker 0.10 mean it is now needed, even if it's as simple as having a call to wait for all other harnesses in the recipe set to check in. Thoughts? -- Dan Callaghan <dcallagh(a)redhat.com> Software Engineer, Infrastructure Engineering and Development Red Hat, Inc.

5 10

A more transparent design process for Beaker
by Nick Coghlan 28 Jan '13

28 Jan '13

One thing I noticed when starting work on Beaker is that a lot of the design rationale for major architectural decisions is hidden in email archives (including Red Hat internal ones), Bugzilla entries and code reviews on Gerrit. To make it easier for new developers to understand past architectural changes/additions, to help users to understand changes coming in future releases, and to help us be confident that our proposed designs are solid, I'm suggesting we adopt a "Beaker Design Proposal" process for major changes (such as the scheduler updates I'm working on, the upcoming integrated dashboard, the "system pools" concept, self-service user groups and group jobs). I am unashamedly stealing this concept from Python's PEP process, because I like it and I think it works pretty well in ensuring that the *rationale* for a design is captured in addition to forcing us to think a bit more about the design itself before diving into code. As with Python, only major changes would need to be written up as a design proposal - bug fixes and smaller features would be handled through Bugzilla as usual. In recent times, the integration with oVirt Engine in 0.10 is probably the only change big enough to qualify (although the Supported Query idea might have been worth writing up as well). In terms of mechanics, I'd largely follow the Python model for numbering: reserve BDP 1 for the meta-proposal describing the process, BDP 2-99 for any other meta-proposals we come up with, and start the actual design proposals at BDP 100. Publication would be via Sphinx on beaker-project.org: the design proposals would just live in another docs directory alongside the existing User and Admin guides. BDP approval rights would rest with the Beaker Development Lead (i.e. currently me) or their delegate (similar to the way PEP approval rights ultimately rest with Guido van Rossum as Python's BDFL, but one of the other senior core developers may take on the role of BDFL-delegate for a particular PEP). Regards, Nick. -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane Python Applications Team Lead Beaker Development Lead (http://beaker-project.org/) PulpDist Development Lead (http://pulpdist.readthedocs.org)

2 2

Updating the home page blurb for Beaker
by Nick Coghlan 16 Jan '13

16 Jan '13

It has been pointed out [1] that the current home page blurb doesn't do a great job of describing Beaker's purpose and capabilities. While I think Ed's suggested wording is an improvement over the status quo, I have a few additional proposed changes of my own. Rather than trying to work out the details in patch comments, I figure it makes more sense to get rough consensus on new wording here, then we can update the home page and any affected project descriptions in the documentation accordingly. Here's my current draft ================================ Beaker is a tool to provide and manage globally distributed testing infrastructure shared by multiple development and quality assurance teams. With Beaker, users and administrators are able to: - Manage systems in a globally distributed network - Maintain an automated inventory of system hardware details - Provision execution environments on systems - Schedule tasks to run on one or more systems - Store and view task results Beaker enables users to create any task they would like (from testing changes to kernel memory management to installing their favourite OS and game server) and run these tasks on any number of machines of any specification located anywhere in the world, and provides a central interface where the outcome of these tasks can be reviewed. Beaker is currently based on: - Anaconda kickstarts for bare metal system provisioning - KVM/virsh for guest system provisioning - oVirt for dynamic virtual system provisioning - RPM for task definition and distribution - the Beah framework for task execution Developed by `Red Hat <http://www.redhat.com>`__, Beaker can be used as a searchable machine inventory, system provisioning engine, task scheduler, test automation framework and hardware lab management tool. ================================ Once we have a more accurate description of the status quo, then I believe it becomes easier to create a description of "things we're interested in doing differently" (such as supporting execution frameworks other than Beah, task distribution with Git, image based provising of guest and dynamic virt systems, etc). Regards, Nick. [1] http://gerrit.beaker-project.org/#/c/1257 -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane Python Applications Team Lead Beaker Development Lead (http://beaker-project.org/) GlobalSync Development Lead (http://pulpdist.readthedocs.org)

5 14

Scheduling recipe sets rather than recipes
by Nick Coghlan 04 Jan '13

04 Jan '13

The current scheduler works almost purely at the recipe level. The extent to which it pays attention to recipesets pretty much amounts to ensuring all recipes in a recipe set are scheduled on the same lab controller. This creates some interesting problems with multi-host testing: - a recipe set with strict host requirements for only some systems may hold on to common systems for a long time while waiting for rare ones (the addition of dynamic virt support opens the door for a recipe set to hold on to dynamic virt resources while waiting for physical hardware for other recipes) - recipe sets scheduled for unique systems may deadlock if a high priority job is competing with a previously queued low priority job which has already claimed some resources To better explain the latter problem, consider a lab with only 2 systems, A and B, containing a particular piece of hardware, and a multi-host recipe set that needs both of them. Queue a low priority version of that job while a test is running on system A, and the job will claim system B immediately for one recipe, while the other will remain in the queue. If a high priority copy of the job is added before the test running on system A completes, then system A will be claimed by Job 2. This leaves the two jobs in a classic ABBA deadlock, as Job 1 has System B and is waiting for System A, while Job 2 has System A and is waiting for System B. Some of the metrics support being added in 0.11 is actually about measuring the overall impact of the first problem (by seeing what proportion of their time systems spend in the Scheduled state). For other reasons to do with being able to effectively partition the scheduling task between multiple schedulers each handling the systems managed by a particular lab controller, I've been considering proposing the inclusion of a "Claimed" state in the recipe lifecycle. The "Claimed" state would fit between "Queued" and "Scheduled", and indicate that the recipe had been assigned to a specific lab controller, but not yet assigned to a specific system (at the moment, this state change is handled implicitly through setting "recipe.recipeset.lab_controller" when the first recipe in the recipeset is scheduled). Furthermore, the scheduler would be updated to work on a *cached* copy of the System status data. This is needed to avoid the current problem where there's a race condition with system status changes occurring during a scheduling pass leading to recipes jumping the queue (I'm interested in hearing about relatively clean ways to this with SQL Alchemy, though: http://stackoverflow.com/questions/13983067/cached-reads-immediate-writes-w…) In combination, these two would allow Claimed recipes to be given priority over Queued recipes on subsequent passes, preventing the deadlock problem and theoretically also improving system utilitization. One social challenge with addressing this is that we don't want to enable/encourage queue jumping for rare systems by scheduling them in a recipe set with a job that will be scheduled quickly, but I'm not sure we can solve that at the technical level. Cheers, Nick. -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane Python Applications Team Lead Beaker Development Lead (http://beaker-project.org/) GlobalSync Development Lead (http://pulpdist.readthedocs.org)

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Beaker-devel January 2013