Adding resource management to beaker
by Bill Peck
Hello Beaker,
One of the jobs we run uses shared storage. For example:
HostA with Driver1
HostB with Driver2
HostC with Driver3
All connected to StorageX.
We can't run the test on more than one host at a time since it would
conflict.
Being able to specify the following in the job xml is the goal:
<recipeSet>
<recipe/> <!-- Require HostA -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostB -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostC -->
<resource/> <!-- Require StorageX -->
</recipeSet>
I know there is a resource type for recipe but this doesn't seem like a
good solution. I did try an experiment where I created StorageX host in
beaker with no power control and scheduled three recipesets like above.
But this hack won't work because a watchdog with a NULL expire time is
set and that means it will sit there forever. Even if it did work,
maybe if we created a dummy power type, we would still end up with an abort.
I don't think it would take much work to support this type of
multi-host/resource requirement.
Right now we have the following model
Object Recipe, which has the following attributes: distro_tree,
resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks,
dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends
Object MachineRecipe which inherits from Recipe, and adds a guests
attribute.
Object GuestRecipe which inherits from Recipe.
A ResourceRecipe object wouldn't need most of these attributes, probably
just the following:
resource, systems, and dyn_systems. (no watchdog, just return when
recipeSet completes or aborts)
And resource currently maps to a RecipeResource which can be one of the
following:
SystemResource
VirtResource
GuestResource
I think adding another Object here would make sense, but the term
Resource is overused and its a little confusing right now.
If done correctly we could re-use the filtering code in needproperty,
but I think the only things we could search on for resources would be
the following: Name, lab_controller, and key_values? (storing how these
shared resources are connected would be another option I suppose, but
maybe too complicated for a first implementation?)
I'm sure there are other things to consider, we could support setting up
and tearing down shared resources (almost like a power on/off command).
I want to be mindful of things like that but I don't want to be
paralysed with what if's and never get anything done.
I know you guys have been working hard on the groups features and want
to re-do the scheduler as well. Maybe this would be something for after
the scheduler re-write? I don't see any mention of this on the roadmap
[1] (although the roadmap is quite full already!).
In any event, I'm interested in hearing your ideas and feedback.
Thanks!
1 - http://beaker-project.org/dev/tech-roadmap.html
9 years, 6 months
Docs/overview for using autotest in Beaker?
by Nick Coghlan
Hi Don, Bill, LMR,
I saw that the pull request for Beaker harness API support in autotest has been merged. I'd like to link to that feature from beaker-project.org, so I was wondering if there was anything published anywhere (even a blog or mailing list post) that covered the details of:
1. What to add to a job/recipe definition to specify autotest as the harness
2. What, if anything, to add to the task list in the recipe to configure autotest appropriately
3. What, if anything, to include in an autotest job to make it suitable for running inside Beaker
4. How the result reporting works when using autotest as a harness inside Beaker (Do the results just go to Beaker? Or does normal autotest reporting still happen as well?)
If there isn't, we'd be happy to accept a patch on Gerrit to add an autotest section to http://beaker-project.org/docs/alternative-harnesses/ :)
Somewhat related, is there a tentative date yet for the next autotest release?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 5 months
New sub-command to list access policy for a system
by Amit Saha
I noticed that there is no sub-command to list the existing policy for a system, so
tried writing one [1]. My primary objective is to of course having it as a sub-command,
unless there was another reason other than time that it wasn't added earlier.
The second objective is to use it as the second example in our developer guide
to show how you expose controller methods and write a client to access it.
Example usage:
# bkr policy-list --system beaker-test-vm1 --hub http://localhost/bkr --username admin --password foobar
{
"rules": [
{
"everybody": true,
"permission": "control_system",
"group": null,
"id": 1,
"user": null
},
{
"everybody": true,
"permission": "reserve",
"group": null,
"id": 4,
"user": null
}
],
"possible_permissions": [
{
"value": "edit_policy",
. ..
. ..
The authentication bit is not required I suspect.
[1] http://paste.fedoraproject.org/41757/13800067/
Thoughts and suggestions welcome.
Best,
Amit.
--
Amit Saha <http://echorand.me>
Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 5 months
First draft of reservations design proposal
by Nick Coghlan
The first draft of the improved reservations and loans design proposal
is up at:
http://gerrit.beaker-project.org/#/c/2302/1/dev/proposals/improved-reserv...
Very rough sketch at the moment, so I'm mostly looking for feedback on
whether or not the general concepts seem sound.
I've quoted the whole thing below, since I'm not really after
line-by-line review comments at this stage, so email seems like a more
appropriate venue.
Cheers,
Nick.
Improved System Reservations and Loans
======================================
:Author: Nick Coghlan
:Status: Proposed
:Target Release: 0.16
Abstract
--------
This proposal supplements the current timed system reservation mechanism
with a new mechanism that is independent of the lab controller's external
watchdog timer. This new reservation timer is then used to offer time
limited manual reservations and automatic reservation of systems when
using an alternative harness or after a recipe is aborted.
A similar time limiting mechanism is proposed for system loans, as well as
the ability to indicate when reserving a system in Manual mode that the
loan should be returned automatically when the reservation is returned.
Finally, a command line interface will be provided for system loan
management.
.. note::
Writing this out, I suspect it needs to be done in two phases, with phase
one covering the reservation expiry times and automatically returning
loans when the reservation is returned, while phase two covers loan
expiry.
Harness independent time limited reservations
---------------------------------------------
A new "expiry date" attribute will be added to reservation records.
For systems in Manual mode, the web UI will be updated to allow
specification
of a duration when taking the system. The default will continue to be no
time limit, but if a time limit is requested, then the default will be
24 hours (configurable as a global Beaker server setting). If a time limit
is requested, then the reservation expiry date will be set appropriately.
For systems in Automated mode, the reservation expiry date will not be
used by default. Instead, job submitters will be able to opt in to the new
mechanism by specifying a new ``reservesys`` element as part of the recipe
definition.
Ignoring things like making the attributes optional and giving them a
specific data types, the new ``reservesys`` element is defined as::
<element name="reservesys">
<attribute name="onabort"/>
<attribute name="onfail"/>
<attribute name="onwarn"/>
<attribute name="onpass"/>
<attribute name="duration"/>
</element>
By default, the system would be reserved regardless of how the recipe
finishes. The duration setting would control how long the automatic
reservation would be for (defaulting to 24 hours)
onabort/fail/warn/pass control whether or not the system is
actually reserved in the relevant situations:
* onabort = reserve system if a task or the overall recipe aborts
* onfail = reserve system if at least one task reports a failed result
* onwarn = reserve system if at least one task reports a warning
* onpass = reserve system if none of the above occur
All four settings would default to "true", so the reservation is
unconditional by default.
If any of the chosen conditions is triggered, the appropriate expiry time
will be recorded on the reservation once the recipe has completed rather
than returning the system immediately.
If the recipe is explicitly cancelled, then the system will not be reserved.
While a reservation is active, it may be extended. The existing watchdog
extension commands will be updated to also extended the reservation when
needed.
The Beaker scheduling daemon will periodically check for expired
reservations
and automatically return them. For systems in Automated mode, the expiry of
the post-completion reservation will not alter the result reported for the
recipe.
.. note::
Perhaps Beaker should send a warning email 25 hours prior to expiry of
the reservation?
(Also see :issue:`639938`)
Automatically returning loans
-----------------------------
When reserving a system, a user will be able to choose to automatically
return a loan. If this option is specified, then when the reservation is
returned, the active system loan will also be returned.
Time limited loans
------------------
System loans will be separated out to be tracked explicitly (similar to
the handling of reservations, only without the "type" field). When loaning
the system, the user granting the loan will be able to specify a duration
of the loan, which will be recorded as a loan expiry time. The default will
continue to be no time limit, but if a time limit is requested, then the
default will be 7 days (configurable as a global Beaker server setting).
While a loan is active, it may be extended. However, the user extending the
loan must have ``loan-self`` or ``loan-any`` permissions on the system.
If a system has a loan expiry time configured, then all reservations made
by that user must be time limited, defaulting to the loan expiry time.
Attempting to extend reservations beyond the end of the loan period will
fail.
The Beaker scheduling daemon will periodically check for expired loans
and automatically return them. For reserved systems where the reservation
is from a user that is no longer permitted to access the system once the
loan has been returned, it will also return the reservation or cancel the
recipe as appropriate.
.. note::
Perhaps Beaker should send a warning email 25 hours prior to expiry of
the loan?
User interface proposals
------------------------
Web UI
~~~~~~
TBD
Command line
~~~~~~~~~~~~
TBD
Deferred features
-----------------
* Updating the scheduled provisioning mechanism for systems in Automated
mode to use the new harness independent mechanism rather than the
reservesys task
* Allowing the ``reservesys`` element to be specified at the recipe set
level
to reserve all systems in the recipe set whenever one or more of them
encounters a problem.
Rejected features
-----------------
* Allowing the ``reservesys`` element to be specified at the job level,
since it isn't clear how that would work when recipe sets are run at
different times.
* Having "onpass" default to false in the reservesys element. While this is
desirable in some respects, having different defaults for one of the
items is difficult to document clearly.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 7 months
Beaker 0.15 released!
by Nick Coghlan
On behalf of the Beaker development team, I'm pleased to announce that
Beaker v0.15 is now available from beaker-project.org [1].
In this release, we have completed the first phase of the "Access
Policies for Systems" design proposal [2], adding more fine-grained
permissions that allow system owners to control how other users and
groups make use of their systems:
http://beaker-project.org/docs-release-0.15/user-guide/systems/sharing.html
As part of providing the policy editor needed for this new feature, the
technology stack for the main Beaker web interface has also seen several
upgrades. The most visible of these updates is the migration of the web
UI style to be based on Bootstrap CSS. The current theme is almost
entirely unmodified Bootstrap, as we plan to work towards supporting
custom site-specific themes in a future release. Under the hood we have
also started the migration from the legacy TurboGears1 framework to
Flask and have adopted Backbone.js to make it easier to create
responsive client side widgets for complex interactions (like the access
policy editor).
Additional details about the above features and smaller changes for this
release can be found in the release notes here:
http://beaker-project.org/docs/whats-new/release-0.15.html
Regards,
Nick
[1] http://beaker-project.org/releases/
[2]
http://beaker-project.org/dev/proposals/access-policies-for-systems.html#...
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 7 months
Automatically updating docs on beaker-project.org?
by Nick Coghlan
Does the last RAM upgrade for beaker-project.org give it enough grunt to
run a Sphinx docs build locally?
It would be nice if we could just have cron jobs on there periodically
updating the various docs* directories directly from git rather than
having to do the git submodules dance to publish the website.
master -> docs/
develop -> docs-develop/
release-0.15 -> docs-release-0.15/
(etc...)
If we didn't want the release docs autoupdating, then we could run those
off tags rather than branches. If the server has the capacity now, it
seems like it would be reasonably straightforward to set that up based
on a data file committed to the main beaker-project.org repo and stop
building from the git submodules.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 7 months
Splitting up nightlies per distro
by Dan Callaghan
Currently the nightlies on Beaker's web site at
http://beaker-project.org/nightlies/
are for RHEL6 only. I am about to move those into
a RedHatEnterpriseLinux6 subdirectory, and add a subdirectory for
Fedora18 (symlinked to Fedora19 and Fedora20) in the same style as the
release yum repos at /yum/.
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 7 months
Distro import error from develop RPMs
by Amit Saha
In my development environment, I upgraded the server and the lab controller packages to the
current version from git. While importing a distro I am getting this error:
root CRITICAL <Fault 1: "<type 'exceptions.TypeError'>:'set' object does not support indexing">
It does seem like the exception is raised form the server side. Here is an example:
# beaker-import http://mirror.aarnet.edu.au/pub/fedora/linux/development/rawhide/i386/os/ --debug
2013-09-09 02:53:01,987 root DEBUG Importer ComposeInfoLegacy does not match
2013-09-09 02:53:02,101 root DEBUG Importer ComposeInfo does not match
2013-09-09 02:53:02,321 root DEBUG Importer TreeInfoLegacy does not match
2013-09-09 02:53:02,576 root DEBUG Importer TreeInfoRhel5 does not match
2013-09-09 02:53:02,782 root DEBUG Importer TreeInfoFedora Matches
2013-09-09 02:53:02,783 root INFO Attempting to import: http://mirror.aarnet.edu.au/pub/fedora/linux/development/rawhide/i386/os/
2013-09-09 02:53:04,091 root DEBUG
{'arch': 'i386',
'arches': [],
'images': [{'path': 'images/pxeboot/vmlinuz', 'type': 'kernel'},
{'path': 'images/pxeboot/initrd.img', 'type': 'initrd'}],
'kernel_options': None,
'kernel_options_post': None,
'ks_meta': None,
'name': 'Fedora-rawhide',
'osmajor': 'Fedorarawhide',
'osminor': '0',
'repos': [{'path': '.', 'repoid': 'Fedora', 'type': 'variant'},
{'path': '../debug',
'repoid': 'Fedora-debuginfo',
'type': 'debug'}],
'tags': [],
'tree_build_time': '1378389545.99',
'urls': ['http://mirror.aarnet.edu.au/pub/fedora/linux/development/rawhide/i386/os/'],
'variant': ''}
2013-09-09 02:53:04,119 root CRITICAL <Fault 1: "<type 'exceptions.TypeError'>:'set' object does not support indexing">
Seeing the 'git log' i am not sure what may be causing this.
Can someone try and see if it can be reproduced?
Thanks,
Amit.
--
Amit Saha <http://echorand.me>
10 years, 7 months
BZ# 998354: Docs for writing flask controllers
by Amit Saha
As per my discussions on IRC with Dan, and looking at some recent patches,
here are the points I plan to include in the doc:
Routing applications: The Flask application instance, 'app'
needs to be imported from bkr.server.wsgi and then the view function
can be exposed by decorating it with @app.route(). You can also
specify the HTTP methods which the view can handle using the methods
keyword argument.
Example: @app.route('/systems/<fqdn>/access-policy', methods=['POST', 'PUT'])
To learn more: http://flask.pocoo.org/docs/api/#url-route-registrations
Identity checking: If a view function needs authentication,
it can be checked using bkr.server.identity
UI Feedback: Although Flask has it's own mechanism for flash messages,
you must use Turbogears's flash() method
Retuning data: Use Flask's jsonify() function to return your response
as JSON data.
To learn more: http://flask.pocoo.org/docs/api/#module-flask.json
Aborting: If something is not right, use the abort() function
to raise a HTTPException. For eg, abort(401) would indicate a
authentication failure.
To learn more: http://flask.pocoo.org/docs/api/#flask.abort
Returning empty response: If the view function has nothing to return,
return an empty string with a status code, like so: return '', 204
Comments, suggestions welcome.
Thanks,
Amit.
10 years, 7 months