Adding resource management to beaker
by Bill Peck
Hello Beaker,
One of the jobs we run uses shared storage. For example:
HostA with Driver1
HostB with Driver2
HostC with Driver3
All connected to StorageX.
We can't run the test on more than one host at a time since it would
conflict.
Being able to specify the following in the job xml is the goal:
<recipeSet>
<recipe/> <!-- Require HostA -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostB -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostC -->
<resource/> <!-- Require StorageX -->
</recipeSet>
I know there is a resource type for recipe but this doesn't seem like a
good solution. I did try an experiment where I created StorageX host in
beaker with no power control and scheduled three recipesets like above.
But this hack won't work because a watchdog with a NULL expire time is
set and that means it will sit there forever. Even if it did work,
maybe if we created a dummy power type, we would still end up with an abort.
I don't think it would take much work to support this type of
multi-host/resource requirement.
Right now we have the following model
Object Recipe, which has the following attributes: distro_tree,
resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks,
dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends
Object MachineRecipe which inherits from Recipe, and adds a guests
attribute.
Object GuestRecipe which inherits from Recipe.
A ResourceRecipe object wouldn't need most of these attributes, probably
just the following:
resource, systems, and dyn_systems. (no watchdog, just return when
recipeSet completes or aborts)
And resource currently maps to a RecipeResource which can be one of the
following:
SystemResource
VirtResource
GuestResource
I think adding another Object here would make sense, but the term
Resource is overused and its a little confusing right now.
If done correctly we could re-use the filtering code in needproperty,
but I think the only things we could search on for resources would be
the following: Name, lab_controller, and key_values? (storing how these
shared resources are connected would be another option I suppose, but
maybe too complicated for a first implementation?)
I'm sure there are other things to consider, we could support setting up
and tearing down shared resources (almost like a power on/off command).
I want to be mindful of things like that but I don't want to be
paralysed with what if's and never get anything done.
I know you guys have been working hard on the groups features and want
to re-do the scheduler as well. Maybe this would be something for after
the scheduler re-write? I don't see any mention of this on the roadmap
[1] (although the roadmap is quite full already!).
In any event, I'm interested in hearing your ideas and feedback.
Thanks!
1 - http://beaker-project.org/dev/tech-roadmap.html
9 years, 7 months
Design proposal status updates
by Nick Coghlan
I just went through and reviewed the current status of the various
design proposals at https://beaker-project.org/dev/proposals/
The main change was to create a new "Deferred" category, and move
several proposals there where we're currently reconsidering our
approach. I also added notes to the proposals themselves to explain why
they're currently deferred.
The rest we're actively working on, so I moved them all to In Progress
(slightly tweaking the definition of that category in the process).
I also updated the Handling Large Installations proposal to reflect the
0.16 release, and updated the 0.17 entry with a new target release time
frame, and some more info on the interesting features that have already
landed, or that we expect to land next week:
https://beaker-project.org/dev/proposals/handling-large-installations.htm...
I haven't updated the tech road map yet - I'll tackle that next week.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
10 years
Usage reminder email redesign
by Nick Coghlan
One of the things that we're looking to (hopefully) have implemented for
Beaker 0.17 is a redesign of the usage reminder email system so that it
sends at most one email per user per day (currently, it can send one
email per reserved system per user per day, and so nobody will ever turn
it on for a large Beaker installation).
The full proposal is available at
https://beaker-project.org/dev/proposals/beaker-usage-report-emails.html, but
the core part is the design of the reminder email itself:
=========
[Beaker] Usage report for <name> (<date>)
=========
Hi <name>,
Your reservations of the following systems in <Beaker instance FQDN> are
going to expire within 24 hours. If you wish to ensure you retain the
contents of these systems, please extend your reservation.
Expiry Date FQDN
2014-05-15 05:18:01 +10:00 host.beakerlab.example.com
The following systems have been allocated to you in <Beaker instance FQDN>
for more than <X> days and have other recipes queued for execution. Please
return them if you are no longer using them.
Duration Waiting FQDN
7 days 1 recipe host2.beakerlab.example.com
The following jobs you submitted to <Beaker instance FQDN> have been queued
for more than <X> days. Please cancel them if they are no longer relevant,
or perhaps arrange a loan of an appropriate system or systems
Start time Delayed Job
2014-05-15 05:18:01 +10:00 https://<Beaker instance FQDN>/jobs/4594
=========
The proposal has more details on what would trigger the specific entries
in the email.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
10 years
Force by default for bkr machine-test
by Amit Saha
We have had some discussion on the bugzilla entry for this bug [1].
So, just to make sure all of us are on the same page, as opposed to my first
plan to implement a "--force" switch, the forced provisioning will always
be the default. Thus, bkr machine-test system1.fqdn.name will provision the system
even if it is manual or broken.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1093224
10 years
API design for harness independent system reservations
by Nick Coghlan
Amit has been working on the implementation of a "<reservesys/>" element
for recipes ([1,2]) and we haven't really liked either of the approaches
we have tried so far. So, bringing it to the list to summarise what
we've tried so far and get feedback on a new proposal.
Background
==========
Status quo:
- system reservation for recipes is handled as a Beaker task
- only works for harnesses that can run Beaker tasks
- doesn't work at all for aborted recipes
- due to the now asynchronous nature of recipe status updates
RESERVE_IF_FAIL may not trigger if the only failing task is
the one immediately before the reservesys task
- this approach will continue to work (and remains the only
way to pause in the *middle* of a recipe)
- the new mechanism for leaving a system provisioned for a
while after the recipe ends is in *addition* to the
reservesys task
First attempt:
- enum for "onpass", "onfail", "onwarn" and "onabort"
- not suitable, since it didn't allow combinations
Second attempt:
- independent boolean flags with those names
- not suitable, since it allows nonsensical combinations and makes the
implementation and testing of the feature annoyingly complicated
- (this was my suggestion in the BZ, but after seeing it implemented,
my reaction was "bleargh")
New proposal
============
- go back to Amit's enum idea, but make it an *ordered* list.
- under this model, we'd have the following enum options:
* never
* onabort
* onfail
* onwarn
* always
- each level would implicitly include the levels above, so if you say
"onwarn", it will also reserve the system for failures and aborts,
and "onfail" will also reserve the system for aborts
The UI in the XML for this would look similar to Amit's original idea:
<reservesys when="onwarn" duration="86400">
If "when" is not given, it would default to "always", if there was no
reservesys element in the recipe at all, this would be recorded on the
server as "never".
Open question: do we allow 'when="never"' in the XML schema? (I say no:
the reservesys element should just be omitted for that case)
The question of "what about kernel panics and installation failures?"
also came up and Dan and I agreed that is covered by "onabort" - if the
recipe is configured to ignore kernel panics and installation failures
in general, then this reservation mechanism will also ignore them,
otherwise it will handle them like any other recipe that is killed by
the external watchdog.
Cheers,
Nick.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=639938
[2] http://gerrit.beaker-project.org/#/c/3075/
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
10 years
diff-cover could be worth a look
by Nick Coghlan
Interesting idea here: applying automated quality checks just to a
*diff*, so you can use them as patch gating criteria *even if* you have
legacy code that doesn't pass them:
http://engineering.edx.org/2013/12/diff-cover-test-coverage-for-git-commits/
This could be particularly useful to us for improving our unit test
coverage - the integration tests run real servers, so collecting
coverage data is a bit of a pain, but unit tests run in the current process.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
10 years
Postponing scheduler related design proposals?
by Nick Coghlan
Ray has been doing some background research on ways to potentially
improve Beaker's scheduling architecture, and that mainly ended with
recommendations to look into taking advantage of the Apache Mesos
meta-scheduling framework (https://mesos.apache.org/). That seems like a
plausible direction for us to go, so I'd like to suggest that we put the
scheduler redesign related design proposals on hold for the time being.
Specifically, I'd like to mark the following proposals as deferred
pending further research into Mesos:
https://beaker-project.org/dev/proposals/system-pools.html
https://beaker-project.org/dev/proposals/event-driven-scheduler.html
https://beaker-project.org/dev/proposals/effective-job-priorities.html
Between OpenStack integration, migrating the inventory task to lshw (and
supporting running it on RHEL7), the system page redesign, improving the
management and reporting of reservations and loans, etc, etc, I think
there are plenty of other proposals to keep Beaker moving forward in the
meantime :)
Thoughts?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
10 years
