1:06 a.m.
Amit has been working on the implementation of a "<reservesys/>" element
for recipes ([1,2]) and we haven't really liked either of the approaches
we have tried so far. So, bringing it to the list to summarise what
we've tried so far and get feedback on a new proposal.
Background
==========
Status quo:
- system reservation for recipes is handled as a Beaker task
- only works for harnesses that can run Beaker tasks
- doesn't work at all for aborted recipes
- due to the now asynchronous nature of recipe status updates
RESERVE_IF_FAIL may not trigger if the only failing task is
the one immediately before the reservesys task
- this approach will continue to work (and remains the only
way to pause in the *middle* of a recipe)
- the new mechanism for leaving a system provisioned for a
while after the recipe ends is in *addition* to the
reservesys task
First attempt:
- enum for "onpass", "onfail", "onwarn" and
"onabort"
- not suitable, since it didn't allow combinations
Second attempt:
- independent boolean flags with those names
- not suitable, since it allows nonsensical combinations and makes the
implementation and testing of the feature annoyingly complicated
- (this was my suggestion in the BZ, but after seeing it implemented,
my reaction was "bleargh")
New proposal
============
- go back to Amit's enum idea, but make it an *ordered* list.
- under this model, we'd have the following enum options:
* never
* onabort
* onfail
* onwarn
* always
- each level would implicitly include the levels above, so if you say
"onwarn", it will also reserve the system for failures and aborts,
and "onfail" will also reserve the system for aborts
The UI in the XML for this would look similar to Amit's original idea:
<reservesys when="onwarn" duration="86400">
If "when" is not given, it would default to "always", if there was no
reservesys element in the recipe at all, this would be recorded on the
server as "never".
Open question: do we allow 'when="never"' in the XML schema? (I say no:
the reservesys element should just be omitted for that case)
The question of "what about kernel panics and installation failures?"
also came up and Dan and I agreed that is covered by "onabort" - if the
recipe is configured to ignore kernel panics and installation failures
in general, then this reservation mechanism will also ignore them,
otherwise it will handle them like any other recipe that is killed by
the external watchdog.
Cheers,
Nick.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=639938
[2] http://gerrit.beaker-project.org/#/c/3075/
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
3:04 a.m.
On 05/14/2014 04:06 PM, Nick Coghlan wrote:
New proposal
============
- go back to Amit's enum idea, but make it an *ordered* list.
- under this model, we'd have the following enum options:
* never
* onabort
* onfail
* onwarn
* always
- each level would implicitly include the levels above, so if you say
"onwarn", it will also reserve the system for failures and aborts,
and "onfail" will also reserve the system for aborts
Amit pointed out this list can be seen as a little odd, since it blurs
the difference between the recipe status and the recipe result. It's
really the amalgamation of two different lists into one.
First list (based on recipe status):
* never
* onabort
* oncompleted
Second list (based on recipe result, only applies when status is Completed):
* onfail
* onwarn
* onpass
These combine into a single list by replacing the single "oncompleted"
entry with the three recipe results of interest:
* never
* onabort
* oncompleted_fail
* oncompleted_warn
* oncompleted_pass
And then my latest proposal simplifies the names to omit the implied
"completed" status from the last 3 entries, and make it explicit that
the final option covers every possible outcome of running a recipe by
referring to it as "always" rather than "onpass":
* never
* onabort
* onfail
* onwarn
* always
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
5:35 p.m.
New subject: API design for harness independent system reservations
----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)redhat.com>
To: beaker-devel(a)lists.fedorahosted.org
Sent: Wednesday, May 14, 2014 6:04:28 PM
Subject: Re: [Beaker-devel] API design for harness independent system reservations
On 05/14/2014 04:06 PM, Nick Coghlan wrote:
> New proposal
> ============
>
> - go back to Amit's enum idea, but make it an *ordered* list.
> - under this model, we'd have the following enum options:
> * never
> * onabort
> * onfail
> * onwarn
> * always
> - each level would implicitly include the levels above, so if you say
> "onwarn", it will also reserve the system for failures and aborts,
> and "onfail" will also reserve the system for aborts
Amit pointed out this list can be seen as a little odd, since it blurs
the difference between the recipe status and the recipe result. It's
really the amalgamation of two different lists into one.
First list (based on recipe status):
* never
* onabort
* oncompleted
Second list (based on recipe result, only applies when status is Completed):
* onfail
* onwarn
* onpass
These combine into a single list by replacing the single "oncompleted"
entry with the three recipe results of interest:
* never
* onabort
* oncompleted_fail
* oncompleted_warn
* oncompleted_pass
And then my latest proposal simplifies the names to omit the implied
"completed" status from the last 3 entries, and make it explicit that
the final option covers every possible outcome of running a recipe by
referring to it as "always" rather than "onpass":
* never
* onabort
* onfail
* onwarn
* always
My confusion was mainly rooted in the ordering idea that you proposed. If we were to
consider each
state either on it's own (first implementation) or a combination (second
implementation),
it is less confusing as far as I see it. The first implementation is not powerful enough.
The second implementation allows dubious combinations which may not have a meaning - we
would then
just have to trust the submitter to do the right thing.
However, if we were to implement the ordering, "onfail" also including
"onabort" muddies things, for example.
"always" is an exception, since it just means always, really and so is
"never". But having a ordering between
"onabort", "onfail" and "onwarn" is what I am not very clear
about.
6:33 p.m.
New subject: API design for harness independent system reservations
----- Original Message -----
From: "Amit Saha" <asaha(a)redhat.com>
To: "Beaker development" <beaker-devel(a)lists.fedorahosted.org>
Sent: Thursday, May 15, 2014 8:35:39 AM
Subject: Re: [Beaker-devel] API design for harness independent system reservations
----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)redhat.com>
To: beaker-devel(a)lists.fedorahosted.org
Sent: Wednesday, May 14, 2014 6:04:28 PM
Subject: Re: [Beaker-devel] API design for harness independent system reservations
On 05/14/2014 04:06 PM, Nick Coghlan wrote:
> New proposal
> ============
>
> - go back to Amit's enum idea, but make it an *ordered* list.
> - under this model, we'd have the following enum options:
> * never
> * onabort
> * onfail
> * onwarn
> * always
> - each level would implicitly include the levels above, so if you say
> "onwarn", it will also reserve the system for failures and aborts,
> and "onfail" will also reserve the system for aborts
Amit pointed out this list can be seen as a little odd, since it blurs
the difference between the recipe status and the recipe result. It's
really the amalgamation of two different lists into one.
First list (based on recipe status):
* never
* onabort
* oncompleted
Second list (based on recipe result, only applies when status is Completed):
* onfail
* onwarn
* onpass
These combine into a single list by replacing the single "oncompleted"
entry with the three recipe results of interest:
* never
* onabort
* oncompleted_fail
* oncompleted_warn
* oncompleted_pass
And then my latest proposal simplifies the names to omit the implied
"completed" status from the last 3 entries, and make it explicit that
the final option covers every possible outcome of running a recipe by
referring to it as "always" rather than "onpass":
* never
* onabort
* onfail
* onwarn
* always
My confusion was mainly rooted in the ordering idea that you proposed. If we were to
consider each
state either on it's own (first implementation) or a combination (second
implementation),
it is less confusing as far as I see it. The first implementation is not powerful enough.
The second implementation allows dubious combinations which may not have a meaning - we
would then
just have to trust the submitter to do the right thing.
However, if we were to implement the ordering, "onfail" also including
"onabort" muddies things, for example.
"always" is an exception, since it just means always, really and so is
"never". But having a ordering between
"onabort", "onfail" and "onwarn" is what I am not very clear
about.
I agree with amit, I donnot see any point of having a ordering between
"onabort", "onfail" and "onwarn here.
Based on my understanding, correct me if I am wrong. To determine whether or not the
system is actually reserved
in the relevant situations:
- never = never reserve system
- onabort = reserve system if the recipe status is onabort
- onfail = reserve system if the recipe status is oncompleted and the recipe result is
onfail
- onwarn = reserve system if the recipe status is oncompleted and the recipe result is
onwarn
- always = reserve system in regardless of the recipe status and result
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
6:36 p.m.
New subject: API design for harness independent system reservations
Sorry, I messed up the comments. I clean it up here.
And then my latest proposal simplifies the names to omit the implied
"completed" status from the last 3 entries, and make it explicit that
the final option covers every possible outcome of running a recipe by
referring to it as "always" rather than "onpass":
* never
* onabort
* onfail
* onwarn
* always
My confusion was mainly rooted in the ordering idea that you
proposed. If we were to consider each
state either on it's own (first implementation) or a combination (second
implementation),
it is less confusing as far as I see it. The first implementation is not powerful
enough.
The second implementation allows dubious combinations which may not have a meaning - we
would then
just have to trust the submitter to do the right thing.
However, if we were to implement the ordering, "onfail"
also including "onabort" muddies things, for example.
"always" is an exception, since it just means always, really and so is
"never". But having a ordering between
"onabort", "onfail" and "onwarn" is what I am not very
clear about.
I agree with amit, I donnot see any point of having a ordering between
"onabort", "onfail" and "onwarn here.
Based on my understanding, correct me if I am wrong. To determine whether or not the
system is actually reserved
in the relevant situations:
- never = never reserve system
- onabort = reserve system if the recipe status is onabort
- onfail = reserve system if the recipe status is oncompleted and the recipe result is
onfail
- onwarn = reserve system if the recipe status is oncompleted and the recipe result is
onwarn
- always = reserve system in regardless of the recipe status and result
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
_______________________________________________
Beaker-devel mailing list
Beaker-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
1:30 a.m.
New subject: API design for harness independent system reservations
On 05/15/2014 09:36 AM, Matt Jia wrote:
I agree with amit, I donnot see any point of having a ordering
between "onabort", "onfail" and "onwarn here.
Based on my understanding, correct me if I am wrong. To determine whether or not the
system is actually reserved
in the relevant situations:
- never = never reserve system
- onabort = reserve system if the recipe status is onabort
- onfail = reserve system if the recipe status is oncompleted and the recipe result is
onfail
- onwarn = reserve system if the recipe status is oncompleted and the recipe result is
onwarn
So how do I express the following?:
- reserve if it fails *or* aborts
- reserve it it warns, fails *or* aborts
More importantly, what is the use case for reserving *only* if it fails,
but not if it aborts, and *only* if it warns, but not if it fails or aborts?
My contention is that the "simple" meanings for "onfail" and
"onwarn"
are actually useless from an end user perspective, and the "implied or"
approach is far more useful.
From a docs perspective, explaining it in terms of implications rather
than "ordering" may be clearer. So:
- never
- onabort
- onfail (implies onabort)
- onwarn (implies onfail and onabort)
- always
And then the individual meanings are still:
- never = never reserve system
- onabort = reserve system if the recipe status is Abort
- onfail = reserve system if the recipe status is Complete and the
recipe result is Fail
- onwarn = reserve system if the recipe status is Complete and the
recipe result is Warn
- always = reserve system if the recipe status is Abort or Complete
From an implementation perspective, those checks then become something
like:
def should_reserve(status, result, when):
# check the recipe is actually finished
if status not in (Abort, Complete):
return False
# check the status and result independent options
if when == ReserveAlways:
return True
if when == ReserveNever:
return False
# check for an aborted recipe
# onfail and onwarn both imply onabort
if status == Abort:
return True
if when == ReserveOnAbort:
return False
# check for a failed recipe
# onwarn implies onfail
if result == Fail:
return True
if when == ReserveOnFail:
return False
# only thing left to do is check for a warning
assert when == ReserveOnWarn
return result == Warn
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
2:01 a.m.
New subject: API design for harness independent system reservations
----- Original Message -----
From: "Nick Coghlan" <ncoghlan(a)redhat.com>
To: beaker-devel(a)lists.fedorahosted.org
Sent: Thursday, May 15, 2014 4:30:34 PM
Subject: Re: [Beaker-devel] API design for harness independent system reservations
On 05/15/2014 09:36 AM, Matt Jia wrote:
> I agree with amit, I donnot see any point of having a ordering between
> "onabort", "onfail" and "onwarn here.
>
> Based on my understanding, correct me if I am wrong. To determine whether
> or not the system is actually reserved
> in the relevant situations:
>
> - never = never reserve system
> - onabort = reserve system if the recipe status is onabort
> - onfail = reserve system if the recipe status is oncompleted and the
> recipe result is onfail
> - onwarn = reserve system if the recipe status is oncompleted and the
> recipe result is onwarn
So how do I express the following?:
- reserve if it fails *or* aborts
- reserve it it warns, fails *or* aborts
More importantly, what is the use case for reserving *only* if it fails,
but not if it aborts, and *only* if it warns, but not if it fails or aborts?
Okay, I think I now understand the real reason behind the ordering idea.
So, here is an idea: what if we expose only the following "When" options:
- Always
- Fail (covering "aborted" recipe state, and "failed", and
"warn" task results)
- Pass
We don't need a "never" (just don't add a "reservesys" element
to the XML).
This covers the following use cases:
- Always reserve irrespective of the task result and recipe state
- Any kind of failure
- Only when all tests pass
Unless we envision use cases which would depend on exactly the kind of failure
(warn, abort, fail), I think this is not a bad idea.
4:27 a.m.
New subject: API design for harness independent system reservations
On 05/15/2014 05:01 PM, Amit Saha wrote:
Okay, I think I now understand the real reason behind the ordering
idea.
So, here is an idea: what if we expose only the following "When" options:
- Always
- Fail (covering "aborted" recipe state, and "failed", and
"warn" task results)
- Pass
I see four problems with that:
1. You may not care about warnings
2. If you're only interested in cases where the test hangs, you may not
care about failures
3. "Fail" already means something else, so you'd have to introduce a new
word like "Error" and explain what it means
4. Reserve "only if it passes" doesn't make much sense (and suggests
you're using Beaker for something where OpenStack might be a better fit)
We don't need a "never" (just don't add a
"reservesys" element to the XML).
Yes, that can just be a NULL in the DB, and None in Python.
This covers the following use cases:
- Always reserve irrespective of the task result and recipe state
- Any kind of failure
- Only when all tests pass
Unless we envision use cases which would depend on exactly the kind of failure
(warn, abort, fail), I think this is not a bad idea.
I picked those three levels because they're the ones that seem useful:
- reserve on installation failure/crash/infinite loop/deadlock/kernel
panic/etc (as indicated by EWD Abort)
- reserve on actual error (as indicated by EWD Abort or Fail result)
- reserve on dubious behaviour (as indicated by EWD Abort, Fail result
or Warn result)
Now, you could come up with new names for them, but "onabort",
"onfail",
"onwarn" cover it without introducing any new concepts other than
"onwarn implies onfail which in turn implies onabort".
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
8:04 p.m.
Since this particular part of the feature (selecting when to reserve) is
proving quite contentious I suggest that we start out implementing the
simplest possible thing: if <reservesys/> is given, reserve
unconditionally after all tasks are finished.
Or we could do the next simplest thing:
<reservesys/> reserve unconditionally
<reservesys onpass="false"/> reserve if there is any result besides Pass
This matches the current functionality of /distribution/reservesys with
RESERVE_IF_FAIL=1.
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Hosted & Shared Services
Red Hat, Inc.
1:16 a.m.
On 05/16/2014 11:04 AM, Dan Callaghan wrote:
Since this particular part of the feature (selecting when to reserve)
is
proving quite contentious I suggest that we start out implementing the
simplest possible thing: if <reservesys/> is given, reserve
unconditionally after all tasks are finished.
Or we could do the next simplest thing:
<reservesys/> reserve unconditionally
<reservesys onpass="false"/> reserve if there is any result besides Pass
This matches the current functionality of /distribution/reservesys with
RESERVE_IF_FAIL=1.
Yeah, that sounds reasonable to me.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
6:11 p.m.
New subject: API design for harness independent system reservations
On 05/15/2014 04:30 PM, Nick Coghlan wrote:
On 05/15/2014 09:36 AM, Matt Jia wrote:
> I agree with amit, I donnot see any point of having a ordering between
"onabort", "onfail" and "onwarn here.
>
> Based on my understanding, correct me if I am wrong. To determine whether or not the
system is actually reserved
> in the relevant situations:
>
> - never = never reserve system
> - onabort = reserve system if the recipe status is onabort
> - onfail = reserve system if the recipe status is oncompleted and the recipe result
is onfail
> - onwarn = reserve system if the recipe status is oncompleted and the recipe result
is onwarn
So how do I express the following?:
- reserve if it fails *or* aborts
- reserve it it warns, fails *or* aborts
More importantly, what is the use case for reserving *only* if it fails,
but not if it aborts, and *only* if it warns, but not if it fails or aborts?
My contention is that the "simple" meanings for "onfail" and
"onwarn"
are actually useless from an end user perspective, and the "implied or"
approach is far more useful.
From a docs perspective, explaining it in terms of implications rather
than "ordering" may be clearer. So:
- never
- onabort
- onfail (implies onabort)
- onwarn (implies onfail and onabort)
- always
And then the individual meanings are still:
- never = never reserve system
- onabort = reserve system if the recipe status is Abort
- onfail = reserve system if the recipe status is Complete and the
recipe result is Fail
- onwarn = reserve system if the recipe status is Complete and the
recipe result is Warn
- always = reserve system if the recipe status is Abort or Complete
From an implementation perspective, those checks then become something like:
def should_reserve(status, result, when):
# check the recipe is actually finished
if status not in (Abort, Complete):
return False
# check the status and result independent options
if when == ReserveAlways:
return True
if when == ReserveNever:
return Falsenick
# check for an aborted recipe
# onfail and onwarn both imply onabort
if status == Abort:
return True
if when == ReserveOnAbort:
return False
# check for a failed recipe
# onwarn implies onfail
if result == Fail:
return True
if when == ReserveOnFail:
return False
# only thing left to do is check for a warning
assert when == ReserveOnWarn
return result == Warn
Cheers,
Nick.
+1, we can take amit's idea to remove "never" here.
Cheers, Matt Jia
3654
days inactive
3656
days old
beaker-devel@lists.fedorahosted.org
10 comments
4 participants
participants (4)
-
Amit Saha -
Dan Callaghan -
Matt Jia -
Nick Coghlan