An important first step towards supporting alternative harnesses and/or the mythical Beaker Simple Harness is coming up with a stable, documented API for the harness to interact with Beaker. So I'm starting this thread now to get the ball rolling.
My first thoughts:
* It should have the smallest possible surface area -- just enough to expose all of Beaker's functionality and nothing more.
* It should be defined from the point of view of lab controller <-> test system, not scheduler <-> test system. Corollary: we might need to start treating the lab controller as a first-class citizen instead of just a dumb proxy to the scheduler.
* Just because we use XML-RPC now doesn't mean it's the best choice, and doesn't mean we need to keep using it.
* In particular, the HTTP protocol probably supports everything we need to build the API (in other words, it can be a "RESTful API" even though I hate that phrase). For example logs could be uploaded using HTTP PUT (with Content-Range), which means if we wanted we could potentially use Apache mod_dav_fs to efficiently write these directly to the filesystem without any intervening Beaker code.
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
* find out what to run * report results and upload logs * extend the watchdog time * synchronize with other recipes in the recipe set
This last area is a new one, it's currently handled entirely in beah and there is no API on the Beaker side for it. But the dynamic FQDNs in Beaker 0.10 mean it is now needed, even if it's as simple as having a call to wait for all other harnesses in the recipe set to check in.
Thoughts?
First sketch of an HTTP API below. I don't think I've forgotten anything...
Excerpts from Dan Callaghan's message of 2012-12-19 16:46:19 +1000:
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
- find out what to run
Get recipe XML:
GET /recipes/{recipeid} Accept: application/xml
- report results and upload logs
Record a task result:
POST /recipes/{recipeid}/tasks/{taskid}/results/ Content-type: application/x-www-form-urlencoded
path=/the/task/result&result=Pass&score=1234&message=It+worked
-> 302 See Other Location: /recipes/{recipeid}/tasks/{taskid}/results/{resultid}
Upload recipe log:
PUT /recipes/{recipeid}/logs/{path...}
Upload task log:
PUT /recipes/{recipeid}/tasks/{taskid}/logs/{path...}
Upload task-result log:
PUT /recipes/{recipeid}/tasks/{taskid}/results/{resultid}/logs/{path...}
These could accept Content-Range (for partial uploads) and Content-Type (which we could store and return later).
- extend the watchdog time
Set watchdog to now + X seconds:
POST /recipes/{recipeid}/watchdog Content-type: application/x-www-form-urlencoded
seconds=X
- synchronize with other recipes in the recipe set
Block until all other recipes in the recipe set (except guest recipes?) have called this, or until X seconds have elapsed:
POST /recipes/{recipeid}/sync-start?timeout=X
On 12/19/2012 11:50 PM, Dan Callaghan wrote:
First sketch of an HTTP API below. I don't think I've forgotten anything...
Hi Dan,
What if a group wants to use an alternate harness? Is the idea that the alternate harness provide a backend plugin which would use this API? or would we be free to communicate directly back to a different results server?
Excerpts from Dan Callaghan's message of 2012-12-19 16:46:19 +1000:
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
- find out what to run
Get recipe XML:
GET /recipes/{recipeid} Accept: application/xml
- report results and upload logs
Record a task result:
POST /recipes/{recipeid}/tasks/{taskid}/results/ Content-type: application/x-www-form-urlencoded path=/the/task/result&result=Pass&score=1234&message=It+worked-> 302 See Other Location: /recipes/{recipeid}/tasks/{taskid}/results/{resultid}
Upload recipe log:
PUT /recipes/{recipeid}/logs/{path...}Upload task log:
PUT /recipes/{recipeid}/tasks/{taskid}/logs/{path...}Upload task-result log:
PUT /recipes/{recipeid}/tasks/{taskid}/results/{resultid}/logs/{path...}These could accept Content-Range (for partial uploads) and Content-Type (which we could store and return later).
- extend the watchdog time
Set watchdog to now + X seconds:
POST /recipes/{recipeid}/watchdog Content-type: application/x-www-form-urlencoded seconds=X
- synchronize with other recipes in the recipe set
Block until all other recipes in the recipe set (except guest recipes?) have called this, or until X seconds have elapsed:
POST /recipes/{recipeid}/sync-start?timeout=X
Beaker-devel mailing list Beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
Excerpts from Bill Peck's message of 2012-12-20 22:51:06 +1000:
On 12/19/2012 11:50 PM, Dan Callaghan wrote:
First sketch of an HTTP API below. I don't think I've forgotten anything...
Hi Dan,
What if a group wants to use an alternate harness? Is the idea that the alternate harness provide a backend plugin which would use this API? or would we be free to communicate directly back to a different results server?
I think if you did have an alternative harness that sent results elsewhere that would be fine, but it would still need some code to at least report the state of what's running and extend the watchdog (even if it just reports one result against one task with no logs, or whatever). So it could use this API to do that.
What I had in mind here was actually the simple_harness.py which is floating around (written by some users) as well as the future versions of beah or Beaker Simple Harness or whatever we develop in future. They would all benefit from having a stable API which stays backwards compatible.
Excerpts from Dan Callaghan's message of 2012-12-20 14:50:15 +1000:
First sketch of an HTTP API below. I don't think I've forgotten anything...
I've just uploaded a patch series to Gerrit which implements most of this draft API:
http://gerrit.beaker-project.org/1703 http://gerrit.beaker-project.org/1704 http://gerrit.beaker-project.org/1705 http://gerrit.beaker-project.org/1706
along with a few supporting changes:
http://gerrit.beaker-project.org/1242 http://gerrit.beaker-project.org/1701 http://gerrit.beaker-project.org/1702
As an added bonus, there are also tests covering the existing XML-RPC API.
A few things are still missing:
* recipe synchronization * "N/A" result type, and attaching URLs to results (as discussed elsewhere on this thread) * API docs for the new harness API * release notes for the log storage change
But, I think it's close enough that we could include it as an "experimental" API in 1.0, which people developing harnesses can start working with. Min, how does that sound to you?
Excerpts from Dan Callaghan's message of 2012-12-19 16:46:19 +1000:
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
- find out what to run
Get recipe XML:
GET /recipes/{recipeid} Accept: application/xml
- report results and upload logs
Record a task result:
POST /recipes/{recipeid}/tasks/{taskid}/results/ Content-type: application/x-www-form-urlencoded path=/the/task/result&result=Pass&score=1234&message=It+worked-> 302 See Other Location: /recipes/{recipeid}/tasks/{taskid}/results/{resultid}
Upload recipe log:
PUT /recipes/{recipeid}/logs/{path...}Upload task log:
PUT /recipes/{recipeid}/tasks/{taskid}/logs/{path...}Upload task-result log:
PUT /recipes/{recipeid}/tasks/{taskid}/results/{resultid}/logs/{path...}These could accept Content-Range (for partial uploads) and Content-Type (which we could store and return later).
- extend the watchdog time
Set watchdog to now + X seconds:
POST /recipes/{recipeid}/watchdog Content-type: application/x-www-form-urlencoded seconds=X
- synchronize with other recipes in the recipe set
Block until all other recipes in the recipe set (except guest recipes?) have called this, or until X seconds have elapsed:
POST /recipes/{recipeid}/sync-start?timeout=X
Hi Dan,
----- Original Message -----
From: "Dan Callaghan" dcallagh@redhat.com To: "Beaker Devel" beaker-devel@lists.fedorahosted.org Cc: "Min Shin" mishin@redhat.com Sent: Wednesday, February 6, 2013 5:59:51 PM Subject: Re: Defining a harness API
Excerpts from Dan Callaghan's message of 2012-12-20 14:50:15 +1000:
First sketch of an HTTP API below. I don't think I've forgotten anything...
I've just uploaded a patch series to Gerrit which implements most of this draft API:
http://gerrit.beaker-project.org/1703 http://gerrit.beaker-project.org/1704 http://gerrit.beaker-project.org/1705 http://gerrit.beaker-project.org/1706along with a few supporting changes:
http://gerrit.beaker-project.org/1242 http://gerrit.beaker-project.org/1701 http://gerrit.beaker-project.org/1702As an added bonus, there are also tests covering the existing XML-RPC API.
A few things are still missing:
* recipe synchronization * "N/A" result type, and attaching URLs to results (as discussed elsewhere on this thread) * API docs for the new harness API * release notes for the log storage changeBut, I think it's close enough that we could include it as an "experimental" API in 1.0, which people developing harnesses can start working with. Min, how does that sound to you?
Thanks for pushing this. I think this and an actual harness implementation project that we are trying to kick off will complement each other well.
Please make sure to add some design documents and minimum API documentation if possible.
Regards, Min
Excerpts from Dan Callaghan's message of 2012-12-19 16:46:19 +1000:
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
- find out what to run
Get recipe XML:
GET /recipes/{recipeid} Accept: application/xml
- report results and upload logs
Record a task result:
POST /recipes/{recipeid}/tasks/{taskid}/results/ Content-type: application/x-www-form-urlencoded path=/the/task/result&result=Pass&score=1234&message=It+worked-> 302 See Other Location: /recipes/{recipeid}/tasks/{taskid}/results/{resultid}
Upload recipe log:
PUT /recipes/{recipeid}/logs/{path...}Upload task log:
PUT /recipes/{recipeid}/tasks/{taskid}/logs/{path...}Upload task-result log:
PUT /recipes/{recipeid}/tasks/{taskid}/results/{resultid}/logs/{path...}These could accept Content-Range (for partial uploads) and Content-Type (which we could store and return later).
- extend the watchdog time
Set watchdog to now + X seconds:
POST /recipes/{recipeid}/watchdog Content-type: application/x-www-form-urlencoded seconds=X
- synchronize with other recipes in the recipe set
Block until all other recipes in the recipe set (except guest recipes?) have called this, or until X seconds have elapsed:
POST /recipes/{recipeid}/sync-start?timeout=X-- Dan Callaghan dcallagh@redhat.com Software Engineer, Infrastructure Engineering and Development Red Hat, Inc.
----- Original Message -----
From: "Dan Callaghan" dcallagh@redhat.com To: "Beaker Devel" beaker-devel@lists.fedorahosted.org Sent: Wednesday, December 19, 2012 4:46:19 PM Subject: [Beaker-devel] Defining a harness API
An important first step towards supporting alternative harnesses and/or the mythical Beaker Simple Harness is coming up with a stable, documented API for the harness to interact with Beaker. So I'm starting this thread now to get the ball rolling.
My first thoughts:
It should have the smallest possible surface area -- just enough to expose all of Beaker's functionality and nothing more.
It should be defined from the point of view of lab controller <->
test system, not scheduler <-> test system. Corollary: we might need to start treating the lab controller as a first-class citizen instead of just a dumb proxy to the scheduler.
Just because we use XML-RPC now doesn't mean it's the best choice, and doesn't mean we need to keep using it.
In particular, the HTTP protocol probably supports everything we
need to build the API (in other words, it can be a "RESTful API" even though I hate that phrase). For example logs could be uploaded using HTTP PUT (with Content-Range), which means if we wanted we could potentially use Apache mod_dav_fs to efficiently write these directly to the filesystem without any intervening Beaker code.
There are four areas the API needs to cover (that I can think of). The harness needs to be able to:
- find out what to run
- report results and upload logs
I don't think results should be a necessity. I wonder if the LC should only require the API to expose a way of setting the status of a task. e.g. Running/Completed.
Because the UI of beaker is pretty tightly coupled to the idea of a result, perhaps we would have to provide some default which indicates no particular result either way, e.g N/A. People who would be comfortable using beaker in this way would just be using it for it's inventory resources, scheduling algorithms, and provisioning capablities, and would be interested in sending their test results and logs elewhere.
Of course I don't imagine this to be the majority of use cases, and people would also be able to upload a result and logs as per normal, but I don't think it should be necessary.
- extend the watchdog time
- synchronize with other recipes in the recipe set
This last area is a new one, it's currently handled entirely in beah and there is no API on the Beaker side for it. But the dynamic FQDNs in Beaker 0.10 mean it is now needed, even if it's as simple as having a call to wait for all other harnesses in the recipe set to check in.
Thoughts?
-- Dan Callaghan dcallagh@redhat.com Software Engineer, Infrastructure Engineering and Development Red Hat, Inc.
Beaker-devel mailing list Beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
On 12/21/2012 12:52 PM, Raymond Mancy wrote:
- report results and upload logs
I don't think results should be a necessity. I wonder if the LC should only require the API to expose a way of setting the status of a task. e.g. Running/Completed.
Because the UI of beaker is pretty tightly coupled to the idea of a result, perhaps we would have to provide some default which indicates no particular result either way, e.g N/A. People who would be comfortable using beaker in this way would just be using it for it's inventory resources, scheduling algorithms, and provisioning capablities, and would be interested in sending their test results and logs elewhere.
Of course I don't imagine this to be the majority of use cases, and people would also be able to upload a result and logs as per normal, but I don't think it should be necessary.
I'd think of this part of the API as "report progress" rather than "report results". Even if they don't tell us whether or not the task succeeded, they still need to tell us when it is done. For navigation purposes, we may want to provide a way for alternate test harnesses/result servers to add an "external results URL" at the job/recipe/task level (with the appropriate granularity being an open question).
I like Dan's idea of using a REST-style API for the stable harness API, and making the lab controller a more active participant in the process. We're already moving that way in 0.11, with the lab controller intercepting the clear netboot calls in order to execute them synchronously.
Cheers, Nick.
Excerpts from Nick Coghlan's message of 2013-01-02 11:53:20 +1000:
On 12/21/2012 12:52 PM, Raymond Mancy wrote:
- report results and upload logs
I don't think results should be a necessity. I wonder if the LC should only require the API to expose a way of setting the status of a task. e.g. Running/Completed.
Because the UI of beaker is pretty tightly coupled to the idea of a result, perhaps we would have to provide some default which indicates no particular result either way, e.g N/A. People who would be comfortable using beaker in this way would just be using it for it's inventory resources, scheduling algorithms, and provisioning capablities, and would be interested in sending their test results and logs elewhere.
Of course I don't imagine this to be the majority of use cases, and people would also be able to upload a result and logs as per normal, but I don't think it should be necessary.
I'd think of this part of the API as "report progress" rather than "report results". Even if they don't tell us whether or not the task succeeded, they still need to tell us when it is done.
Okay, we could fully de-couple reporting progress (task X is completed) from reporting results (task X result Y is pass). Actually now that I look at it, they are already completely de-coupled -- task_stop is separate from task_result. I missed that in my first draft.
I think right now if the harness stops a task without recording any results then it will just become Completed/New, which might be reasonable if the harness doesn't want to say whether the task passed or failed. We could also introduce some new result distinct from New, such as None or Unspecified, if we wanted to make it more explicit.
For navigation purposes, we may want to provide a way for alternate test harnesses/result servers to add an "external results URL" at the job/recipe/task level (with the appropriate granularity being an open question).
Yes I like this idea of allowing the harness to submit arbitrary URLs as results. On the Beaker side we could probably just treat that the same as a recipe-task log (or recipe-task-result log) but instead of being stored by Beaker it is just a reference to a URL.
On 01/02/2013 07:40 PM, Dan Callaghan wrote:
Excerpts from Nick Coghlan's message of 2013-01-02 11:53:20 +1000:
On 12/21/2012 12:52 PM, Raymond Mancy wrote:
- report results and upload logs
I don't think results should be a necessity. I wonder if the LC should only require the API to expose a way of setting the status of a task. e.g. Running/Completed.
Because the UI of beaker is pretty tightly coupled to the idea of a result, perhaps we would have to provide some default which indicates no particular result either way, e.g N/A. People who would be comfortable using beaker in this way would just be using it for it's inventory resources, scheduling algorithms, and provisioning capablities, and would be interested in sending their test results and logs elewhere.
Of course I don't imagine this to be the majority of use cases, and people would also be able to upload a result and logs as per normal, but I don't think it should be necessary.
I'd think of this part of the API as "report progress" rather than "report results". Even if they don't tell us whether or not the task succeeded, they still need to tell us when it is done.
Okay, we could fully de-couple reporting progress (task X is completed) from reporting results (task X result Y is pass). Actually now that I look at it, they are already completely de-coupled -- task_stop is separate from task_result. I missed that in my first draft.
I think right now if the harness stops a task without recording any results then it will just become Completed/New, which might be reasonable if the harness doesn't want to say whether the task passed or failed. We could also introduce some new result distinct from New, such as None or Unspecified, if we wanted to make it more explicit.
I would prefer N/A (not applicable)
For navigation purposes, we may want to provide a way for alternate test harnesses/result servers to add an "external results URL" at the job/recipe/task level (with the appropriate granularity being an open question).
Yes I like this idea of allowing the harness to submit arbitrary URLs as results. On the Beaker side we could probably just treat that the same as a recipe-task log (or recipe-task-result log) but instead of being stored by Beaker it is just a reference to a URL.
This is a great idea.
Beaker-devel mailing list Beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
On 01/03/2013 11:27 PM, Bill Peck wrote:
On 01/02/2013 07:40 PM, Dan Callaghan wrote:
I think right now if the harness stops a task without recording any results then it will just become Completed/New, which might be reasonable if the harness doesn't want to say whether the task passed or failed. We could also introduce some new result distinct from New, such as None or Unspecified, if we wanted to make it more explicit.
I would prefer N/A (not applicable)
N/A sounds good to me, as it makes it clear this is a deliberate omission.
For navigation purposes, we may want to provide a way for alternate test harnesses/result servers to add an "external results URL" at the job/recipe/task level (with the appropriate granularity being an open question).
Yes I like this idea of allowing the harness to submit arbitrary URLs as results. On the Beaker side we could probably just treat that the same as a recipe-task log (or recipe-task-result log) but instead of being stored by Beaker it is just a reference to a URL.
This is a great idea.
Ah, URLs, the magic fairy dust of system integration tasks :)
Cheers, Nick.
beaker-devel@lists.fedorahosted.org
-
Bill Peck -
Dan Callaghan -
Minjung Shin -
Nick Coghlan -
Raymond Mancy