[Libcloud-list] API XML and Client Examples

[Bob McWhirter]

Howdy guys--

I've made some good progress on the API, and Hugh suggested I punt  
some examples to the list.

First, we have "Flavors" which denote types of VM that can be spun up,  
with their capabilities:

<flavor href="http://localhost:3000/api/flavors/m1-large">
     <id>m1-large</id>
     <memory>7.5</memory>
     <storage>850</storage>
     <architecture>x86_64</architecture>
   </flavor>

Next, we have images (templates in RHEV-M terminology):

<image href="http://localhost:3000/api/images/img1">
     <id>img1</id>
     <description>Fedora 10</description>
     <owner_id>fedoraproject</owner_id>
     <architecture>x86_64</architecture>
   </image>

From there we can spin up instances, actual running VMs based upon an  
image:

<instance href="http://localhost:3000/api/instances/inst1">
     <id>inst1</id>
     <public-address>img3.inst1.public.com</public-address>
     <image>http://localhost:3000/api/images/img3</image>
     <private-address>img3.inst1.private.com</private-address>
     <state>RUNNING</state>
     <owner_id>mockuser</owner_id>
     <flavor>http://localhost:3000/api/flavors/m1-small</flavor>
   </instance>

Storage volumes and snapshots are still works-in-progress.

The whole API is based upon discovery from a root URL, which gives  
entry-points into the sub-collections.

<api version="1.0">
     <link rel="flavors">http://localhost:3000/api/flavors</link>
     <link rel="images">http://localhost:3000/api/images</link>
     <link rel="instances">http://localhost:3000/api/instances</link>
     <link rel="storage_volumes">http://localhost:3000/api/storage/volumes 
</link>
     <link rel="storage_snapshots">http://localhost:3000/api/storage/snapshots 
</link>
</api>

I'm writing a Ruby client library also, which tries to be easy to use,  
and exposes a rich object model to the caller, with navigable and lazy- 
loaded references between

it "should allow navigation from instance to image" do
     DeltaCloud.new( API_NAME, API_PASSWORD, API_URL ) do |client|
       instances = client.instances
       instances.should_not be_empty
       instance = instances.first
       instance.image.should_not be_nil
       instance.image.description.should_not be_nil
       instance.image.description.should be_a(String)
     end
   end

Models are defined here:

http://github.com/bobmcwhirter/deltacloud-client-ruby/blob/master/lib/models...

For instance, instance:

http://github.com/bobmcwhirter/deltacloud-client-ruby/blob/master/lib/models...

I'm also hacking together a "mock" driver, so that even folks without  
EC2 access can fire up a driver to test against.  This is still a work- 
in-progress also (isn't everything?)

-Bob

[Sam Johnston]

Bob,

Very interesting. Are you desigining a client or an API or both? A lot of
this looks *very* similar to the current thinking around the OGF's Open
Cloud Computing Interface (OCCI), right down to the discovery document and
use of LINK elements (and HTTP Link: headers). More info at
http://www.occi-wg.org/ but be warned that the documentation currently lags
the discussions... it looks like there's a good opportunity for us to work
together here (both client libraries like libcloud and standard interfaces
like OCCI will be required).

Sam
Secretary, OCCI-WG etc.

On Sat, Aug 1, 2009 at 4:25 PM, Bob McWhirter  wrote:
> Howdy guys--
>
> I've made some good progress on the API, and Hugh suggested I punt some
> examples to the list.
>
> First, we have "Flavors" which denote types of VM that can be spun up, with
> their capabilities:
>
>  <flavor href="http://localhost:3000/api/flavors/m1-large">
>    <id>m1-large</id>
>    <memory>7.5</memory>
>    <storage>850</storage>
>    <architecture>x86_64</architecture>
>  </flavor>
>
> Next, we have images (templates in RHEV-M terminology):
>
>  <image href="http://localhost:3000/api/images/img1">
>    <id>img1</id>
>    <description>Fedora 10</description>
>    <owner_id>fedoraproject</owner_id>
>    <architecture>x86_64</architecture>
>  </image>
>
> From there we can spin up instances, actual running VMs based upon an
> image:
>
>  <instance href="http://localhost:3000/api/instances/inst1">
>    <id>inst1</id>
>    <public-address>img3.inst1.public.com</public-address>
>    <image>http://localhost:3000/api/images/img3</image>
>    <private-address>img3.inst1.private.com</private-address>
>    <state>RUNNING</state>
>    <owner_id>mockuser</owner_id>
>    <flavor>http://localhost:3000/api/flavors/m1-small</flavor>
>  </instance>
>
> Storage volumes and snapshots are still works-in-progress.
>
> The whole API is based upon discovery from a root URL, which gives
> entry-points into the sub-collections.
>
> <api version="1.0">
>    <link rel="flavors">http://localhost:3000/api/flavors</link>
>    <link rel="images">http://localhost:3000/api/images</link>
>    <link rel="instances">http://localhost:3000/api/instances</link>
>    <link rel="storage_volumes">http://localhost:3000/api/storage/volumes
> </link>
>    <link rel="storage_snapshots">
> http://localhost:3000/api/storage/snapshots</link>
> </api>
>
> I'm writing a Ruby client library also, which tries to be easy to use, and
> exposes a rich object model to the caller, with navigable and lazy-loaded
> references between
>
> it "should allow navigation from instance to image" do
>    DeltaCloud.new( API_NAME, API_PASSWORD, API_URL ) do |client|
>      instances = client.instances
>      instances.should_not be_empty
>      instance = instances.first
>      instance.image.should_not be_nil
>      instance.image.description.should_not be_nil
>      instance.image.description.should be_a(String)
>    end
>  end
>
> Models are defined here:
>
> http://github.com/bobmcwhirter/deltacloud-client-ruby/blob/master/lib/models...
>
> For instance, instance:
>
> http://github.com/bobmcwhirter/deltacloud-client-ruby/blob/master/lib/models...
>
> I'm also hacking together a "mock" driver, so that even folks without EC2
> access can fire up a driver to test against.  This is still a
> work-in-progress also (isn't everything?)
>    -Bob
>
> Libcloud-list mailing list
> [e-mail]
> https://www.redhat.com/mailman/listinfo/libcloud-list

[Bob McWhirter]

Hiya Sam--

Yah, I saw that Rackspace also uses Link: type of headers, but I thus  
far haven't seen the need in what I'm doing.  Perhaps next week I'll  
come to the realization as to why they're useful. :)

We're designing an API primarily, along with some implementations, and  
aim to have various language-binding clients for Ruby, Python, Java,  
etc.

I'm sure there's some overlap with everyone doing cloud stuff.

At this point, we're marching forward as fast as we can.  I'll take a  
look at the OCCI stuff, though, and see if we can't find some common  
ground.

Thanks for the pointers,

-Bob

On Aug 1, 2009, at 11:05 AM, Sam Johnston wrote:
> Bob,
>
> Very interesting. Are you desigining a client or an API or both? A  
> lot of this looks very similar to the current thinking around the  
> OGF's Open Cloud Computing Interface (OCCI), right down to the  
> discovery document and use of LINK elements (and HTTP Link:  
> headers). More info at http://www.occi-wg.org/ but be warned that  
> the documentation currently lags the discussions... it looks like  
> there's a good opportunity for us to work together here (both client  
> libraries like libcloud and standard interfaces like OCCI will be  
> required).

[Sam Johnston]

Bob,

So the story behind OCCI is that I had planned to follow Google's example
with GData 2.0 - it's essentially Atom Publishing Protocol (APP) extended
for something like 16 different data types (everything from contacts to
mails to videos) and it works nicely and has a bunch of clients already. Of
course that ended in a religious war about the format-du-jour (XML vs JSON
vs plain text) and we now actually have to do some real work (which is fine
- the result should be better as a result). I wanted a draft by May (and
still plan to deliver a final version by October) so now it's down to
putting the current thinking on paper.

I've spent a few months reading up on the various options, studying great
books like RESTful Web Services and examining many existing APIs. I'm now
trying to get as close to HTTP as possible, essentially making the API an
"overlay" that brings resources (e.g. vmx, vmdk, ovf files) to life. It will
likely be detected via /.well-known/occi/ (ala robots.txt) and should be
compatible with everything from basic HTTP to Subversion, Git and WebDAV
(with HTTP PUT et al required for interop). Alternative representations
(e.g. console screenshots, OVF files, etc.) and interfaces (e.g. to the
state machine) are advertised by Link: headers and a category scheme lifted
from Atom allows for flexible categorisation of an arbitrary number of
resources (as well as questions like "show me the windows servers in the US
without patch KB123456 applied).

We're also set to define a simple descriptor format following the footsteps
of VMware's VMX (namespaced attribute value pairs - e.g "cpu.cores: 2")
which will expose the usual IaaS functionality without the overhead of
OVF/CIM (while retaining compatibility). This could prove particularly
interesting for open source hypervisors.

All of this is subject to change - right now I'm working on making it mesh
together and tacking difficult problems like enumeration, templates, etc.

Cheers,

Sam

On Sat, Aug 1, 2009 at 8:09 PM, Bob McWhirter  wrote:
> Hiya Sam--
>
> Yah, I saw that Rackspace also uses Link: type of headers, but I thus far
> haven't seen the need in what I'm doing.  Perhaps next week I'll come to the
> realization as to why they're useful. :)
>
> We're designing an API primarily, along with some implementations, and aim
> to have various language-binding clients for Ruby, Python, Java, etc.
>
> I'm sure there's some overlap with everyone doing cloud stuff.
>
> At this point, we're marching forward as fast as we can.  I'll take a look
> at the OCCI stuff, though, and see if we can't find some common ground.
>
> Thanks for the pointers,
>
> -Bob
>
> On Aug 1, 2009, at 11:05 AM, Sam Johnston wrote:
>
> Bob,
>
> Very interesting. Are you desigining a client or an API or both? A lot of
> this looks *very* similar to the current thinking around the OGF's Open
> Cloud Computing Interface (OCCI), right down to the discovery document and
> use of LINK elements (and HTTP Link: headers). More info at
> http://www.occi-wg.org/ but be warned that the documentation currently
> lags the discussions... it looks like there's a good opportunity for us to
> work together here (both client libraries like libcloud and standard
> interfaces like OCCI will be required).

[David Lutterkort]

On Sat, 2009-08-01 at 20:37 +0200, Sam Johnston wrote:
> We're also set to define a simple descriptor format following the
> footsteps of VMware's VMX (namespaced attribute value pairs - e.g
> "cpu.cores: 2") which will expose the usual IaaS functionality without
> the overhead of OVF/CIM (while retaining compatibility). This could
> prove particularly interesting for open source hypervisors.

Personally, I would stay away from that particular format, since there's
no reasonable tools to describe what is and is not valid. I'd highly
recommend XML, since there are plenty of tools around writing schemas,
validating documents etc. not to speak of parsers and the like.

David

[David Lutterkort]

On Sat, 2009-08-01 at 10:25 -0400, Bob McWhirter wrote:
> I've made some good progress on the API, and Hugh suggested I punt  
> some examples to the list.

Very cool.
> <api version="1.0">
>      <link rel="flavors">http://localhost:3000/api/flavors</link>
>      <link rel="images">http://localhost:3000/api/images</link>
>      <link rel="instances">http://localhost:3000/api/instances</link>
>      <link rel="storage_volumes">http://localhost:3000/api/storage/volumes 
> </link>
>      <link rel="storage_snapshots">http://localhost:3000/api/storage/snapshots 
> </link>
> </api>

One nit: put the URL's into an 'href' or similar attribute of the link
element; attribute values are easier to handle than text nodes.

David

[Bob McWhirter]

> One nit: put the URL's into an 'href' or similar attribute of the link
> element; attribute values are easier to handle than text nodes.

Thanks, I'd been waffling on that. I'll shuffle back to @href  
attributes, and see how that feels for a while.

-Bob

> From: [e-mail] [mailto:libcloud-list-
>
> Howdy guys--
>
> I've made some good progress on the API, and Hugh suggested I punt
> some examples to the list.
>
> First, we have "Flavors" which denote types of VM that can be spun up,
> with their capabilities:
>
>    <flavor href="http://localhost:3000/api/flavors/m1-large">
>      <id>m1-large</id>
>      <memory>7.5</memory>
>      <storage>850</storage>
>      <architecture>x86_64</architecture>
>    </flavor>

[IH] 
Bob - great to have a start to work with.
I know this is work in progress - just wondering about your take of items
like number of cores? Support of more than a single disk/nic?

Thanks,
   Itamar

[Bob McWhirter]

> Bob - great to have a start to work with.
> I know this is work in progress - just wondering about your take of  
> items
> like number of cores? Support of more than a single disk/nic?

Yah, definitely.  I'll get number of cores added in.

In terms of NICs, I'll have to see what's reasonable for EC2.  For  
RHEV-M, it's a configurable option?

Disks are separate, at least from the EC2 POV.  You have the local  
disk, and then you can have zero-or-more additional storage volumes  
mounted to an instance.

How's that map to RHEV-M?

-Bob

> From: Bob McWhirter [mailto:[e-mail]]

...
> Yah, definitely.  I'll get number of cores added in.
>
> In terms of NICs, I'll have to see what's reasonable for EC2.  For
> RHEV-M, it's a configurable option?

[IH] you can define nic's per template and per vm. You can add/remove
nic's as you want. You don't realy assign nic's though, rather logical
networks which are exposed as nic's in the vm, and mapped to logical
networks on the hosts.
> Disks are separate, at least from the EC2 POV.  You have the local
> disk, and then you can have zero-or-more additional storage volumes
> mounted to an instance.
>
> How's that map to RHEV-M?

[IH] you can add/remove disks from a VM as you want. Disks can be sparse
or preallocated, have different formats (cow, raw), be snapshots from the
shared templates (cow only). There isn't exactly the notion of 'local
disk', but this can probably be abstracted for template driven cloud vm's.
> -Bob