deltacloud-devel July 2009

deltacloud-devel@lists.fedorahosted.org

7 participants
11 discussions

Re: [Libcloud-list] First shot at verbs (nouns to follow)
by Bob McWhirter 17 Jul '09

17 Jul '09

For anyone external w/o access to our clearspace, here's the content. -Bob Authentication Sessions * Create a new authentication session - POST /sessions * Destroy an authentication session - DELETE /sessions/${id} Images * Query collection of images - GET /images o ec2-describe-images * Retrieve a single image - GET /images/${id} o ec2-describe-images * Create an image [NOT SUPPORTED] o ec2-bundle o ec2-register * Modify an image [NOT SUPPORTED] * Delete an image [NOT SUPPORTED] o ec2-deregister Instances * Query collection of instances - GET /instances o ec2-describe-instances * Retrieve a single instance - GET /instances/${id} o ec2-describe-instances * Create an instance - POST /instances o ec2-run-instances * Modify an instance - PUT /instances/${id} o ec2-reboot-instances * Delete an image - DELETE /instances/${id} o ec2-terminate-instances Security Policies * Query collection of policies - GET /security/policies o ec2-describe-group o ec2-describe-keypairs * Retrieve a single policy - GET /security/policies/${id} o ec2-describe-group o ec2-describe-keypairs o ec2-fingerprint-key * Create a policy - POST /security/policies o ec2-add-group o ec2-add-keypair * Modify a policy - PUT /security/policies/${id} o ec2-authorize o ec2-revoke * Delete a policy - DELETE /security/policies/${id} o ec2-delete-group o ec2-delete-keypair Addresses * Query collection of addresses - GET /network/addresses o ec2-describe-address * Retrieve a single address - GET /network/addresses/${id} o ec2-describe-address * Create a new address - POST /network/addresses o ec2-allocate-address * Modify an address - PUT /network/addresses/${id} o ec2-associate-address o ec2-deassociate-address * Delete an address - DELETE /network/addresses/${id} o ec2-release-address Storage Volumes * Query a collection of storage volumes - GET /storage/volumes o ec2-describe-volumes * Query a single storage volume - GET /storage/volumes/${id} o ec2-describe-volumes * Create a new storage volume - POST /storage/volumes o ec2-create-volume * Modify a storage volume - PUT /storage/volumes/${id} o ec2-attach-volume o ec2-detach-volume * Delete a storage volume - DELETE /storage/volumes/${id} o ec2-delete-volume Storage Volume Snapshots * Query a collection of storage volume snapshots - GET /storage/ snapshots o ec2-describe-snapshots * Query a single storage volume snapshot - GET /storage/snapshots/$ {id} o ec2-describe-snapshots * Create a new storage volume snapshot - POST /storage/snapshots o ec2-create-snapshot * Delete a storage volume - DELETE /storage/snapshots/${id} o ec2-destroy-snapshot On Jul 14, 2009, at 2:19 PM, Bob McWhirter wrote: > Here's a first shot at delineating the verbs for libcloud. There's > a big lack of what the contents of the nouns are (attributes on the > resources) so far. > > But, this includes a mapping to the EC2 commands, at least initially. > > https://docspace.corp.redhat.com/clearspace/docs/DOC-22758 > > Still not sure about docspace, but trying it anyhow. > > -Bob > > _______________________________________________ > Libcloud-list mailing list > Libcloud-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/libcloud-list

5 9

[Libcloud-list] Security, stateless; discuss!
by Bob McWhirter 16 Jul '09

16 Jul '09

I know it's probably an implementation detail, but always better to consider security sooner rather than later. Also, given that security is one way that cloud vendors differ, it's something we should bake into the API correctly the first time through. And yet also, given the desire for stateless drivers where possible, it may affect our security decisions, depending on how we define "stateless" or "security". Three main models if security seen by me thus far are: a) Some login() type of call, which takes a name and a password. Many times the "password" is not your personal password associated with web-console logins, but another, randomly-generated (hah!) key that you can use for just API calls. The API key can be regenerated if it's been compromised, without affecting your web-UI login credentials. This login() returns a session token which is then passed through to every other call. This is basically normal web workflow with session ID in query string or a cookie, but with a separate password. b) no login(), but an API key passed with each request. The ultimate service has no concept of sessions. c) no login(), but each request is cryptographically signed. Once again, no concept of sessions. Does our desire for stateless drivers also include a desire to have no session state? Or would that be an acceptable level of state the implementor of the API needs to concern himself with? If we insist on no sessions, and we're using, let's say, the adapter that talks EC2 signed SOAP out the backside, how do we communicate our EC2 credentials to the libcloud receiver? If we're doing credential mapping, then I can login to the proxy as "bob", and it's already in possession of my EC2 X.509 credentials, provided through some out-of- band method (ie, a portal). But, if I'm using the driver directly, on port 80, over REST... Do we still insist on the driver having a store of credentials local to it? Or would the client be expected to ship his X.509 certificate to the driver with every request, so that the driver can sign the ultimate EC2 SOAP requests on his behalf? If we allow/require stateful sessions, then at least login(...) could take a driver-specific hash of information to set up the security context. ie: login ( :x509_cert=>'.....', :access_key=>'...', :secret_access_key=>'....' ) for the EC2 driver, which would expect those 3 members. Yes, login() would actually be a PUT to /sessions, and be RESTful... But basically, leave the schema for the Session resource as void*, or implementation-defined, to allow for flexibility. For name/password, login( :name=>'...', :password=>'...'), and for API key-based, just login( :key=>'...' ) or such. So, is session state too much state? I think it'd be okay to store in RAM, no guarantee on longevity, and requires no local long-term storage. Thanks, -Bob

5 7

[Libcloud-list] Authentication & Sessions with ∂-API
by Bob McWhirter 15 Jul '09

15 Jul '09

Howdy everyone-- Wrote up some thoughts and examples about authentication and sessions within ∂-API. https://docspace.corp.redhat.com/clearspace/docs/DOC-22760 I've also coined some terminology for discussing the various topologies a user might experience. -Bob

1 0

[Libcloud-list] First shot at verbs (nouns to follow)
by Bob McWhirter 14 Jul '09

14 Jul '09

Here's a first shot at delineating the verbs for libcloud. There's a big lack of what the contents of the nouns are (attributes on the resources) so far. But, this includes a mapping to the EC2 commands, at least initially. https://docspace.corp.redhat.com/clearspace/docs/DOC-22758 Still not sure about docspace, but trying it anyhow. -Bob

1 0

[Libcloud-list] another libcloud
by Alex Polvi 13 Jul '09

13 Jul '09

Hi, I was excited when I found your project. Not only does it have similar goals, but it has the same name: http://github.com/polvi/libcloud/ We're essentially trying to do the same thing ... that is, develop a unified interface, plus a set of drivers, so that we only have to develop against one API. I read through the archives, and I think we're on the same track. Right now, my approach has been to keep things as simple as possible. The library will give you an api client for whatever providers (only ec2 and a dummy provider, atm) it supports. The methods (only list_nodes, so far) on the client will be identical. The end result will be a library, written in python, that you could expose via a REST API or whatever you wanted. Here is a very basic example: http://github.com/polvi/libcloud/blob/4ea159380dfa966f15711602c47d37a37f669… Anyway, next week I'm speaking at OSCON about this topic (cloud standards, portability). Would love to mention this if possible. Also, if anyone is going to be around OSCON, we should team up and try to get some more momentum! -Alex -- co-founder, cloudkick.com twitter.com/cloudkick 541 231 0624

1 0

[Libcloud-list] libcloud as SOA
by Bob McWhirter 10 Jul '09

10 Jul '09

Further discussion in #libcloud has lead us to think (but no decisions yet!) about instead of "drivers" being implementation details, but rather have them be running services that directly speak libcloud-API on one side, and the cloud-native API on the other. In this case, Steve the Scripter would fire up the driver service he's using today, perhaps on localhost, and work with a cloudsh or other client that talks REST to the localhost-hosted service. For the value-add of the previously discussed generic libcloud- service, which might do credential mapping and cloud multiplexing, it would be another cloud that speaks libcloud-API on the front, but also talks libcloud-API out the back ultimately to the real driver services. In this case, libcloud-service would act more as a proxy/aggregator, and not directly drive the drivers. One benefit is the RHEV-M driver would be able to run fully on the win32 machine, colocated with RHEV-M. It'd be a separate process from RHEV-M, but would not require a "driver" to be placed in libcloud- service AND a shim placed on the win32 machine. The full driver would be a single process running on the win32 box, aggregated by the libcloud-service. If libcloud-service is a proxy, we *could* end up chaining them together, such as a libcloud-service for each department, which talks to its own local clouds AND also the libcloud-service for the corporation. The libcloud-service for the corporation would then also talk to its own clouds. If each driver is really a service, and we stitch them together, we're SOA buzzword compliant. And each driver can be authored in the language most appropriate for that driver. In previous discussion, if libcloud-service directly spoke to the drivers in-process, that severely restricts the implementation choices. Here's some diagrams: http://fnokd.com/~bob/libcloud-soa-1.png The first represents Steve the Scripter who just wants a homogenous API he can script against regardless of which cloud he's really using. It would require 1 service running, possibly on localhost (but not required!). The bottom half of that diagram shows a more complex configuration, where a proxying/aggregating value-add libcloud-service stands between Don the Developer and several clouds available simultaneously through the service. Each driver may be on the same host, or may be exposed by the cloud provider natively, or distributed within the organization as appropriate (ie, RHEV-M's driver sitting directly on a win32 machine, everything else is scattered on linux boxen). To make life easy for folks who are happy to use whatever language we've picked, we can also provide a libcloud-service-framework. This would be a template implementation of a driver, lacking the actual guts that do real work, but otherwise handling the HTTP and REST aspects. http://fnokd.com/~bob/libcloud-framework-1.png Folks could simply ignore it, though, and do their own implementation between libcloud-API and native-API. It's just an optional chunk to help driver implementors if they so choose. -Bob

4 3

[Libcloud-list] Stateless v. stateful
by David Lutterkort 10 Jul '09

10 Jul '09

Hi, I was just chatting with Hugh, and we went another round or two on the 'Is libcloud stateless or stateful?' question. Mostly to clear my head, some thoughts on that: there's two usage scenarios I'd want to capture with libcloud 1. Steve the Scripter writes a few Python scripts to make his miserable, awful, pitisome existence as a developer easier; some of these scripts need to start/stop virtual machines in the cloud (maybe a JBoss cluster from Ivan the Integrator) - Steve uses libcloud to make sure that he can move from cloud A to cloud B without having to rewrite all those scripts. 2. Don the Developer is writing a cloud portal UI. He uses libcloud in the backend of his portal so that he can claim 100% compatibility with N (fslvo of N) popular clouds. I think these two uses cover a large percentage of the current EC2 API uses; they differ tremendously in the amount of deployment pain Don and Steve are willing to go through - Steve is perfectly happy with a very basic API, but would prefer libcloud setup be as simple as possible, ideally just installing a library, while Don wants to enable lots of spiffy features in his cloud portal UI (SSO, accounting, image mgmt, ...) and is happy with a more complex setup involving lots of auxiliary services (DB, web server, ...). To serve Steve, we'd want a stateless, thin library; Steve might have to use different ways to authenticate from his script for different clouds, and therefore change that part of his script when he switches to a different cloud backend, but he'll be happy with that as long as those changes are localized and don't require a big rewrite of his scripts. IOW, authentication should be the only area where he needs to be aware of the concrete backend he's using - this usage is very close to what libvirt does. Ideally, libcloud deployment for Steve involves little more than installing an RPM (and maybe starting a service) - but it should definitely not require setting up a fullblown DB server, manually loading schema etc. How should such a stateless API be exposed ? I think an in-process DSO-style API is out of the question, since it would either require doing SOAP calls from C (yuck) or tying the API to a specific implementation language. That means that it needs to be exposed by some RPC mechanism, i.e. will require running a server; and since we already want to expose a REST API, some sort of web server makes the most sense. To make both Don and Steve happy, we should consider either two different ways to run the server: stateless with reduced functionality as a fairly direct API-passthrough, and stateful with value-add features like accounting etc. 'Stateless' might actually not be completely stateless: we could still offer simple deployment if we use a sqlite backend to store some basic data like auth credentials for different clouds. (Completely stateless has the disadvantage that backend cloud credentials only live for the lifetime of the server, which leads to interesting error handling for the client) For both Don and Steve a very basic session to start a bunch of VM's might look like 1. Authenticate to libcloud 2. Register credentials for backend clouds 3. List available backend clouds[1] (Steve has Alzheimer's and forgot what happened in step (2)) 4. List available templates[2] for each backend cloud 5. Run some VM's based on the available templates 6. Shut the VM's down David [1] At that point, a cloud is an abstraction that consists of things like a cost/billing model and resource quotas; what specific cloud we are talking about should be immaterial [2] This is one place in the API where we need to decide whether we want to expose the concept of 'VM image' as distinct from 'VM template' in the API

3 3

[Libcloud-list] Stateful vs Stateless VMs
by Bob McWhirter 10 Jul '09

10 Jul '09

Not to confuse with stateful vs stateless API.... As we know, EC2 has no state for the instances. If you power-off and terminate a machine, it's gone. You can't start up the exact same VM again. You *can* start up another copy of the same image, but it'll lack any changes you made, of course. And I consider that to be a Good Thing, when doing cloud computing. If some virtualization fabric provides stateful machines (ie, RHEV-M, VMware), I think it's still best to ignore that if you're truly wanting to deploy using "the cloud methodology". My view of the cloud is based upon homogeneity, particularly if you're wanting to scale up/down over time, or have instances simply fail. For example, in our JBoss demo, we start with a cluster of 2 AS nodes. As demand increases, we want to add 2 more node *just like* the existing 2 nodes. As demand falls, we want to be able to kill off any 2 of the now 4 nodes, to conserve resources. If we assume statefulness of the VMs, it seems we're no longer thinking in terms of commodity/utility computing. I do like EC2's EBS, though, for maintaining data, which should not be ephemeral. Glue that to your DB node. If the DB node fails, mount the storage partition on another copy of the DB node. Or mount it on an updated version of the DB node. Ultimately, if you develop your images (or portfolio of cooperating images) to behave well in a stateless environment, I think you'll end up follow more "best practices" in terms of fail-over, scalability and management. If you develop assuming your nodes are stateful, you're also more apt to "park" them somewhere "just in case". Or end up being too reliant upon a very particular instance as it's evolved over time. Assuming statelessness helps ensure you separate compute resources from storage resources, and have something you can spin up multiple times in a meaningful way. Makes it more cloudy above and beyond just virtualizationy. But that's just me, feel free to disagree. -Bob

4 7

[Libcloud-list] front-end/back-end authentication
by Bob McWhirter 09 Jul '09

09 Jul '09

wrt authentication... I think users should authenticate against libcloud, using a pluggable auth method. CompanyOne might have it wired up to LDAP for users to auth. CompanyTwo might use Kerb or RSA or whatever. CompanyThree might use a libcloud-specific Users table with local password hashes stored for use only with libcloud. Then a strategy to map inbound consumers of libcloud to outbound credentials for the various backend drivers. CompanyOne might have a 1:1 mapping of libcloud user to back-end user. For instance, sending *my* particularly EC2 credentials out with commands to EC2. CompanyOne handles 230 people sending in monthly expense reports for their individual AWS bills. CompanyTwo might have a many:1 mapping, where "bob" logs into libcloud, but uses a singular corporate-wide credential to fire up stuff on EC2. libcloud tracks the accounting for that. While A/P pays a single bill for $2000, they know that "bob" used $42 of it. CompanyThree might have a dept:1 mapping, where "bob" is a member of JBoss, and uses JBoss EC2 creds, and "bryan" is a member of 'Red Hat' and uses Red Hat creds. That results in a bill from AWS for each department, instead of individual. Some backends may require no authentication, where libcloud is the sole source of accounting. For internal datacenter clouds, for instance, running VMware ESXi of vSphere or whatever. "bob" used 400 hours of CPU time, and his department's budget is reduced by some arbitrarily determined "cost" per CPU hour. Depending on the proxying strategy chosen, all users may already be enrolled in the clouds, or they may have to inform libcloud "here are *my* credentials for a particular backend" before that provider is enabled for them. Next layer up, libcloud could restrict some users from using some backends. Maybe everyone can use the EC2 backend, but only IT can use the internal production cloud backend. Which also brings up another point, which I think might relate to "clusters" in RHEV-M if I understand them correctly. While we might have "the RHEV-M driver", which may have different clouds powered by that driver. The QA cluster, the IT production cluster, etc. That can wait for v2 of libcloud, though. :) -Bob

4 4

[Libcloud-list] High Level Use Cases
by Bryan Kearney 08 Jul '09

08 Jul '09

Here are some high level use cases to get us discussing the boundaries of libcloud. There are some things in here which may or may not be part of libcloud, and the anbiguity should be good to help drive the boundaries. Actor: Jerome the Application Deployer: Owns an applications, responsible for the SLA of the application Use Case: Enroll with the Cloud ------------------------------- Jerome chooses to use CloudCo as his cloud provider. Jerome enrolls with CloudCo providing them his credentials and his billing information. Jerome is provided with an additional set of credentials which he must use when authenticating with CloudCo. Use Case: Test Application ---------------------------- Jerome bundles an application with Fedora to create his test image. He loads the image into the cloud. His application is database intensive, so he launches the image and associates a data disk to the instance where DB files are to be located. He sets an FQDN for the instance, and uses this FQDN to do testing. Jerome finds an issue with the image, so he updates the image in the cloud as well as his internal gold copy. Once testing is complete, Jerome generates a new gold image and reloads a second version of the image. He then performs another round of testing. Once satisfied, Jerome leaves the instance running for 1 week for his users to sanity test the application. Use Case: Simple Deployment -------------------- Jerome is satisfied with the gold copy of the instance. Jerome clones the image to create a new image for production, and assigns a new FQDN. Jerome associates new storage with the production image for the production database. Jerome configures automated backup of the database, and sets up monitoring to inform him when the image stops working. Note: Not dealing with multi-tier applications with firewalls or clustering. Use Case: Reporting ----------------------- Jerome is required to maintain a 3 nines uptime. On the first of each month, Jerome queries for the uptime of the image in order to generate his statistics. Jerome also queries disk, bandwidth, and memory usage in order to generate his chargeback reports. Use Case: Decommissioning ------------------------ CloudCo has only allowed Jerome to achieve a 95% uptime for 3 months. Jerome decides to go with iCloud instead of CloudCo. Jerome has enrolled with iCloud,and exports his image and data files from CloudCo and loads them into iCloud. Other Ideas Not covered here ---------------------------- - granting access to images. - firewalls, LANS, - clustering - Recovery -- bk

2 5

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

deltacloud-devel July 2009