Hi everyone,
I want to announce a new gogrid driver for Deltacloud. Gogrid.com provide a Cloud hosting and it's fully supported by 'right' gems (which we already using in EC2).
Patch contains a simple unit test file to test basic functionality. (cd server/libexec && ruby lib/deltacloud/drivers/gogrid_test.rb)
Unfortunately, this test will fail in destroy, reboot and stop action, because instance isn't ready for this actions yet. To test those functions, you need wait for deployment and then explicitly write a name of instance and run it again.
You could simply use this driver by applying a path (see attachment) and then start Deltacloud API server with './bin/deltacloudd gogrid'.
Of course you need install a 'right_gogrid' gem first ;-)
- Michal
On Mon, 2010-01-18 at 13:51 +0100, Michal Fojtik wrote:
I want to announce a new gogrid driver for Deltacloud. Gogrid.com provide a Cloud hosting and it's fully supported by 'right' gems (which we already using in EC2).
Very cool. Some comments inline in the patch at the end. BTW, it's best to prepare patches for submission with git-format-patch and then send them with git-send-email - inline patches make it very easy to review patches and talk about them.
Patch contains a simple unit test file to test basic functionality. (cd server/libexec && ruby lib/deltacloud/drivers/gogrid_test.rb)
This is more a general thing: I think we should get to a point where we have tests based on mock cloud interfaces, i.e. mock out right_gogrid for this driver. Also, how about using a BDD framework like rspec or cucumber (I've used rspec some, but not cucumber) - any opinions ?
We definitely need a good suite of unit tests, particulalry to ensure that drivers return XML that is right.
Some comments on the driver itself:
# Storage capacity is same on all machines (10gb), it could be extended using 'Cloud Storage' define_hardware_profile('server-with-512mb-ram') do cpu 2 memory 0.5 storage 10 architecture 'i386' end
I haven't quite made my way through the GoGrid docs, but they also support x86_64 images.
# gogrid doesn't supports realm yet (it's US/CA from whois)
Heh .. yeah, from their website they're right around the corner from where I live.
define_instance_states do start.to( :off ) .on( :create ) pending.to( :on ) .automatically running.to( :off ) .on( :reboot ) running.to( :off ) .on( :stop ) shutting_down.to( :off ) .automatically stopped.to( :off ) .automatically end
This isn't right. For example, you have a transition from start to off on create. I assume you mean start.to(:pending).on(:create) etc.
Can you include a link to the GoGrid docs that describe instance states ? (Have I mentioned I find their docs hard to follow)
David
On Tue, Jan 19, 2010 at 6:04 PM, David Lutterkort lutter@redhat.com wrote:
This is more a general thing: I think we should get to a point where we have tests based on mock cloud interfaces, i.e. mock out right_gogrid for this driver. Also, how about using a BDD framework like rspec or cucumber (I've used rspec some, but not cucumber) - any opinions ?
+1 to BDD - I know bob has things set up in dc-framework (core) for rspec already, but I don't have a strong opinion on which framework we use going forward - so long as we use one.
We definitely need a good suite of unit tests, particulalry to ensure that drivers return XML that is right.
+1. In fact, I will repeat my earlier suggestion that no patch is accepted w/o accompanying tests.
On 19/01/10 18:12 -0500, Jason Guiditta wrote:
On Tue, Jan 19, 2010 at 6:04 PM, David Lutterkort lutter@redhat.com wrote:
This is more a general thing: I think we should get to a point where we have tests based on mock cloud interfaces, i.e. mock out right_gogrid for this driver. Also, how about using a BDD framework like rspec or cucumber (I've used rspec some, but not cucumber) - any opinions ?
+1 to BDD - I know bob has things set up in dc-framework (core) for rspec already, but I don't have a strong opinion on which framework we use going forward - so long as we use one.
+1 to BDD as well. We need to choose one framework and the use it. Tests are extremely usefull especially if we do API programming.
We definitely need a good suite of unit tests, particulalry to ensure that drivers return XML that is right.
+1. In fact, I will repeat my earlier suggestion that no patch is accepted w/o accompanying tests.
+1 for your suggestion.
- Michal
On 19/01/10 15:04 -0800, David Lutterkort wrote:
Some comments on the driver itself:
# Storage capacity is same on all machines (10gb), it could be extended using 'Cloud Storage' define_hardware_profile('server-with-512mb-ram') do cpu 2 memory 0.5 storage 10 architecture 'i386' end
I haven't quite made my way through the GoGrid docs, but they also support x86_64 images.
Indeed.
So my question here is, what exactly means 'architecture' in hardware_profile ? Because in GoGrid, architecture is part of image description (32-bit|64-bit). So I just guessing, but hardware_profiles will be something like a filters for images ?
define_instance_states do start.to( :off ) .on( :create ) pending.to( :on ) .automatically running.to( :off ) .on( :reboot ) running.to( :off ) .on( :stop ) shutting_down.to( :off ) .automatically stopped.to( :off ) .automatically end
This isn't right. For example, you have a transition from start to off on create. I assume you mean start.to(:pending).on(:create) etc.
I'm sorry it was my mistake.
Can you include a link to the GoGrid docs that describe instance states ? (Have I mentioned I find their docs hard to follow)
http://wiki.gogrid.com/wiki/index.php/API:Server_(Object)
state option 1.0 The state of this server.
* To list option values, call common.lookup.list[1] with lookup set to * server.state
Returns these statuses:
On Off Starting Stopping Restarting Saving Restoring
But unfortunatelly, the only statuses that API shows are 'On' and 'Off'. I tried it several times, added sleep, forcing it manually but it always returns On or Off.
[1] http://wiki.gogrid.com/wiki/index.php/API:common.lookup.list
- Michal
On Wed, 2010-01-20 at 12:08 +0100, Michal Fojtik wrote:
On 19/01/10 15:04 -0800, David Lutterkort wrote:
Some comments on the driver itself:
# Storage capacity is same on all machines (10gb), it could be extended using 'Cloud Storage' define_hardware_profile('server-with-512mb-ram') do cpu 2 memory 0.5 storage 10 architecture 'i386' end
I haven't quite made my way through the GoGrid docs, but they also support x86_64 images.
Indeed.
So my question here is, what exactly means 'architecture' in hardware_profile ? Because in GoGrid, architecture is part of image description (32-bit|64-bit). So I just guessing, but hardware_profiles will be something like a filters for images ?
Yes, one (or the main) purpose of HWP's is to make it possible to decide whether a certain image can be run on a certain cloud. Architecture will really only become important when we have tools that can transform an image from one cloud to another, which is a ways off.
define_instance_states do start.to( :off ) .on( :create ) pending.to( :on ) .automatically running.to( :off ) .on( :reboot ) running.to( :off ) .on( :stop ) shutting_down.to( :off ) .automatically stopped.to( :off ) .automatically end
This isn't right. For example, you have a transition from start to off on create. I assume you mean start.to(:pending).on(:create) etc.
I'm sorry it was my mistake.
Can you include a link to the GoGrid docs that describe instance states ? (Have I mentioned I find their docs hard to follow)
http://wiki.gogrid.com/wiki/index.php/API:Server_(Object)
state option 1.0 The state of this server.
* To list option values, call common.lookup.list[1] with lookup set to * server.stateReturns these statuses:
On Off Starting Stopping Restarting Saving Restoring
But unfortunatelly, the only statuses that API shows are 'On' and 'Off'. I tried it several times, added sleep, forcing it manually but it always returns On or Off.
Aah .. yeah, it's even in the FAQ: http://wiki.gogrid.com/wiki/index.php/Frequently_Asked_Questions#API
For now, they only have 'On' and 'Off' as states.
David
On Wed, Jan 20, 2010 at 1:54 PM, David Lutterkort lutter@redhat.com wrote:
On Wed, 2010-01-20 at 12:08 +0100, Michal Fojtik wrote:
On 19/01/10 15:04 -0800, David Lutterkort wrote:
Some comments on the driver itself:
# Storage capacity is same on all machines (10gb), it could be extended using 'Cloud Storage' define_hardware_profile('server-with-512mb-ram') do cpu 2 memory 0.5 storage 10 architecture 'i386' end
I haven't quite made my way through the GoGrid docs, but they also support x86_64 images.
Indeed.
So my question here is, what exactly means 'architecture' in hardware_profile ? Because in GoGrid, architecture is part of image description (32-bit|64-bit). So I just guessing, but hardware_profiles will be something like a filters for images ?
Yes, one (or the main) purpose of HWP's is to make it possible to decide whether a certain image can be run on a certain cloud. Architecture will really only become important when we have tools that can transform an image from one cloud to another, which is a ways off.
define_instance_states do start.to( :off ) .on( :create ) pending.to( :on ) .automatically running.to( :off ) .on( :reboot ) running.to( :off ) .on( :stop ) shutting_down.to( :off ) .automatically stopped.to( :off ) .automatically end
This isn't right. For example, you have a transition from start to off on create. I assume you mean start.to(:pending).on(:create) etc.
I'm sorry it was my mistake.
Can you include a link to the GoGrid docs that describe instance states ? (Have I mentioned I find their docs hard to follow)
http://wiki.gogrid.com/wiki/index.php/API:Server_(Object)
state option 1.0 The state of this server.
* To list option values, call common.lookup.list[1] with lookup set to * server.state
Returns these statuses:
On Off Starting Stopping Restarting Saving Restoring
But unfortunatelly, the only statuses that API shows are 'On' and 'Off'. I tried it several times, added sleep, forcing it manually but it always returns On or Off.
Aah .. yeah, it's even in the FAQ: http://wiki.gogrid.com/wiki/index.php/Frequently_Asked_Questions#API
For now, they only have 'On' and 'Off' as states.
David
Here is another option for BDD frameworks that I have seen on several blogs, and was mentioned on the Ruby5 podcast: http://evan.tiggerpalace.com/articles/2009/09/12/coulda/
On Wed, 2010-01-20 at 12:08 +0100, Michal Fojtik wrote:
So my question here is, what exactly means 'architecture' in hardware_profile ?
Forgot to mention: in some clouds (e.g., EC2), some HWP's are only available in some architectures. For example, small instances on EC2 are 32bit, others are 64bit.
David
In my mind, a hardware profile represents the virtual hardware you can construct given a box of virtual parts.
The Virtual IT guys like to limit the possibilities, to make the virtual cost more calculable, and to limit the amount of work they have to do.
Some virtual IT shops only pre-built 5 configurations, with no options or add-ons available.
Others give you a handful of basic types, which you can customize.
The "architecture" of any HWP offered represents the architecture of the CPU. Each chunk of hardware (real or virtual) produced has a CPU (real or virtual) which supports an architecture.
The software produced (the virtual images, or appliances, or whatnot) expects to be run on a certain architecture of CPU.
So, we need to be able to map an image somehow desires to run, onto a bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
On IRC we've discussed backwards compatibility (running 32-bit images on 64-bit virtual hardware) but I think that's a distraction. Since the hardware is virtual, and we're not just picking chips out of a physical bone-yard, there is no reason to configure a 64-bit virtual hardware to execute a 32-bit image. Instead, just spin up a 32-bit virtual server to run it.
So, if you have 2 identical servers, with different virtual CPUs, I'd consider them to be 2 hardware profiles.
I don't think architecture needs to be multi-valued.
It will result in a possible doubling of available hardware profiles, but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't see too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
But I could be wrong.
-Bob
On Jan 20, 2010, at 5:16 PM, David Lutterkort wrote:
On Wed, 2010-01-20 at 12:08 +0100, Michal Fojtik wrote:
So my question here is, what exactly means 'architecture' in hardware_profile ?
Forgot to mention: in some clouds (e.g., EC2), some HWP's are only available in some architectures. For example, small instances on EC2 are 32bit, others are 64bit.
David
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
On Thu, 2010-01-21 at 11:34 -0500, Bob McWhirter wrote:
So, we need to be able to map an image somehow desires to run, onto a bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
I completely agree - I didn't mean to give the impression that we don't need arch in HWP.
It will result in a possible doubling of available hardware profiles, but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't see too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
Even if it does double the number of HWP, I don't think that's a big deal. We'll need a way to express those things in HWP's more concisely, anyway (as was discussed with the memory ballooning that Rimu supports), and in that revamped HWP description, arch will just become an enumeration.
David
Not sure if it was mentioned, but number of CPUs/cores in a given profile would be nice (it is potentially of similar importance to the architecture). Whilst with clouds as of now are not too interesting in this regard, surely that will change as more cores are crammed onto the harware that people are putting in datacentres.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it? (or am I wrong and virtualisation tech is that efficient??). If so, then # of cpus as a first class value would be nice from an apps point of view when deciding what to run/target.
On Fri, Jan 22, 2010 at 5:17 AM, David Lutterkort lutter@redhat.com wrote:
On Thu, 2010-01-21 at 11:34 -0500, Bob McWhirter wrote:
So, we need to be able to map an image somehow desires to run, onto a bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
I completely agree - I didn't mean to give the impression that we don't need arch in HWP.
It will result in a possible doubling of available hardware profiles, but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't see too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
Even if it does double the number of HWP, I don't think that's a big deal. We'll need a way to express those things in HWP's more concisely, anyway (as was discussed with the memory ballooning that Rimu supports), and in that revamped HWP description, arch will just become an enumeration.
David
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
Another good thing to add to this profile concept is location. Any serious use will eventually require resolution of resources that are local to eachother. Ex. AMIs in Amazon must be in the same region as the nodes.
I hope this helps, -Adrian
On Thu, Jan 21, 2010 at 2:17 PM, Michael Neale michael.neale@gmail.comwrote:
Not sure if it was mentioned, but number of CPUs/cores in a given profile would be nice (it is potentially of similar importance to the architecture). Whilst with clouds as of now are not too interesting in this regard, surely that will change as more cores are crammed onto the harware that people are putting in datacentres.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it? (or am I wrong and virtualisation tech is that efficient??). If so, then # of cpus as a first class value would be nice from an apps point of view when deciding what to run/target.
On Fri, Jan 22, 2010 at 5:17 AM, David Lutterkort lutter@redhat.comwrote:
On Thu, 2010-01-21 at 11:34 -0500, Bob McWhirter wrote:
So, we need to be able to map an image somehow desires to run, onto a bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
I completely agree - I didn't mean to give the impression that we don't need arch in HWP.
It will result in a possible doubling of available hardware profiles, but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't see too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
Even if it does double the number of HWP, I don't think that's a big deal. We'll need a way to express those things in HWP's more concisely, anyway (as was discussed with the memory ballooning that Rimu supports), and in that revamped HWP description, arch will just become an enumeration.
David
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
-- Michael D Neale home: www.michaelneale.net blog: michaelneale.blogspot.com
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
We have the "realm" concept which matches the datacenter concept.
But we have not linked images to realms. Perhaps we should, eh?
-Bob
On Jan 21, 2010, at 5:34 PM, Adrian Cole wrote:
Another good thing to add to this profile concept is location. Any serious use will eventually require resolution of resources that are local to eachother. Ex. AMIs in Amazon must be in the same region as the nodes.
I hope this helps, -Adrian
On Thu, Jan 21, 2010 at 2:17 PM, Michael Neale <michael.neale@gmail.com
wrote:
Not sure if it was mentioned, but number of CPUs/cores in a given profile would be nice (it is potentially of similar importance to the architecture). Whilst with clouds as of now are not too interesting in this regard, surely that will change as more cores are crammed onto the harware that people are putting in datacentres.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it? (or am I wrong and virtualisation tech is that efficient??). If so, then # of cpus as a first class value would be nice from an apps point of view when deciding what to run/target.
On Fri, Jan 22, 2010 at 5:17 AM, David Lutterkort lutter@redhat.com wrote: On Thu, 2010-01-21 at 11:34 -0500, Bob McWhirter wrote:
So, we need to be able to map an image somehow desires to run,
onto a
bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
I completely agree - I didn't mean to give the impression that we don't need arch in HWP.
It will result in a possible doubling of available hardware
profiles,
but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't
see
too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
Even if it does double the number of HWP, I don't think that's a big deal. We'll need a way to express those things in HWP's more concisely, anyway (as was discussed with the memory ballooning that Rimu supports), and in that revamped HWP description, arch will just become an enumeration.
David
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
-- Michael D Neale home: www.michaelneale.net blog: michaelneale.blogspot.com
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
+1 to linking images to "realm", The fact that amazon can't/won't use their CDN to overcome this limitation is beyond me, but in the mean time location of the image is very relevant.
I don't know how this feels to you, but another way is to make this a query.
ex in java: image.supportsRealm(realm)
wdyt?
-Adrian
On Thu, Jan 21, 2010 at 2:49 PM, Bob McWhirter bmcwhirt@redhat.com wrote:
We have the "realm" concept which matches the datacenter concept.
But we have not linked images to realms. Perhaps we should, eh?
-Bob
On Jan 21, 2010, at 5:34 PM, Adrian Cole wrote:
Another good thing to add to this profile concept is location. Any serious use will eventually require resolution of resources that are local to eachother. Ex. AMIs in Amazon must be in the same region as the nodes.
I hope this helps, -Adrian
On Thu, Jan 21, 2010 at 2:17 PM, Michael Neale michael.neale@gmail.comwrote:
Not sure if it was mentioned, but number of CPUs/cores in a given profile would be nice (it is potentially of similar importance to the architecture). Whilst with clouds as of now are not too interesting in this regard, surely that will change as more cores are crammed onto the harware that people are putting in datacentres.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it? (or am I wrong and virtualisation tech is that efficient??). If so, then # of cpus as a first class value would be nice from an apps point of view when deciding what to run/target.
On Fri, Jan 22, 2010 at 5:17 AM, David Lutterkort lutter@redhat.comwrote:
On Thu, 2010-01-21 at 11:34 -0500, Bob McWhirter wrote:
So, we need to be able to map an image somehow desires to run, onto a bag of virtual hardware to execute it.
And the Hardware Profile describes that bag.
I completely agree - I didn't mean to give the impression that we don't need arch in HWP.
It will result in a possible doubling of available hardware profiles, but I don't imagine it will. In Amazon's case, they have exactly 1 low-end profile for 32bit, and everything else is 64bit. I don't see too many 32-bit huge machines with 1TB storage and 16 virtual CPUs.
Even if it does double the number of HWP, I don't think that's a big deal. We'll need a way to express those things in HWP's more concisely, anyway (as was discussed with the memory ballooning that Rimu supports), and in that revamped HWP description, arch will just become an enumeration.
David
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
-- Michael D Neale home: www.michaelneale.net blog: michaelneale.blogspot.com
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
deltacloud-devel mailing list deltacloud-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/deltacloud-devel
On Fri, 2010-01-22 at 09:17 +1100, Michael Neale wrote:
Not sure if it was mentioned, but number of CPUs/cores in a given profile would be nice (it is potentially of similar importance to the architecture). Whilst with clouds as of now are not too interesting in this regard, surely that will change as more cores are crammed onto the harware that people are putting in datacentres.
The cpu's in HW profiles are _virtual CPU's_ - we will never have any control over how they relate to HW cores/sockets etc. I am not sure how different hypervisor's present vcpu's to the guest, but I doubt that any of it has much of an influence on how that guest performs.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it?
That's entirely up to the cloud provider - and not something we can ever expect to control over the provider's API.
David
On Fri, Jan 22, 2010 at 10:32 AM, David Lutterkort lutter@redhat.comwrote:
The cpu's in HW profiles are _virtual CPU's_ - we will never have any control over how they relate to HW cores/sockets etc. I am not sure how different hypervisor's present vcpu's to the guest, but I doubt that any of it has much of an influence on how that guest performs.
Well - the fact is they matter to a running application - I would run things differently if I know I have N cpu's available, even if they are virtual.
I am not a virtualisation expert - but is it reasonable to expect that if a chip has 20 cores, one would not create 20 VMs to dole out based on it?
That's entirely up to the cloud provider - and not something we can ever expect to control over the provider's API.
I am not saying we should prescribe, I am saying it is likely in the near future that the hardware flavours are likely to have a wider range of virtual CPUs available - and will not doubt want to offer this choice to the consumers of the clouds. There are many apps that exist to day that would prefer fewer faster CPUs, and others that would work nicely with more numerous but less powerful (virtual) CPUs. But I don't think this is a big issue, its just another field in the flavour/profile etc...
David
deltacloud-devel@lists.fedorahosted.org
-
Adrian Cole -
Bob McWhirter -
David Lutterkort -
jason.guiditta -
Michael Neale -
Michal Fojtik