tracking cloud instances

[Seth Vidal]

So - as we noodle around with cloud instances more the most obvious 
problem I have seen is getting a list of instance ids like:

i-F7AA3F96
i-77B34039
i-B1EB403D
i-2C294684

and then trying to figure out which ones are jenkins slaves, the torrent 
seed test and the fedocal instance. And which can be nuked safely or not.

I'm thinking we need a tool that would poll the cloudlet(s), retrieve all 
the basic, available, info about the running instances.



Then admins could either add metadata to any given instance id to know 
whence things come.

Data I'd be interested in having:
- who owns it - not just the account/tenant
- what it is for
- expected expiration date (if any)
- who should have access to it (usernames from fas and or group names from 
fas, ideally) - this will make keeping ssh keys on it somewhat sane
- what, if any, configuration script was run on it (eg: an ansible 
playbook)
- published urls and where they should alias from?


Now we probably also need something that keeps a list of persistent 
instances we should always restart and register them.

for example: let's say we want one instance always running as a simple 
webserver - maybe as a touchstone to verify the cloud is always working. 
So we should be able to register this instance. Say which img it should 
use, what security group, etc and note that it should ALWAYS be running. 
Then when that instance is running its instance id/public ip should be 
registered in the db listed above.

We can use the data in the db to generate aliases, perhaps.

still fleshing out these ideas.

-sv

[Kevin Fenzi]

On Tue, 16 Oct 2012 17:02:00 -0400 (EDT)
Seth Vidal <skvidal(a)fedoraproject.org&gt; wrote:

>  So - as we noodle around with cloud instances more the most obvious 
>  problem I have seen is getting a list of instance ids like:
>  
>  i-F7AA3F96
>  i-77B34039
>  i-B1EB403D
>  i-2C294684
>  
>  and then trying to figure out which ones are jenkins slaves, the
>  torrent seed test and the fedocal instance. And which can be nuked
>  safely or not.
>  
>  I'm thinking we need a tool that would poll the cloudlet(s), retrieve
>  all the basic, available, info about the running instances. 
I agree. 

>  Then admins could either add metadata to any given instance id to
>  know whence things come.
>  
>  Data I'd be interested in having:
>  - who owns it - not just the account/tenant  
Ideally I'd like an email address we can use to contact the responsible
party. Or perhaps a primary and a backup. 

>  - what it is for
>  - expected expiration date (if any)
>  - who should have access to it (usernames from fas and or group names
>  from fas, ideally) - this will make keeping ssh keys on it somewhat
>  sane
>  - what, if any, configuration script was run on it (eg: an ansible 
>  playbook) 
If this thing ends up having a database, perhaps we could just store
history on it? when started, what playbooks, etc. 
We might also be able to get some accounting data... BW used, etc. 

>  - published urls and where they should alias from?
>
>  
>  Now we probably also need something that keeps a list of persistent 
>  instances we should always restart and register them.
>  
>  for example: let's say we want one instance always running as a
>  simple webserver - maybe as a touchstone to verify the cloud is
>  always working. So we should be able to register this instance. Say
>  which img it should use, what security group, etc and note that it
>  should ALWAYS be running. Then when that instance is running its
>  instance id/public ip should be registered in the db listed above. 
Yeah, I think that would be nice/good. 

>  We can use the data in the db to generate aliases, perhaps.
>  
>  still fleshing out these ideas. 
yeah, there's a ton of ways we could work things. I think we are just
going to have to feel them out and adjust over time as we get better at
seeing whats where and how we can use it. 

kevin

[Matthew Miller]

On Tue, Oct 16, 2012 at 05:02:00PM -0400, Seth Vidal wrote:
>  I'm thinking we need a tool that would poll the cloudlet(s),
>  retrieve all the basic, available, info about the running instances. 
Unfortunately, neither Eucalpytus or OpenStack support 'tags' in the EC2
api. That wouldn't solve the entire problem, but it's the right way to do
part of it.

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/Using_Tags.html

I bring this up not just to be annoying :) but wanted to make sure you're
aware of it when designing anything. (I think practically that means: store
things as key value pair per resource, and try to keep below Amazon's
arbitrary limit of 10 tags per resource.)


-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Seth Vidal]

On Thu, 18 Oct 2012, Matthew Miller wrote:

>  On Tue, Oct 16, 2012 at 05:02:00PM -0400, Seth Vidal wrote:
> > I'm thinking we need a tool that would poll the cloudlet(s),
> > retrieve all the basic, available, info about the running instances.
>
>  Unfortunately, neither Eucalpytus or OpenStack support 'tags' in the EC2
>  api. That wouldn't solve the entire problem, but it's the right way to do
>  part of it.
>
>  http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/Using_Tags.html
>
>  I bring this up not just to be annoying :) but wanted to make sure you're
>  aware of it when designing anything. (I think practically that means: store
>  things as key value pair per resource, and try to keep below Amazon's
>  arbitrary limit of 10 tags per resource.) 
I can say right now that keeping below 10 tags per resource is unlikely to 
happen.

And I've talked to the euca folks about tags that's not available and 
unlikely to be in 3.2

-sv

[Matthew Miller]

On Thu, Oct 18, 2012 at 06:24:10PM -0400, Seth Vidal wrote:
>  >I bring this up not just to be annoying :) but wanted to make
> sure you're
>  >aware of it when designing anything. (I think practically that means:
>  >store things as key value pair per resource, and try to keep below
>  >Amazon's arbitrary limit of 10 tags per resource.)
>  I can say right now that keeping below 10 tags per resource is
>  unlikely to happen. 
That's ten _keys_ per resource. You can have many more than ten values. Is
that still a problem?

>  And I've talked to the euca folks about tags that's not
> available
>  and unlikely to be in 3.2 
Yeah, crickets for openstack too.



-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Seth Vidal]

On Thu, 18 Oct 2012, Matthew Miller wrote:

>  On Thu, Oct 18, 2012 at 06:24:10PM -0400, Seth Vidal wrote:
> >> I bring this up not just to be annoying :) but wanted to make sure you're
> >> aware of it when designing anything. (I think practically that means:
> >> store things as key value pair per resource, and try to keep below
> >> Amazon's arbitrary limit of 10 tags per resource.)
> > I can say right now that keeping below 10 tags per resource is
> > unlikely to happen.
>
>  That's ten _keys_ per resource. You can have many more than ten values. Is
>  that still a problem? 

Ultimately, yes - and of course my other problem is I need this data 
available ACROSS reboots/rebuilds of the cloud cluster.


For our persistent instances plan I need to be able to know that I need to 
bring up 10 instances as m1.xlarge. Instance 1 needs to get this ip: 
allocate it accordingly, instance 1 needs this volume attached, etc, etc.

See where I'm going? If the instance is down that's not very functional

>  Yeah, crickets for openstack too. 
maybe cloudstack? David was telling me about their options.

-sv

[Matthew Miller]

On Thu, Oct 18, 2012 at 11:57:26PM -0400, Seth Vidal wrote:
>  For our persistent instances plan I need to be able to know that I
>  need to bring up 10 instances as m1.xlarge. Instance 1 needs to get
>  this ip: allocate it accordingly, instance 1 needs this volume
>  attached, etc, etc.
>  See where I'm going? If the instance is down that's not very functional 
Yeah. It's just particularly nice because people could easily tag their own
instances with euca-create-tags.

Have you looked at Scalr? It might be overkill here -- I don't know how well
it scales _down_.

-- 
Matthew Miller  ☁☁☁  Fedora Cloud Architect  ☁☁☁  <mattdm(a)fedoraproject.org&gt;

[Garrett Holmstrom]

On 2012-10-18 15:24, Seth Vidal wrote:
>  On Thu, 18 Oct 2012, Matthew Miller wrote:
> > Unfortunately, neither Eucalpytus or OpenStack support 'tags' in the EC2
> > api. That wouldn't solve the entire problem, but it's the right way to do
> > part of it.
> >
> > http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/Using_Tags.html
> >
> > I bring this up not just to be annoying :) but wanted to make sure you're
> > aware of it when designing anything. (I think practically that means:
> > store
> > things as key value pair per resource, and try to keep below Amazon's
> > arbitrary limit of 10 tags per resource.)
>
>  I can say right now that keeping below 10 tags per resource is unlikely
>  to happen.
>
>  And I've talked to the euca folks about tags that's not available and
>  unlikely to be in 3.2 
Indeed.  It is slated for 3.3, AFAIK.

If these tags need to persist longer than instances do then resource 
tags on individual instances probably aren't the right solution anyway 
-- they evaporate when the instances do.  :-\  Tags on auto-scaling 
groups can be more useful, but that isn't going to land until eucalyptus 
3.3 either.