"What would you say ya *do* here?"

[Greg DeKoenigsberg]

As has been pointed out, "cloud" is a very large topic.  Which means that, 
IMO, there are two directions we can go on this list that are not mutually 
exclusive: long-winded, bloviating discussions about what we *could* do, 
and detailed plans for what we *will* do.

Let's focus on the second for now, since the first is going to happen 
anyways.

There are many potential cloud goals that we could be tackling, but I 
believe that a lot of people would agree with this proposed initial goal:

LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK.

:)

Right now we face a dilemma:

   * The latest Fedora kernel available on Amazon EC2 is Fedora 8;
   * Various people are creating "Fedora images" based on that AKI;
   * Confusion results, e.g.: 
http://stackoverflow.com/questions/448794/how-does-an-amazon-ec2-instance...
   * Which (a) doesn't work and (b) gives Fedora a bad name in the process.

We have an opportunity to fix this.  Justin Forbes has been working with 
some folks at Amazon to make this better, and it looks like we'll have an 
"official" account that will allow us to do something clever, like put a 
Gold Star next to "official" Fedora EC2 images.

Once we actually get them up and running, of course.  Which, painted in 
broad strokes, maybe looks something like this:

1. Getting a basic image uploaded and working for anybody (Justin is on 
this, but maybe could use some help?);

2. Getting a basic image uploaded and working for everybody (which means 
coordinating a testing account for Fedora people to use free-of-charge, 
which we have funding for, and then finding actual people to test);

3. Getting that basic image created as part of the Fedora release process 
(which was discussed at FUDCon, but I don't know if any of those 
conversations continued);

4. Providing proper tools for people to create their own images, since 
creation of a custom "spin" is basically an essential activity in the 
cloud.

It may be that some of this activity is going on already, and that's a 
large part of what this SIG is about: to find those people and get them 
talking in one place, away from the clutter of fedora-devel and other 
lists.  So if you're working on tools that will help with this goal, by 
all means, speak up.

And now, a couple of questions.

* For Justin.  How's your current image looking?  Does it work well enough 
for other folks to try it out?  If not, what's the issue, and can anyone 
help?

* For everyone.  Have you played with EC2 yet?  If not, do you have a 
side project that could benefit from its use, that furthers the goals of 
Fedora / free software / etc., and are willing to test our images?  If so, 
let us know -- we've got an account for this purpose, and can send you the 
info needed to get started.  Even if we don't have the latest Fedora up 
yet, there are plenty of images that you can play with to get a sense of 
how EC2 works.

* For everyone.  Would an IRC meeting to talk through some of this stuff 
be useful -- sort of a kickoff where everyone takes an hour out of their 
life to think about all this -- or should we stick to the mailing list for 
now?

--g

--
Computer Science professors should be teaching open source.
Help make it happen.   Visit http://teachingopensource.org.

[David Huff]

On 01/20/2010 08:50 AM, Greg DeKoenigsberg wrote:
>  LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK.
>
>  :)
>
>  Right now we face a dilemma:
>
>      * The latest Fedora kernel available on Amazon EC2 is Fedora 8;
>      * Various people are creating "Fedora images" based on that AKI;
>      * Confusion results, e.g.:
>
> http://stackoverflow.com/questions/448794/how-does-an-amazon-ec2-instance...
>      * Which (a) doesn't work and (b) gives Fedora a bad name in the process.
>
>  We have an opportunity to fix this.  Justin Forbes has been working with
>  some folks at Amazon to make this better, and it looks like we'll have an
>  "official" account that will allow us to do something clever, like put a
>  Gold Star next to "official" Fedora EC2 images.
>
>  Once we actually get them up and running, of course.  Which, painted in
>  broad strokes, maybe looks something like this:
>
>  1. Getting a basic image uploaded and working for anybody (Justin is on
>  this, but maybe could use some help?); 
So there are really several parts here....

a. create ami: Amazon Machine Image -> pre installed fedora image.  This 
is the easy part.  There are several tools for building pre-installed 
images form a ks file.  I have been working with the SPIN sig to define 
a minimal ks file (AOS) that describes minimal Fedora Image with dhcp, 
yum, selinux, that would work here.  We are also currently working to 
include a fedora-mini.ks file that can me extended for other "SPINS".

b. create aki, Amazon Machine Image, In the past I have just copied the 
vmlinuz file form a running image.  Is there a better more reliable way 
to do this?

c. create ari, Amazon Ramdisk Image. this needs a custom fstab per 
machine type, m1.small(32bit) vs. m1.large (64bit) etc, and needs the 
xen modules preloaded. Is there a good way to automate this?

d. edit image attributes of all of the above to 1)link ramdisk and 
kernel to ami, and 2)share with the world

I would be willing to help out here a much as possible however we need 
suggestions of good ways of automating the above.

>  2. Getting a basic image uploaded and working for everybody (which
> means
>  coordinating a testing account for Fedora people to use free-of-charge,
>  which we have funding for, and then finding actual people to test); 
I can help test if needed

>  3. Getting that basic image created as part of the Fedora release process
>  (which was discussed at FUDCon, but I don't know if any of those
>  conversations continued); 
This is just a matter of getting the stuff I described above into the 
Fedora release process which I am not real familiar with.  I have some 
suggestions here, related to how we build the ovirt node, but will wait 
to see if others have better ideas first.

>  4. Providing proper tools for people to create their own images,
> since
>  creation of a custom "spin" is basically an essential activity in the
>  cloud. 
Im not going to touch this yet however I have some good ideas here as well.

>  It may be that some of this activity is going on already, and
> that's a
>  large part of what this SIG is about: to find those people and get them
>  talking in one place, away from the clutter of fedora-devel and other
>  lists.  So if you're working on tools that will help with this goal, by
>  all means, speak up.
>
>  And now, a couple of questions.
>
>  * For Justin.  How's your current image looking?  Does it work well enough
>  for other folks to try it out?  If not, what's the issue, and can anyone
>  help?
>
>  * For everyone.  Have you played with EC2 yet?  If not, do you have a
>  side project that could benefit from its use, that furthers the goals of
>  Fedora / free software / etc., and are willing to test our images?  If so,
>  let us know -- we've got an account for this purpose, and can send you the
>  info needed to get started.  Even if we don't have the latest Fedora up
>  yet, there are plenty of images that you can play with to get a sense of
>  how EC2 works. 
Yes, and I would like to see the AOS/thincrust project be the base of 
these images.

>  * For everyone.  Would an IRC meeting to talk through some of this
> stuff
>  be useful -- sort of a kickoff where everyone takes an hour out of their
>  life to think about all this -- or should we stick to the mailing list for
>  now? 
I could go either way here

-D

[Mike McGrath]

On Wed, 20 Jan 2010, David Huff wrote:

>  On 01/20/2010 08:50 AM, Greg DeKoenigsberg wrote:
>  > LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK.
>  >
>  > :)
>  >
>  > Right now we face a dilemma:
>  >
>  >     * The latest Fedora kernel available on Amazon EC2 is Fedora 8;
>  >     * Various people are creating "Fedora images" based on that AKI;
>  >     * Confusion results, e.g.:
>  >
> http://stackoverflow.com/questions/448794/how-does-an-amazon-ec2-instance...
>  >     * Which (a) doesn't work and (b) gives Fedora a bad name in the process.
>  >
>  > We have an opportunity to fix this.  Justin Forbes has been working with
>  > some folks at Amazon to make this better, and it looks like we'll have an
>  > "official" account that will allow us to do something clever, like put a
>  > Gold Star next to "official" Fedora EC2 images.
>  >
>  > Once we actually get them up and running, of course.  Which, painted in
>  > broad strokes, maybe looks something like this:
>  >
>  > 1. Getting a basic image uploaded and working for anybody (Justin is on
>  > this, but maybe could use some help?);
>
>  So there are really several parts here....
>
>  a. create ami: Amazon Machine Image -> pre installed fedora image.  This
>  is the easy part.  There are several tools for building pre-installed
>  images form a ks file.  I have been working with the SPIN sig to define
>  a minimal ks file (AOS) that describes minimal Fedora Image with dhcp,
>  yum, selinux, that would work here.  We are also currently working to
>  include a fedora-mini.ks file that can me extended for other "SPINS".
>
>  b. create aki, Amazon Machine Image, In the past I have just copied the
>  vmlinuz file form a running image.  Is there a better more reliable way
>  to do this?
>
>  c. create ari, Amazon Ramdisk Image. this needs a custom fstab per
>  machine type, m1.small(32bit) vs. m1.large (64bit) etc, and needs the
>  xen modules preloaded. Is there a good way to automate this?
>
>  d. edit image attributes of all of the above to 1)link ramdisk and
>  kernel to ami, and 2)share with the world
>
>  I would be willing to help out here a much as possible however we need
>  suggestions of good ways of automating the above.
>
>  > 2. Getting a basic image uploaded and working for everybody (which means
>  > coordinating a testing account for Fedora people to use free-of-charge,
>  > which we have funding for, and then finding actual people to test);
>
>  I can help test if needed
>
>  >
>  > 3. Getting that basic image created as part of the Fedora release process
>  > (which was discussed at FUDCon, but I don't know if any of those
>  > conversations continued);
>
>  This is just a matter of getting the stuff I described above into the
>  Fedora release process which I am not real familiar with.  I have some
>  suggestions here, related to how we build the ovirt node, but will wait
>  to see if others have better ideas first.
>  
Lets start with the first thing that's blocked.  How far can we get with
the above list today before the first "I don't know how to.." gets hit?

	-Mike

[Mike McGrath]

On Wed, 20 Jan 2010, Mike McGrath wrote:

>  On Wed, 20 Jan 2010, David Huff wrote:
>
>  > On 01/20/2010 08:50 AM, Greg DeKoenigsberg wrote:
>  > > LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK.
>  > >
>  > > :)
>  > >
>  > > Right now we face a dilemma:
>  > >
>  > >     * The latest Fedora kernel available on Amazon EC2 is Fedora 8;
>  > >     * Various people are creating "Fedora images" based on that AKI;
>  > >     * Confusion results, e.g.:
>  > >
> http://stackoverflow.com/questions/448794/how-does-an-amazon-ec2-instance...
>  > >     * Which (a) doesn't work and (b) gives Fedora a bad name in the
> process.
>  > >
>  > > We have an opportunity to fix this.  Justin Forbes has been working with
>  > > some folks at Amazon to make this better, and it looks like we'll have an
>  > > "official" account that will allow us to do something clever, like
> put a
>  > > Gold Star next to "official" Fedora EC2 images.
>  > >
>  > > Once we actually get them up and running, of course.  Which, painted in
>  > > broad strokes, maybe looks something like this:
>  > >
>  > > 1. Getting a basic image uploaded and working for anybody (Justin is on
>  > > this, but maybe could use some help?);
>  >
>  > So there are really several parts here....
>  >
>  > a. create ami: Amazon Machine Image -> pre installed fedora image.  This
>  > is the easy part.  There are several tools for building pre-installed
>  > images form a ks file.  I have been working with the SPIN sig to define
>  > a minimal ks file (AOS) that describes minimal Fedora Image with dhcp,
>  > yum, selinux, that would work here.  We are also currently working to
>  > include a fedora-mini.ks file that can me extended for other "SPINS".
>  >
>  > b. create aki, Amazon Machine Image, In the past I have just copied the
>  > vmlinuz file form a running image.  Is there a better more reliable way
>  > to do this?
>  >
>  > c. create ari, Amazon Ramdisk Image. this needs a custom fstab per
>  > machine type, m1.small(32bit) vs. m1.large (64bit) etc, and needs the
>  > xen modules preloaded. Is there a good way to automate this?
>  >
>  > d. edit image attributes of all of the above to 1)link ramdisk and
>  > kernel to ami, and 2)share with the world
>  >
>  > I would be willing to help out here a much as possible however we need
>  > suggestions of good ways of automating the above.
>  >
>  > > 2. Getting a basic image uploaded and working for everybody (which means
>  > > coordinating a testing account for Fedora people to use free-of-charge,
>  > > which we have funding for, and then finding actual people to test);
>  >
>  > I can help test if needed
>  >
>  > >
>  > > 3. Getting that basic image created as part of the Fedora release process
>  > > (which was discussed at FUDCon, but I don't know if any of those
>  > > conversations continued);
>  >
>  > This is just a matter of getting the stuff I described above into the
>  > Fedora release process which I am not real familiar with.  I have some
>  > suggestions here, related to how we build the ovirt node, but will wait
>  > to see if others have better ideas first.
>  >
>
>  Lets start with the first thing that's blocked.  How far can we get with
>  the above list today before the first "I don't know how to.." gets hit?
>  
Related to this...

http://fedoraproject.org/wiki/Features
http://fedoraproject.org/wiki/Schedule

If this is going to be a feature (and require any changes to our process
or features) It needs to be submitted as a feature no later then the 26th.
That's pretty close.

	-Mike

[Mike McGrath]

On Wed, 20 Jan 2010, Mike McGrath wrote:

>  On Wed, 20 Jan 2010, Mike McGrath wrote:
>
>  > On Wed, 20 Jan 2010, David Huff wrote:
>  >
>  > > On 01/20/2010 08:50 AM, Greg DeKoenigsberg wrote:
>  > > > LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK.
>  > > >
>  > > > :)
>  > > >
>  > > > Right now we face a dilemma:
>  > > >
>  > > >     * The latest Fedora kernel available on Amazon EC2 is Fedora 8;
>  > > >     * Various people are creating "Fedora images" based on that
> AKI;
>  > > >     * Confusion results, e.g.:
>  > > >
> http://stackoverflow.com/questions/448794/how-does-an-amazon-ec2-instance...
>  > > >     * Which (a) doesn't work and (b) gives Fedora a bad name in the
> process.
>  > > >
>  > > > We have an opportunity to fix this.  Justin Forbes has been working with
>  > > > some folks at Amazon to make this better, and it looks like we'll have
> an
>  > > > "official" account that will allow us to do something clever,
> like put a
>  > > > Gold Star next to "official" Fedora EC2 images.
>  > > >
>  > > > Once we actually get them up and running, of course.  Which, painted in
>  > > > broad strokes, maybe looks something like this:
>  > > >
>  > > > 1. Getting a basic image uploaded and working for anybody (Justin is on
>  > > > this, but maybe could use some help?);
>  > >
>  > > So there are really several parts here....
>  > >
>  > > a. create ami: Amazon Machine Image -> pre installed fedora image.  This
>  > > is the easy part.  There are several tools for building pre-installed
>  > > images form a ks file.  I have been working with the SPIN sig to define
>  > > a minimal ks file (AOS) that describes minimal Fedora Image with dhcp,
>  > > yum, selinux, that would work here.  We are also currently working to
>  > > include a fedora-mini.ks file that can me extended for other
> "SPINS".
>  > >
>  > > b. create aki, Amazon Machine Image, In the past I have just copied the
>  > > vmlinuz file form a running image.  Is there a better more reliable way
>  > > to do this?
>  > >
>  > > c. create ari, Amazon Ramdisk Image. this needs a custom fstab per
>  > > machine type, m1.small(32bit) vs. m1.large (64bit) etc, and needs the
>  > > xen modules preloaded. Is there a good way to automate this?
>  > >
>  > > d. edit image attributes of all of the above to 1)link ramdisk and
>  > > kernel to ami, and 2)share with the world
>  > >
>  > > I would be willing to help out here a much as possible however we need
>  > > suggestions of good ways of automating the above.
>  > >
>  > > > 2. Getting a basic image uploaded and working for everybody (which means
>  > > > coordinating a testing account for Fedora people to use free-of-charge,
>  > > > which we have funding for, and then finding actual people to test);
>  > >
>  > > I can help test if needed
>  > >
>  > > >
>  > > > 3. Getting that basic image created as part of the Fedora release process
>  > > > (which was discussed at FUDCon, but I don't know if any of those
>  > > > conversations continued);
>  > >
>  > > This is just a matter of getting the stuff I described above into the
>  > > Fedora release process which I am not real familiar with.  I have some
>  > > suggestions here, related to how we build the ovirt node, but will wait
>  > > to see if others have better ideas first.
>  > >
>  >
>  > Lets start with the first thing that's blocked.  How far can we get with
>  > the above list today before the first "I don't know how to.." gets
> hit?
>  >
>
>  Related to this...
>
>  http://fedoraproject.org/wiki/Features
>  http://fedoraproject.org/wiki/Schedule
>
>  If this is going to be a feature (and require any changes to our process
>  or features) It needs to be submitted as a feature no later then the 26th.
>  That's pretty close.
>  
Have we determined we don't need to go through the Features process?
Who's actually owning this task to verify that?

	-Mike

[Jeremy Katz]

On Wed, Jan 20, 2010 at 10:45 AM, David Huff <dhuff(a)redhat.com&gt; wrote:
>  On 01/20/2010 08:50 AM, Greg DeKoenigsberg wrote:
> > LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK. 
[snip]
> > 1. Getting a basic image uploaded and working for anybody (Justin
> is on
> > this, but maybe could use some help?);
>
>  So there are really several parts here....
>
>  a. create ami: Amazon Machine Image -> pre installed fedora image.  This
>  is the easy part.  There are several tools for building pre-installed
>  images form a ks file.  I have been working with the SPIN sig to define
>  a minimal ks file (AOS) that describes minimal Fedora Image with dhcp,
>  yum, selinux, that would work here.  We are also currently working to
>  include a fedora-mini.ks file that can me extended for other "SPINS". 
Yep, this is relatively straight-forward and we have all the tools
that are really needed to create the filesystem image that runs
between python-imgcreate/livecd-creator and the things built on top of
it.  (Note: I'd really like the whole AOS/JEOS nomenclature to go
away; it's an image just like any other image.  Maybe one of them is
minimal, maybe another is not.  There's nothing particularly
"appliance" about it)

>  b. create aki, Amazon Machine Image, In the past I have just copied
> the
>  vmlinuz file form a running image.  Is there a better more reliable way
>  to do this? 
This is basically what I gather

>  c. create ari, Amazon Ramdisk Image. this needs a custom fstab per
>  machine type, m1.small(32bit) vs. m1.large (64bit) etc, and needs the
>  xen modules preloaded. Is there a good way to automate this? 
The ramdisk _should_ just be able to be a standard Fedora ramdisk.  If
not, we have a problem.  The whole custom fstab and module preloading
blah blah blah is a side-effect of the pretty poor way the stock
Amazon images are constructed without partitioning, etc.  All of our
standard system tools can bring up Xen guests and once it's booted, it
should look like a regular system.

>  d. edit image attributes of all of the above to 1)link ramdisk and
>  kernel to ami, and 2)share with the world
>
>  I would be willing to help out here a much as possible however we need
>  suggestions of good ways of automating the above. 
Once we're just using standard things built by our standard tools,
this is straight-forward enough to automate :-)

> > 2. Getting a basic image uploaded and working for everybody
> (which means
> > coordinating a testing account for Fedora people to use free-of-charge,
> > which we have funding for, and then finding actual people to test);
>
>  I can help test if needed 
I can also help testing and have real systems where I'm using the
ancient Fedora stuff in production on EC2 that I'd be happy to have
running on a newer Fedora :)


- Jeremy

[Jesse Keating]

On Wed, 2010-01-20 at 08:50 -0500, Greg DeKoenigsberg wrote:
>  * For everyone.  Have you played with EC2 yet?  
Not yet.

>   If not, do you have a 
>  side project that could benefit from its use, that furthers the goals of 
>  Fedora / free software / etc., and are willing to test our images?  
Not yet.

>   If so, 
>  let us know -- we've got an account for this purpose, and can send you the 
>  info needed to get started.  Even if we don't have the latest Fedora up 
>  yet, there are plenty of images that you can play with to get a sense of 
>  how EC2 works. 
I'm a bit more interested in setting up my own clouds, rather than using
existing ones.  But we would still have to generate stuff and work with
a cloud in any case.

>  
>  * For everyone.  Would an IRC meeting to talk through some of this stuff 
>  be useful -- sort of a kickoff where everyone takes an hour out of their 
>  life to think about all this -- or should we stick to the mailing list for 
>  now?  
I'm down with either, although IRC would be much higher bandwidth.

-- 
Jesse Keating
Fedora -- Freedom² is a feature!
identi.ca: http://identi.ca/jkeating

[Greg DeKoenigsberg]

On Wed, 20 Jan 2010, Jesse Keating wrote:

>  I'm a bit more interested in setting up my own clouds, rather
> than using
>  existing ones.  But we would still have to generate stuff and work with
>  a cloud in any case. 
So this is a complicated question -- one that I think deserves some frank 
discussion.

We have a project called oVirt, but no one is actively working on it, 
because...

We've got a product called RHEV-M that came from the Qumranet acquisition, 
but it's not yet open source.  A large team is actively working on this 
problem, but it's going to take them a while.  Maybe quite a while.

We've got a project called Deltacloud, the purpose of which is to create 
an abstraction layer around cloud management, much as libvirt creates an 
abstraction layer around virt management -- and drivers still need to be 
written for both public cloud providers and private cloud tools.  One of 
the reasons for understanding EC2 really well is to gain a deep 
understanding of the use cases for Deltacloud; a useful goal is to manage 
all Fedora/community EC2 interactions with the Deltacloud API instead of 
the Amazon EC2 API, and then test our ability to move workloads from EC2 
to, say, Rackspace or IBM or whomever.

We've got some homegrown stuff in Fedora that might be useful for 
management of private clouds -- I know mmcgrath has some tricks up his 
sleeve.  Some subset of that functionality could sit beneath Deltacloud 
drivers -- it might be worthwhile for someone to start looking at that.

None of this stuff is yet critical path, IMO -- but once we've got "Fedora 
for EC2" solved, those questions become immediately Very Large.  And right 
now, we've got way more questions than answers.

I hope this list is a place where we can talk through some of this stuff.

--g

--
Computer Science professors should be teaching open source.
Help make it happen.   Visit http://teachingopensource.org.

[Jan Pazdziora]

On Wed, Jan 20, 2010 at 08:50:04AM -0500, Greg DeKoenigsberg wrote:
>  
>  There are many potential cloud goals that we could be tackling, but I 
>  believe that a lot of people would agree with this proposed initial goal:
>  
>  LET'S CREATE FEDORA EC2 IMAGES THAT DON'T SUCK. 
Yes.

>  * For everyone.  Have you played with EC2 yet?  If not, do you have a
>
>  side project that could benefit from its use, that furthers the goals of 
>  Fedora / free software / etc., and are willing to test our images?  If so,  
I played with it a bit, and I have a project which would benefit from
EC2.

>  * For everyone.  Would an IRC meeting to talk through some of this
> stuff 
>  be useful -- sort of a kickoff where everyone takes an hour out of their 
>  life to think about all this -- or should we stick to the mailing list for 
>  now? 
I prefer to avoid IRC.

-- 
Jan Pazdziora
Principal Software Engineer, Satellite Engineering, Red Hat