4:22 a.m.
// Deploying fedora infrastructure (koji) across clouds
// As promised, steps to deploy kojihub to an openstack instance
communicating with builders on ec2
// Any provider supported by deltacloud will work
// (lots of em: http://deltacloud.apache.org/drivers.html)
// A short video of these steps in action can be seen here:
// http://youtu.be/qF2ctg7ItNc
// install deltacloud & drivers
$ sudo yum install wget deltacloud-core deltacloud-core-openstack
deltacloud-core-ec2
// start it
$ deltacloudd -i mock
// small wrapper script around deltacloud:
$ wget https://raw.github.com/movitto/mycloud/master/mycloud.rb
$ chmod +x mycloud.rb
// template describing kojihub cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// template describing kojibuild cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// edit kojihub deployment to contain openstack credentials
$ vim koji_f17.xml
// startup an instance on openstack w/ kojihub setup togo
$ ./mycloud.rb koji_f17.xml
// grab public address from output, grab kojibuild ssl credentials from
new instance
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/kojiadmin.pem .
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/koji_ca_cert.crt .
// edit kojibuild deployment to deploy to ec2 w/ correct koji
credentials & hub address
$ vim koji_builder_f17.xml
// startup an instance on ec2 w/ kojibuild communicating w/ the hub
$ ./mycloud.rb koji_builder_f17.xml
// open up a webbrowser, navigating to http://kojihub/koji to use your
Koji instance!
-Mo
11:40 a.m.
----- Original Message -----
From: "Mo Morsi" <mmorsi(a)redhat.com>
To: "aeolus-devel" <aeolus-devel(a)lists.fedorahosted.org>, "Fedora
Cloud SIG" <cloud(a)lists.fedoraproject.org>,
"Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Wednesday, October 31, 2012 2:22:47 AM
Subject: Deploying fedora infrastructure (koji) across clouds
// Deploying fedora infrastructure (koji) across clouds
// As promised, steps to deploy kojihub to an openstack instance
communicating with builders on ec2
// Any provider supported by deltacloud will work
// (lots of em: http://deltacloud.apache.org/drivers.html)
// A short video of these steps in action can be seen here:
// http://youtu.be/qF2ctg7ItNc
// install deltacloud & drivers
$ sudo yum install wget deltacloud-core deltacloud-core-openstack
deltacloud-core-ec2
// start it
$ deltacloudd -i mock
// small wrapper script around deltacloud:
$ wget https://raw.github.com/movitto/mycloud/master/mycloud.rb
$ chmod +x mycloud.rb
// template describing kojihub cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// template describing kojibuild cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// edit kojihub deployment to contain openstack credentials
$ vim koji_f17.xml
// startup an instance on openstack w/ kojihub setup togo
$ ./mycloud.rb koji_f17.xml
// grab public address from output, grab kojibuild ssl credentials
from
new instance
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/kojiadmin.pem .
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/koji_ca_cert.crt .
// edit kojibuild deployment to deploy to ec2 w/ correct koji
credentials & hub address
$ vim koji_builder_f17.xml
// startup an instance on ec2 w/ kojibuild communicating w/ the hub
$ ./mycloud.rb koji_builder_f17.xml
// open up a webbrowser, navigating to http://kojihub/koji to use
your
Koji instance!
A few questions:
#1: How does this take things like ARM or other archs into the picture - ie: I am guessing
we can't really build ARM on ec2? :)
#2: Could there be a way to take a (working) nightly build, build one's package
against that nightly in a personal build of some sort, and somehow have a verification
process that it built in that "personal build" before it goes into rawhide, etc?
(or even... unit tests, etc.)?
I'm mostly thinking about things like "how to not have perpetually broken
rawhide" (avoid checking in things that will likely break the build in the first
place).
Re: the endless anaconda-killing-f18 thread: I know there's been some discussion about
whether or not the devel process really accommodates what needs to happen with something
like a full-blown anaconda re-write - and while I know that "THE CLOUD" is not
the entire solution to that (there are obviously, as others have graciously pointed out,
many other feature/fesco/planning/etc processes intertwined here that also need love) - it
seems like having these capabilities might fit in to a solution, or at least, something on
the road to a better devel/build process.
There are plenty of projects doing CI/CD - and having cloud infra makes this significantly
easier to enable - though obviously not a lot of cases of people doing an OS.
-robyn
-Mo
_______________________________________________
cloud mailing list
cloud(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/cloud
11:59 a.m.
On 10/31/2012 12:40 PM, Robyn Bergeron wrote:
----- Original Message -----
> From: "Mo Morsi" <mmorsi(a)redhat.com>
> To: "aeolus-devel" <aeolus-devel(a)lists.fedorahosted.org>,
"Fedora Cloud SIG" <cloud(a)lists.fedoraproject.org>,
> "Development discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
> Sent: Wednesday, October 31, 2012 2:22:47 AM
> Subject: Deploying fedora infrastructure (koji) across clouds
>
> // Deploying fedora infrastructure (koji) across clouds
> // As promised, steps to deploy kojihub to an openstack instance
> communicating with builders on ec2
> // Any provider supported by deltacloud will work
> // (lots of em: http://deltacloud.apache.org/drivers.html)
>
> // A short video of these steps in action can be seen here:
> // http://youtu.be/qF2ctg7ItNc
>
A few questions:
#1: How does this take things like ARM or other archs into the picture - ie: I am
guessing we can't really build ARM on ec2? :)
No ARM on EC2, but you can mix and match clouds as you will. Providers
are now just config options so you can easily add and change the
environments and resources that you leverage anytime you want.
There's really a world of possibilities available, we just have to
identify the correct level of abstraction the Fedora community is
looking for.
#2: Could there be a way to take a (working) nightly build, build one's package
against that nightly in a personal build of some sort, and somehow have a verification
process that it built in that "personal build" before it goes into rawhide, etc?
(or even... unit tests, etc.)?
Absolutely, repositories and packages can be added on the fly, and we
can incorporate any image already pushed to the cloud as well as build
new ones for our purposes.
Even what I demoed in the screencast (just the tip of the iceburg
concerning Deltacloud's capabilities) can be automated such that new
cloud instances w/ any stack we want can be provisioned on demand
anywhere we want.
I'm mostly thinking about things like "how to not have perpetually broken
rawhide" (avoid checking in things that will likely break the build in the first
place).
Really the workflow at this point is the only thing vague to me, once
that's down I'm sure it can be implemented.
Perhaps people can send their thoughts or we can discuss it at the next
cloud sig meeting? (Wednesdays at 10AM EST on #fedora-meeting-1 for
those that are interested)
Re: the endless anaconda-killing-f18 thread: I know there's been some discussion
about whether or not the devel process really accommodates what needs to happen with
something like a full-blown anaconda re-write - and while I know that "THE
CLOUD" is not the entire solution to that (there are obviously, as others have
graciously pointed out, many other feature/fesco/planning/etc processes intertwined here
that also need love) - it seems like having these capabilities might fit in to a solution,
or at least, something on the road to a better devel/build process.
There are plenty of projects doing CI/CD - and having cloud infra makes this
significantly easier to enable - though obviously not a lot of cases of people doing an
OS.
-robyn
I've said it before and I'll say it again, the cloud is _not_ the be all
end all. :-) Its a great tool, and brings a powerful computational
resource to the table, but it's never going to completely mitigate local
infrastructure.
I think hybrid technologies is the future, being able to leverage cloud
resources in addition to and along side of local ones seemlessly in an
open manner will really enable us to do some really cool things. I
believe Fedora can make great headway and lead on this front, we just
have to find what works for people and do it! :-)
-Mo
12:10 p.m.
On Wed, 31 Oct 2012, Mo Morsi wrote:
> #2: Could there be a way to take a (working) nightly build, build
one's
> package against that nightly in a personal build of some sort, and
> somehow have a verification process that it built in that "personal
> build" before it goes into rawhide, etc? (or even... unit tests, etc.)?
Absolutely, repositories and packages can be added on the fly, and we
can incorporate any image already pushed to the cloud as well as build
new ones for our purposes.
Is this a new feature in koji, then? B/c in the past adding repositories
has been a major limiting factor in koji. Especially untrusted, remote
repositories.
I think hybrid technologies is the future, being able to leverage
cloud
resources in addition to and along side of local ones seemlessly in an
open manner will really enable us to do some really cool things. I
believe Fedora can make great headway and lead on this front, we just
have to find what works for people and do it! :-)
Have you been following what the infrastructure team has been doing with
our private cloud instances and deployment/provisioning there?
some info here if you're interested:
https://fedoraproject.org/wiki/Infrastructure_private_cloud
-sv
7:33 a.m.
On 10/31/2012 01:10 PM, Seth Vidal wrote:
On Wed, 31 Oct 2012, Mo Morsi wrote:
>> #2: Could there be a way to take a (working) nightly build, build
>> one's package against that nightly in a personal build of some sort,
>> and somehow have a verification process that it built in that
>> "personal build" before it goes into rawhide, etc? (or even... unit
>> tests, etc.)?
>
> Absolutely, repositories and packages can be added on the fly, and we
> can incorporate any image already pushed to the cloud as well as build
> new ones for our purposes.
Is this a new feature in koji, then? B/c in the past adding
repositories has been a major limiting factor in koji. Especially
untrusted, remote repositories.
Hrm I was refering more to the repos necessary to bootstrap the cloud
instance for the koji builder. Which component imposes this limitation
on koji, the hub or the builder? Would being able to build custom cloud
images on the fly for different clouds assist with this? We are able to
do that w/ our imagefactory / oz tools (written in Python incidently [1][2])
-Mo
[1] https://github.com/aeolusproject/imagefactory
[2] https://github.com/clalancette/oz
7:34 a.m.
On 11/01/2012 08:33 AM, Mo Morsi wrote:
On 10/31/2012 01:10 PM, Seth Vidal wrote:
>
>
> On Wed, 31 Oct 2012, Mo Morsi wrote:
>
>>> #2: Could there be a way to take a (working) nightly build, build
>>> one's package against that nightly in a personal build of some sort,
>>> and somehow have a verification process that it built in that
>>> "personal build" before it goes into rawhide, etc? (or even...
unit
>>> tests, etc.)?
>> Absolutely, repositories and packages can be added on the fly, and we
>> can incorporate any image already pushed to the cloud as well as build
>> new ones for our purposes.
> Is this a new feature in koji, then? B/c in the past adding
> repositories has been a major limiting factor in koji. Especially
> untrusted, remote repositories.
>
Hrm I was refering more to the repos necessary to bootstrap the cloud
instance for the koji builder. Which component imposes this limitation
on koji, the hub or the builder? Would being able to build custom cloud
images on the fly for different clouds assist with this? We are able to
do that w/ our imagefactory / oz tools (written in Python incidently [1][2])
-Mo
[1] https://github.com/aeolusproject/imagefactory
[2] https://github.com/clalancette/oz
Heh, speak about timing, imagefactory was just hosted on the latest
FLOSS Weekly:
http://www.youtube.com/watch?v=H5z-tpYS0Ng
-Mo
8:40 a.m.
On Thu, 1 Nov 2012, Mo Morsi wrote:
Hrm I was refering more to the repos necessary to bootstrap the cloud
instance for the koji builder.
I see - I thought you were referring to package repos.
Which component imposes this limitation
on koji, the hub or the builder?
the hub.
Would being able to build custom cloud
images on the fly for different clouds assist with this?
No - where it is built has nothing to do with it.
-sv
12:07 p.m.
// small wrapper script around deltacloud:
$ wget https://raw.github.com/movitto/mycloud/master/mycloud.rb
$ chmod +x mycloud.rb
// template describing kojihub cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// template describing kojibuild cloud deployment
$ wget
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
// edit kojihub deployment to contain openstack credentials
$ vim koji_f17.xml
// startup an instance on openstack w/ kojihub setup togo
$ ./mycloud.rb koji_f17.xml
// grab public address from output, grab kojibuild ssl credentials from
new instance
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/kojiadmin.pem .
$ scp -i ~/os.key ec2-user@kojihub:/etc/pki/koji/koji_ca_cert.crt .
// edit kojibuild deployment to deploy to ec2 w/ correct koji
credentials & hub address
$ vim koji_builder_f17.xml
// startup an instance on ec2 w/ kojibuild communicating w/ the hub
$ ./mycloud.rb koji_builder_f17.xml
// open up a webbrowser, navigating to http://kojihub/koji to use your
Koji instance!
Mo,
Interesting!
You can orchestrate all of these steps across/between multiple systems
using ansible: http://ansible.cc - I've been documenting spinning up and
provisioning instances on my blog in the last week or so. You might take a
look - it should solve the problem of the above needing to be so manual
of a process and it requires nothing other than ssh be installed on the
machine you're trying to configure/control.
The problems we have encountered in the fedora infrastructure with koji
builders are:
1. having to bring them up and down while waiting for them to complete
build process (koji enable-host/disable host)
2. them having no way to recover a build w/o manual intervention if a
builder were to crash or go dead during a build
3. the way the builders connect to the hub to get jobs rather than pushing
out from the hub to the builders. Mainly this makes it a pain to deal with
1 and 2 above.
4. For completely arbitrary repositories the tagging process for koji
seems pretty cumbersome, especially for transient scratch/chain builds.
I've talked about some of these on the buildsys list
(buildsys(a)lists.fedoraproject.org)
If you're interested in discussing this further drop by there.
Thanks,
-sv
7:32 a.m.
On 10/31/2012 01:07 PM, Seth Vidal wrote:
> // small wrapper script around deltacloud:
> $ wget https://raw.github.com/movitto/mycloud/master/mycloud.rb
> $ chmod +x mycloud.rb
>
> // template describing kojihub cloud deployment
> $ wget
>
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
>
>
> // template describing kojibuild cloud deployment
> $ wget
>
https://raw.github.com/aeolus-incubator/templates/master/fedora_infra/koj...
>
>
> // edit kojihub deployment to contain openstack credentials
> $ vim koji_f17.xml
>
> // startup an instance on openstack w/ kojihub setup togo
> $ ./mycloud.rb koji_f17.xml
>
Mo,
Interesting!
You can orchestrate all of these steps across/between multiple systems
using ansible: http://ansible.cc - I've been documenting spinning up and
provisioning instances on my blog in the last week or so. You might
take a
look - it should solve the problem of the above needing to be so manual
of a process and it requires nothing other than ssh be installed on the
machine you're trying to configure/control.
Cool thanks for the info Seth. Ansible looks interesting, its a
configuration orchestration component akin to Puppet / Chef is it not?
Does it do any provisioning in itself?
The ec2_create utility on your blog seems to call out to euca2ools to do
the actual provisioning on ec2 correct? You'd still want a component
such as Deltacloud to abstractify commands to different cloud providers
would you not?
-Mo
8:24 a.m.
Le jeudi 01 novembre 2012 à 08:32 -0400, Mo Morsi a écrit :
On 10/31/2012 01:07 PM, Seth Vidal wrote:
> You can orchestrate all of these steps across/between multiple
systems
> using ansible: http://ansible.cc - I've been documenting spinning up and
> provisioning instances on my blog in the last week or so. You might
> take a
> look - it should solve the problem of the above needing to be so manual
> of a process and it requires nothing other than ssh be installed on the
> machine you're trying to configure/control.
Cool thanks for the info Seth. Ansible looks interesting, its a
configuration orchestration component akin to Puppet / Chef is it not?
Does it do any provisioning in itself?
It can be used for remote and parallel job executation ( like run uptime
on all servers ).
But it can also be used with playbook, to describe the state of a server
and make sure it is compliant. For example, "make sure service is
started", "restart if config was changed" ( using a notification system
).
See http://ansible.cc/docs/playbooks.html
--
Michael Scherer
12:32 a.m.
On Thu, 1 Nov 2012, Michael Scherer wrote:
Le jeudi 01 novembre 2012 à 08:32 -0400, Mo Morsi a écrit :
> On 10/31/2012 01:07 PM, Seth Vidal wrote:
>> You can orchestrate all of these steps across/between multiple systems
>> using ansible: http://ansible.cc - I've been documenting spinning up and
>> provisioning instances on my blog in the last week or so. You might
>> take a
>> look - it should solve the problem of the above needing to be so manual
>> of a process and it requires nothing other than ssh be installed on the
>> machine you're trying to configure/control.
>
>
> Cool thanks for the info Seth. Ansible looks interesting, its a
> configuration orchestration component akin to Puppet / Chef is it not?
> Does it do any provisioning in itself?
It can be used for remote and parallel job executation ( like run uptime
on all servers ).
But it can also be used with playbook, to describe the state of a server
and make sure it is compliant. For example, "make sure service is
started", "restart if config was changed" ( using a notification system
).
Most importantly to me is the ability to orchestrate between servers in a
single playbook or via the api:
Ex: "Take data from running this on system 1 and put that data into the
config on system 2, notify system 3 you've done this"
-sv
8:31 a.m.
On Thu, 1 Nov 2012, Mo Morsi wrote:
Cool thanks for the info Seth. Ansible looks interesting, its a
configuration orchestration component akin to Puppet / Chef is it not?
Does it do any provisioning in itself?
Ansible is more like this:
Combining puppet and chef in one item. Then making it so you can run the
entire tool w/o having to first setup the config mgmt system on the node
and not requiring any sort of central server.
So - ansible is those things and it doesn't make install a giant ruby blob
on a system.
The ec2_create utility on your blog seems to call out to euca2ools to
do
the actual provisioning on ec2 correct? You'd still want a component
such as Deltacloud to abstractify commands to different cloud providers
would you not?
I doubt it. It's easier to simply use the euca2ools w/the ec2 api against
openstack/cloudstack/euca/etc. Or if need be write an ostack_create to use
nova.
I'm not interested in supporting the whole world of clouds with this - we
only have euca and openstack setup - nothing else.
-sv
4190
days inactive
4192
days old
11 comments
4 participants
participants (4)
-
Michael Scherer -
Mo Morsi -
Robyn Bergeron -
Seth Vidal