[PATCH aeolus] Return 401 on unauthorized xml request
by Martyn Taylor
From: Martyn Taylor <mtaylor(a)redhat.com>
---
src/app/controllers/application_controller.rb | 13 +++++++++----
1 files changed, 9 insertions(+), 4 deletions(-)
diff --git a/src/app/controllers/application_controller.rb b/src/app/controllers/application_controller.rb
index cb4a1d3..c9b2949 100644
--- a/src/app/controllers/application_controller.rb
+++ b/src/app/controllers/application_controller.rb
@@ -212,10 +212,15 @@ class ApplicationController < ActionController::Base
end
def require_user
- return if current_user
- store_location
- flash[:notice] = "You must be logged in to access this page"
- redirect_to login_url
+ respond_to do |format|
+ format.html do
+ return if current_user
+ store_location
+ flash[:notice] = "You must be logged in to access this page"
+ redirect_to login_url
+ end
+ format.xml { head :unauthorized }
+ end
end
def require_no_user
--
1.7.4
12 years, 11 months
Image building - cli, factory, conductor and iwhd plan
by Mark McLoughlin
Hey,
Apologies for another long email, but here's my proposal for how to
proceed with the image building CLI etc.
Cheers,
Mark.
= Terminology =
- Provider - a cloud instance, e.g. a RHEV-M installation or an EC2
region.
- Target/Provider Type - the cloud type, irrespective of the specific
instance e.g. RHEV-M vs EC2.
- Provider Account - a set of user credentials for a specific
provider.
- Provider Image - these are images which have been pushed to, or
imported from, a provider and are available for use by instances.
- Target Image - these are images which have been built for a
specific target and are available to be pushed to multiple
providers.
- Image Build - these are objects which track all target and
provider images built from a template at a certain time, all of
which should be equivalent to each other.
- Image - these are metadata objects which describe the purpose of
the image, the parameters it takes, the set of available builds of
the image and the latest build.
= Overview =
As described previously[1], the short/medium term plan is that Aeolus
will:
- Have well documented deployable and template description formats
with plenty of examples. Deployable authors will create these
descriptions manually or with the help of simple command line
tools and store them as they fit (e.g. in their own git repo)
- Support launching deployables in conductor simply by allowing the
user to supply a URL to a deployable description, or choosing a
URL from a list populated by an admin.
- Include a set of command line tools for building images from a
template description. This tools will also allow deployable
authors to list available images available to be referenced in
their deployables.
- Store images and their related metadata in IWHD. Conductor will
use IWHD queries to resolve deployables' image references and
allow users to launch individual images directly.
- Use image factory to build and import images, including the
updating of IWHD metadata about the images.
= IHWD Metadata =
The following IWHD buckets will be used, each one for a different
object type:
images:
- object_type == "image"
- object_body: XML document with image description and parameters,
for deployable authors and simple instance launch from UI
- uuid: UUID
- latest_build: build UUID
builds:
- object_type == "build"
- object_body: empty
- uuid: UUID
- image: UUID of the image object
- parent: UUID of previous build
target_images:
- object_type == "target_image"
- object_body: target specific image, for upload builds; empty for
snapshot builds
- uuid: UUID
- build: UUID of the build object
- icicle: UUID of icicle object
- template: UUID of template object
- target: ec2, rhev-m, etc.
- target_parameters: target specific data stashed here for the push
stage
provider_images:
- object_type == "provider_image"
- object_body: empty
- uuid: UUID
- target_image: UUID of image
- provider: e.g. ec2-us-east-1, rhev-m site
- icicle == "none"
- target_identifier: target specific ID for the image
templates:
- object_type == "template"
- object_body: <template/> document supplied for image build
- uuid: UUID
icicles:
- object_type == "icicle"
- object_body: <icicle/> document
- uuid: UUID
This fairly closely reflects the metadata currently stored by image
factory in IWHD, with the following changes:
- the current "image" type is renamed to "target_type", along with
the bucket and image reference on provider images[2]
- new image and build object types and buckets are introduced
- a build reference is added to the target image type
= Image Factory APIs =
Image factory currently supports building target images and pushing
provider images, along with the appropriate updating of image
warehouse metadata.
In order to allow the CLI to fire-and-forget, we will need to add new
build and push APIs which can handle multiple images at once.
These would look like:
- image(image_id, template, targets[])
+ builds an image for the supplied targets
+ image_id should be ommitted, if a new image is required
+ template and targets may be ommitted if a previous build
exists and the previous template and targets will be used
- push(image_id, providers, credentials)
+ push the image the supplied providers
+ assumes factory can figure out which target image is
appropriate for each provider
+ credentials argument is a <provider_credentials/> document
Also, we'll need to rename image factory's current concept of an image
as a target image. Some initial work on that here[3].
= Conductor APIs =
Conductor needs APIs to support the following:
1) List the available provider types in an XML doc
2) List the available providers in an XML doc
3) Dump a <provider_credentials/> document encapsulating all of the
available provider accounts
The latter API would be subject to authentication and access control.
= Conductor IWHD Queries =
Conductor has basically two use cases for querying IWHD:
1) As a user, I want to launch a deployable:
2) As a user, I want to launch an instance of an image
For (1), conductor needs to resolve each deployable image reference to
a set of provider images. This can be done with something like:
$> $build_id = curl http://iwhd/$image_id/latest_build
$> curl -d '$build=="'$build_id'"' http://iwhd/target_images/_query
$> foreach $target_image_id; do curl -d '$target_image=="'$target_image_id'"' http://iwhd/provider_images; done
For (2), conductor needs to list all images available for launching a
standalone instance and, when the user launches the image, it needs to
list the parameters for the image. Both are easily achieved by
querying the images bucket.
= Image Building CLI =
The image building CLI is used by Aeolus users to build and upload
images from templates. It is also used by deployable authors to list
the available images.
The use cases are:
1) As an image builder or deployable author, I want to list all
images
2) As an image builder or deployable author, I want to list all
builds of an image
3) As an image builder or deployable author, I want to list all the
targets and providers an image has been built for
4) As an image builder, I want to build an image
5) As an image builder, I want to push an image to a provider
6) As an image builder, I want to import an image
7) As an image builder, I want to delete an image
8) As an image builder, I want to delete old versions of an image
9) As an image builder, I want to delete a provider image
10) As an image builder, I want to delete an image
The tool needs to interact with IWHD for listing, image factory for
building and conductor for provider/account details.
It might look like:
$> aeolus-image images # list available images
$> aeolus-image builds $image_id # list the builds of an image
$> aeolus-image target-images $build_id # list the target images from a build
$> aeolus-image provider-images $target_image_id # list the provider images from a target image
$> aeolus-image build --target ec2 --template my.tmpl # build a new image for ec2 from the template
$> aeolus-image build --image $image_id # rebuild the image template and targets from latest build
$> aeolus-image build --target ec2 --target rackspace \ #
--image $image_id \ # rebuild the image with a new template and set of targets
--template my.tmpl #
$> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target image to the specified provider
$> aeolus-image push $build_id # push all target images for a build, to same providers as previously
$> aeolus-image push --account $provider_account $build_id # ditto, using a specific provider account
$> aeolus-image push $image_id # push all the target images for the latest build
$> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from the specified provider
$> aeolus-image delete --image $image_id # deletes all builds, target images and provider images
$> aeolus-image delete --build $build_id # deletes a build, updating latest/parent references as appropriate
$> aeolus-image delete --target-image $target_image # deletes a target image and its provider images
$> aeolus-image delete --provider-image $provider_image # deletes a provider image
$> aeolus-image targets # list the values available for the --target parameter
$> aeolus-image providers # list the values available for the --provider parameter
$> aeolus-image accounts #
list the values available for the --account parameter
Some other notes:
- The tool will need to authenticate at least against conductor, so
--user/--pass arguments will be needed. We may also want to support
supplying these via environment variables and/or a dotfile.
- The build/push/import commands are long running, so we should
support displaying the progress of the builds and have a
--background parameter for fire-and-forget.
= Footnotes =
[1] - https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001640.html
[2] - to migrate existing data, you can do:
$> cat > update.js <<EOF
db.main.update({ _bucket : "images", object_type : "image" }, { \$set: { _bucket : "target_images", object_type : "target_image" } }, false, true)
db.main.find({ _bucket : "provider_images", object_type : "provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image; db.main.save(pi); })
db.main.update({ _bucket : "provider_images", object_type : "provider_image" }, { \$unset: { image : 1 } }, false, true)
EOF
$> mongo --port 27018 repo update.js
$> mv _fs/images _fs/provider_images
This clearly only works for IWHD with the fs_autostart, but
should be easy to adjust.
[3] - https://github.com/markmc/image_factory/commits/master
12 years, 11 months
[PATCH conductor] TASK 1601 implements RESTclient iwhd backend
by Tomas Hrcka
---
src/app/models/image.rb | 30 ++++++++++++++++++++++++++++++
src/app/models/image_build.rb | 36 ++++++++++++++++++++++++++++++++++++
src/app/models/provider_image.rb | 28 ++++++++++++++++++++++++++++
src/app/models/target_image.rb | 26 ++++++++++++++++++++++++++
src/lib/warehouse_client.rb | 2 +-
src/lib/warehouse_model.rb | 20 ++++++++++++++++++++
src/spec/spec_helper.rb | 7 +++++++
7 files changed, 148 insertions(+), 1 deletions(-)
create mode 100644 src/app/models/image.rb
create mode 100644 src/app/models/image_build.rb
create mode 100644 src/app/models/provider_image.rb
create mode 100644 src/app/models/target_image.rb
create mode 100644 src/lib/warehouse_model.rb
diff --git a/src/app/models/image.rb b/src/app/models/image.rb
new file mode 100644
index 0000000..a4e69b6
--- /dev/null
+++ b/src/app/models/image.rb
@@ -0,0 +1,30 @@
+class Image < WarehouseModel
+
+ def initialize(attrs)
+ attrs.each do |k,v|
+ self.class.send(:attr_accessor, k.to_sym) unless respond_to?(:"#{k}=")
+ send(:"#{k}=", v)
+ end
+ end
+
+ def self.all
+ self.set_warehouse_and_bucket('images') if self.bucket.nil?
+
+ self.bucket.objects.map do |wh_object|
+ wh_object.attrs(wh_object.attr_list)
+ end
+ end
+
+ def self.find(uuid)
+ self.set_warehouse_and_bucket('images') if self.bucket.nil?
+ if self.bucket.include?(uuid)
+ Image.new(self.bucket.object(uuid).attrs(self.bucket.object(uuid).attr_list))
+ else
+ raise BucketObjectNotFound
+ end
+ end
+
+ def image_builds
+ ImageBuild.find_all_by_image_uuid(self.uuid)
+ end
+end
\ No newline at end of file
diff --git a/src/app/models/image_build.rb b/src/app/models/image_build.rb
new file mode 100644
index 0000000..a98cdd8
--- /dev/null
+++ b/src/app/models/image_build.rb
@@ -0,0 +1,36 @@
+require 'warehouse_client'
+
+class ImageBuild < WarehouseModel
+
+ def initialize(attrs)
+ attrs.each do |k,v|
+ self.class.send(:attr_accessor, k.to_sym) unless respond_to?(:"#{k}=")
+ send(:"#{k}=", v)
+ end
+ end
+
+ def self.all
+ self.set_warehouse_and_bucket('builds') if self.bucket.nil?
+ self.bucket.objects.map do |wh_object|
+ wh_object.attrs(wh_object.attr_list)
+ end
+ end
+
+ def self.find(uuid)
+ self.set_warehouse_and_bucket('builds') if self.bucket.nil?
+ if self.class.bucket.include?(uuid)
+ self.class.bucket.object(uuid).attrs(self.class.bucket.object(uuid).attr_list)
+ else
+ raise BucketObjectNotFound
+ end
+ end
+
+ def self.find_all_by_image_uuid(uuid)
+ self.set_warehouse_and_bucket('builds') if self.bucket.nil?
+ self.bucket.objects.map do |wh_object|
+ if wh_object.attr('image')[1] == uuid
+ ImageBuild.new(wh_object.attrs(wh_object.attr_list))
+ end
+ end.compact
+ end
+end
\ No newline at end of file
diff --git a/src/app/models/provider_image.rb b/src/app/models/provider_image.rb
new file mode 100644
index 0000000..2aac4e4
--- /dev/null
+++ b/src/app/models/provider_image.rb
@@ -0,0 +1,28 @@
+require 'warehouse_client'
+
+class ProviderImage< WarehouseModel
+
+ def initialize(attrs)
+ attrs.each do |k,v|
+ self.class.send(:attr_accessor, k.to_sym) unless respond_to?(:"#{k}=")
+ send(:"#{k}=", v)
+ end
+ end
+
+ def self.all
+ self.set_warehouse_and_bucket('provider_images') if self.bucket.nil?
+
+ self.bucket.objects.map do |wh_object|
+ wh_object.attrs(wh_object.attr_list)
+ end
+ end
+
+ def self.find(uuid)
+ self.set_warehouse_and_bucket('provider_images') if self.bucket.nil?
+ if self.bucket.include?(uuid)
+ ProviderImage.new(self.bucket.object(uuid).attrs(self.bucket.object(uuid).attr_list))
+ else
+ raise BucketObjectNotFound
+ end
+ end
+end
\ No newline at end of file
diff --git a/src/app/models/target_image.rb b/src/app/models/target_image.rb
new file mode 100644
index 0000000..d6380c9
--- /dev/null
+++ b/src/app/models/target_image.rb
@@ -0,0 +1,26 @@
+class TargetImage < WarehouseModel
+
+ def initialize(attrs)
+ attrs.each do |k,v|
+ self.class.send(:attr_accessor, k.to_sym) unless respond_to?(:"#{k}=")
+ send(:"#{k}=", v)
+ end
+ end
+
+ def self.all
+ self.set_warehouse_and_bucket('target_images') if self.bucket.nil?
+
+ self.bucket.objects.map do |wh_object|
+ wh_object.attrs(wh_object.attr_list)
+ end
+ end
+
+ def self.find(uuid)
+ self.set_warehouse_and_bucket('target_images') if self.bucket.nil?
+ if self.bucket.include?(uuid)
+ TargetImage.new(self.bucket.object(uuid).attrs(self.bucket.object(uuid).attr_list))
+ else
+ raise BucketObjectNotFound
+ end
+ end
+end
\ No newline at end of file
diff --git a/src/lib/warehouse_client.rb b/src/lib/warehouse_client.rb
index 12b8d9d..51d256f 100644
--- a/src/lib/warehouse_client.rb
+++ b/src/lib/warehouse_client.rb
@@ -133,7 +133,7 @@ module Warehouse
end
def create_bucket(bucket)
- @connection.do_request "/#{bucket}", :method => :put
+ @connection.do_request "/#{bucket}", :method => :put rescue RestClient::InternalServerError
Bucket.new(bucket, @connection)
end
diff --git a/src/lib/warehouse_model.rb b/src/lib/warehouse_model.rb
new file mode 100644
index 0000000..3c246df
--- /dev/null
+++ b/src/lib/warehouse_model.rb
@@ -0,0 +1,20 @@
+require 'warehouse_client'
+class BucketObjectNotFound < Exception;end
+class BucketNotFound < Exception;end
+
+class WarehouseModel
+
+ WAREHOUSE_CONFIG = YAML.load_file("#{RAILS_ROOT}/config/image_warehouse.yml")
+ class << self
+ attr_accessor :warehouse, :bucket
+
+ def set_warehouse_and_bucket(bucket_name)
+ begin
+ self.warehouse = Warehouse::Client.new(WAREHOUSE_CONFIG['baseurl'])
+ self.bucket = self.warehouse.bucket(bucket_name)
+ rescue
+ raise BucketNotFound
+ end
+ end
+ end
+end
\ No newline at end of file
diff --git a/src/spec/spec_helper.rb b/src/spec/spec_helper.rb
index 88413c0..e61f649 100644
--- a/src/spec/spec_helper.rb
+++ b/src/spec/spec_helper.rb
@@ -74,4 +74,11 @@ Spec::Runner.configure do |config|
config.after(:each) do
DatabaseCleaner.clean
end
+
+ def create_test_iwhd_data_for(bucket_name)
+ require 'warehouse_client'
+ warehouse_client = Warehouse::Client.new(YAML.load_file("#{RAILS_ROOT}/config/image_warehouse.yml")['baseurl'])
+ bucket = warehouse_client.create_bucket(bucket_name)
+ bucket.create_object("#{bucket_name}_testing_uuid","",{:uuid => "#{bucket_name}_testing_uuid"})
+ end
end
--
1.7.4.4
12 years, 11 months
[PATCH 00/11][image_factory] Support building for multiple providers
by Mark McLoughlin
Hey,
Here's a more complete effort at the image factory part of:
https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001667.html
In summary, it adds two new QMF methods (build_image and push_image) which allow
building an image for multiple targets, and pushing the image to multiple
providers, at once. It also adds new IWHD metadata objects and attributes which
maintain the relationships between images, builds, target images and provider
images.
The only significant issue that remains to be thought through is how to map a
given provider name to a target name - i.e. if the user supplies "ec2-us-east-1"
as a provider name, how does image factory know that the "ec2" target image is
the one that needs to be uploaded.
One possible solution to this would be to require provider names to be prefixed
with the target name - i.e. "ec2:ec2-us-east-1" would be the provider name.
Apart from that, I'm pretty happy with the state of the patches. Please review
and ACK.
Cheers,
Mark.
12 years, 12 months
Warning: your existing iwhd image object need migrating
by Mark McLoughlin
Hi,
I hate making incompatible changes, but we've had to do it with the
latest set of patches pushed to image_factory. Basically, what we were
calling 'image' objects in iwhd are now called 'target_image' objects
If you want to try and migrating your existing images to this new
format, you can try something like:
$> cat > update.js <<EOF
db.main.update({ _bucket : "images" }, { \$set: { _bucket : "target_images" } }, false, true)
db.main.update({ _bucket : "images", object_type : "image" }, { \$set: { object_type : "target_image" } }, false, true)
db.main.find({ _bucket : "provider_images", object_type : "provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image; db.main.save(pi); })
db.main.update({ _bucket : "provider_images", object_type : "provider_image" }, { \$unset: { image : 1 } }, false, true)
EOF
$> mongo --port 27018 repo update.js
$> mv _fs/images _fs/provider_images
If anyone wants to dig into creating a more useful migration
instructions, I'm happy to help.
Cheers,
Mark.
12 years, 12 months
Image permissions
by Mark McLoughlin
Hey,
We had a chat earlier medium term plans for image permissions. John is
going to write up some more detailed design thoughts, but I thought I'd
write down my understanding of the basic requirements before I forget:
1) Access control
We need users to be able to restrict access to images they create
or own - e.g. if you've got sensitive data in an image, or you
just want to prevent others from being able to delete your images
(This sounds to me like posix filesystem style permissions on
IWHD objects)
2) Quotas
When an administrator adds a provider account in conductor, she
needs to be able to set a per-user quota for that account - e.g.
Mary can only use 20Gb of S3 storage on this EC2 account
(This sounds to me like a policy stored in Conductor, enforced
either by conductor or image factory. If the latter, the quota
could be passed to image factory via the credentials XML)
3) Environment/pool family policies
Based on the environment a user is launching an instance, a
different set of images should be available to the user.
(This sounds to me like a policy managed by the image tools and
enforced by Conductor)
4) Entitlements/slots
This I'm less clear on. Take RHEL entitlements. When a RHEL
instance is started, it should automatically consume an
entitlement. However, does an image consume an entitlement? If so,
how do we make that happen?
(I think I got this one totally wrong)
Cheers,
Mark.
12 years, 12 months
#1605 - Launch a deployment
by Matt Wagner
Hi,
I'm attaching a patch that implements #1605, "As a user, I want to launch a deployment." This happens within the context of a pool, as opposed to a dedicated Deployments tab. (View a pool and press "New Deployment".)
There are two things you should be aware of:
1.) It doesn't actually work. We need to parse the deployable definition (as XML loaded from the provided URL) and then find or build matching images based on the enclosed data. But since that isn't implemented just yet, we just store the contents of the URL in the database and move you to the next page, from which you could launch an image, if only we had any of the needed data.
2.) It breaks existing deployments, which are going away at some point. I get a slew of failing tests as a result. (I'm happy to remove the no-longer-applicable tests, but I first wanted someone to confirm that I'm not going crazy and removing functionality that we actually want to keep.)
So why send it? Because it implements a feature that we're going to need (and soon!), and because I'd rather have eyes on it now than wait until next week when it might have something to hook into to be fully-functional.
-- Matt
12 years, 12 months
1604 - Deployable model and UI
by Jan Provazník
Hi,
this patchset adds deployable model and managing UI. With this admin is able to
select set of deployables which users are allowed to launch in conductor.
Note that for now full list of deployables is not fetched from warehouse but
LegacyDeployable list is used. This is because deployables are not saved in
warehouse yet. Switching to warehouse listing will be very simple.
Jan
12 years, 12 months
