So this is a point that we've talked about in several meetings, on IRC, etc but haven't really nailed it down as a specific set of tasks to implement.
Right now, in order to add a new provider to Conductor -- say you've just spun up another RHEV instance on a new host -- there are 3 different places in the Aeolus infrastructure that you've got to configure with certain information duplicated:
* First you've got to start up a new deltacloud-core instance with the RHEV provider URL as the default value for X-Deltacloud-Provider (There's no need to reply here that "deltacloud has supported multi-provider core for ages -- why isn't conductor using it? -- as that is _exactly_ what I'm addressing here) * You've then got to update the rhevm.json (or whatever it's named) config file with various information about the new RHEV provider * Finally, the Conductor admin adds a new provider via the Conductor Provider UI, making sure to point to the newly-created deltacloud-core server and using a provider name that matches the configuration added rhevm.json
We need to reduce this to _one_ place -- once a new RHEV server is up and running the Conductor admin should be able to go to the Conductor Provider UI and configure the provider here with no need to mess with deltacloud config files (or new deltacloud-core servers) or with factory config files.
The redmine story for this should read something like: "As an administrator, when bringing up a new private cloud instance, I can configure the new provider in a single Aeolus location"
There are 2 main "sub-stories" 1) "As an administrator, when bringing up a new private cloud instance, it is not necessary to launch a new deltacloud-core instance or configure any existing deltacloud-core instances with details on this new provider; the 'new provider' UI in Conductor provides everything needed for an existing deltacloud-core." 2) "As an administrator, when bringing up a new private cloud instance, it is not necessary to edit the imagefactory config files with details on this new provider; the 'new provider' UI in Conductor provides everything needed by imagefactory."
In more detail, I'll consider them separately:
So the first is relatively straightforward -- I'll be starting to implement it as part of this sprint. The tasks are as follows: a) add a deltacloud_driver field to the ProviderType model (no UI for this model -- it's currently loaded in seeds.rb b) add a deltacloud_provider field to the Provider model c) in the UI, make 2 changes to the 'new provider' form: 1) validate the provider type -- i.e. for the chosen provider type, query the given dc core URL and make sure that this core supports this provider type, initially as part of form validation, eventually w/ javascript checking 2) add a deltacloud_provider field to the form. This is a free-form textfield, but we could make a future enhancement where for ec2 the user gets a picklist of supported regions (internally saved as text like for other provider types) d) anywhere in conductor that we make deltacloud calls, update the code to add in these new fields, either as URL params (http://localhost:3001/api;driver=ec2;provider=us-east-1) or by setting request headers for X-Deltacloud-Driver and X-Deltacloud-Provider, whichever is easier e) same as d) but for condor too
One open question here is whether we should make deltacloud_provider optional -- i.e. will we let people defer to the prior behavior of taking the default provider for a given deltacloud-core instance.
For the second sub-story, there are a lot more open issues. The tasks are as follows: a) augment the Provider model and UI with whatever fields are needed to reproduce the contents of the factory config files (rhevm.json, etc) b) make sure that the providers.xml (or providers.json) API calls include these in a format that factory is prepared to deal with -- either just like rhevm.json now, or alter factory to deal with the conductor API format. c) aeolus-image should be altered to make this API call and pass the xml (or json) onto factory d) imagefactory should be altered to accept the providers.json or providers.xml content and override its local rhevm.json, etc files with this content
So there you have it. I expect most of the discussion/disagreement to occur on the second sub-story, but feel free to respond with comments/corrections/etc for either.
Scott
Hi Scott,
For reference, here was my previous attempt to dig through this:
https://fedorahosted.org/pipermail/aeolus-devel/2011-June/002120.html
On Thu, 2011-08-04 at 01:06 -0400, Scott Seago wrote:
So this is a point that we've talked about in several meetings, on IRC, etc but haven't really nailed it down as a specific set of tasks to implement.
Right now, in order to add a new provider to Conductor -- say you've just spun up another RHEV instance on a new host -- there are 3 different places in the Aeolus infrastructure that you've got to configure with certain information duplicated:
- First you've got to start up a new deltacloud-core instance with the
RHEV provider URL as the default value for X-Deltacloud-Provider (There's no need to reply here that "deltacloud has supported multi-provider core for ages -- why isn't conductor using it? -- as that is _exactly_ what I'm addressing here)
- You've then got to update the rhevm.json (or whatever it's named)
config file with various information about the new RHEV provider
- Finally, the Conductor admin adds a new provider via the Conductor
Provider UI, making sure to point to the newly-created deltacloud-core server and using a provider name that matches the configuration added rhevm.json
We need to reduce this to _one_ place -- once a new RHEV server is up and running the Conductor admin should be able to go to the Conductor Provider UI and configure the provider here with no need to mess with deltacloud config files (or new deltacloud-core servers) or with factory config files.
Yes!
The redmine story for this should read something like: "As an administrator, when bringing up a new private cloud instance, I can configure the new provider in a single Aeolus location"
There are 2 main "sub-stories"
I'd call these "anti-stories" :)
- "As an administrator, when bringing up a new private cloud instance,
it is not necessary to launch a new deltacloud-core instance or configure any existing deltacloud-core instances with details on this new provider; the 'new provider' UI in Conductor provides everything needed for an existing deltacloud-core." 2) "As an administrator, when bringing up a new private cloud instance, it is not necessary to edit the imagefactory config files with details on this new provider; the 'new provider' UI in Conductor provides everything needed by imagefactory."
In more detail, I'll consider them separately:
So the first is relatively straightforward -- I'll be starting to implement it as part of this sprint. The tasks are as follows: a) add a deltacloud_driver field to the ProviderType model (no UI for this model -- it's currently loaded in seeds.rb b) add a deltacloud_provider field to the Provider model c) in the UI, make 2 changes to the 'new provider' form: 1) validate the provider type -- i.e. for the chosen provider type, query the given dc core URL and make sure that this core supports this provider type, initially as part of form validation, eventually w/ javascript checking 2) add a deltacloud_provider field to the form. This is a free-form textfield, but we could make a future enhancement where for ec2 the user gets a picklist of supported regions (internally saved as text like for other provider types)
In the case of RHEV, this is a URL - we should provide people with as much help as possible when filling out this field.
d) anywhere in conductor that we make deltacloud calls, update the code to add in these new fields, either as URL params (http://localhost:3001/api;driver=ec2;provider=us-east-1) or by setting request headers for X-Deltacloud-Driver and X-Deltacloud-Provider, whichever is easier e) same as d) but for condor too
Before I get to your open question below, what I don't like about this is that the deltacloudd URL is duplicated between all providers and something that users need to fill in.
Entering the deltacloudd URL is not the common case. Conductor should be configured with a deltacloudd URL in the same way it would be configured with the address of IWHD, Image Factory or Condor.
One open question here is whether we should make deltacloud_provider optional -- i.e. will we let people defer to the prior behavior of taking the default provider for a given deltacloud-core instance.
I'd see this as a special case where we allow users to add a "deltacloud provider" - i.e. you just give it a deltacloud URL, rather than a X-Deltacloud-Provider value to pass to our embedded deltacloudd.
For the second sub-story, there are a lot more open issues. The tasks are as follows: a) augment the Provider model and UI with whatever fields are needed to reproduce the contents of the factory config files (rhevm.json, etc) b) make sure that the providers.xml (or providers.json) API calls include these in a format that factory is prepared to deal with -- either just like rhevm.json now, or alter factory to deal with the conductor API format. c) aeolus-image should be altered to make this API call and pass the xml (or json) onto factory d) imagefactory should be altered to accept the providers.json or providers.xml content and override its local rhevm.json, etc files with this content
Right, exactly. We already pass the credentials XML to image factory, so perhaps the providers details could just be included there?
Finally, to repeat a point I made before - with this new setup, having separate UIs for configuring providers and provider accounts doesn't really make much sense to me. It makes most sense to me for the user to point Conductor at a provider and supply the credentials for it at the same time.
Right now, the design seems to be optimized for the case where you're entering many accounts for the one provider. IMHO, that's not the important case and we could tackle that with some "this is another account for a provider I added earlier" UI trickery
Cheers, Mark.
On 08/04/2011 08:00 AM, Mark McLoughlin wrote:
Hi Scott,
For reference, here was my previous attempt to dig through this:
https://fedorahosted.org/pipermail/aeolus-devel/2011-June/002120.html
Ahh, yes I remember that thread now. You're tackling a much larger scope with that document, though, and I'm not sure we'd want to handle all of that in this one iteration. In particular, some of the proposals around the more fundamental rethinking of providers and provider accounts still need a fair amount more thinking/design/etc which shouldn't necessarily hold back the changes I'm proposing here, which can probably be more readily put in place. In this larger context, the separate chunks of work would break out roughly like this: 1) Multi-provider core refactoring (proposed below; relatively few remaining issues to work out; ready to start implementation this week) 2) All imagefactory provider config comes via conductor api along with credentials (proposed below; need to coordinate design/api/changes with aeolus-image and factory team) 3) Hash out requirements/design for the more fundamental changes Mark proposed in the linked message (can be done in parallel w/ impl for 1-2) 4) Implement desired changes to Provider/Account/etc models.
On Thu, 2011-08-04 at 01:06 -0400, Scott Seago wrote:
So this is a point that we've talked about in several meetings, on IRC, etc but haven't really nailed it down as a specific set of tasks to implement.
Right now, in order to add a new provider to Conductor -- say you've just spun up another RHEV instance on a new host -- there are 3 different places in the Aeolus infrastructure that you've got to configure with certain information duplicated:
- First you've got to start up a new deltacloud-core instance with the
RHEV provider URL as the default value for X-Deltacloud-Provider (There's no need to reply here that "deltacloud has supported multi-provider core for ages -- why isn't conductor using it? -- as that is _exactly_ what I'm addressing here)
- You've then got to update the rhevm.json (or whatever it's named)
config file with various information about the new RHEV provider
- Finally, the Conductor admin adds a new provider via the Conductor
Provider UI, making sure to point to the newly-created deltacloud-core server and using a provider name that matches the configuration added rhevm.json
We need to reduce this to _one_ place -- once a new RHEV server is up and running the Conductor admin should be able to go to the Conductor Provider UI and configure the provider here with no need to mess with deltacloud config files (or new deltacloud-core servers) or with factory config files.
Yes!
The redmine story for this should read something like: "As an administrator, when bringing up a new private cloud instance, I can configure the new provider in a single Aeolus location"
There are 2 main "sub-stories"
I'd call these "anti-stories" :)
- "As an administrator, when bringing up a new private cloud instance,
it is not necessary to launch a new deltacloud-core instance or configure any existing deltacloud-core instances with details on this new provider; the 'new provider' UI in Conductor provides everything needed for an existing deltacloud-core." 2) "As an administrator, when bringing up a new private cloud instance, it is not necessary to edit the imagefactory config files with details on this new provider; the 'new provider' UI in Conductor provides everything needed by imagefactory."
In more detail, I'll consider them separately:
So the first is relatively straightforward -- I'll be starting to implement it as part of this sprint. The tasks are as follows: a) add a deltacloud_driver field to the ProviderType model (no UI for this model -- it's currently loaded in seeds.rb b) add a deltacloud_provider field to the Provider model c) in the UI, make 2 changes to the 'new provider' form: 1) validate the provider type -- i.e. for the chosen provider type, query the given dc core URL and make sure that this core supports this provider type, initially as part of form validation, eventually w/ javascript checking 2) add a deltacloud_provider field to the form. This is a free-form textfield, but we could make a future enhancement where for ec2 the user gets a picklist of supported regions (internally saved as text like for other provider types)
In the case of RHEV, this is a URL - we should provide people with as much help as possible when filling out this field.
OK, so I see a two phase approach here -- first we get the field out there so it's functional, then we need some sort of UI helper component that's driver-specific. For ec2 it's a picklist, for RHEV perhaps some boilerplate URL text "https://%5BRHEV-HOST%5D/what-we-expect-here", for eucalyptus we'd start with "|ec2=[EC2-IP]:8773;s3=[S3-IP]:8773" in the textfield, etc. |
d) anywhere in conductor that we make deltacloud calls, update the code to add in these new fields, either as URL params (http://localhost:3001/api;driver=ec2;provider=us-east-1) or by setting request headers for X-Deltacloud-Driver and X-Deltacloud-Provider, whichever is easier e) same as d) but for condor too
Before I get to your open question below, what I don't like about this is that the deltacloudd URL is duplicated between all providers and something that users need to fill in.
Entering the deltacloudd URL is not the common case. Conductor should be configured with a deltacloudd URL in the same way it would be configured with the address of IWHD, Image Factory or Condor.
So how about this compromise: 1) we add a conf value for default deltacloud-core URL -- possibly to the metadata_objects table so it can be edited via admin UI, or as you say we could do it in the same way we do IWHD/condor/etc 2) for 'new provider', the normal workflow just shows this as plain text in HTML (with a hidden form element actually specifying the value for the object) -- we provide the user with a button to override default provider -- if the click that, the plain text is replaced with an editable textfield with the default deltacloud URL pre-filled.
I'd say that, for 2), for the first iteration we just use a regular textfield but we pre-fill the data with the default value so in most cases. We then enhance it with hiding the formfield by default after we get some UI design on this. In particular, I'd say that the "hide the formfield until the user clicks the 'override' button should be js-only. The plain HTML fallback just provides the textfield with default filled-in.
So the main thing here is that we're providing the multi-provider core for convenience -- users shouldn't have to worry about where deltacloud-core is when adding providers. However, there may be cases where the user wants to use a different core. For example, what if the user has aeolus admin privileges on the web UI but not shell access to mess with deltacloud-core config/packages and wants to test out a new driver that's not loaded on the default core instance? If he can override the core URL, he can set up a separate core instance and point there.
One open question here is whether we should make deltacloud_provider optional -- i.e. will we let people defer to the prior behavior of taking the default provider for a given deltacloud-core instance.
I'd see this as a special case where we allow users to add a "deltacloud provider" - i.e. you just give it a deltacloud URL, rather than a X-Deltacloud-Provider value to pass to our embedded deltacloudd.
Well even when you give a deltacloud URL (as suggested above) this "custom" deltacloud-core instance may still be set up for multiple providers, so we don't want to preclude them choosing the X-Deltacloud-Provider field as well. Perhaps we say that X-Deltacloud-Provider is required _unless_ the user overrides the default deltacloud-core URL, in which case it's optional?
For the second sub-story, there are a lot more open issues. The tasks are as follows: a) augment the Provider model and UI with whatever fields are needed to reproduce the contents of the factory config files (rhevm.json, etc) b) make sure that the providers.xml (or providers.json) API calls include these in a format that factory is prepared to deal with -- either just like rhevm.json now, or alter factory to deal with the conductor API format. c) aeolus-image should be altered to make this API call and pass the xml (or json) onto factory d) imagefactory should be altered to accept the providers.json or providers.xml content and override its local rhevm.json, etc files with this content
Right, exactly. We already pass the credentials XML to image factory, so perhaps the providers details could just be included there?
Finally, to repeat a point I made before - with this new setup, having separate UIs for configuring providers and provider accounts doesn't really make much sense to me. It makes most sense to me for the user to point Conductor at a provider and supply the credentials for it at the same time.
Yeah I think some of that's on Ken's roadmap for the admin UI. At a minimum we need to connect the Provider and ProviderAccount UIs. At the moment we're even limiting to one account per provider, so it's even easier to consolidate these two. However, long-term I don't think we can continue with this restriction as-is.
Right now, the design seems to be optimized for the case where you're entering many accounts for the one provider. IMHO, that's not the important case and we could tackle that with some "this is another account for a provider I added earlier" UI trickery
Right -- and, see above, you can't even add multiple accounts with the present UI.
Cheers, Mark.
aeolus-devel@lists.fedorahosted.org