taskomatic plans

[Ian Main]

So I've been looking over some things and doing a lot of pondering on
how to move forward with taskomatic for deltacloud.

The plan I'm working with now is to:

- Finish the infrastructure for dependency information between tasks
  and implement threading.  I think it would be wise to just do this
  up front so that as we implement tasks we can figure out the
  dependencies and implement them.  The task dependencies will work as
  David and I worked out before for ovirt, using a class for each task
  type, an instance for each action will be added to a list which can
  be scanned by each instance to determine if it can run. I actually
  don't think this will add too much difficulty given the limited number
  of tasks in dcloud.

- Implement a seperate thread that deals with updates from the service
  providers.  This will basically be the "dbomatic" for dcloud.  The
  poll interval will be based on an XML config value in each driver and
  will likely be fairly longish (I'm thinking 60sec or so depending).

- Plumb in a system that allows quicker updates to objects on which
  recent actions have taken place.  For example when an instance is
  started we want to have more frequent polling of that instance until
  its state changes so that we can more quickly inform the user.  I am
  unsure whether I will just use a seperate thread (or even be part of
  the task) to do the updates or somehow notify the main update
  thread. I suspect another thread will be easier..

- Work on/test task implementations to make them reliable.

- I'm not going to worry about torqeubox for now.  If we do end up
  using it it may be useful to use the queuing system as a method to
  notify taskomatic of new tasks.

Sound reasonable to everyone?

Ian

> -----Original Message-----
> From: [e-mail]

[mailto:deltacloud-
> [e-mail]] On Behalf Of Ian Main
> Sent: Monday, September 21, 2009 16:15 PM
> To: [e-mail]
> Subject: [deltacloud-devel] taskomatic plans
>
> So I've been looking over some things and doing a lot of pondering on
> how to move forward with taskomatic for deltacloud.
>
> The plan I'm working with now is to:
> Finish the infrastructure for dependency information between tasks
> and implement threading.  I think it would be wise to just do this
> up front so that as we implement tasks we can figure out the
> dependencies and implement them.  The task dependencies will work as
> David and I worked out before for ovirt, using a class for each task
> type, an instance for each action will be added to a list which can
> be scanned by each instance to determine if it can run. I actually
> don't think this will add too much difficulty given the limited number
> of tasks in dcloud.

[IH] how do you deal with transactions? Transactions in the db are easy,
but what if the dependency implies an action on a provider, then the
taskomatic crashes - how do we recover post crash / from a failure in a
driver action?
> Implement a seperate thread that deals with updates from the service
> providers.  This will basically be the "dbomatic" for dcloud.  The
> poll interval will be based on an XML config value in each driver and
> will likely be fairly longish (I'm thinking 60sec or so depending).

IH] rhev-m supports async notifications (callback) for events. So far this
has been based on WCF callback capabilities, but can also be implemented
by a web service callback, or via QMF.
Please design with room for a callback, instead of polling.

In any case, polling/callback sounds like a driver implementation issue,
but I understand we don't want to require a daemon for each driver just
for polling. but since a callback is so much more efficient than polling,
I think we should think this through.

Btw, even in polling there is a difference between a provider that
supports it and a provider that doesn't. for example, in rhev-m you can
poll if you don't have a callback, but you poll to know if there were
changes (in state/statistics) to objects you care about (registered a
query/search for) - you don't have to poll all info on each object just to
compare if it was changed.
> Plumb in a system that allows quicker updates to objects on which
> recent actions have taken place.  For example when an instance is
> started we want to have more frequent polling of that instance until
> its state changes so that we can more quickly inform the user.  I am
> unsure whether I will just use a seperate thread (or even be part of
> the task) to do the updates or somehow notify the main update
> thread. I suspect another thread will be easier..
>
> Work on/test task implementations to make them reliable.
>
> I'm not going to worry about torqeubox for now.  If we do end up
> using it it may be useful to use the queuing system as a method to
> notify taskomatic of new tasks.
>
> Sound reasonable to everyone?
>
> Ian
> _______________________________________________
> deltacloud-devel mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/deltacloud-devel

[David Lutterkort]

On Mon, 2009-09-21 at 13:14 -0700, Ian Main wrote:
> So I've been looking over some things and doing a lot of pondering on
> how to move forward with taskomatic for deltacloud.
>
> The plan I'm working with now is to:
> Finish the infrastructure for dependency information between tasks
> and implement threading.  I think it would be wise to just do this
> up front so that as we implement tasks we can figure out the
> dependencies and implement them.  The task dependencies will work as
> David and I worked out before for ovirt, using a class for each task
> type, an instance for each action will be added to a list which can
> be scanned by each instance to determine if it can run. I actually
> don't think this will add too much difficulty given the limited number
> of tasks in dcloud.

Sounds good - the devil is in the details with this, but that's a good
start. And deltacloud desperately needs queue parallelization, since we
can't have the queue stall just because we lost connection to EC2 (but
still have connections to RHEV-M)
> Implement a seperate thread that deals with updates from the service
> providers.  This will basically be the "dbomatic" for dcloud.  The
> poll interval will be based on an XML config value in each driver and
> will likely be fairly longish (I'm thinking 60sec or so depending).
>
> Plumb in a system that allows quicker updates to objects on which
> recent actions have taken place.  For example when an instance is
> started we want to have more frequent polling of that instance until
> its state changes so that we can more quickly inform the user.  I am
> unsure whether I will just use a seperate thread (or even be part of
> the task) to do the updates or somehow notify the main update
> thread. I suspect another thread will be easier..

How about just sticking a 'frequency' and a 'next_poll' property into
each task, so that your dbomatic thread polls for status updates for all
tasks whose 'next_poll' is in the past and sets next_poll to next_poll +
frequency after each successful poll ? State changes (or any change) to
an action can adjust frequency. That way, you don't need to distinguish
between slow and fast polling tasks.
> I'm not going to worry about torqeubox for now.  If we do end up
> using it it may be useful to use the queuing system as a method to
> notify taskomatic of new tasks.

Absolutely agreed. Getting something working with whatever is familiar
is the top priority.

David