Taskotron and Cloud Image tests

[Tim Flink]

On Tue, 08 Apr 2014 13:51:10 +0200
Vitaly Kuznetsov <vitty at redhat.com> wrote:

> Hi Tim & all,
> 
> in Cloud WG we have 'Automatic Smoketests on Image Build' task:
> https://fedorahosted.org/cloud/ticket/38
> AFAIK you have somehting similar among future Taskotron goals. It
> would be nice if we can collaborate here. By writing this message I
> just want to start the conversation :-)

I'm all for collaboration. I do have some ideas on how things would
work but the whole point of this is to make something that's useful -
if that means deviating from what I originally had in mind, that's fine
with me.

> For RHEL cloud image checks we have special tool called 'valid':
> https://github.com/RedHatQE/valid , it can be reused for Fedora as
> well. The questions are: how do you see the integration and how can
> someone (me?) work on this (non-existent atm) part of Taskotron.

For now, I think that the best place to start would be to figure out
what's missing from taskotron in order for the integration to happen.
 - would the cloud image startup be done by valid or in taskotron?
 - where would the cloud images be run?
 - what other image work would be done in the task?


Our documentation is pretty much non-existent at the moment and I was
hoping to get depcheck written as a task before writing too much of that
in case we run into problems that require syntax changes to the yaml or
interface changes to the python code. However, if you're looking to
start soon, I can try to get some of the docs done sooner. In the
meantime, there are a couple examples on how to interface with python
code:

 - https://bitbucket.org/fedoraqa/task-examplebodhi
 - https://bitbucket.org/fedoraqa/task-examplelong
 - https://bitbucket.org/fedoraqa/task-rpmlint


Outside of the python task work that would be needed to get valid
working, there are a couple of things I can think of that would be
likely be needed to do cloud image testing:

 - write a directive for finding and dealing with images
   * likely some combination of "find the new image", "upload it to a
     cloud system", "launch the image with a given configuration"

 - figure out how to record results
   * this should be pretty simple because I think cloud image results
     will be similar but not quite the same as package results.

 - determine what fedmsgs to trigger off of

> I can see two possible approaches:
> 1) 'Easy'. We create special type of task (sorry if I'm not that
> precise with terminology) which is being executed on static slave
> node which runs valid (in black-box mode) on newly uploaded image,
> grabs the result and returns it back.

I suspect that this will be the best way to get started. The
ephemeral client work you mention below is going to be a lot of work
and there are a bunch of unknowns (use openstack or not, how isolated
should the clients be etc.). If we want to have a shot at having some
of the automated cloud tests done for f21, I think that starting with
the easiest route would likely be best

> 2) 'Hard'. We work on 'ephemeral task clients' (term from
> http://tirfa.com/an-initial-idea-for-taskbot.html). The benefit here
> is that in that case we can run any test on a cloud VM. This approach
> will require some optimization from the very beginning, at least:
> 1. One VM should be user for multiple tests (so several dozens of
> image tests won't require several dozens of VMs created)
> 2. Tests should be able to 'control' VM (e.g. use cloud-specific
> features: attach device, reboot VM,...) - that can be workarounded by
> providing cloud credential inside the VM itself but that doesn't sound
> that attractive.

I suspect that some variation on this is going to be a better solution
in the long run but as I mentioned above, I think that there is little
chance of getting it working in time to be useful for f21. With the
easier option, there is a decent chance of getting something running in
the not-too-distant future.

> Yesterday I've tried setting up Taskotron (according to
> http://roshi.fedorapeople.org/dexy-themed/) on 3 F20 VMs in EC2 and
> actually I've failed (some ansible playbooks are out-of-sync with some
> git repos, e.g. 'master playbook' is failing with 'stderr: cp: cannot
> stat '/home/qaadmin/trigger/jobtrigger.py': No such file or directory'
> in 'TASK: [copy fedmsg config]'. I totally understand these
> instructions and repos are rather proof-of-concept and can be
> slightly outdated but I want to ask about 'minimalistic one-host dev
> environment' setup anyway. Can 'Host' be eliminated from the setup
> completely and can we merge Master and Slave (in case we'll need it
> for 'cloud') on one host? That would help a lot.

Yeah, that documentation is really out of date at this point. It was
mostly written as a proof of concept and I've been holding off on
updating it until the new playbooks are done (and they're starting to
get close - you can see them in the 'rolerefactoring' branch)

For a small setup, master and slave should be able to co-exist on the
same machine. I know buildbot supports it but I'm not sure that the
playbooks are written in a way that eliminating the 'Host' like that
would actually work.

To make things even easier, you can write tasks without doing the
entire taskotron setup process by using the runner in libtaskotron
(almost identical to how it's executed in production, just without all
the extra systems for scheduling, delegation and reporting). The
reporting functionality isn't quite done yet but the python execution
module is solid and as far as I know, it's ready to be used.

> I'm looking forward to hearing from you.

Apologies for the delayed response. Looking forward to working with you
on cloud image testing.

Tim

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20140415/aa1e316f/attachment.sig>