RFC: Taskotron task description format

[Tim Flink]

I've been working on this for a bit but finally have enough together
that we can start talking about the task description format for
taskotron.

The code to actually run tasks (well, task. for now, rpmlint is all
that's supported) is up at:

https://bitbucket.org/fedoraqa/libtaskotron-demo

The runner code isn't very good yet but I didn't want to wait until it
was done before starting to discuss the task description format.

Continuing the use of rpmlint as an example, I have a task repository
set up on bitbucket:

https://bitbucket.org/fedoraqa/task-rpmlint

At the moment, that task yaml file for rpmlint [1] looks like:

dependencies:
      - rpmlint
      - libtaskbot

input:
      args: envr,arch

preparation:
      koji: download $envr

execution:
      python: run_rpmlint.py $workdir

post:
      shell: clean $workdir

report:
      resultdb: something

[1]https://bitbucket.org/fedoraqa/task-rpmlint/src/be90c723a11f3cec3a7ee72e378a...

In a nutshell, the idea is to keep repetitive code out of the tasks
without requiring everything to be written in the same language or
porting the library to any language people want to use (perl, c, bash
etc.).

The execution of the task is described in the yml file and eventually
delegates actual execution to the python file described in that yml
file. Output, for now, is in TAP format. The runner just spits it out
as text for the moment but eventually, that TAP output will be used for
reporting the results.

Items prepended with '$' represent variables - either things determined
at runtime (the working directory) or things that are arguments to the
specific task run (the envr of packages to check).

Examples of what future features could look like:

execution:
      beaker: job=somejob.xml distro=$latestfedora

preparation:
      bodhi: download $updates

preparation:
      openstack: image=imagename config=someconfigtype size=m1.small

execution:
       infinity: config=test/sometest.cfg

Any thoughts on whether this direction is a good direction to take or
suggestions on how the format could be improved? Eventually, I want to
send this out to devel@ for input but figured that the smaller group of
folks here would be a good start.

Thanks,

Tim

[Josef Skladanka]

Hi Tim,

sorry for the late reply, this somewhat slipped my mind :(

Overall, I like the concept, and although I understand that this is Proof-of-Concept, I'm a bit worried about the get_argparser() method https://bitbucket.org/fedoraqa/libtaskotron-demo/src/8e6c39c8cfb2fb659888b00f42c70b4c4e1a4c0d/libtaskotron/runner.py?at=master#cl-86.

Would it mean, that we need to know all the possible arguments in advance? Or is this just a simple piece of code intended for easy-to-use demo?

J.

[Tim Flink]

On Mon, 6 Jan 2014 08:04:06 -0500 (EST)
Josef Skladanka  wrote:
> Hi Tim,
>
> sorry for the late reply, this somewhat slipped my mind :(
>
> Overall, I like the concept, and although I understand that this is
> Proof-of-Concept, I'm a bit worried about the get_argparser() method
> https://bitbucket.org/fedoraqa/libtaskotron-demo/src/8e6c39c8cfb2fb659888b00f42c70b4c4e1a4c0d/libtaskotron/runner.py?at=master#cl-86.
>
> Would it mean, that we need to know all the possible arguments in
> advance? Or is this just a simple piece of code intended for
> easy-to-use demo?

It's just demo code - I wanted to get something working to demonstrate
the description format and cut a few corners.

Assuming that I understand your concern, my thought is to have a
set of standard args like envr, release, arch etc. but allow for other
args on a more-or-less per-task basis. The actual number of args will
be limited by the types of input we trigger on, though.

That being said, I'm not 100% sure that I understand what your concern
is. Could you be a bit more specific about what you're concerned about?

Tim

[Kamil Paral]

I've finally read it as well :) Comments below.
> At the moment, that task yaml file for rpmlint [1] looks like:
>
> dependencies:
>       - rpmlint
>       - libtaskbot
>
> input:
>       args: envr,arch

These arguments are provided by the event trigger, right? So, the event trigger offers a set of arguments and we pick some of them inside "input" section; they can be then used in other sections. Is that right? ('arch' is not actually used anywhere in this demo file).
> preparation:
>       koji: download $envr

This is great, a preparation phase is exactly what I wanted.
> execution:
>       python: run_rpmlint.py $workdir

1) You don't actually use $workdir in run_rpmlint.py. I guess that's a bug? Or will we guarantee that CWD is set to $workdir?

2) Can we have multiple execution lines that are run sequentially (or in parallel?) and then consider the aggregate result as a test result? One example is in our outdated rats_install test:
for ks in ['fedora','updates','updates-testing']:
    job.run_test('rats_install', tag=ks, ks=ks, **autoqa_args)
We can of course define this as three separate tests, but sometimes running a few different variants together in a single test run might make sense.
> post:
>       shell: clean $workdir

How is this supposed to work? 'clean' is not an existing command, is that a keyword?

Will we support running arbitrary shell commands or custom scripts (e.g. './myscript') in this or other phases (like preparation)?
> report:
>       resultdb: something

I guess it's possible to run and report multiple results from a single test run? I.e. upgradepath checking 20 builds and reporting 20 separate test results.

Can I also report a single result from several execution lines?

I know this is just a demo and we will probably need to discover the answers for many of the questions above collectively. I was trying to think about different use cases a maybe discover some missing ones.

[Tim Flink]

On Tue, 7 Jan 2014 11:23:02 -0500 (EST)
Kamil Paral  wrote:
> I've finally read it as well :) Comments below.

Thanks for the comments, I really appreciate it.
> > At the moment, that task yaml file for rpmlint [1] looks like:
> >
> > dependencies:
> >       - rpmlint
> >       - libtaskbot
> >
> > input:
> >       args: envr,arch
>
> These arguments are provided by the event trigger, right? So, the
> event trigger offers a set of arguments and we pick some of them
> inside "input" section; they can be then used in other sections. Is
> that right? ('arch' is not actually used anywhere in this demo file).

At the moment, it's mostly verification that the correct args were
passed in but that sums the idea up pretty well.

Yeah, rpmlint isn't arch specific - I had it in there as a test and
probably should have taken it out.
> > preparation:
> >       koji: download $envr
>
> This is great, a preparation phase is exactly what I wanted.
> > execution:
> >       python: run_rpmlint.py $workdir
> You don't actually use $workdir in run_rpmlint.py. I guess that's
>
> a bug? Or will we guarantee that CWD is set to $workdir?

Yeah, the variables haven't been implemented yet. I want to start using
something like jinja2 for variable substitution (which would change
variables to {{ workdir }}, but I see that as a minor change) but
haven't implemented the code for that yet.

In the case of $workdir, that variable would be expanded to the
directory where the task is being run by the runner @ runtime.
> Can we have multiple execution lines that are run sequentially (or
>
> in parallel?) and then consider the aggregate result as a test
> result? One example is in our outdated rats_install test: for ks in
> ['fedora','updates','updates-testing']: job.run_test('rats_install',
> tag=ks, ks=ks, **autoqa_args) We can of course define this as three
> separate tests, but sometimes running a few different variants
> together in a single test run might make sense.

At the moment, no. I hadn't really thought about that, to be honest.

The problem that I see with multiple steps is how to express
interactions with yaml. How would we send output from one step to
another? Would we need to exchange data between steps?

If we're talking about multiple, independent (ie, don't require
communication between them) steps that each produce results which are
aggregated, I don't see much of an issue. If we're talking about
combining results or communication between execution steps, that could
end up being a bit more complicated.
> > post:
> >       shell: clean $workdir
>
> How is this supposed to work? 'clean' is not an existing command, is
> that a keyword?

It's a potentially outdated concept that I've been using for the
previous taskotron demo deployments - it deletes any downloaded files
and stuff created while running.

If we do end up needing a 'clean', it should be a command in itself
instead of a shell command.
> Will we support running arbitrary shell commands or custom scripts
> (e.g. './myscript') in this or other phases (like preparation)?

Eventually, yes. The code is not implemented in the demo yet, though.
> > report:
> >       resultdb: something
>
> I guess it's possible to run and report multiple results from a
> single test run? I.e. upgradepath checking 20 builds and reporting 20
> separate test results.

That's the use case that I had in mind initially, yes. The execution
phase would output 1 or more results which would each be fed into the
report section.
> Can I also report a single result from several execution lines?

Do we have a use case for this yet? I'd be tempted to put any multiple
execution steps into a single python (or other supported language)
script.

The flow I had in mind is admittedly rather not-just-stdout centric.
Execution would produce logging messages on stdout and return
TAP/subunit separately. Each TAP/subunit result would be reported,
assuming that a report section exists.

I suppose that we could add a "resultsmunging" section where all
results are fed in and the results are used for reporting but I'm not
sure that's a route we really want to take.

Off the top of my head example:
----------------------------------------------------------------------
...
execution:
  python: run_something.py method=first_part workdir=$workdir
  python: run_something.py method=second_part workdir=$workdir

resultmunging:
  python: munge_results.py output=mungedresults

report:
  resultsdb: $mungedresults
----------------------------------------------------------------------

At what point do we just limit the functionality of the runner and tell
folks that they need to do the fancy stuff in the task code called in
the execution phase?
> I know this is just a demo and we will probably need to discover the
> answers for many of the questions above collectively. I was trying to
> think about different use cases a maybe discover some missing ones.

That's pretty much what I was hoping to do with the demo - spark
discussion. At some point, we should probably take this to devel@ to
see what people would do with taskotron but for now, I'm tempted to
limit the functionality of the yaml based descriptions and wait until
we have a working system before soliciting feature requests.

Another question is whether this yaml based task description is worth
doing or if we should just use a python-based task format similar to
what autotest does (inherit from a python class and override methods as
needed).

Tim

[Kamil Paral]

> Another question is whether this yaml based task description is worth
> doing or if we should just use a python-based task format similar to
> what autotest does (inherit from a python class and override methods as
> needed).

I find this yaml format much more readable and simple. It's definitely more restricted than what you can do in a full programming language. But unless we really need the full programming power, I'd rather stay on KISS side.

[Nick Coghlan]

On 01/08/2014 09:51 PM, Kamil Paral wrote:
> > Another question is whether this yaml based task description is worth
> > doing or if we should just use a python-based task format similar to
> > what autotest does (inherit from a python class and override methods as
> > needed).
>
> I find this yaml format much more readable and simple. It's definitely more restricted than what you can do in a full programming language. But unless we really need the full programming power, I'd rather stay on KISS side.

And so long as you parse it in safe mode, YAML is much easier to trust
as a configuration format.
> From LCA last week, two OpenStack infrastructure projects potentially

worth mining for inspiration in relation to Taskotron (including
definition formats) would be:

Jenkins Job Builder:
http://ci.openstack.org/jenkins-job-builder/configuration.html
Zuul (project gating system):
http://ci.openstack.org/zuul/zuul.html#layout-yaml

Kamil already linked to my talk on Beaker from the Monday automation
miniconf, but there are some good OpenStack ones from the Friday as part
of the main conf: http://mirror.linux.org.au/linux.conf.au/2014/Friday/

Cheers,
Nick.