RFC: Taskotron task description format

[Tim Flink]

On Tue, 7 Jan 2014 11:23:02 -0500 (EST)
Kamil Paral <kparal at redhat.com> wrote:

> I've finally read it as well :) Comments below.

Thanks for the comments, I really appreciate it.

> > At the moment, that task yaml file for rpmlint [1] looks like:
> > 
> >   dependencies:
> >       - rpmlint
> >       - libtaskbot
> > 
> >   input:
> >       args: envr,arch
> 
> These arguments are provided by the event trigger, right? So, the
> event trigger offers a set of arguments and we pick some of them
> inside "input" section; they can be then used in other sections. Is
> that right? ('arch' is not actually used anywhere in this demo file).

At the moment, it's mostly verification that the correct args were
passed in but that sums the idea up pretty well.

Yeah, rpmlint isn't arch specific - I had it in there as a test and
probably should have taken it out.

> > 
> >   preparation:
> >       koji: download $envr
> 
> This is great, a preparation phase is exactly what I wanted.
> 
> > 
> >   execution:
> >       python: run_rpmlint.py $workdir
> 
> 1) You don't actually use $workdir in run_rpmlint.py. I guess that's
> a bug? Or will we guarantee that CWD is set to $workdir?

Yeah, the variables haven't been implemented yet. I want to start using
something like jinja2 for variable substitution (which would change
variables to {{ workdir }}, but I see that as a minor change) but
haven't implemented the code for that yet.

In the case of $workdir, that variable would be expanded to the
directory where the task is being run by the runner @ runtime.

> 2) Can we have multiple execution lines that are run sequentially (or
> in parallel?) and then consider the aggregate result as a test
> result? One example is in our outdated rats_install test: for ks in
> ['fedora','updates','updates-testing']: job.run_test('rats_install',
> tag=ks, ks=ks, **autoqa_args) We can of course define this as three
> separate tests, but sometimes running a few different variants
> together in a single test run might make sense.

At the moment, no. I hadn't really thought about that, to be honest.

The problem that I see with multiple steps is how to express
interactions with yaml. How would we send output from one step to
another? Would we need to exchange data between steps?

If we're talking about multiple, independent (ie, don't require
communication between them) steps that each produce results which are
aggregated, I don't see much of an issue. If we're talking about
combining results or communication between execution steps, that could
end up being a bit more complicated.

> > 
> >   post:
> >       shell: clean $workdir
> 
> How is this supposed to work? 'clean' is not an existing command, is
> that a keyword?

It's a potentially outdated concept that I've been using for the
previous taskotron demo deployments - it deletes any downloaded files
and stuff created while running.

If we do end up needing a 'clean', it should be a command in itself
instead of a shell command.

> Will we support running arbitrary shell commands or custom scripts
> (e.g. './myscript') in this or other phases (like preparation)?

Eventually, yes. The code is not implemented in the demo yet, though.

> > 
> >   report:
> >       resultdb: something
> 
> I guess it's possible to run and report multiple results from a
> single test run? I.e. upgradepath checking 20 builds and reporting 20
> separate test results.

That's the use case that I had in mind initially, yes. The execution
phase would output 1 or more results which would each be fed into the
report section.

> Can I also report a single result from several execution lines?

Do we have a use case for this yet? I'd be tempted to put any multiple
execution steps into a single python (or other supported language)
script.

The flow I had in mind is admittedly rather not-just-stdout centric.
Execution would produce logging messages on stdout and return
TAP/subunit separately. Each TAP/subunit result would be reported,
assuming that a report section exists.

I suppose that we could add a "resultsmunging" section where all
results are fed in and the results are used for reporting but I'm not
sure that's a route we really want to take.

Off the top of my head example:
----------------------------------------------------------------------
...
execution:
  python: run_something.py method=first_part workdir=$workdir
  python: run_something.py method=second_part workdir=$workdir

resultmunging:
  python: munge_results.py output=mungedresults

report:
  resultsdb: $mungedresults
----------------------------------------------------------------------

At what point do we just limit the functionality of the runner and tell
folks that they need to do the fancy stuff in the task code called in
the execution phase?

> I know this is just a demo and we will probably need to discover the
> answers for many of the questions above collectively. I was trying to
> think about different use cases a maybe discover some missing ones.

That's pretty much what I was hoping to do with the demo - spark
discussion. At some point, we should probably take this to devel@ to
see what people would do with taskotron but for now, I'm tempted to
limit the functionality of the yaml based descriptions and wait until
we have a working system before soliciting feature requests.

Another question is whether this yaml based task description is worth
doing or if we should just use a python-based task format similar to
what autotest does (inherit from a python class and override methods as
needed).

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20140107/be6d9ed9/attachment.sig>