Taskotron Data Interfaces

[Kamil Paral]

> Some liberties will likely need to be taken with this concept when
> interfacing with external systems, the important part is that any
> single component from the trigger to the runner and reporting systems
> should be isolate-able by using a simple mock object/system fed by
> dictionaries.

This is a good goal.

> Trigger
> =======
> 
> The trigger is a piece of code which is responsible for starting the
> process of task execution. 

In this whole email you say 'task'. Do you mean 'check' in our usual terminology, or something else? Especially when describing the data structures.

> In our initial implementation, it will
> listen for fedmsgs and based on the content of those messages, schedule
> jobs with buildbot. The interface in this case is:
> 
> {
>     'item': <name of item - envr, update id, koji tag, etc.>,
>     'item_type': <koji_build, koji_tag, bodhi_update>,
>     'taskname': <task to be scheduled>,
>     'arch': <arch needed for execution, x86_64, i386 etc.>
> }

So this is data sent from Trigger to Buildbot, right?

> 
> 
> Buildbot
> ========
> 
> When jobs are scheduled by the trigger using the above data, buildbot
> will queue the task and delegate execution to an appropriate client
> machine based on the input data. The git repo corresponding to the
> taskname will be cloned on the client 

We will need to do some optimization here, otherwise the git servers won't be happy.

> but buildbot will add some input
> data. The resultant interface will be:
> 
> {
>     'item': <name of item>,
>     'item_type': <type of item>,
>     'build_id': <uuid of buildbot build, used for logs and results>,
>     'yamlfile': <taskname>.yml
> }

The yaml file is in the git repo, so I assume all the git repos are going to be cloned on the server as well, not just clients, right?

It makes sense, because the Trigger also needs to know what checks are available, therefore it must be available on the server.

> 
> 
> Runner
> ======
> 
> The runner is responsible for taking the specified yaml file and
> executing the directives contained within. In the case of CLI operation
> where there is no build_id, a sensible default will be used instead to
> indicate a filler value (-1000 for example).

What is -1000? Maybe just use None? Or don't include the field at all.

> 
> When running a task, the runner will copy most of its own input as the
> task's input:
> 
> {
>     'item': <name of item>,
>     'item_type': <type of item>,
>     'build_id': <uuid of buildbot build, used for logs and results>
> }
> 
> While the runner is responsible for running a task, each task is
> responsible for its own reporting mechanism. The output from the runner
> itself is in the form of execution information only and does not
> necessarily parallel the result of the executed task.
> 
> {
>     'execution_status': <OK or NOT_OK>,

Is this determined purely from the check exit code, or somehow else?

>     'status': <status message, possibly including any exeptions>

What goes here?

> }
> 
> 
> Task
> ====
> 
> I won't go too far into detail for this as it is being discussed in
> phabricator:
>  - https://phab.qadevel.cloud.fedoraproject.org/T84
> 
> I think that I've captured all that we've been talking about but I'm
> sure that Josef or Kamil will correct me if I've missed something :)
> 
> The primary idea is that tasks will generate a list of TAP results
> which can be farther processed into ResultsDB submissions or Bodhi
> comments (for now). While TAP isn't scrictly a dictionary, it is a
> reasonably well defined output format and when TAP13 is used with
> embedded YAML, we end up with something that doesn't diverge too much
> from a dictionary.
> 
> Example TAP output (stolen from D26):
> 
> ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
>   ---
>   details:
>     output: |-
>       xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
>       xchat.x86_64: W: non-conffile-in-etc
>                        /etc/gconf/schem/apps_xchat_url_handler.schemas
>       xchat.x86_64: W: no-manual-page-for-binary xchat
>   item: xchat-0.5-1.fc20
>   outcome: PASSED
>   summary: 5 ERRORS, 10 WARNINGS
>   type: koji_build
>   ...
>   not ok - $CHECKNAME for Bodhi update foobar-1.2-3.fc20  # FAIL

Just a note, this line and the "ok" line below should be aligned with the very first line.

>   ---
>   item: foobar-1.2-3.fc20
>   outcome: FAILED
>   summary: dependency error
>   type: bodhi_update
>   ...
>   ok - $CHECKNAME for Yum repository f20-updates
>   ---
>   item: f20-updates
>   outcome: INFO
>   summary: 2 stale updates
>   type: yum_repository
>   ...

We will need to add some additional lines here, e.g. log_url (computed from build_id). I'd like to do that transparently in our library without the user needing to care about anything.

> 
> Example ResultsDB post (stolen from T84):
> 
> {
>     "outcome": 'NEEDS_INSPECTION',
>     "testcase_name": 'upgradepath',
>     "summary": 'not ok upgradepath for build foo-1.2-3.fc20'
>     "result_data": {
>                      'item': 'foo-1.2-3.fc20',
>                      'type': 'koji_build',
>                      },
>     "job_id": <resultsdb id>,
>     "log_url": <path to log files>
> }

Thanks for the overview, Tim. I think the overall idea is good, and this is the way forward.