Taskotron Data Interfaces

Tue Mar 25 07:34:58 UTC 2014

On Thu, 20 Mar 2014 09:47:22 -0400 (EDT)
Kamil Paral <kparal at redhat.com> wrote:

> > Some liberties will likely need to be taken with this concept when
> > interfacing with external systems, the important part is that any
> > single component from the trigger to the runner and reporting
> > systems should be isolate-able by using a simple mock object/system
> > fed by dictionaries.
> 
> This is a good goal.
> 
> > Trigger
> > =======
> > 
> > The trigger is a piece of code which is responsible for starting the
> > process of task execution. 
> 
> In this whole email you say 'task'. Do you mean 'check' in our usual
> terminology, or something else? Especially when describing the data
> structures.
> 
> > In our initial implementation, it will
> > listen for fedmsgs and based on the content of those messages,
> > schedule jobs with buildbot. The interface in this case is:
> > 
> > {
> >     'item': <name of item - envr, update id, koji tag, etc.>,
> >     'item_type': <koji_build, koji_tag, bodhi_update>,
> >     'taskname': <task to be scheduled>,
> >     'arch': <arch needed for execution, x86_64, i386 etc.>
> > }
> 
> So this is data sent from Trigger to Buildbot, right?

With the current setup, yes.

> > 
> > 
> > Buildbot
> > ========
> > 
> > When jobs are scheduled by the trigger using the above data,
> > buildbot will queue the task and delegate execution to an
> > appropriate client machine based on the input data. The git repo
> > corresponding to the taskname will be cloned on the client 
> 
> We will need to do some optimization here, otherwise the git servers
> won't be happy.
> 
> > but buildbot will add some input
> > data. The resultant interface will be:
> > 
> > {
> >     'item': <name of item>,
> >     'item_type': <type of item>,
> >     'build_id': <uuid of buildbot build, used for logs and results>,
> >     'yamlfile': <taskname>.yml
> > }
> 
> The yaml file is in the git repo, so I assume all the git repos are
> going to be cloned on the server as well, not just clients, right?

Yeah, I had the repos mirrored on the master using gitolite but that
was causing as many problems as it was solving. I've switched over to
using grokmirror which doesn't have the same level of setup issues but
either way, the slaves will clone from the master instead of hitting
bitbucket, github or wherever the tasks are stored every time.

> It makes sense, because the Trigger also needs to know what checks
> are available, therefore it must be available on the server.

The initial deployment is going to be a bit primitive. Martin's working
on it but I believe the plan is to keep task names in the trigger's
config file for now.

I suspect that we'll want/need something more complicated when we start
taking user-submitted tasks, though.

> > 
> > 
> > Runner
> > ======
> > 
> > The runner is responsible for taking the specified yaml file and
> > executing the directives contained within. In the case of CLI
> > operation where there is no build_id, a sensible default will be
> > used instead to indicate a filler value (-1000 for example).
> 
> What is -1000? Maybe just use None? Or don't include the field at all.

-1000 was a number that I pulled out of nowhere which would be an
invalid build id. The idea was that it would be easy to determine that
"we're running standalone at the moment, this is a dummy value".

I don't have my heart set on any particular implementation/value as long
as the runner functions equally well in both production-ish and
standlone environments.

> > 
> > When running a task, the runner will copy most of its own input as
> > the task's input:
> > 
> > {
> >     'item': <name of item>,
> >     'item_type': <type of item>,
> >     'build_id': <uuid of buildbot build, used for logs and results>
> > }
> > 
> > While the runner is responsible for running a task, each task is
> > responsible for its own reporting mechanism. The output from the
> > runner itself is in the form of execution information only and does
> > not necessarily parallel the result of the executed task.
> > 
> > {
> >     'execution_status': <OK or NOT_OK>,
> 
> Is this determined purely from the check exit code, or somehow else?

Effectively, yes. It's determined by the execution engine, so any
not-ok exit code from the repo clone or task execution would affect the
execution_status.

> >     'status': <status message, possibly including any exeptions>
> 
> What goes here?

Error messages which would indicate why the execution failed if that
information is available.

> > }
> > 
> > 
> > Task
> > ====
> > 
> > I won't go too far into detail for this as it is being discussed in
> > phabricator:
> >  - https://phab.qadevel.cloud.fedoraproject.org/T84
> > 
> > I think that I've captured all that we've been talking about but I'm
> > sure that Josef or Kamil will correct me if I've missed something :)
> > 
> > The primary idea is that tasks will generate a list of TAP results
> > which can be farther processed into ResultsDB submissions or Bodhi
> > comments (for now). While TAP isn't scrictly a dictionary, it is a
> > reasonably well defined output format and when TAP13 is used with
> > embedded YAML, we end up with something that doesn't diverge too
> > much from a dictionary.
> > 
> > Example TAP output (stolen from D26):
> > 
> > ok - $CHECKNAME for Koji build xchat-0.5-1.fc20
> >   ---
> >   details:
> >     output: |-
> >       xchat.x86_64: W: file-not-utf8 /usr/share/doc/xchat/ChangeLog
> >       xchat.x86_64: W: non-conffile-in-etc
> >                        /etc/gconf/schem/apps_xchat_url_handler.schemas
> >       xchat.x86_64: W: no-manual-page-for-binary xchat
> >   item: xchat-0.5-1.fc20
> >   outcome: PASSED
> >   summary: 5 ERRORS, 10 WARNINGS
> >   type: koji_build
> >   ...
> >   not ok - $CHECKNAME for Bodhi update foobar-1.2-3.fc20  # FAIL
> 
> Just a note, this line and the "ok" line below should be aligned with
> the very first line.

A side effect of the manual formatting I did.

> >   ---
> >   item: foobar-1.2-3.fc20
> >   outcome: FAILED
> >   summary: dependency error
> >   type: bodhi_update
> >   ...
> >   ok - $CHECKNAME for Yum repository f20-updates
> >   ---
> >   item: f20-updates
> >   outcome: INFO
> >   summary: 2 stale updates
> >   type: yum_repository
> >   ...
> 
> We will need to add some additional lines here, e.g. log_url
> (computed from build_id). I'd like to do that transparently in our
> library without the user needing to care about anything.

Yeah, agreed on making the log url easy. I just wasn't sure if that
needed to be in the TAP output or not.

> > 
> > Example ResultsDB post (stolen from T84):
> > 
> > {
> >     "outcome": 'NEEDS_INSPECTION',
> >     "testcase_name": 'upgradepath',
> >     "summary": 'not ok upgradepath for build foo-1.2-3.fc20'
> >     "result_data": {
> >                      'item': 'foo-1.2-3.fc20',
> >                      'type': 'koji_build',
> >                      },
> >     "job_id": <resultsdb id>,
> >     "log_url": <path to log files>
> > }
> 
> Thanks for the overview, Tim. I think the overall idea is good, and
> this is the way forward.

Glad to hear that there weren't any huge discrepancies between what we
were thinking.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20140325/e12a4ca7/attachment.sig>