Taskbot: TAP vs Subunit

[Josef Skladanka]

Tim (and of course the rest of the gang ;)),

During our chat with Tim, we agreed that we'd really like to use some standardized 'output format' for the tests in Taskbot, to be a bit more programming-language/results-store-implementation agnostic.
We knew about two options - TAP <http://testanything.org/wiki/index.php/Main_Page> and Subunit <https://pypi.python.org/pypi/python-subunit>.

= Subunit =

At least in Python is quite tightly coupled with unittest, both ideologically and practically. I was unable to find a way to just produce a Subunit stream without the need of actually running a testsuite. 

The format is (basically) just plain PASS/FAIL/INFO/..., and there is possibility to add some 'details'results. It should also be possible to add an attachment to the end of the stream, but no result-specific data can be added (IMO).

Also Subunit is now in the process of transition to new implementation, which now should fix some issues with concurrency, add more result-states, etc., but will be binary, so human readability would quite suffer.

I do not really feel that this is a good match for our needs (at least without any significant hacking)

= TAP =

TAP is not unittest-specific, and is human-readable plaintext format.

It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).

The real issue with TAP is Python support. 
There is a TAP-consumer library created as an example for PyParsing <http://pyparsing.wikispaces.com/file/detail/TAP.py>, but it does not support the v13 protocol, and is quite useless as such.
TAP-producer library for Python <https://github.com/rjbs/pytap> is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.


= Result =

Although neither choice is ideal, I feel that TAP would be the better choice, even though it would mean implementing our own producer/parser.
Also the TAP is really simplistic format, so creating a TAP output should be quite easy even in any programming language.

It is possible that I somehow utterly misunderstood the Subunit concept, so it might be useful to contact some QAs currently using it (I thing Tim mentioned something about OpenStack?), or contact the developer directly.


J.