Taskbot: TAP vs Subunit

[Tim Flink]

On Thu, 31 Oct 2013 07:17:03 -0400 (EDT)
Josef Skladanka <jskladan at redhat.com> wrote:

Bah, just realized this only went to Josef the first time I sent it. He
gets two copies!

> Tim (and of course the rest of the gang ;)),
> 
> During our chat with Tim, we agreed that we'd really like to use some
> standardized 'output format' for the tests in Taskbot, to be a bit
> more programming-language/results-store-implementation agnostic. We
> knew about two options - TAP
> <http://testanything.org/wiki/index.php/Main_Page> and Subunit
> <https://pypi.python.org/pypi/python-subunit>.
> 
> = Subunit =
> 
> At least in Python is quite tightly coupled with unittest, both
> ideologically and practically. I was unable to find a way to just
> produce a Subunit stream without the need of actually running a
> testsuite. 
> 
> The format is (basically) just plain PASS/FAIL/INFO/..., and there is
> possibility to add some 'details'results. It should also be possible
> to add an attachment to the end of the stream, but no result-specific
> data can be added (IMO).
> 
> Also Subunit is now in the process of transition to new
> implementation, which now should fix some issues with concurrency,
> add more result-states, etc., but will be binary, so human
> readability would quite suffer.
> 
> I do not really feel that this is a good match for our needs (at
> least without any significant hacking)  

Yeah, this matches what I remember finding when I looked into this a
couple of months ago. It looks like a better protocol overall but it's
going to be quite a bit more upfront work to support than TAP. I know
that openstack's test suite uses subunit for nova and neutron [1]

[1] https://wiki.openstack.org/wiki/Testr

Subunit has been recently added to the fedora repos but I suppose
that's of little utility for us if we end up having to re-implement
parts to get away from UnitTest.

> = TAP =
> 
> TAP is not unittest-specific, and is human-readable plaintext format.
> 
> It also has just PASS/FAIL logic, but there is a possibility to add
> YAML 'metadata' to any result (since TAP v. 13).
> 
> The real issue with TAP is Python support. 
> There is a TAP-consumer library created as an example for PyParsing
> <http://pyparsing.wikispaces.com/file/detail/TAP.py>, but it does not
> support the v13 protocol, and is quite useless as such. TAP-producer
> library for Python <https://github.com/rjbs/pytap> is also using the
> old version (i.e. no YAML extensions), and seems to be dead (2 years
> since last update). It is also quite badly written.  

Another option for TAP emission is bayeux [2] which was created for a
better, less perl-ish interface to tap [3]. It might be worth asking
mcepl if he has any thoughts on TAP vs. subunit vs. something else.

[2] http://luther.ceplovi.cz/git/bayeux.git/
[3]http://lists.idyll.org/pipermail/testing-in-python/2012-March/004881.html

> = Result =
> 
> Although neither choice is ideal, I feel that TAP would be the better
> choice, even though it would mean implementing our own
> producer/parser. Also the TAP is really simplistic format, so
> creating a TAP output should be quite easy even in any programming
> language.  

I think that TAP is certainly appears to be the simpler solution for
now and I suspect it's a bit more widely used than subunit. As long as
the amount of work required to have a TAP emitter and parser isn't
crazy, I agree that TAP is a better choice.

> It is possible that I somehow utterly misunderstood the Subunit
> concept, so it might be useful to contact some QAs currently using it
> (I thing Tim mentioned something about OpenStack?), or contact the
> developer directly.  

I suspect that you found this when searching but there's a direct
comparison of the two by someone with more of a TAP background:
http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-protocol-and-subunit.html

Thanks for looking into this,

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20131031/ee93adc0/attachment.sig>