Tim (and of course the rest of the gang ;)),
During our chat with Tim, we agreed that we'd really like to use some standardized 'output format' for the tests in Taskbot, to be a bit more programming-language/results-store-implementation agnostic. We knew about two options - TAP http://testanything.org/wiki/index.php/Main_Page and Subunit https://pypi.python.org/pypi/python-subunit.
= Subunit =
At least in Python is quite tightly coupled with unittest, both ideologically and practically. I was unable to find a way to just produce a Subunit stream without the need of actually running a testsuite.
The format is (basically) just plain PASS/FAIL/INFO/..., and there is possibility to add some 'details'results. It should also be possible to add an attachment to the end of the stream, but no result-specific data can be added (IMO).
Also Subunit is now in the process of transition to new implementation, which now should fix some issues with concurrency, add more result-states, etc., but will be binary, so human readability would quite suffer.
I do not really feel that this is a good match for our needs (at least without any significant hacking)
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support. There is a TAP-consumer library created as an example for PyParsing http://pyparsing.wikispaces.com/file/detail/TAP.py, but it does not support the v13 protocol, and is quite useless as such. TAP-producer library for Python https://github.com/rjbs/pytap is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.
= Result =
Although neither choice is ideal, I feel that TAP would be the better choice, even though it would mean implementing our own producer/parser. Also the TAP is really simplistic format, so creating a TAP output should be quite easy even in any programming language.
It is possible that I somehow utterly misunderstood the Subunit concept, so it might be useful to contact some QAs currently using it (I thing Tim mentioned something about OpenStack?), or contact the developer directly.
J.
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support. There is a TAP-consumer library created as an example for PyParsing http://pyparsing.wikispaces.com/file/detail/TAP.py, but it does not support the v13 protocol, and is quite useless as such. TAP-producer library for Python https://github.com/rjbs/pytap is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.
Doesn't autotest support TAP results storage? How do they do it?
Lucas, can you comment? Thanks.
On 10/31/2013 01:23 PM, Kamil Paral wrote:
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support. There is a TAP-consumer library created as an example for PyParsing http://pyparsing.wikispaces.com/file/detail/TAP.py, but it does not support the v13 protocol, and is quite useless as such. TAP-producer library for Python https://github.com/rjbs/pytap is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.
Doesn't autotest support TAP results storage? How do they do it?
Autotest does support recording results in TAP format, due to the work made by a guy from AMD. Breaking the expectations that a TAP enabled test tool would just output tap on standard output, the autotest client can optionally produce files with the results in tap format, and those files can be further processed by other systems.
Let me make some runs here and show you examples.
Lucas, can you comment? Thanks.
On 10/31/2013 01:23 PM, Kamil Paral wrote:
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support. There is a TAP-consumer library created as an example for PyParsing http://pyparsing.wikispaces.com/file/detail/TAP.py, but it does not support the v13 protocol, and is quite useless as such. TAP-producer library for Python https://github.com/rjbs/pytap is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.
Doesn't autotest support TAP results storage? How do they do it?
Lucas, can you comment? Thanks.
Here you can see an example:
$ client/autotest-local --tap client/tests/sleeptest/control 14:11:58 INFO | Writing results to /home/lmr/Code/autotest.git/autotest/client/results/default 14:11:58 INFO | Could not symlink init scripts (lack of permissions) 14:11:58 INFO | Not logging /proc/slabinfo (lack of permissions) 14:11:58 INFO | START ---- ---- timestamp=1383235918 localtime=Oct 31 14:11:58 14:11:58 INFO | START sleeptest sleeptest timestamp=1383235918 localtime=Oct 31 14:11:58 14:11:58 INFO | Not logging /proc/slabinfo (lack of permissions) 14:11:59 INFO | Not logging /proc/slabinfo (lack of permissions) 14:11:59 INFO | Not logging /var/log/messages (lack of permissions) 14:11:59 INFO | GOOD sleeptest sleeptest timestamp=1383235919 localtime=Oct 31 14:11:59 completed successfully 14:11:59 INFO | END GOOD sleeptest sleeptest timestamp=1383235919 localtime=Oct 31 14:11:59 14:11:59 INFO | END GOOD ---- ---- timestamp=1383235919 localtime=Oct 31 14:11:59 14:11:59 INFO | Report successfully generated at /home/lmr/Code/autotest.git/autotest/client/results/default/job_report.html
$ cd /home/lmr/Code/autotest.git/autotest/client/results/default/
$ ls control control.state debug job_report.html meta.yml sleeptest status status.json status.tap sysinfo tap.tar.gz
$ cat meta.yml file_order: - status.tap - sleeptest/status.tap - sleeptest/keyval.tap
$ cat status.tap 1..2 ok 1 - sleeptest ok 2 - job
$ cat sleeptest/status.tap 1..1 ok 1 - sleeptest
$ cat sleeptest/keyval.tap 1..2 ok 1 - results --- param-seconds: 1 version: 1 ... ok 2 - results --- sysinfo-cmdline: BOOT_IMAGE=/vmlinuz-3.11.6-301.fc20.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.md=0 rd.dm=0 rd.luks.uuid=luks-8c032338-feaa-456a-873c-a08c768b8861 vconsole.keymap=us rd.lvm.lv=fedora/root rhgb quiet LANG=en_US.utf8 sysinfo-memtotal-in-kb: 8061220 sysinfo-phys-mbytes: 8192 sysinfo-uname: 3.11.6-301.fc20.x86_64 #1 SMP Mon Oct 21 21:54:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux ...
Lucas,
do you use any library for producing TAP format? Also, do you have any TAP parser, or do you just emit it? I was looking for something in Python, but all I got is either outdated, or non-complete.
Thanks,
Joza
On 10/31/2013 02:27 PM, Josef Skladanka wrote:
Lucas,
do you use any library for producing TAP format?
No, at least not in the sense of an external project. Producing TAP in the client is an autotest specific implementation. It is the TAPReport() object defined in:
https://github.com/autotest/autotest/blob/master/client/shared/base_job.py#L...
Also, do you have any TAP parser, or do you just emit it?
Currently, autotest only outputs TAP, there's no parser available.
Cleber from my time was evaluating writing one, but we didn't go forward with it.
Matěj Cepl, also a Red Hat employee, implemented one parser at some point, but I think he felt discouraged because people did not give feedback to him (he did ask, though). I for one am also guilty of not following up with his work. He used to have it on his github account, but I can't seem to find it.
https://github.com/mcepl?tab=repositories
I was looking for something in Python, but all I got is either outdated, or non-complete.
I wish we had a better answer to give you, but unfortunately we don't.
On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka jskladan@redhat.com wrote:
Bah, just realized this only went to Josef the first time I sent it. He gets two copies!
Tim (and of course the rest of the gang ;)),
During our chat with Tim, we agreed that we'd really like to use some standardized 'output format' for the tests in Taskbot, to be a bit more programming-language/results-store-implementation agnostic. We knew about two options - TAP http://testanything.org/wiki/index.php/Main_Page and Subunit https://pypi.python.org/pypi/python-subunit.
= Subunit =
At least in Python is quite tightly coupled with unittest, both ideologically and practically. I was unable to find a way to just produce a Subunit stream without the need of actually running a testsuite.
The format is (basically) just plain PASS/FAIL/INFO/..., and there is possibility to add some 'details'results. It should also be possible to add an attachment to the end of the stream, but no result-specific data can be added (IMO).
Also Subunit is now in the process of transition to new implementation, which now should fix some issues with concurrency, add more result-states, etc., but will be binary, so human readability would quite suffer.
I do not really feel that this is a good match for our needs (at least without any significant hacking)
Yeah, this matches what I remember finding when I looked into this a couple of months ago. It looks like a better protocol overall but it's going to be quite a bit more upfront work to support than TAP. I know that openstack's test suite uses subunit for nova and neutron [1]
[1] https://wiki.openstack.org/wiki/Testr
Subunit has been recently added to the fedora repos but I suppose that's of little utility for us if we end up having to re-implement parts to get away from UnitTest.
= TAP =
TAP is not unittest-specific, and is human-readable plaintext format.
It also has just PASS/FAIL logic, but there is a possibility to add YAML 'metadata' to any result (since TAP v. 13).
The real issue with TAP is Python support. There is a TAP-consumer library created as an example for PyParsing http://pyparsing.wikispaces.com/file/detail/TAP.py, but it does not support the v13 protocol, and is quite useless as such. TAP-producer library for Python https://github.com/rjbs/pytap is also using the old version (i.e. no YAML extensions), and seems to be dead (2 years since last update). It is also quite badly written.
Another option for TAP emission is bayeux [2] which was created for a better, less perl-ish interface to tap [3]. It might be worth asking mcepl if he has any thoughts on TAP vs. subunit vs. something else.
[2] http://luther.ceplovi.cz/git/bayeux.git/ [3]http://lists.idyll.org/pipermail/testing-in-python/2012-March/004881.html
= Result =
Although neither choice is ideal, I feel that TAP would be the better choice, even though it would mean implementing our own producer/parser. Also the TAP is really simplistic format, so creating a TAP output should be quite easy even in any programming language.
I think that TAP is certainly appears to be the simpler solution for now and I suspect it's a bit more widely used than subunit. As long as the amount of work required to have a TAP emitter and parser isn't crazy, I agree that TAP is a better choice.
It is possible that I somehow utterly misunderstood the Subunit concept, so it might be useful to contact some QAs currently using it (I thing Tim mentioned something about OpenStack?), or contact the developer directly.
I suspect that you found this when searching but there's a direct comparison of the two by someone with more of a TAP background: http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-pro...
Thanks for looking into this,
Tim
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/01/2013 03:53 AM, Tim Flink wrote:
On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka jskladan@redhat.com wrote:
I do not really feel that this is a good match for our needs (at least without any significant hacking)
Yeah, this matches what I remember finding when I looked into this a couple of months ago. It looks like a better protocol overall but it's going to be quite a bit more upfront work to support than TAP. I know that openstack's test suite uses subunit for nova and neutron [1]
[1] https://wiki.openstack.org/wiki/Testr
Subunit has been recently added to the fedora repos but I suppose that's of little utility for us if we end up having to re-implement parts to get away from UnitTest.
As Tim noted, subunit development is tied in to the OpenStack project's "test repository" (testr) project.
So, if you were going to go down the subunit path, I'd suggest looking at leveraging (at least pieces of) testr for result storage as well.
While it's not an urgent requirement, we also plan to explore adding subunit support to Beaker.
Robert Collins gave a decent talk on testr (et al) at the PyCon AU 2013 OpenStack miniconf: https://www.youtube.com/watch?v=Oe_HhBBbqbw
Looking at the list of Producers and Consumers on the TAP wiki doesn't give me any confidence that *anyone* has successfully used TAP at scale or as part of a distributed continuous integration system, nor does it look like it has anyone providing the level of investment HP and others are putting into the OpenStack testing infrastructure (Robert works for HP, mostly on the OpenStack-on-OpenStack project).
Even if subunit is more work initially, you'll likely enjoy flow-on effects in the future from those OpenStack investments (similar to the way LMR decided to outsource the "inventory management and scaling to multiple labs" problems for autotest to Beaker, so autotest gets to avoid doubling up on work we were going to do anyway).
With subunit as the more expressive protocol, it also means it would be easier to convert TAP streams to subunit streams than the other way around, so building the back end to be subunit based doesn't rule out the possibility of permitting TAP based input in the future. By contrast, using TAP as the base *will* almost certainly rule out accepting subunit streams from testr or Beaker.
Regards, Nick.
- -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead Beaker Development Lead (http://beaker-project.org/)
On Fri, 01 Nov 2013 13:45:08 +1000 Nick Coghlan ncoghlan@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/01/2013 03:53 AM, Tim Flink wrote:
On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka jskladan@redhat.com wrote:
I do not really feel that this is a good match for our needs (at least without any significant hacking)
Yeah, this matches what I remember finding when I looked into this a couple of months ago. It looks like a better protocol overall but it's going to be quite a bit more upfront work to support than TAP. I know that openstack's test suite uses subunit for nova and neutron [1]
[1] https://wiki.openstack.org/wiki/Testr
Subunit has been recently added to the fedora repos but I suppose that's of little utility for us if we end up having to re-implement parts to get away from UnitTest.
As Tim noted, subunit development is tied in to the OpenStack project's "test repository" (testr) project.
To add confusion, they're using an upstream project called testrepository:
https://testrepository.readthedocs.org/en/latest/MANUAL.html
I remember running into problems actually running testr a month or two ago but I got pulled away by something else and never finished looking into it. I just ran it again and it didn't have any issues, so I'm either mis-remembering or my OS upgrade since then ended up fixing the issue.
So, if you were going to go down the subunit path, I'd suggest looking at leveraging (at least pieces of) testr for result storage as well.
While it's not an urgent requirement, we also plan to explore adding subunit support to Beaker.
Ah, I seem to recall talking about that at Flock and it slipped my mind. Thanks for the reminder as this could end up being important for integration.
Robert Collins gave a decent talk on testr (et al) at the PyCon AU 2013 OpenStack miniconf: https://www.youtube.com/watch?v=Oe_HhBBbqbw
Looking at the list of Producers and Consumers on the TAP wiki doesn't give me any confidence that *anyone* has successfully used TAP at scale or as part of a distributed continuous integration system, nor does it look like it has anyone providing the level of investment HP and others are putting into the OpenStack testing infrastructure (Robert works for HP, mostly on the OpenStack-on-OpenStack project).
I think that AMD uses or was using TAP but yeah, I'm not really seeing much of anyone else who is using TAP on a large scale.
My main concern is how tightly coupled SubUnit and testrepository are to UnitTest. I took another quick look at their code and one of the first things I noticed is the description in a header [1]:
# subunit: extensions to Python unittest to get test results from subprocesses.
There are other things that raise flags for our usage at first glance, but I only looked at it for a few minutes and could easily be very wrong :)
I'm intrigued by some of the things that subunit can do/support (embedding files and the idea of replaying tests in particular - I think they have the potential to solve some problems we're eventually going to hit). I think that the short term question may end up being whether we want to focus on better stuff from the beginning or to "rough in" semi-disposable components (knowing or ignoring the fact that those are the types of things which tend to live longer than they should) to get a better idea of where the unknown pain points are.
I can't really argue with xUnit's success as a unit testing paradigm but I'm not crazy about getting too locked into that right now. One of the goals that I have for taskbot is to make it flexible enough to be able to do stuff that isn't traditionally in the domain of a test system. However, I'm not quite crazy enough to try being everything to everyone so that goal may not end up feasible.
[1]http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/python/subunit...
Even if subunit is more work initially, you'll likely enjoy flow-on effects in the future from those OpenStack investments (similar to the way LMR decided to outsource the "inventory management and scaling to multiple labs" problems for autotest to Beaker, so autotest gets to avoid doubling up on work we were going to do anyway).
Assuming that what we're trying to do is similar enough, yes. If it's different enough that we end up modifying their code or re-implementing it ourselves, that big investment could end up being a pain point. The documentation states that v2 is not yet a stable protocol and current implementers should be prepared to deal with non-backwards-compatible changes.
I'm not that familiar with what exactly OpenStack is doing with subunit and testr, so I can't say how much overlap there actually is.
With subunit as the more expressive protocol, it also means it would be easier to convert TAP streams to subunit streams than the other way around, so building the back end to be subunit based doesn't rule out the possibility of permitting TAP based input in the future. By contrast, using TAP as the base *will* almost certainly rule out accepting subunit streams from testr or Beaker.
Yeah, it does sound like this topic needs more investigation and maybe some example/poc code. Most of my concerns will end up being academic if using subunit ends up being easier than or at least no more difficult than the alternatives.
Thanks for the discussion, I really appreciate your input. This has shaken loose a few things from the back of my mind from when I was looking into this a couple of months ago before handing the task off to Josef.
Tim
Regards, Nick.
Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead Beaker Development Lead (http://beaker-project.org/) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJScyPEAAoJEHEkJo9fMO/LR8QH/iLHZCA0f87FT1nWzZe9UKOM H0V2Pd+NdubHrHruxsNAhzwdfgZhc3JhW+515hR53LkYti5XlceP/cqUtS1o2YfQ U2qFjkEJ1Fu4+jTVD6a7XO1CejeDHN9SJ+lAKQr+yw7lzCrsmb+GkfMkYPhw/8/5 4B0S9soOCs9Jasl3PwhFvx+1eso3r/+ZYUt6w8IpcDM4PSHhgjoZdQ/Xg7Tmi/js 3qcpEDS9eJCpKJ1KlCG01gFEAxmKBC86dku/Rb3Dy7FYPCBwOFkerYAD89QLaPrX 1pEbIKM77F0e6FoxthEjweALAztdI20uuH0mYisd6HfQAThPPyzOmjuRNY9VJVE= =pNIi -----END PGP SIGNATURE----- _______________________________________________ qa-devel mailing list qa-devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/01/2013 04:50 PM, Tim Flink wrote:
I can't really argue with xUnit's success as a unit testing paradigm but I'm not crazy about getting too locked into that right now. One of the goals that I have for taskbot is to make it flexible enough to be able to do stuff that isn't traditionally in the domain of a test system. However, I'm not quite crazy enough to try being everything to everyone so that goal may not end up feasible.
I think we're probably stuck with xUnit as the core interoperability format for *finished* tasks (e.g. I know there are various XSLT scripts kicking around to transform Beaker's result XML to standard xUnit format). However, it simply doesn't work for streaming due to the requirement to state the number of results up front.
I also realised that the YAML support in TAP likely gives you the ability to embed whatever you want if you discover the need, so it may make sense to start with the simpler format and make use of that embedded capability to transport other things.
Cheers, Nick.
- -- Nick Coghlan Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead Beaker Development Lead (http://beaker-project.org/)
----- Original Message -----
From: "Nick Coghlan" ncoghlan@redhat.com
I also realised that the YAML support in TAP likely gives you the ability to embed whatever you want if you discover the need, so it may make sense to start with the simpler format and make use of that embedded capability to transport other things.
Yup, that was exactly my plan :)
j.
qa-devel@lists.fedoraproject.org
-
Josef Skladanka -
Kamil Paral -
Lucas Meneghel Rodrigues -
Nick Coghlan -
Tim Flink