Taskbot: TAP vs Subunit

Fri Nov 1 06:50:55 UTC 2013

On Fri, 01 Nov 2013 13:45:08 +1000
Nick Coghlan <ncoghlan at redhat.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 11/01/2013 03:53 AM, Tim Flink wrote:
> > On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka 
> > <jskladan at redhat.com> wrote:
> >> I do not really feel that this is a good match for our needs (at
> >>  least without any significant hacking)
> > 
> > Yeah, this matches what I remember finding when I looked into this 
> > a couple of months ago. It looks like a better protocol overall
> > but it's going to be quite a bit more upfront work to support than
> > TAP. I know that openstack's test suite uses subunit for nova and 
> > neutron [1]
> > 
> > [1] https://wiki.openstack.org/wiki/Testr
> > 
> > Subunit has been recently added to the fedora repos but I suppose 
> > that's of little utility for us if we end up having to re-implement
> > parts to get away from UnitTest.
> 
> As Tim noted, subunit development is tied in to the OpenStack
> project's "test repository" (testr) project.

To add confusion, they're using an upstream project called
testrepository:

https://testrepository.readthedocs.org/en/latest/MANUAL.html

I remember running into problems actually running testr a month or two
ago but I got pulled away by something else and never finished looking
into it. I just ran it again and it didn't have any issues, so I'm
either mis-remembering or my OS upgrade since then ended up fixing the
issue.

> So, if you were going to go down the subunit path, I'd suggest looking
> at leveraging (at least pieces of) testr for result storage as well.
> 
> While it's not an urgent requirement, we also plan to explore adding
> subunit support to Beaker.

Ah, I seem to recall talking about that at Flock and it slipped my
mind. Thanks for the reminder as this could end up being important for
integration.

> Robert Collins gave a decent talk on testr (et al) at the PyCon AU
> 2013 OpenStack miniconf: https://www.youtube.com/watch?v=Oe_HhBBbqbw
> 
> Looking at the list of Producers and Consumers on the TAP wiki doesn't
> give me any confidence that *anyone* has successfully used TAP at
> scale or as part of a distributed continuous integration system, nor
> does it look like it has anyone providing the level of investment HP
> and others are putting into the OpenStack testing infrastructure
> (Robert works for HP, mostly on the OpenStack-on-OpenStack project).

I think that AMD uses or was using TAP but yeah, I'm not really seeing
much of anyone else who is using TAP on a large scale.

My main concern is how tightly coupled SubUnit and testrepository are
to UnitTest. I took another quick look at their code and one of the
first things I noticed is the description in a header [1]:

#  subunit: extensions to Python unittest to get test results from
subprocesses.

There are other things that raise flags for our usage at first glance,
but I only looked at it for a few minutes and could easily be very
wrong :)

I'm intrigued by some of the things that subunit can do/support
(embedding files and the idea of replaying tests in particular - I
think they have the potential to solve some problems we're
eventually going to hit). I think that the short term question may end
up being whether we want to focus on better stuff from the beginning or
to "rough in" semi-disposable components (knowing or ignoring the fact
that those are the types of things which tend to live longer than they
should) to get a better idea of where the unknown pain points are.

I can't really argue with xUnit's success as a unit testing paradigm
but I'm not crazy about getting too locked into that right now. One of
the goals that I have for taskbot is to make it flexible enough to be
able to do stuff that isn't traditionally in the domain of a test
system. However, I'm not quite crazy enough to try being everything to
everyone so that goal may not end up feasible.

[1]http://bazaar.launchpad.net/~subunit/subunit/trunk/view/head:/python/subunit/v2.py

> Even if subunit is more work initially, you'll likely enjoy flow-on
> effects in the future from those OpenStack investments (similar to the
> way LMR decided to outsource the "inventory management and scaling to
> multiple labs" problems for autotest to Beaker, so autotest gets to
> avoid doubling up on work we were going to do anyway).

Assuming that what we're trying to do is similar enough, yes. If
it's different enough that we end up modifying their code or
re-implementing it ourselves, that big investment could end up being a
pain point. The documentation states that v2 is not yet a stable
protocol and current implementers should be prepared to deal with
non-backwards-compatible changes.

I'm not that familiar with what exactly OpenStack is doing with subunit
and testr, so I can't say how much overlap there actually is.

> With subunit as the more expressive protocol, it also means it would
> be easier to convert TAP streams to subunit streams than the other way
> around, so building the back end to be subunit based doesn't rule out
> the possibility of permitting TAP based input in the future. By
> contrast, using TAP as the base *will* almost certainly rule out
> accepting subunit streams from testr or Beaker.

Yeah, it does sound like this topic needs more investigation and
maybe some example/poc code. Most of my concerns will end up being
academic if using subunit ends up being easier than or at least no more
difficult than the alternatives.

Thanks for the discussion, I really appreciate your input. This has
shaken loose a few things from the back of my mind from when I was
looking into this a couple of months ago before handing the task off to
Josef.

Tim

> Regards,
> Nick.
> 
> - -- 
> Nick Coghlan
> Red Hat Infrastructure Engineering & Development, Brisbane
> 
> Testing Solutions Team Lead
> Beaker Development Lead (http://beaker-project.org/)
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> 
> iQEcBAEBAgAGBQJScyPEAAoJEHEkJo9fMO/LR8QH/iLHZCA0f87FT1nWzZe9UKOM
> H0V2Pd+NdubHrHruxsNAhzwdfgZhc3JhW+515hR53LkYti5XlceP/cqUtS1o2YfQ
> U2qFjkEJ1Fu4+jTVD6a7XO1CejeDHN9SJ+lAKQr+yw7lzCrsmb+GkfMkYPhw/8/5
> 4B0S9soOCs9Jasl3PwhFvx+1eso3r/+ZYUt6w8IpcDM4PSHhgjoZdQ/Xg7Tmi/js
> 3qcpEDS9eJCpKJ1KlCG01gFEAxmKBC86dku/Rb3Dy7FYPCBwOFkerYAD89QLaPrX
> 1pEbIKM77F0e6FoxthEjweALAztdI20uuH0mYisd6HfQAThPPyzOmjuRNY9VJVE=
> =pNIi
> -----END PGP SIGNATURE-----
> _______________________________________________
> qa-devel mailing list
> qa-devel at lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/qa-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20131101/854b3363/attachment.sig>