Re: Randomness in Testing
On Wed, 2010-08-25 at 14:08 -0400, Justin Harris wrote:
All,
I have seen some use of random number/name generation used in tests
recently and want to discuss it further here. I am
going to go out on a limb and give the opinion that this is not a good
thing, and sets a
bad precedent. I think that the real value of tests is that they are an
easily repeatable, consistent assessment of where
the code is at a certain point in time. Tests really start to lose their
value when two runs of the same test on the same
code base can result in two different outcomes. I know that for things
like name generation this probably won't happen,
but as I said it is starting down a slippery slope, and IMHO we should
avoid it if possible.
Note that there is nothing wrong with *arbitrary* values, or values
determined at random and hard coded into tests - let's
just be sure that they are the same on subsequent runs of the same test.
Thoughts?
- Justin
_______________________________________________
candlepin mailing list
candlepin(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/candlepin
We use timestamps like System.currentTimeMillis() in our tests - here's
why.
1) It makes tests re-runnable. When tests create "stuff" on the
software under test, you often can't rerun the test - because it will
fail since the resource name already exists.
2) It allows you to figure out when resources were created just based on
the name. Sometimes this is helpful in debugging.
timestamps work better than random because randoms only fulfill #1
above.
Yes, there is a minor repeatability issue with using timestamps (or any
calculated value), but I think the value of re-runnability far outweighs
the loss here. The whole reason randoms are used in the first place is
to avoid these collisions that will cause false positives in the test
results.
Jeff