test vs check naming consistency

[Tim Flink]

On Thu, 13 Feb 2014 03:42:03 -0500 (EST)
Kamil Paral <kparal at redhat.com> wrote:

> I've noticed that we started to use the word 'check' for naming the
> scripts performing package-level or system-level checks/tests. I'd
> like to be consistent in our code and in our documentation, so that
> our users are not confused. Can somebody (probably Tim) clarify what
> should we call a 'check' and what should we call a 'test'? Do I
> understand correctly that 'checks' are anything provided by the users
> (the scripts), and 'tests/testing' will be used mainly for denoting
> unit/functional tests?

It's an attempt by me to differentiate between testing and checking but
choosing my battles.

Michael Bolton and James Bach have spent far more time on
differentiating between checking and testing [1] than I have but as a
summary (borrowing some stuff verbatim from their article):

[1] http://www.satisfice.com/blog/archives/856

From their definitions:

 - Testing is the process of evaluating a product by learning about it
   through experimentation, which includes to some degree: questioning,
   study, modeling, observation and inference.

 - Checking is the process of making evaluations by applying algorithmic
   decision rules to specific observations of a product.


Within the scope of this specific discussion, the two key implications
I see are:

 - Testing encompasses checking, whereas checking cannot encompass
   testing.

 - Checking is a process that can, in principle be performed by
   a tool instead of a human, whereas testing can only be supported by
   tools. Nevertheless, tools can be used for much more than checking.


Using these definitions, anything run in taskotron that reports a
PASS/FAIL/OTHER status (ie, everything) would be a check. I'm not
saying that checks are bad or anything like that, just that there are
limitations to what a check (and an automated check, in particular) can
do.

In the context of testing Fedora, I see a value in differentiating
between checking and testing because it highlights the value of human
testing contributions. Having automated checks for various parts of
Fedora is great and I'm all for the attempt to do get them running but
I do not believe that the ideas of automating all testing for Fedora or
that all tests could/should be automated are worth considering because
they are _impossible_ without incredible advances in AI or the
singularity.

While automated checking is a useful and important part of a good
overall testing strategy, those automated checks are only a portion of
said testing strategy. There is a place for sapient, (semi-)exploratory
testing in Fedora and there will be a place for it in the foreseeable
future.

That being said, I'm not sure there is enough benefit to fight the
common usage of the term "unit testing". While they are checks by the
definitions I listed above, the cost of re-defining all "unit tests" as
"unit checks" would be rather high and I'm not convinced that there's
enough benefit there to justify the attempt.

> At the moment, I'm working on porting autoqa.test.TestDetail class
> into Taskotron. When following these directions, I guess it should be
> called libtaskotron.check.CheckDetail then?

Yes, pretty much anything that was a test in autoqa would be a check in
the context of taskotron.

Tim
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20140224/3ded4753/attachment.sig>