[Fedora-packaging] Introducing a directory for installed test programs

[Michael Schwendt]

On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:

> The purpose of this long email is to justify the need for a /usr/tests
> directory 

> To put this in the context of a Linux distribution, let's consider a
> fictitious example: imagine that glib and gtk came with ATF-based tests.
> These tests would get installed into testsdir/glib/... and
> testsdir/gtk/...  The user would then do "cd testsdir/gtk/ && atf-run"
> to execute the tests on his machine, 

> 2) The second is that ATF is not encoded anywhere in the path.  I.e. the
> directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
> /usr/tests/atf/pkg*.  The reason for this is that the package-specific
> tests needn't be implemented using the ATF libraries.

Still, the tests must be compatible with the testing framework's set of
tools. That's enough of a reason to add some sort of namespace to the
root directory.

I also find it contradictory how you refer to "atf-run", "atf-report"
and "a collection of tools", which have "atf-" in their name. Apparently,
they are stored in global search path for executables (and therefore it
is reasonable to prefix them with something to avoid creating too generic
names, and hopefully you wouldn't propose dropping the "atf-" prefix from
their names either).

Surely the testing framework would not be renamed regularly. And
software authors, who would use this framework for writing own tests,
would need to be able to rely on an interface for retrieving filesystem
path details anyway. For example, a pkg-config file that can be queried.
Or a similar atf-config script that returns various paths.

> In fact, Kyua,
> the replacement for ATF, is able to execute these same unmodified tests
> and can also execute tests written other frameworks.  In a future world,
> packages would install arbitrary tests into /usr/tests written using
> their preferred libraries (e.g. pyunit, autotest, etc.) and a single
> tool (Kyua) would run them all.  Encoding the tool name in the directory
> introduces a dependency on the tool that does not necessarily exist.

Implementation details here would need to be discussed. The names of the
tools are irrelevant so far. The various testing frameworks would need to
be compatible somehow and meet the requirements of the controlling tools.
One couldn't dump arbitrary tests into a shared directory and e.g. mix
automated, semi-automated and non-automated tests.

Leaving FHS issues aside, in general it is not nice to let any package
"occupy" directories, which have very generic names, without very good
reason. That's not limited to /usr, and it's not a "first come, first served"
thing either. Imagine, a second group of testing framework authors would
also want to use a global /usr/tests in order to store incompatible tests
in there.