[Fedora-packaging] Introducing a directory for installed test programs

[Julio Merino]

Hello,

The purpose of this long email is to justify the need for a /usr/tests
directory and to ask for approval for its creation, with the final goal
of packaging ATF, its tests and the tests provided by arbitrary
applications (e.g. the just-approved lutok package[1]).  This email is
long because I feel the need to provide tons of background to explain
why this directory is required.  I hope the text is clear and that it
makes enough sense to convince you of its approval.  Here we go!

I am the author of the Automated Testing Framework[2] (ATF for short).
ATF is the testing framework currently used in the NetBSD operating
system, although the ATF project itself is OS-agnostic.  In other words,
ATF builds and runs fine on multiple Linux distributions, BSD systems
and OS X.  (ATF is being replaced by a new project called Kyua[3]; this
will come into play later in the text.)

I am now interested in creating an RPM package of ATF (and Kyua) for
Fedora and there is a "little detail" in ATF that will surely be
controversial.  This is why I am reaching the FPC before sending any
spec file for review, as this particular point will need discussion and
approval.

ATF is made up of two parts: 1) a collection of libraries (C, C++ and
shell) that enable developers to write test programs for their
applications and libraries; and 2) a collection of tools (atf-run,
atf-report) that allow the developers and *end users* to execute these
tests.

To re-emphasize the point above, note that I said "end users".  One of
the major features that differentiate ATF from other testing frameworks
is that *the tests written using ATF are supposed to be installed in the
system* as part of the regular application or library.  ATF then
provides standalone tools to execute these tests and to generate reports
of their results, all without the need of the original program sources
nor any development tools (read: no gcc, and specially no make).  These
tests can be run by any user as they don't require write access to the
directory in which the tests live.

To put this in the context of a Linux distribution, let's consider a
fictitious example: imagine that glib and gtk came with ATF-based tests.
These tests would get installed into testsdir/glib/... and
testsdir/gtk/...  The user would then do "cd testsdir/gtk/ && atf-run"
to execute the tests on his machine, without the need for any -devel
packages.  The key point of this idea is that this allows end users to
gain confidence in their systems, because they can validate that a
particular piece of software runs fine on their OS / hardware
combination.  What is more, suppose that glib got upgraded to a new
version but not gtk.  If there were any incompatibilities between the
two, the installed gtk tests would highlight it because they'd start
failing.  The packager may never have noticed that fact himself if he
always kept his build environment up-to-date (as is commonly the case).
(Also, consider creating a cron job to run all these tests nightly!)

Well, imagine no more.  We already do something like this in NetBSD.
NetBSD defines the testsdir above as "/usr/tests".  This tree[4]
contains a subdirectory for each package that provides ATF-based tests,
and there is a top-level Atffile file that allows atf-run to recurse
into all the package subdirectories.  (I.e. executing atf-run within
/usr/tests runs *all* the tests available in the system for all
packages.)  You can see in [5] a more detailed description as to how
this comes into practice.

There are two important things to consider about /usr/tests:

1) The first is that there is a single directory acting as the "root"
for the tests of arbitrary packages because this is what allows all
these tests to be run at once.  In other words, instead of having
/usr/pkg1/tests /usr/pkg2/tests, we have /usr/tests/pkg1 and
/usr/tests/pkg2.

2) The second is that ATF is not encoded anywhere in the path.  I.e. the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
/usr/tests/atf/pkg*.  The reason for this is that the package-specific
tests needn't be implemented using the ATF libraries.  In fact, Kyua,
the replacement for ATF, is able to execute these same unmodified tests
and can also execute tests written other frameworks.  In a future world,
packages would install arbitrary tests into /usr/tests written using
their preferred libraries (e.g. pyunit, autotest, etc.) and a single
tool (Kyua) would run them all.  Encoding the tool name in the directory
introduces a dependency on the tool that does not necessarily exist.

So, to sum it up: the whole point of this email is to discuss what the
"testsdir" above should be in Fedora.  I think it makes a lot of sense
to keep the default of /usr/tests.

However, of course, this does not complain with any the LSB policies.
The problem is that no other already-existing directory seems to suit
the goals of /usr/tests.  For example: /usr/share/tests is unsuitable
because the tests are obviously executables and thus not shareable.
/usr/libexec/tests also appears unsuitable because these programs are
not helper binaries internally used by other programs: they are supposed
to be exposed to the end user, who may choose to run them by hand.
/usr/lib/tests is even worse because these are not libraries and also
has the same problem of libexec in the sense that the files are hidden
from end users.  End users need to be encouraged to "cd /usr/tests" and
to run stuff from there, just as they do with /usr/bin or /usr/sbon.

Therefore, I believe that introducing a new directory is sensible
enough, particularly because it is potentially optional (e.g. could be
made part of a -filesystem package).  Actually, the way I envision the
packaging to work is the following.  Let's consider lutok again, as its
distribution file contains ATF-based tests.  lutok would come in these
packages: lutok, lutok-devel, lutok-tests.  The latter package would
place files in /usr/tests/lutok only and it would depend on libatf-c++
(because lutok's tests are implemented in C++) and maybe some
"tests-filesystem" package.  lutok-tests needn't depend on the atf
tools.  Then, the user could just install the 'atf' package providing
atf-run and atf-report to execute these tests from within /usr/tests or
/usr/tests/lutok.  Similarly, other tools could place their installable
tests in /usr/tests/<pkg>.

So, what packages would make use of /usr/tests if it were introduced
now?  As far as I can tell, these would be atf, lutok and kyua
(including all of its subpackages for the libraries, etc.).  Bind 9
recently adopted ATF too, although I'm not sure if they install the
tests.

This should be all for now.  Is it OK to introduce this optional
/usr/tests directory?

Please let me know if you have any questions or concerns.

Thank you,

Julio Merino

1: https://bugzilla.redhat.com/show_bug.cgi?id=785619
2: http://www.NetBSD.org/~jmmv/atf/
3: http://code.google.com/p/kyua/
4: http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/?only_with_tag=MAIN
5: http://blog.netbsd.org/tnf/entry/testing_netbsd_easy_does_it