[Fedora-packaging] Introducing a directory for installed test programs

Mon Feb 6 14:58:32 UTC 2012

On 2/6/12 7:27 AM, Tom Callaway wrote:
> On 02/06/2012 04:59 AM, Julio Merino wrote:
>> Hello,
[...]

Hello Tom,

Thanks for the detailed reply.  A longer reply comes below ;)

> So, the first thought I had was "these binaries won't be in the $PATH".
> You say above that "End users need to be encouraged to 'cd /usr/tests'
> and to run stuff from there, just as they do with /usr/bin or
> /usr/sbin." ... but end users do not cd into /usr/bin to run a binary
> from there, they simply call the binary name and expect it to work.
> I'm assuming that the fact that these tests are not in the $PATH is a
> feature, not a bug. :)

Correct.  They are not supposed to be in the PATH.  (They could be 
though, but their utility would be reduced and they'd just screw up 
autocompletion.  Also, this might not be totally doable given that 
testsdir is a tree with multiple nested subdirectories; see below for a 
detailed example showcasing corner cases.)

It'd be argued that the tools to run the tests (atf-run in this case) 
should automatically look for stuff in /usr/tests unless told otherwise, 
thus hiding the details of where the tests are stored.  I'll grant you 
that this is a nice idea to pursue, but that's not how things work at 
the moment and I'm not sure about what the ramifications of such change 
would be.

(The idea of having to cd into the tests directory before running the 
tests is that you can have tests in multiple "roots".  One root is the 
testsdir, where the tests for all installed packages live.  But other 
valid roots are the top directory of your source trees, so that you can 
execute the tests while doing development before doing an actual 
installation!)

> If this is the case, then it may be possible after all to use
> /usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
> area, the FHS doesn't mention it at anywhere (despite repeated pleading
> to them to do so), it comes from the GNU Coding Standards and Fedora has
> adopted it as an exception. The Fedora Packaging Guidelines say:
>
>    Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
>    as the directory for executable programs that are designed primarily
>    to be run by other programs rather than by users.
>
> That said, I could definitely see the argument being made that
> /usr/libexec/tests is a perfectly acceptable use case for binaries that
> are "designed primarily to be run by testing frameworks, but also
> available outside of $PATH for users to manually execute".

I see.  /usr/libexec/tests doesn't sound too bad considering this 
description!  However, the data files issue you raised could get in the 
way.  Some more details below.

> However, whether it is /usr/tests or /usr/libexec/tests, there is going
> to be a need to tightly constrain the acceptable files that could go
> into /usr/tests and document exactly how they are allowed to go into it.
>
> For example, my experience with test suites is almost certainly far less
> than yours, but there seems to also be a fair amount of non-executable
> test data content that comes along with the executable tests. E.g.
> test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
> non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
> /usr/share/tests/$program/ ?

As it turns out, I forgot to mention this exact point (data files) in my 
original email.  The way things work at the moment within ATF is to have 
data files and helper binaries in the same directory as the test program 
that needs them.  The test programs can then dynamically query what 
their "source directory" is so that they can locate these files 
regardless of where the test program lives.

The rationale for this was two-folded.  First, it allows the test 
programs to run both from the source directory and the installed tree 
because they don't hardcode absolute paths to their helpers.  And 
second, the whole tests tree lives in a self-contained directory that 
can either be pruned from the system (which is convenient due to the way 
NetBSD works) or shared on the network.  Looking at existing practice, a 
test tree is composed mostly of binaries or scripts; the percentage of 
data files is very low.  (There was also an idea of being able to 
provide such programs as "bundles" that could be copied to slave 
machines for remote testing, but has never been implemented yet.)

In other words, the layout for your example would be:

testsdir/
testsdir/xpdf/
testsdir/xpdf/parser_test - Binary; atf-run executes it from within
     a temporary directory to "isolate" it from the read-only testsdir
     and other parts of the system.  parser_test can query its "srcdir",
     which in this case is "testsdir/xpdf/", and thus locate where the
     00-broken.pdf file is.
testsdir/xpdf/00-broken.pdf - The auxiliary data file for the xpdf
     tests.

This may rule out the suitability of /usr/libexec/tests as the location 
for these files, given that your description seems to imply that libexec 
is for binaries only.  But if it isn't, I'd be OK with using this directory.

> It is also possible that introducing a new home for binaries would cause
> multi-lib heartburn in RPM, but I suspect that is a simple enough
> problem to fix, since we haven't really hit it with our use of
> /usr/libexec to date.

What do you mean by "multi-lib heartburn in RPM"?  (Sorry, not familiar 
from the term and a couple of Google searches did't provide much 
insight.)  FWIW, I have a preliminary atf.spec file here that provides 
subpackages for libraries, tests and binaries and things seem to be fine 
to my untrained eye.  I can share this file if it helps at all.

> You probably should open a ticket with the FHS working group:
> https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
> to them so that it has a chance of being officially added in the next
> major FHS revision, whenever that may be.

OK, I can do that.  Wouldn't it be easier to illustrate the point / 
convince them if there was existing practice somewhere?  (Not sure if 
BSD flavors count in this case!)

> Above and beyond your proposal, the only other way I can think of
> handling this would be an elaborate namespacing of the test binaries in
> /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
> solution that would permit a strict compliance with the FHS as it exists
> today, I'm not very excited about it and am not seriously recommending
> it as an alternative.

I don't think this would be an appropriate layout, and it doesn't play 
well with the structure of atf (let aside the clutter that would appear 
in /usr/bin).  Let me provide some additional details of how the tests 
tree is structured.

Each directory in testsdir has an Atffile that indicates which test 
programs exist in that directory, which subdirectories to recurse into, 
and additional properties that indicate meta-data for the test programs. 
  So, for example, the testsdir would look like this:

testsdir/ - The top-level directory.  This directory would be owned
     by a filesystem package and be a dependency of anything that places
     files in testsdir/.

testsdir/Atffile - The main control file.  This has the knowledge to
     recurse into the direct subdirectories of testsdir/ looking for
     other Atffiles to execute.  E.g. this would find testsdir/a/Atffile
     and testsdir/b/Atffile to source tests, but it would not find
     testsdir/c/Notatf nor testsdir/d/e/Atffile.  This file would be
     installed by the "atf" tools package, as it does not make any
     sense to have it without the corresponding tools.

testsdir/lutok/ - Owned by a lutok-tests package, including all the
     subfiles and subdirectories.

testsdir/lutok/Atffile - The control file for the tests provided by
     lutok.  Contains the list of test programs to run and any optional
     meta-data for them.  It also includes other subdirectories to
     "recurse" into.

testsdir/lutok/operations_test - A test program for the "operations"
     module of the source tree.  (The name of the binary is arbitrary.)

testsdir/lutok/state_test - Another test program.

testsdir/lutok/detail/ - A subdirectory of lutok's tests containing
     tests for internal modules only.  (This would mimic the structure
     of the source package.)

testsdir/lutok/detail/Atffile - Same as above, but for the tests in
     this subdirectory.

testsdir/lutok/detail/internal_test - Yet another test program.

testsdir/lutok/detail/state_test - Another test program.  Note that
     this can have the same name as the program in the parent
     directory!  There is nothing to prevent this, and thus flattening
     the test programs into /usr/bin as you propose would be very
     complicated.

testsdir/lutok/foo/ - Another subdirectory for additional tests, etc.

testsdir/foobar/ - Another package, with the same structure as above.

testsdir/baz/ - Another package, but this time without any Atffile in
     it.  This would not be executed by atf-run because the top-level
     Atffile would discard this subdirectory.  (Kyua, the replacement
     of atf, could run these tests though provided that a Kyuafile was
     created within the subdirectory.  This is out of scope now though.)

A little detail to note above is that in a project with a good amount of 
test coverage, there will potentially be a test program per each source 
file.  This means that the testsdir contains lots of binaries, and you 
don't want that many, potentially with conflicting names, anywhere in 
your path!  To illustrate the point, check out this output page which 
contains the results of the execution of the NetBSD test suite:

     http://www.whooppee.com/~paul/amd64-results/last_atf.html

Anyway.  This layout may remind you a lot of make(1) and a traditional 
source tree populated with Makefiles.  And actually, this is the main 
idea of how the tests tree is organized and is an artifact of ATF's 
origins (NetBSD's source tree).  However, having played with other large 
source code bases, I can say that this organization is not unique to NetBSD.

We'd discuss extensively whether this layout is suboptimal for Fedora's 
guidelines or not...  But please note that all I'm trying to achieve 
here is to figure out how to package an existing piece of software, not 
how to re-architect it!

If you made it this far again, thank you.

-- 
Julio Merino