[Fedora-packaging] Introducing a directory for installed test programs

[Tom Callaway]

On 02/06/2012 04:59 AM, Julio Merino wrote:
> Hello,
> 
> The purpose of this long email is to justify the need for a /usr/tests
> directory and to ask for approval for its creation, with the final goal
> of packaging ATF, its tests and the tests provided by arbitrary
> applications (e.g. the just-approved lutok package[1]).  This email is
> long because I feel the need to provide tons of background to explain
> why this directory is required.  I hope the text is clear and that it
> makes enough sense to convince you of its approval.  Here we go!

[LOTS OF GOOD RATIONALE SNIPPED]

So, the first thought I had was "these binaries won't be in the $PATH".
You say above that "End users need to be encouraged to 'cd /usr/tests'
and to run stuff from there, just as they do with /usr/bin or
/usr/sbin." ... but end users do not cd into /usr/bin to run a binary
from there, they simply call the binary name and expect it to work.
I'm assuming that the fact that these tests are not in the $PATH is a
feature, not a bug. :)

If this is the case, then it may be possible after all to use
/usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
area, the FHS doesn't mention it at anywhere (despite repeated pleading
to them to do so), it comes from the GNU Coding Standards and Fedora has
adopted it as an exception. The Fedora Packaging Guidelines say:

  Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
  as the directory for executable programs that are designed primarily
  to be run by other programs rather than by users.

That said, I could definitely see the argument being made that
/usr/libexec/tests is a perfectly acceptable use case for binaries that
are "designed primarily to be run by testing frameworks, but also
available outside of $PATH for users to manually execute".

However, whether it is /usr/tests or /usr/libexec/tests, there is going
to be a need to tightly constrain the acceptable files that could go
into /usr/tests and document exactly how they are allowed to go into it.

For example, my experience with test suites is almost certainly far less
than yours, but there seems to also be a fair amount of non-executable
test data content that comes along with the executable tests. E.g.
test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
/usr/share/tests/$program/ ?

It is also possible that introducing a new home for binaries would cause
multi-lib heartburn in RPM, but I suspect that is a simple enough
problem to fix, since we haven't really hit it with our use of
/usr/libexec to date.

You probably should open a ticket with the FHS working group:
https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
to them so that it has a chance of being officially added in the next
major FHS revision, whenever that may be.

Above and beyond your proposal, the only other way I can think of
handling this would be an elaborate namespacing of the test binaries in
/usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
solution that would permit a strict compliance with the FHS as it exists
today, I'm not very excited about it and am not seriously recommending
it as an alternative.

Thanks for your message,

~tom

==
Fedora Project