9:59 p.m.
Hello,
The purpose of this long email is to justify the need for a /usr/tests
directory and to ask for approval for its creation, with the final goal
of packaging ATF, its tests and the tests provided by arbitrary
applications (e.g. the just-approved lutok package[1]). This email is
long because I feel the need to provide tons of background to explain
why this directory is required. I hope the text is clear and that it
makes enough sense to convince you of its approval. Here we go!
I am the author of the Automated Testing Framework[2] (ATF for short).
ATF is the testing framework currently used in the NetBSD operating
system, although the ATF project itself is OS-agnostic. In other words,
ATF builds and runs fine on multiple Linux distributions, BSD systems
and OS X. (ATF is being replaced by a new project called Kyua[3]; this
will come into play later in the text.)
I am now interested in creating an RPM package of ATF (and Kyua) for
Fedora and there is a "little detail" in ATF that will surely be
controversial. This is why I am reaching the FPC before sending any
spec file for review, as this particular point will need discussion and
approval.
ATF is made up of two parts: 1) a collection of libraries (C, C++ and
shell) that enable developers to write test programs for their
applications and libraries; and 2) a collection of tools (atf-run,
atf-report) that allow the developers and *end users* to execute these
tests.
To re-emphasize the point above, note that I said "end users". One of
the major features that differentiate ATF from other testing frameworks
is that *the tests written using ATF are supposed to be installed in the
system* as part of the regular application or library. ATF then
provides standalone tools to execute these tests and to generate reports
of their results, all without the need of the original program sources
nor any development tools (read: no gcc, and specially no make). These
tests can be run by any user as they don't require write access to the
directory in which the tests live.
To put this in the context of a Linux distribution, let's consider a
fictitious example: imagine that glib and gtk came with ATF-based tests.
These tests would get installed into testsdir/glib/... and
testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run"
to execute the tests on his machine, without the need for any -devel
packages. The key point of this idea is that this allows end users to
gain confidence in their systems, because they can validate that a
particular piece of software runs fine on their OS / hardware
combination. What is more, suppose that glib got upgraded to a new
version but not gtk. If there were any incompatibilities between the
two, the installed gtk tests would highlight it because they'd start
failing. The packager may never have noticed that fact himself if he
always kept his build environment up-to-date (as is commonly the case).
(Also, consider creating a cron job to run all these tests nightly!)
Well, imagine no more. We already do something like this in NetBSD.
NetBSD defines the testsdir above as "/usr/tests". This tree[4]
contains a subdirectory for each package that provides ATF-based tests,
and there is a top-level Atffile file that allows atf-run to recurse
into all the package subdirectories. (I.e. executing atf-run within
/usr/tests runs *all* the tests available in the system for all
packages.) You can see in [5] a more detailed description as to how
this comes into practice.
There are two important things to consider about /usr/tests:
1) The first is that there is a single directory acting as the "root"
for the tests of arbitrary packages because this is what allows all
these tests to be run at once. In other words, instead of having
/usr/pkg1/tests /usr/pkg2/tests, we have /usr/tests/pkg1 and
/usr/tests/pkg2.
2) The second is that ATF is not encoded anywhere in the path. I.e. the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
/usr/tests/atf/pkg*. The reason for this is that the package-specific
tests needn't be implemented using the ATF libraries. In fact, Kyua,
the replacement for ATF, is able to execute these same unmodified tests
and can also execute tests written other frameworks. In a future world,
packages would install arbitrary tests into /usr/tests written using
their preferred libraries (e.g. pyunit, autotest, etc.) and a single
tool (Kyua) would run them all. Encoding the tool name in the directory
introduces a dependency on the tool that does not necessarily exist.
So, to sum it up: the whole point of this email is to discuss what the
"testsdir" above should be in Fedora. I think it makes a lot of sense
to keep the default of /usr/tests.
However, of course, this does not complain with any the LSB policies.
The problem is that no other already-existing directory seems to suit
the goals of /usr/tests. For example: /usr/share/tests is unsuitable
because the tests are obviously executables and thus not shareable.
/usr/libexec/tests also appears unsuitable because these programs are
not helper binaries internally used by other programs: they are supposed
to be exposed to the end user, who may choose to run them by hand.
/usr/lib/tests is even worse because these are not libraries and also
has the same problem of libexec in the sense that the files are hidden
from end users. End users need to be encouraged to "cd /usr/tests" and
to run stuff from there, just as they do with /usr/bin or /usr/sbon.
Therefore, I believe that introducing a new directory is sensible
enough, particularly because it is potentially optional (e.g. could be
made part of a -filesystem package). Actually, the way I envision the
packaging to work is the following. Let's consider lutok again, as its
distribution file contains ATF-based tests. lutok would come in these
packages: lutok, lutok-devel, lutok-tests. The latter package would
place files in /usr/tests/lutok only and it would depend on libatf-c++
(because lutok's tests are implemented in C++) and maybe some
"tests-filesystem" package. lutok-tests needn't depend on the atf
tools. Then, the user could just install the 'atf' package providing
atf-run and atf-report to execute these tests from within /usr/tests or
/usr/tests/lutok. Similarly, other tools could place their installable
tests in /usr/tests/<pkg>.
So, what packages would make use of /usr/tests if it were introduced
now? As far as I can tell, these would be atf, lutok and kyua
(including all of its subpackages for the libraries, etc.). Bind 9
recently adopted ATF too, although I'm not sure if they install the
tests.
This should be all for now. Is it OK to introduce this optional
/usr/tests directory?
Please let me know if you have any questions or concerns.
Thank you,
Julio Merino
1: https://bugzilla.redhat.com/show_bug.cgi?id=785619
2: http://www.NetBSD.org/~jmmv/atf/
3: http://code.google.com/p/kyua/
4: http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/?only_with_tag=MAIN
5: http://blog.netbsd.org/tnf/entry/testing_netbsd_easy_does_it
6:27 a.m.
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
The purpose of this long email is to justify the need for a /usr/tests
directory and to ask for approval for its creation, with the final goal
of packaging ATF, its tests and the tests provided by arbitrary
applications (e.g. the just-approved lutok package[1]). This email is
long because I feel the need to provide tons of background to explain
why this directory is required. I hope the text is clear and that it
makes enough sense to convince you of its approval. Here we go!
[LOTS OF GOOD RATIONALE SNIPPED]
So, the first thought I had was "these binaries won't be in the $PATH".
You say above that "End users need to be encouraged to 'cd /usr/tests'
and to run stuff from there, just as they do with /usr/bin or
/usr/sbin." ... but end users do not cd into /usr/bin to run a binary
from there, they simply call the binary name and expect it to work.
I'm assuming that the fact that these tests are not in the $PATH is a
feature, not a bug. :)
If this is the case, then it may be possible after all to use
/usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
area, the FHS doesn't mention it at anywhere (despite repeated pleading
to them to do so), it comes from the GNU Coding Standards and Fedora has
adopted it as an exception. The Fedora Packaging Guidelines say:
Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
as the directory for executable programs that are designed primarily
to be run by other programs rather than by users.
That said, I could definitely see the argument being made that
/usr/libexec/tests is a perfectly acceptable use case for binaries that
are "designed primarily to be run by testing frameworks, but also
available outside of $PATH for users to manually execute".
However, whether it is /usr/tests or /usr/libexec/tests, there is going
to be a need to tightly constrain the acceptable files that could go
into /usr/tests and document exactly how they are allowed to go into it.
For example, my experience with test suites is almost certainly far less
than yours, but there seems to also be a fair amount of non-executable
test data content that comes along with the executable tests. E.g.
test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
/usr/share/tests/$program/ ?
It is also possible that introducing a new home for binaries would cause
multi-lib heartburn in RPM, but I suspect that is a simple enough
problem to fix, since we haven't really hit it with our use of
/usr/libexec to date.
You probably should open a ticket with the FHS working group:
https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
to them so that it has a chance of being officially added in the next
major FHS revision, whenever that may be.
Above and beyond your proposal, the only other way I can think of
handling this would be an elaborate namespacing of the test binaries in
/usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
solution that would permit a strict compliance with the FHS as it exists
today, I'm not very excited about it and am not seriously recommending
it as an alternative.
Thanks for your message,
~tom
==
Fedora Project
8:58 a.m.
On 2/6/12 7:27 AM, Tom Callaway wrote:
On 02/06/2012 04:59 AM, Julio Merino wrote:
> Hello,
[...]
Hello Tom,
Thanks for the detailed reply. A longer reply comes below ;)
So, the first thought I had was "these binaries won't be in
the $PATH".
You say above that "End users need to be encouraged to 'cd /usr/tests'
and to run stuff from there, just as they do with /usr/bin or
/usr/sbin." ... but end users do not cd into /usr/bin to run a binary
from there, they simply call the binary name and expect it to work.
I'm assuming that the fact that these tests are not in the $PATH is a
feature, not a bug. :)
Correct. They are not supposed to be in the PATH. (They could be
though, but their utility would be reduced and they'd just screw up
autocompletion. Also, this might not be totally doable given that
testsdir is a tree with multiple nested subdirectories; see below for a
detailed example showcasing corner cases.)
It'd be argued that the tools to run the tests (atf-run in this case)
should automatically look for stuff in /usr/tests unless told otherwise,
thus hiding the details of where the tests are stored. I'll grant you
that this is a nice idea to pursue, but that's not how things work at
the moment and I'm not sure about what the ramifications of such change
would be.
(The idea of having to cd into the tests directory before running the
tests is that you can have tests in multiple "roots". One root is the
testsdir, where the tests for all installed packages live. But other
valid roots are the top directory of your source trees, so that you can
execute the tests while doing development before doing an actual
installation!)
If this is the case, then it may be possible after all to use
/usr/libexec/tests for this. After all, /usr/libexec is a bit of a grey
area, the FHS doesn't mention it at anywhere (despite repeated pleading
to them to do so), it comes from the GNU Coding Standards and Fedora has
adopted it as an exception. The Fedora Packaging Guidelines say:
Libexecdir (aka, /usr/libexec on Fedora systems) should only be used
as the directory for executable programs that are designed primarily
to be run by other programs rather than by users.
That said, I could definitely see the argument being made that
/usr/libexec/tests is a perfectly acceptable use case for binaries that
are "designed primarily to be run by testing frameworks, but also
available outside of $PATH for users to manually execute".
I see. /usr/libexec/tests doesn't sound too bad considering this
description! However, the data files issue you raised could get in the
way. Some more details below.
However, whether it is /usr/tests or /usr/libexec/tests, there is
going
to be a need to tightly constrain the acceptable files that could go
into /usr/tests and document exactly how they are allowed to go into it.
For example, my experience with test suites is almost certainly far less
than yours, but there seems to also be a fair amount of non-executable
test data content that comes along with the executable tests. E.g.
test-xpdf reads in 00-Broken.pdf with XFAIL as the result. Would
non-executable test data live in /usr/[libexec/]tests/$program/ ? Or
/usr/share/tests/$program/ ?
As it turns out, I forgot to mention this exact point (data files) in my
original email. The way things work at the moment within ATF is to have
data files and helper binaries in the same directory as the test program
that needs them. The test programs can then dynamically query what
their "source directory" is so that they can locate these files
regardless of where the test program lives.
The rationale for this was two-folded. First, it allows the test
programs to run both from the source directory and the installed tree
because they don't hardcode absolute paths to their helpers. And
second, the whole tests tree lives in a self-contained directory that
can either be pruned from the system (which is convenient due to the way
NetBSD works) or shared on the network. Looking at existing practice, a
test tree is composed mostly of binaries or scripts; the percentage of
data files is very low. (There was also an idea of being able to
provide such programs as "bundles" that could be copied to slave
machines for remote testing, but has never been implemented yet.)
In other words, the layout for your example would be:
testsdir/
testsdir/xpdf/
testsdir/xpdf/parser_test - Binary; atf-run executes it from within
a temporary directory to "isolate" it from the read-only testsdir
and other parts of the system. parser_test can query its "srcdir",
which in this case is "testsdir/xpdf/", and thus locate where the
00-broken.pdf file is.
testsdir/xpdf/00-broken.pdf - The auxiliary data file for the xpdf
tests.
This may rule out the suitability of /usr/libexec/tests as the location
for these files, given that your description seems to imply that libexec
is for binaries only. But if it isn't, I'd be OK with using this directory.
It is also possible that introducing a new home for binaries would
cause
multi-lib heartburn in RPM, but I suspect that is a simple enough
problem to fix, since we haven't really hit it with our use of
/usr/libexec to date.
What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.
You probably should open a ticket with the FHS working group:
https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
to them so that it has a chance of being officially added in the next
major FHS revision, whenever that may be.
OK, I can do that. Wouldn't it be easier to illustrate the point /
convince them if there was existing practice somewhere? (Not sure if
BSD flavors count in this case!)
Above and beyond your proposal, the only other way I can think of
handling this would be an elaborate namespacing of the test binaries in
/usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
solution that would permit a strict compliance with the FHS as it exists
today, I'm not very excited about it and am not seriously recommending
it as an alternative.
I don't think this would be an appropriate layout, and it doesn't play
well with the structure of atf (let aside the clutter that would appear
in /usr/bin). Let me provide some additional details of how the tests
tree is structured.
Each directory in testsdir has an Atffile that indicates which test
programs exist in that directory, which subdirectories to recurse into,
and additional properties that indicate meta-data for the test programs.
So, for example, the testsdir would look like this:
testsdir/ - The top-level directory. This directory would be owned
by a filesystem package and be a dependency of anything that places
files in testsdir/.
testsdir/Atffile - The main control file. This has the knowledge to
recurse into the direct subdirectories of testsdir/ looking for
other Atffiles to execute. E.g. this would find testsdir/a/Atffile
and testsdir/b/Atffile to source tests, but it would not find
testsdir/c/Notatf nor testsdir/d/e/Atffile. This file would be
installed by the "atf" tools package, as it does not make any
sense to have it without the corresponding tools.
testsdir/lutok/ - Owned by a lutok-tests package, including all the
subfiles and subdirectories.
testsdir/lutok/Atffile - The control file for the tests provided by
lutok. Contains the list of test programs to run and any optional
meta-data for them. It also includes other subdirectories to
"recurse" into.
testsdir/lutok/operations_test - A test program for the "operations"
module of the source tree. (The name of the binary is arbitrary.)
testsdir/lutok/state_test - Another test program.
testsdir/lutok/detail/ - A subdirectory of lutok's tests containing
tests for internal modules only. (This would mimic the structure
of the source package.)
testsdir/lutok/detail/Atffile - Same as above, but for the tests in
this subdirectory.
testsdir/lutok/detail/internal_test - Yet another test program.
testsdir/lutok/detail/state_test - Another test program. Note that
this can have the same name as the program in the parent
directory! There is nothing to prevent this, and thus flattening
the test programs into /usr/bin as you propose would be very
complicated.
testsdir/lutok/foo/ - Another subdirectory for additional tests, etc.
testsdir/foobar/ - Another package, with the same structure as above.
testsdir/baz/ - Another package, but this time without any Atffile in
it. This would not be executed by atf-run because the top-level
Atffile would discard this subdirectory. (Kyua, the replacement
of atf, could run these tests though provided that a Kyuafile was
created within the subdirectory. This is out of scope now though.)
A little detail to note above is that in a project with a good amount of
test coverage, there will potentially be a test program per each source
file. This means that the testsdir contains lots of binaries, and you
don't want that many, potentially with conflicting names, anywhere in
your path! To illustrate the point, check out this output page which
contains the results of the execution of the NetBSD test suite:
http://www.whooppee.com/~paul/amd64-results/last_atf.html
Anyway. This layout may remind you a lot of make(1) and a traditional
source tree populated with Makefiles. And actually, this is the main
idea of how the tests tree is organized and is an artifact of ATF's
origins (NetBSD's source tree). However, having played with other large
source code bases, I can say that this organization is not unique to NetBSD.
We'd discuss extensively whether this layout is suboptimal for Fedora's
guidelines or not... But please note that all I'm trying to achieve
here is to figure out how to package an existing piece of software, not
how to re-architect it!
If you made it this far again, thank you.
--
Julio Merino
9:30 a.m.
On 02/06/2012 03:58 PM, Julio Merino wrote:
What do you mean by "multi-lib heartburn in RPM"? (Sorry,
not familiar
from the term and a couple of Google searches did't provide much
insight.) FWIW, I have a preliminary atf.spec file here that provides
subpackages for libraries, tests and binaries and things seem to be fine
to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
I'm not sure how much value there is in running the test suites for
supported architectures aside from the default though. If there isn't a
desire for this, we just need to explicitly state that packages
containing test-suites cannot be multi-arch.
> You probably should open a ticket with the FHS working group:
> https://bugs.linuxfoundation.org/ (product: FHS) and explain all of this
> to them so that it has a chance of being officially added in the next
> major FHS revision, whenever that may be.
OK, I can do that. Wouldn't it be easier to illustrate the point /
convince them if there was existing practice somewhere? (Not sure if
BSD flavors count in this case!)
Honestly, I don't know. Common sense says yes, but common sense has yet
to appear in the FHS revision process. ;)
Fedora does not have to wait for the FHS to approve it, but we do want
the things that we adopt to also be driven through the FHS approval
process with the hopes that we can "merge" these exceptions
"upstream".
> Above and beyond your proposal, the only other way I can think
of
> handling this would be an elaborate namespacing of the test binaries in
> /usr/bin, e.g. /usr/bin/test-$pkg-$testname , and while that would be a
> solution that would permit a strict compliance with the FHS as it exists
> today, I'm not very excited about it and am not seriously recommending
> it as an alternative.
I don't think this would be an appropriate layout, and it doesn't play
well with the structure of atf (let aside the clutter that would appear
in /usr/bin). Let me provide some additional details of how the tests
tree is structured.
[LAYOUT INFO SNIPPED]
Yes, I understand this. If we were to propose such a layout, we'd put
all the other stuff in say, /usr/share/tests/$pkg/, then provide
symlinks for the binaries into that hierarchy.
But like I said, I don't really like it.
A little detail to note above is that in a project with a good amount
of
test coverage, there will potentially be a test program per each source
file. This means that the testsdir contains lots of binaries, and you
don't want that many, potentially with conflicting names, anywhere in
your path! To illustrate the point, check out this output page which
contains the results of the execution of the NetBSD test suite:
This further illustrates why I don't really like it. I was simply
stating it because inevitably, someone else would, and I wanted it on
the record has having been considered and discarded. ;)
~tom
==
Fedora Project
3:36 a.m.
On 02/06/2012 05:30 PM, Tom Callaway wrote:
On 02/06/2012 03:58 PM, Julio Merino wrote:
> What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
> from the term and a couple of Google searches did't provide much
> insight.) FWIW, I have a preliminary atf.spec file here that provides
> subpackages for libraries, tests and binaries and things seem to be fine
> to my untrained eye. I can share this file if it helps at all.
No, what I meant here is that RPM, when dealing with multi-arch support
(essentially the ability to simultaneously foo.i686.rpm and
foo.x86_64.rpm containing the same set of files, just built for the
different architecture targets), won't understand /usr/tests (or
/usr/libexec/tests for that matter).
However, if you want to be able to support this use-case, you'll need to
do something here to enable it, such as requiring that architecture be
embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
Um, rpm doesn't do multilib conflict resolution based on specific
directories but file type, essentially 32bit vs 64bit ELF. So if
foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
/usr/libexec/tests/foo, the file from the preferred arch (normally the
64bit one) will "win", the other file is not installed at all and no
conflict is raised.
Whether you actually *want* that behavior is another question: tests and
their associated data can just as well be arch-specific or
arch-independent. The only safe assumption is to assume arch-specific
and put all test executables and associated data into arch-specific
paths (whether some variant of lib vs lib64 style differentation or by
actually embedding the arch string).
- Panu -
4:04 a.m.
On 02/07/2012 10:36 AM, Panu Matilainen wrote:
On 02/06/2012 05:30 PM, Tom Callaway wrote:
> On 02/06/2012 03:58 PM, Julio Merino wrote:
>> What do you mean by "multi-lib heartburn in RPM"? (Sorry, not familiar
>> from the term and a couple of Google searches did't provide much
>> insight.) FWIW, I have a preliminary atf.spec file here that provides
>> subpackages for libraries, tests and binaries and things seem to be fine
>> to my untrained eye. I can share this file if it helps at all.
>
> No, what I meant here is that RPM, when dealing with multi-arch support
> (essentially the ability to simultaneously foo.i686.rpm and
> foo.x86_64.rpm containing the same set of files, just built for the
> different architecture targets), won't understand /usr/tests (or
> /usr/libexec/tests for that matter).
>
> However, if you want to be able to support this use-case, you'll need to
> do something here to enable it, such as requiring that architecture be
> embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
Um, rpm doesn't do multilib conflict resolution based on specific
directories but file type, essentially 32bit vs 64bit ELF. So if
foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
/usr/libexec/tests/foo, the file from the preferred arch (normally the
64bit one) will "win", the other file is not installed at all and no
conflict is raised.
Whether you actually *want* that behavior is another question: tests and
their associated data can just as well be arch-specific or
arch-independent. The only safe assumption is to assume arch-specific
and put all test executables and associated data into arch-specific
paths (whether some variant of lib vs lib64 style differentation or by
actually embedding the arch string).
%{_libdir}/<package>/<somewhere> avoids all these issues.
Ralf
4:42 a.m.
On 02/07/2012 12:04 PM, Ralf Corsepius wrote:
On 02/07/2012 10:36 AM, Panu Matilainen wrote:
> On 02/06/2012 05:30 PM, Tom Callaway wrote:
>> On 02/06/2012 03:58 PM, Julio Merino wrote:
>>> What do you mean by "multi-lib heartburn in RPM"? (Sorry, not
familiar
>>> from the term and a couple of Google searches did't provide much
>>> insight.) FWIW, I have a preliminary atf.spec file here that provides
>>> subpackages for libraries, tests and binaries and things seem to be
>>> fine
>>> to my untrained eye. I can share this file if it helps at all.
>>
>> No, what I meant here is that RPM, when dealing with multi-arch support
>> (essentially the ability to simultaneously foo.i686.rpm and
>> foo.x86_64.rpm containing the same set of files, just built for the
>> different architecture targets), won't understand /usr/tests (or
>> /usr/libexec/tests for that matter).
>>
>> However, if you want to be able to support this use-case, you'll need to
>> do something here to enable it, such as requiring that architecture be
>> embedded in the naming scheme, e.g. /usr/tests/xpdf-x86_64/
>
> Um, rpm doesn't do multilib conflict resolution based on specific
> directories but file type, essentially 32bit vs 64bit ELF. So if
> foo-test.i686 and too-test.x86_64 both place an ELF binary into eg
> /usr/libexec/tests/foo, the file from the preferred arch (normally the
> 64bit one) will "win", the other file is not installed at all and no
> conflict is raised.
>
> Whether you actually *want* that behavior is another question: tests and
> their associated data can just as well be arch-specific or
> arch-independent. The only safe assumption is to assume arch-specific
> and put all test executables and associated data into arch-specific
> paths (whether some variant of lib vs lib64 style differentation or by
> actually embedding the arch string).
%{_libdir}/<package>/<somewhere> avoids all these issues.
Yes, except for noarch packages where %{_libdir} is not meaningful.
- Panu -
8:01 p.m.
On 02/08/2012 05:10 PM, Tom Callaway wrote:
On 02/07/2012 05:04 AM, Ralf Corsepius wrote:
> %{_libdir}/<package>/<somewhere> avoids all these issues.
No it doesn't.
Surely it is. It's what autoconf calls pkglibdir:
http://sourceware.org/autobook/autobook/autobook_106.html
Not to mention the fact that it isn't really appropriate
to be putting binaries and architecture independent files there.
pkglibdir has been common practice for more than a decade. It's a
packages private playground, a package can play almost all games it
wants to.
Using it would duplicate "shareable" binaries, but it at the same time
allows parallel installation of multiarched binaries.
As the binaries we are talking about here are "test" binaries, this
would be exactly what the OS asked for.
He could use $libdir/<package>/bin or $libdir/<package>/tests at his
personal preference.
Ralf
6:37 a.m.
On 02/06/2012 04:59 AM, Julio Merino wrote:
Hello,
The purpose of this long email is to justify the need for a /usr/tests
directory and to ask for approval for its creation, with the final goal
of packaging ATF, its tests and the tests provided by arbitrary
applications (e.g. the just-approved lutok package[1]). This email is
long because I feel the need to provide tons of background to explain
why this directory is required. I hope the text is clear and that it
makes enough sense to convince you of its approval. Here we go!
<snip>
The first thing coming in my mind was using something like
/usr/pkg1/test and put a configuration file for each package
(containing path and other stuff) into something like
/etc/tests/conf.d/. The benefit of this would be that additional
information can be stored for each test.
Just my 2 cents
regards
Markus
6:46 a.m.
On 02/06/2012 01:37 PM, Markus Mayer wrote:
The first thing coming in my mind was using something like
/usr/pkg1/test and put a configuration file for each package
(containing path and other stuff) into something like
/etc/tests/conf.d/. The benefit of this would be that additional
information can be stored for each test.
The FHS explicitly forbids that:
Large software packages must not use a direct subdirectory under the
/usr hierarchy.
~tom
==
Fedora Project
9:12 a.m.
On 02/06/2012 01:46 PM, Tom Callaway wrote:
On 02/06/2012 01:37 PM, Markus Mayer wrote:
> The first thing coming in my mind was using something like
> /usr/pkg1/test and put a configuration file for each package
> (containing path and other stuff) into something like
> /etc/tests/conf.d/. The benefit of this would be that additional
> information can be stored for each test.
The FHS explicitly forbids that:
Large software packages must not use a direct subdirectory under the
/usr hierarchy.
I'd use a directory under %{libdir}/<pkg> (aka. pkglibdir).
Ralf
5:26 a.m.
On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:
The purpose of this long email is to justify the need for a
/usr/tests
directory
To put this in the context of a Linux distribution, let's
consider a
fictitious example: imagine that glib and gtk came with ATF-based tests.
These tests would get installed into testsdir/glib/... and
testsdir/gtk/... The user would then do "cd testsdir/gtk/ && atf-run"
to execute the tests on his machine,
2) The second is that ATF is not encoded anywhere in the path. I.e.
the
directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
/usr/tests/atf/pkg*. The reason for this is that the package-specific
tests needn't be implemented using the ATF libraries.
Still, the tests must be compatible with the testing framework's set of
tools. That's enough of a reason to add some sort of namespace to the
root directory.
I also find it contradictory how you refer to "atf-run", "atf-report"
and "a collection of tools", which have "atf-" in their name.
Apparently,
they are stored in global search path for executables (and therefore it
is reasonable to prefix them with something to avoid creating too generic
names, and hopefully you wouldn't propose dropping the "atf-" prefix from
their names either).
Surely the testing framework would not be renamed regularly. And
software authors, who would use this framework for writing own tests,
would need to be able to rely on an interface for retrieving filesystem
path details anyway. For example, a pkg-config file that can be queried.
Or a similar atf-config script that returns various paths.
In fact, Kyua,
the replacement for ATF, is able to execute these same unmodified tests
and can also execute tests written other frameworks. In a future world,
packages would install arbitrary tests into /usr/tests written using
their preferred libraries (e.g. pyunit, autotest, etc.) and a single
tool (Kyua) would run them all. Encoding the tool name in the directory
introduces a dependency on the tool that does not necessarily exist.
Implementation details here would need to be discussed. The names of the
tools are irrelevant so far. The various testing frameworks would need to
be compatible somehow and meet the requirements of the controlling tools.
One couldn't dump arbitrary tests into a shared directory and e.g. mix
automated, semi-automated and non-automated tests.
Leaving FHS issues aside, in general it is not nice to let any package
"occupy" directories, which have very generic names, without very good
reason. That's not limited to /usr, and it's not a "first come, first
served"
thing either. Imagine, a second group of testing framework authors would
also want to use a global /usr/tests in order to store incompatible tests
in there.
5:28 p.m.
On 2/7/12 6:26 AM, Michael Schwendt wrote:
On Sun, 05 Feb 2012 22:59:10 -0500, JM (Julio) wrote:
> The purpose of this long email is to justify the need for a /usr/tests
> directory
> To put this in the context of a Linux distribution, let's consider a
> fictitious example: imagine that glib and gtk came with ATF-based tests.
> These tests would get installed into testsdir/glib/... and
> testsdir/gtk/... The user would then do "cd testsdir/gtk/&&
atf-run"
> to execute the tests on his machine,
> 2) The second is that ATF is not encoded anywhere in the path. I.e. the
> directories are /usr/tests/pkg*, not /usr/atf/pkg* nor
> /usr/tests/atf/pkg*. The reason for this is that the package-specific
> tests needn't be implemented using the ATF libraries.
Still, the tests must be compatible with the testing framework's set of
tools. That's enough of a reason to add some sort of namespace to the
root directory.
If you created "unknown" subdirectories under /usr/tests/, they would be
skipped by the tools. (Actually, the goal would be for the tools to
eventually run arbitrary test programs regardless of whichever libraries
they use.)
I also find it contradictory how you refer to "atf-run",
"atf-report"
and "a collection of tools", which have "atf-" in their name.
Contradictory with what? You certainly don't want a tool named "run" in
your path. There, of course, would be an "atf" dispatcher that accepted
subcommands and executed atf-run and atf-report on your back (much like
git does). But there isn't.
Surely the testing framework would not be renamed regularly. And
software authors, who would use this framework for writing own tests,
would need to be able to rely on an interface for retrieving filesystem
path details anyway. For example, a pkg-config file that can be queried.
Or a similar atf-config script that returns various paths.
Such a thing (atf-config) already exists, although it cannot tell what
the testsdir is. It actually should not, because the location of the
testsdir is a packaging issue and not something the tools care about at
all. atf-run cares about what's in the current directory and won't try
to load stuff from /usr/tests by itself, for example.
> In fact, Kyua,
> the replacement for ATF, is able to execute these same unmodified tests
> and can also execute tests written other frameworks. In a future world,
> packages would install arbitrary tests into /usr/tests written using
> their preferred libraries (e.g. pyunit, autotest, etc.) and a single
> tool (Kyua) would run them all. Encoding the tool name in the directory
> introduces a dependency on the tool that does not necessarily exist.
Implementation details here would need to be discussed. The names of the
tools are irrelevant so far.
Exactly; the names are irrelevant.
Let's say you start by putting all the tests into
/usr/tests/atf/<package>, which would be the case if the name mattered.
This directory contains the top-level Atffile, and each subdirectory
contains an Atffile of its own. atf-run can only be executed from
within /usr/tests/atf.
Now, a new tool comes along (let's call it "magic") that is able to
interpret atf-based test cases but can also run pyunit tests. This tool
needs its own Magicfiles. How do you make the previous /usr/tests/atf/
directory tree work? Surely putting Magicfiles within /usr/tests/atf/
would not be appropriate, because this is in the atf namespace. But you
actually need to have those files in there to let the magic tool know
what to do.
Before you say "well, the tools should auto-discover the tests instead
of reading Magicfiles et. al.": no, they should not. These files
include much more than just a list of test programs. They provide
metadata, such as what "framework" the test programs are written with,
or arbitrary configuration properties for the programs.
The various testing frameworks would need to
be compatible somehow and meet the requirements of the controlling tools.
One couldn't dump arbitrary tests into a shared directory and e.g. mix
automated, semi-automated and non-automated tests.
Why not? As I said above, what the "controlling tool" runs is defined
by a file in the tests directory (Atffile). Any of these incompatible
files would not have a matching Atffile, and thus atf-run would not see
them.
Leaving FHS issues aside, in general it is not nice to let any
package
"occupy" directories, which have very generic names, without very good
reason. That's not limited to /usr, and it's not a "first come, first
served"
thing either. Imagine, a second group of testing framework authors would
also want to use a global /usr/tests in order to store incompatible tests
in there.
Such other systems do not exist, and I seriously doubt any will. If
they ever do, they'd have to play well with existing practice, just as
you ask any package to respect the subpackage-directories layout in
/usr/share. And asking such imaginary systems to respect the proposed
layout would be trivial for them, so it doesn't sound too unreasonable.
Lastly: please keep in mind that all I want to do is package an existing
piece of software, not redesign it. Comments of how things should be or
should not be done within the tool are not relevant in this discussion.
(But I'll gladly take suggestions if they are raised where they
belong: in the upstream mailing lists.)
4455
days inactive
4462
days old
packaging@lists.fedoraproject.org
13 comments
6 participants
participants (6)
-
Julio Merino -
Markus Mayer -
Michael Schwendt -
Panu Matilainen -
Ralf Corsepius -
Tom Callaway