Regression testing

[Peter Jones]

Chris Adams wrote:
> Some regression tests are designed to be run in the build tree, so
> building a package is probably not the best way to do this (the OpenSSH
> tests are this way for example).
>
> A better idea would be for the %build or %install stage (as appropriate)
> to optionally run such tests, so if they fail, the RPM doesn't build.

I think this is a bad answer.  If we're trying to save cycles at build
time, it'd be better to split them into the set of "critical" tests and
"really important" tests.  The "critical" ones would test the
functionality that the package *must* provide for the rest of the
distro, and would run during the build.  The "really important" ones
would be packaged in a separate file (like -debuginfo is) and could be
run out of band.

On Tue, 2005-03-22 at 07:01 -0500, Mike A. Harris wrote:
> The openssh tests also consume 99.9% of all CPUs even in 8way boxes, and
> do this for like 16 hours straight.  This has caused massive buildsystem
> contention in the past when we did have it enabled.

Again, this is a great example where should run a limited test during
the build, and a more elaborate test later.

I think Greg Maxwell said (sorry if I get the attributions wrong, your
indentation isn't obvious):
> One way to accomplish this would be to have the spec optionally build
> a packagename-regression rpm, which includes the scripts and data
> needed to perform the automated regression testing.

This is a good idea.  It'd be ideal to make sure the tests are packaged
in such a way that they could be run later, against a built package.
That would allow:

1) the ability to ship the packages on CDs and such without including
   the extra bloat of the test code.  This is also true of tests that
   run during the build, of course.
2) the ability not to run the tests during the build.  Clearly, there
   are some tests that should be run at build time, but a full test
   suite is not really required.  With openssl, being able to generate
   and use keys needed for ssh and https is probably near all you
   really need to do to say the build completed ok.
3) the ability for moderately-trusted individuals in the community to
   help with the burden of running test suites, which also allows them
   to show that they're making a useful contribution.  People who do a
   good job of making sure tests are run correctly, and figuring out
   why failures happen would be good candidates to invite to the
   maintainers mailing list.

On Tue, 2005-03-22 at 07:01 -0500, Mike A. Harris wrote:
> I agree that enabling post build regression tests is in general a good
> idea, but before anyone enables tests in a given package, they should
> determine how much buildsystem load the tests may cause, and for what
> length of time, and also be mindful that the buildsystem is a shared
> resource used by numerous people.

They should also take into account what rate of false-negative results
is acceptable before the tests become too much of a burden.  For
example, several months ago I built a big pile of CPAN modules on
x86_64, and a very large number of their self-tests failed for
essentially bogus reasons.  I can't imagine ppc/ppc64/s390/s390x is any
_better_.

> For tests that pound the buildsystem(s), they should be disabled by
> default IMHO, and only enabled every now and then when the buildsystem
> is very idle, and never at peak times of building.

I don't think we should have to *change* the package (contents or .spec)
to determine if the tests run.  Those that test for functionality that's
critical to other Fedora packages should be run at build time if at all
possible, other tests are nice to run, but can really be delayed until
later.

-- 
        Peter