Regression testing

[Gregory Maxwell]

On Sat, 19 Mar 2005 21:05:04 -0600, Chris Adams <cmadams at hiwaay.net> wrote:
> Once upon a time, Gregory Maxwell <gmaxwell at gmail.com> said:
> > > Since OpenSSH includes both a server and a client, it can test both
> > > against each other.
> >
> > And it would be fantastic for it to do that...
> 
> It does.
> 
> When I build my OpenSSH RPMs, I do a "rpmbuild -ba", and when that is
> done I "cd ../BUILD/openssh-*; make tests" (on Tru64 I have to
> "SUDO=`which sudo` make tests").  If there's a failure, I remove the
> built RPMs and work on it.
> 
> In an ideal world, regression tests wouldn't be a part the RPM or .spec
> file; they'd be done by the packager (I understand that that isn't very
> realistic though).

Thats a fine for testing on a specific example on each platform,  but
I think an ideal world where any user at any time could request tests
be performed on their software...  not just to catch bugs in the
packages directly, but to catch bugs that arise from other parts of
their system. (i.e. libc and kernel bugs found due to irregularities
in application level tests)

Testing at the packaging point is important.. No package with tests
should be distributed if the tests fail in any important way, and just
about all packages have tests....  Putting the tests in the build
process would help keep them from being missed, ... It would be a rosy
future indeed if all package had a good set of tests, the binary RPMs
had flags to indicate they were run and passed at build, and distro
policy was to never ship (whatever that means in a world moving more
and more to online distribution) an untested package.   ... We're a
long way off from that, but it's certainly possible.

Writing tests could well work out to be another task performed by
those who want to contribute to free software but whom can't, for
whatever reason, contribute substantially to the coding itself.

But testing comes in many kinds and should be done in many places... 
If more packages had 'random input workout' test rigs, I'd gladly run
them in the idle time of quite a few boxes,  and potentially catch a
bug or two.