[Fedora-packaging] How to handle tests in %check

[Michael Schwendt]

On Thu, 05 Sep 2013 09:21:31 +0200, Miroslav Suchy wrote:

> Hi,
> I would like to ask on your opinion on
> https://bugzilla.redhat.com/show_bug.cgi?id=912960#c25
> 
> Mamoru have in spec in %check section:
>    ruby -Ilib:test ./test/run-test.rb || echo "Please investigate this"
> 
> I'm saying that it should not be waived this way as it waive all failure.
> I'm saying that if there is no failure, it should be just:
>    ruby -Ilib:test ./test/run-test.rb
> and if there happened some problem which need upstream attention and 
> could not be fixed immediatelly, it should be replaced with something like:
>    FILE=$(mktemp)
>    ruby -Ilib:test ./test/run-test.rb | tee $FILE || :
>    # test foo is failing, reported as http://foo/issue/1
>    cat $FILE | grep "4 tests, 4 assertions, 1 failures, 0 errors, 0 
> pendings, 0 omissions, 0 notifications
> 
> Mamoru disagree. For me it is blocker.
> I would like to ask for your opinions. Which way is correct and should 
> it be blocker for review?

So far, it's not a blocker.

There are no guidelines for the %check section other than this:
https://fedoraproject.org/wiki/Packaging:Guidelines#Test_Suites

Some packages don't execute test-suites at all, because they would fail.
That's not a blocker either. In some cases, upstream is aware of the 
tests being out-of-sync with the rest of the code.

Once you go down the '|| :' road, at least the test results will be
available in the build.log. That's acceptable and a bit better than
not running the tests at all, but may be dangerous depending on what
fails.

Anything beyond that is up the maintainer. How often would you look at
the output from the "grep"? How often will it change with new releases?
I could imagine someone grepping for the number of known failures and
adding a guard that would make the build fail.