SI - clarification about exit codes

[Kamil Paral]

Hello,

the testing system is required to inspect the exit code of the test suite
[1]:

To invoke the playbook, the *testing system*:
>  6. MUST examine the exit code of the playbook. A zero exit code is
>  successful *test result*, non-zero is failure.
>  
However, the test suite is not required to return the correct exit code.
Under

>  The playbook and its *test suite* or *test framework*:
>  
there is no mention of returning the exit code. I think that should be
clarified, because it can be expected that people implementing SI will only
read those parts relevant for them. I believe this should be added:

>  The playbook and its *test suite* or *test framework*:
>  
> 6. MUST return a zero exit code of the playbook if the test result is a
> pass, or a non-zero exit code if the test result is a fail.
>
Thoughts?


[1] https://fedoraproject.org/wiki/Changes/InvokingTests

[Pierre-Yves Chibon]

On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote:
>     Hello,
>  
>     the testing system is required to inspect the exit code of the test suite
>     [1]:
>  
>       To invoke the playbook, the testing system:
>       6. MUST examine the exit code of the playbook. A zero exit code is
>       successful test result, non-zero is failure.
>  
>     However, the test suite is not required to return the correct exit code.
>     Under
>  
>       The playbook and its test suite or test framework:
>  
>     there is no mention of returning the exit code. I think that should be
>     clarified, because it can be expected that people implementing SI will
>     only read those parts relevant for them. I believe this should be added:
>  
>       The playbook and its test suite or test framework:
>  
>       6. MUST return a zero exit code of the playbook if the test result is a
>       pass, or a non-zero exit code if the test result is a fail.
>  
>     Thoughts?
>  
>     [1] https://fedoraproject.org/wiki/Changes/InvokingTests 
This sounds fair to me

+1


Pierre

[Kamil Paral]

On Mon, Sep 4, 2017 at 12:01 PM, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
wrote:

>  On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote:
>  >    Hello,
>  >
>  >    the testing system is required to inspect the exit code of the test
>  suite
>  >    [1]:
>  >
>  >      To invoke the playbook, the testing system:
>  >      6. MUST examine the exit code of the playbook. A zero exit code is
>  >      successful test result, non-zero is failure.
>  >
>  >    However, the test suite is not required to return the correct exit
>  code.
>  >    Under
>  >
>  >      The playbook and its test suite or test framework:
>  >
>  >    there is no mention of returning the exit code. I think that should be
>  >    clarified, because it can be expected that people implementing SI will
>  >    only read those parts relevant for them. I believe this should be
>  added:
>  >
>  >      The playbook and its test suite or test framework:
>  >
>  >      6. MUST return a zero exit code of the playbook if the test result
>  is a
>  >      pass, or a non-zero exit code if the test result is a fail.
>  >
>  >    Thoughts?
>  >
>  >    [1] https://fedoraproject.org/wiki/Changes/InvokingTests
>
>  This sounds fair to me
>
>  +1
>
>
>  Pierre
>  
I've implemented the change here:
https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTests&d...

[Stef Walter]

On 07.09.2017 10:20, Kamil Paral wrote:
>  On Mon, Sep 4, 2017 at 12:01 PM, Pierre-Yves Chibon
> <pingou(a)pingoured.fr
>  <mailto:[e-mail]>> wrote:
>  
>      On Fri, Sep 01, 2017 at 02:02:16PM +0200, Kamil Paral wrote:
>      >    Hello,
>      >
>      >    the testing system is required to inspect the exit code of the
>      test suite
>      >    [1]:
>      >
>      >      To invoke the playbook, the testing system:
>      >      6. MUST examine the exit code of the playbook. A zero exit
>      code is
>      >      successful test result, non-zero is failure.
>      >
>      >    However, the test suite is not required to return the correct
>      exit code.
>      >    Under
>      >
>      >      The playbook and its test suite or test framework:
>      >
>      >    there is no mention of returning the exit code. I think that
>      should be
>      >    clarified, because it can be expected that people implementing
>      SI will
>      >    only read those parts relevant for them. I believe this should
>      be added:
>      >
>      >      The playbook and its test suite or test framework:
>      >
>      >      6. MUST return a zero exit code of the playbook if the test
>      result is a
>      >      pass, or a non-zero exit code if the test result is a fail.
>      >
>      >    Thoughts?
>      >
>      >    [1] https://fedoraproject.org/wiki/Changes/InvokingTests
>      <https://fedoraproject.org/wiki/Changes/InvokingTests>
>  
>      This sounds fair to me
>  
>      +1
>  
>  
>      Pierre
>  
>  
>  I've implemented the change here:
>
> https://fedoraproject.org/w/index.php?title=Changes%2FInvokingTests&d...
Makes sense to me. To be clear this is a spec only change and does not
affect any already ported/wrapped/written tests. Correct?

Stef

[Kamil Paral]

On Mon, Sep 11, 2017 at 2:09 PM, Stef Walter <stefw(a)redhat.com&gt; wrote:

>  Makes sense to me. To be clear this is a spec only change and does
> not
>  affect any already ported/wrapped/written tests. Correct?
>  
>  Stef
>  
Yes, I only adjust the wiki page, nothing else.