a few questions regarding the example playbook for running tests

[Kamil Paral]

Hello,

I'd like to get a few clarifications regarding the default test playbook
example here:
https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test

First, the example seems to suggest that all binaries should be placed in
/usr/local/bin:

  - name: Create the folder where we will store the tests
    action: file state=directory path={{ item }} owner=root group=root
    with_items:
    - /usr/local/bin

  - name: Install the test files
    copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
    with_items:
    - {file: test-simple, dest: test-simple }


But I don't see such a requirement in the SI. Why shouldn't I run the test
executable from its current location (i.e. the git checkout directory)? And
if they can, why make the example unnecessarily complex?


Second, I'm a bit confused by the exec command in the test execution line:

    - name: Execute the tests
      shell: exec > {{ artifacts }}/output && /usr/local/bin/test-simple
2>&1


I've never seen this usage so I had to study what this does exactly in the
bash man page. I believe it's the same thing as this:

shell: /usr/local/bin/test-simple >{{ artifacts }}/output 2>&1


So why exactly don't we use this more readable and more often used syntax
to redirect a process output? An example should be as simple as possible.


Third, SI says:

>  The playbook and its test suite or test framework:
>   MUST place the main readable output of the test suite into a test.log
>  file in the artifacts variable folder. This MUST happen even if some of the
>  test suites fail.
>  
However, the default example doesn't contain any reference to *test.log*
and uses *output* file instead. Shouldn't those examples above redirect to
*test.log* instead? Or is *test.log* not supposed to be an stdout+stderr
output of the main test process? I don't exactly understand the definition
from SI, what *main readable output* is supposed to be. When the test suite
consists of X individual tests, are they supposed to concatenate their
output to *test.log*?


It would be great to have these clarified.

Thanks,
Kamil

[Kamil Paral]

On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  Third, SI says:
>
> > The playbook and its test suite or test framework:
> >  MUST place the main readable output of the test suite into a test.log
> > file in the artifacts variable folder. This MUST happen even if some of the
> > test suites fail.
> >
>
>  However, the default example doesn't contain any reference to *test.log*
>  and uses *output* file instead. Shouldn't those examples above redirect
>  to *test.log* instead?
>  
It turns out I've been looking at old cached wiki content. After
refreshing, I see that *test.log* is referenced instead.



>  Or is *test.log* not supposed to be an stdout+stderr output of the
> main
>  test process? I don't exactly understand the definition from SI, what *main
>  readable output* is supposed to be. When the test suite consists of X
>  individual tests, are they supposed to concatenate their output to
>  *test.log*?
>  
These questions still stand, though.

[Frederic Lepied]

On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral
> <kparal(a)redhat.com&gt; wrote:
>
> > Third, SI says:
> >
> >> The playbook and its test suite or test framework:
> >>  MUST place the main readable output of the test suite into a test.log
> >> file in the artifacts variable folder. This MUST happen even if some of the
> >> test suites fail.
> >>
> >
> > However, the default example doesn't contain any reference to *test.log*
> > and uses *output* file instead. Shouldn't those examples above redirect
> > to *test.log* instead?
> >
>
>  It turns out I've been looking at old cached wiki content. After
>  refreshing, I see that *test.log* is referenced instead.
>
>
>
> > Or is *test.log* not supposed to be an stdout+stderr output of the main
> > test process? I don't exactly understand the definition from SI, what *main
> > readable output* is supposed to be. When the test suite consists of X
> > individual tests, are they supposed to concatenate their output to
> > *test.log*?
> >
>
>  These questions still stand, though.
>
>  
You should see the test.log file as the main file to debug the run so it
should contain the information to debug issues. So it's up to the test
implementer to decide if it's stdout or stderr or both according to what he
needs to debug. Does it make sense?

Fred

[Kamil Paral]

On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied <flepied(a)redhat.com&gt; wrote:

>  On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal(a)redhat.com&gt;
> wrote:
>
> > On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal(a)redhat.com&gt; wrote:
> >
> >> Third, SI says:
> >>
> >>> The playbook and its test suite or test framework:
> >>>  MUST place the main readable output of the test suite into a test.log
> >>> file in the artifacts variable folder. This MUST happen even if some of the
> >>> test suites fail.
> >>>
> >>
> >> However, the default example doesn't contain any reference to *test.log*
> >> and uses *output* file instead. Shouldn't those examples above redirect
> >> to *test.log* instead?
> >>
> >
> > It turns out I've been looking at old cached wiki content. After
> > refreshing, I see that *test.log* is referenced instead.
> >
> >
> >
> >> Or is *test.log* not supposed to be an stdout+stderr output of the main
> >> test process? I don't exactly understand the definition from SI, what *main
> >> readable output* is supposed to be. When the test suite consists of X
> >> individual tests, are they supposed to concatenate their output to
> >> *test.log*?
> >>
> >
> > These questions still stand, though.
> >
> >
>
>  You should see the test.log file as the main file to debug the run so it
>  should contain the information to debug issues. So it's up to the test
>  implementer to decide if it's stdout or stderr or both according to what he
>  needs to debug. Does it make sense?
>
>  Fred
>  
It does, thanks. Will you include this clarification in the spec? The main
important information for me here is that the purpose of that file is to
*debug* the execution, and most often it'll be stdout+stderr. Because
"readable output", as it currently written, might be understood as human or
machine readable, i.e. a json with detailed results, which is something
completely different.

[Frederic Lepied]

On Wed, Sep 6, 2017 at 8:30 AM, Kamil Paral <kparal(a)redhat.com&gt; wrote:

>  On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied
> <flepied(a)redhat.com&gt;
>  wrote:
>
> > On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal(a)redhat.com&gt; wrote:
> >
> >> On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal(a)redhat.com&gt; wrote:
> >>
> >>> Third, SI says:
> >>>
> >>>> The playbook and its test suite or test framework:
> >>>>  MUST place the main readable output of the test suite into a test.log
> >>>> file in the artifacts variable folder. This MUST happen even if some of
> the
> >>>> test suites fail.
> >>>>
> >>>
> >>> However, the default example doesn't contain any reference to
> >>> *test.log* and uses *output* file instead. Shouldn't those examples
> >>> above redirect to *test.log* instead?
> >>>
> >>
> >> It turns out I've been looking at old cached wiki content. After
> >> refreshing, I see that *test.log* is referenced instead.
> >>
> >>
> >>
> >>> Or is *test.log* not supposed to be an stdout+stderr output of the
> >>> main test process? I don't exactly understand the definition from SI,
> what *main
> >>> readable output* is supposed to be. When the test suite consists of X
> >>> individual tests, are they supposed to concatenate their output to
> >>> *test.log*?
> >>>
> >>
> >> These questions still stand, though.
> >>
> >>
> >
> > You should see the test.log file as the main file to debug the run so it
> > should contain the information to debug issues. So it's up to the test
> > implementer to decide if it's stdout or stderr or both according to what he
> > needs to debug. Does it make sense?
> >
> > Fred
> >
>
>  It does, thanks. Will you include this clarification in the spec? The main
>  important information for me here is that the purpose of that file is to
>  *debug* the execution, and most often it'll be stdout+stderr. Because
>  "readable output", as it currently written, might be understood as human or
>  machine readable, i.e. a json with detailed results, which is something
>  completely different.
>
>  I'll update the specs if no one objects in the coming days. 
Fred

[Merlin Mathesius]

On 09/06/2017 07:07 AM, Frederic Lepied wrote:
>  On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal(a)redhat.com
>  <mailto:[e-mail]>> wrote:
>
>      On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal(a)redhat.com
>      <mailto:[e-mail]>> wrote:
>
>          Third, SI says:
>
>              The playbook and its test suite or test framework:
>               MUST place the main readable output of the test suite
>              into a test.log file in the artifacts variable folder.
>              This MUST happen even if some of the test suites fail.
>
>
>          However, the default example doesn't contain any reference to
>          /test.log/ and uses /output/ file instead. Shouldn't those
>          examples above redirect to /test.log/ instead?
>
>
>      It turns out I've been looking at old cached wiki content. After
>      refreshing, I see that /test.log/ is referenced instead.
>
>       
>
>          Or is /test.log/ not supposed to be an stdout+stderr output of
>          the main test process? I don't exactly understand the
>          definition from SI, what /main readable output/ is supposed to
>          be. When the test suite consists of X individual tests, are
>          they supposed to concatenate their output to /test.log/?
>
>
>      These questions still stand, though.
>       
>
>
>  You should see the test.log file as the main file to debug the run so
>  it should contain the information to debug issues. So it's up to the
>  test implementer to decide if it's stdout or stderr or both according
>  to what he needs to debug. Does it make sense?
>
>  Fred 
For what it's worth, the standard-test-beakerlib and standard-test-rhts
included in the standard-test-roles package [1] currently write test.log
as a brief summary of the results of individual tests. For example,

    PASS test-1
    PASS test-2
    FAIL test-3
    ...

Beyond that, standard-test-beakerlib captures a test./test-name/.log
artifact containing the stdout+stderr for each individual test. (eg.,
test.test-1.log, test.test-2.log, etc.), whereas standard-test-rhts
includes the entire job./XX/ tree (generated from restraint) as an artifact.

[1] https://pagure.io/standard-test-roles

Merlin

[Kamil Paral]

On Wed, Sep 6, 2017 at 7:18 PM, Merlin Mathesius <mmathesi(a)redhat.com&gt;
wrote:

>  For what it's worth, the standard-test-beakerlib and
> standard-test-rhts
>  included in the standard-test-roles package [1] currently write test.log as
>  a brief summary of the results of individual tests. For example,
>
>  PASS test-1
>  PASS test-2
>  FAIL test-3
>  ...
>
>  Beyond that, standard-test-beakerlib captures a test.*test-name*.log
>  artifact containing the stdout+stderr for each individual test. (eg.,
>  test.test-1.log, test.test-2.log, etc.), whereas standard-test-rhts
>  includes the entire job.*XX* tree (generated from restraint) as an
>  artifact.
>
>  [1] https://pagure.io/standard-test-roles
>
>  Merlin
>  
Thanks Merlin. That's exactly what I wanted to avoid, each author
understanding the spec differently.

I think there are two major kinds of artifacts - an stdout+stderr output
used for debugging problems, and (optionally, but frequently) a result
summary. I wouldn't probably try to standardize the result summary file
name, it's not strictly required and the file type can vary a lot -
results.html, summary.txt, results.tap, results/index.html, etc. But it
would be useful to have a standard name for the stdout+stderr output, so
that you always know where to look when debugging problems (test.log is as
good name for it as any).

So if we go this way, some of the roles in standard-test-roles should get
updated.

[Stef Walter]

On 06.09.2017 12:33, Kamil Paral wrote:
>  Hello,
>  
>  I'd like to get a few clarifications regarding the default test playbook
>  example here:
>  https://fedoraproject.org/wiki/CI/Tests#Wrapping_a_script_test
>  
>  First, the example seems to suggest that all binaries should be placed
>  in /usr/local/bin:
>  
>    - name: Create the folder where we will store the tests
>      action: file state=directory path={{ item }} owner=root group=root
>      with_items:
>      - /usr/local/bin
>  
>    - name: Install the test files
>      copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
>      with_items:
>      - {file: test-simple, dest: test-simple }
>  
>  
>  But I don't see such a requirement in the SI. Why shouldn't I run the
>  test executable from its current location (i.e. the git checkout
>  directory)? And if they can, why make the example unnecessarily complex? 
I looked through the thread but didn't find an answer to this part of
your question.

This is because when running against a docker container or atomic host
test subject (qcow2 and docker respectively) such shell script tests
(ditto for beakerlib) need to be placed inside the test subject in order
to be run.

Cheers,

Stef

[Kamil Paral]

On Mon, Sep 11, 2017 at 2:11 PM, Stef Walter <stefw(a)redhat.com&gt; wrote:

>  I looked through the thread but didn't find an answer to this
> part of
>  your question.
>
>  This is because when running against a docker container or atomic host
>  test subject (qcow2 and docker respectively) such shell script tests
>  (ditto for beakerlib) need to be placed inside the test subject in order
>  to be run.
>
>  Cheers,
>
>  Stef
>  
I see, that makes sense. Thanks for explanation. I'm looking forward to
replies to my other questions as well :) Thanks.