On Wed, Sep 6, 2017 at 8:30 AM, Kamil Paral <kparal(a)redhat.com> wrote:
On Wed, Sep 6, 2017 at 2:07 PM, Frederic Lepied
<flepied(a)redhat.com>
wrote:
> On Wed, Sep 6, 2017 at 7:12 AM, Kamil Paral <kparal(a)redhat.com> wrote:
>
>> On Wed, Sep 6, 2017 at 12:33 PM, Kamil Paral <kparal(a)redhat.com> wrote:
>>
>>> Third, SI says:
>>>
>>>> The playbook and its test suite or test framework:
>>>> MUST place the main readable output of the test suite into a test.log
>>>> file in the artifacts variable folder. This MUST happen even if some of
the
>>>> test suites fail.
>>>>
>>>
>>> However, the default example doesn't contain any reference to
>>> *test.log* and uses *output* file instead. Shouldn't those examples
>>> above redirect to *test.log* instead?
>>>
>>
>> It turns out I've been looking at old cached wiki content. After
>> refreshing, I see that *test.log* is referenced instead.
>>
>>
>>
>>> Or is *test.log* not supposed to be an stdout+stderr output of the
>>> main test process? I don't exactly understand the definition from SI,
what *main
>>> readable output* is supposed to be. When the test suite consists of X
>>> individual tests, are they supposed to concatenate their output to
>>> *test.log*?
>>>
>>
>> These questions still stand, though.
>>
>>
>
> You should see the test.log file as the main file to debug the run so it
> should contain the information to debug issues. So it's up to the test
> implementer to decide if it's stdout or stderr or both according to what he
> needs to debug. Does it make sense?
>
> Fred
>
It does, thanks. Will you include this clarification in the spec? The main
important information for me here is that the purpose of that file is to
*debug* the execution, and most often it'll be stdout+stderr. Because
"readable output", as it currently written, might be understood as human or
machine readable, i.e. a json with detailed results, which is something
completely different.
I'll update the specs if no one objects in the coming days.
Fred