Clarify test result reporting in Standard Test Interface

[Petr Šplíchal]

Hi!

Currently the Standard Test Interface does not define how to
distinguish test failure from an infrastructure error. When
ansible playbook is run it usually does some environment
preparation and only then executes tests. Upon a problem
encountered during provisioning or configuration of the test
environment ansible returns non-zero exit code but there is not an
easy way how to detect such error from a successful test execution
of test which failed (returns non-zero exit code as well).

Let's make this part of the specification more clear so that tools
can detect infrastructure errors and report them as such to the
end user. This will help to direct problems (errors versus
failures) to the right audience (infrastructure staff versus
developer/tester).

See the following ticket for proposed solution and share your
thoughts there:

    https://pagure.io/fedora-ci/general/issue/18

Thanks.

psss...

[Petr Šplíchal]

Hi,

here's an update for those who were not involved in the issue:
After a bit longer discussion we've come to the following
agreement:

 * Introduce a new "results.yml" file which should contain
   results of individual tests in a defined format

 * Ansible playbook should return non-zero status on
   infrastructure error and not on test failures

This should allow the pipeline to clearly detect infrastructure
errors and report them accordingly. See the ticket for more
details and the list of proposed changes:

 * https://pagure.io/fedora-ci/general/issue/18

The solution is introducing new requirement for tests which are
not using Standard Test Interface, so please review this and let
us know if you have any concerns. Thanks.

psss...

On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
>
> Hi!
>
> Currently the Standard Test Interface does not define how to
> distinguish test failure from an infrastructure error. When
> ansible playbook is run it usually does some environment
> preparation and only then executes tests. Upon a problem
> encountered during provisioning or configuration of the test
> environment ansible returns non-zero exit code but there is not
> an easy way how to detect such error from a successful test
> execution of test which failed (returns non-zero exit code as
> well).
>
> Let's make this part of the specification more clear so that
> tools can detect infrastructure errors and report them as such
> to the end user. This will help to direct problems (errors
> versus failures) to the right audience (infrastructure staff
> versus developer/tester).
>
> See the following ticket for proposed solution and share your
> thoughts there:
>
>     https://pagure.io/fedora-ci/general/issue/18
>
> Thanks.
>
> psss...

[Petr Šplíchal]

On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
>  Hi,
>
>  here's an update for those who were not involved in the issue:
>  After a bit longer discussion we've come to the following
>  agreement:
>
>   * Introduce a new "results.yml" file which should contain
>     results of individual tests in a defined format
>
>   * Ansible playbook should return non-zero status on
>     infrastructure error and not on test failures
>
>  This should allow the pipeline to clearly detect infrastructure
>  errors and report them accordingly. See the ticket for more
>  details and the list of proposed changes:
>
>   * https://pagure.io/fedora-ci/general/issue/18
>
>  The solution is introducing new requirement for tests which are
>  not using Standard Test Interface, so please review this and let
>  us know if you have any concerns. Thanks. 
This should be "introducing a new requirement for tests which are
not using Standard Test Roles". Sorry for the confusion.

> psss...
>
> On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
> >
> > Hi!
> >
> > Currently the Standard Test Interface does not define how to
> > distinguish test failure from an infrastructure error. When
> > ansible playbook is run it usually does some environment
> > preparation and only then executes tests. Upon a problem
> > encountered during provisioning or configuration of the test
> > environment ansible returns non-zero exit code but there is not
> > an easy way how to detect such error from a successful test
> > execution of test which failed (returns non-zero exit code as
> > well).
> >
> > Let's make this part of the specification more clear so that
> > tools can detect infrastructure errors and report them as such
> > to the end user. This will help to direct problems (errors
> > versus failures) to the right audience (infrastructure staff
> > versus developer/tester).
> >
> > See the following ticket for proposed solution and share your
> > thoughts there:
> >
> >     https://pagure.io/fedora-ci/general/issue/18
> >
> > Thanks.
> >
> > psss...

[Petr Šplíchal]

Hi,

an update of the Standard Test Interface based on the changes
agreed in [1] is now available for review:

    https://pagure.io/fedora-ci/docs/pull-request/5

Review is appreciated because this change is introducing a
backward incompatible behaviour. Please, share your thoughts
and concerns directly in the pull request. Thanks.

psss...

[1] https://pagure.io/fedora-ci/general/issue/18

On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
>
> Hi,
>
> here's an update for those who were not involved in the issue:
> After a bit longer discussion we've come to the following
> agreement:
>
>  * Introduce a new "results.yml" file which should contain
>    results of individual tests in a defined format
>
>  * Ansible playbook should return non-zero status on
>    infrastructure error and not on test failures
>
> This should allow the pipeline to clearly detect infrastructure
> errors and report them accordingly. See the ticket for more
> details and the list of proposed changes:
>
>  * https://pagure.io/fedora-ci/general/issue/18
>
> The solution is introducing new requirement for tests which are
> not using Standard Test Roles, so please review this and let
> us know if you have any concerns. Thanks.
>
> psss...
>
> On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
> >
> > Hi!
> >
> > Currently the Standard Test Interface does not define how to
> > distinguish test failure from an infrastructure error. When
> > ansible playbook is run it usually does some environment
> > preparation and only then executes tests. Upon a problem
> > encountered during provisioning or configuration of the test
> > environment ansible returns non-zero exit code but there is not
> > an easy way how to detect such error from a successful test
> > execution of test which failed (returns non-zero exit code as
> > well).
> >
> > Let's make this part of the specification more clear so that
> > tools can detect infrastructure errors and report them as such
> > to the end user. This will help to direct problems (errors
> > versus failures) to the right audience (infrastructure staff
> > versus developer/tester).
> >
> > See the following ticket for proposed solution and share your
> > thoughts there:
> >
> >     https://pagure.io/fedora-ci/general/issue/18
> >
> > Thanks.
> >
> > psss...

[Petr Šplíchal]

Hi!

The proposed changes have been implemented in the
standard-test-roles and will soon land in updates:

https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-ae9c582836
https://bodhi.fedoraproject.org/updates/FEDORA-MODULAR-2019-c28ebce16b

Let us know if you encounter any problems. Thanks.

psss...

On Thu, 28 Feb 2019 at 15:52, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
>
> Hi,
>
> an update of the Standard Test Interface based on the changes
> agreed in [1] is now available for review:
>
>     https://pagure.io/fedora-ci/docs/pull-request/5
>
> Review is appreciated because this change is introducing a
> backward incompatible behaviour. Please, share your thoughts
> and concerns directly in the pull request. Thanks.
>
> psss...
>
> [1] https://pagure.io/fedora-ci/general/issue/18
>
> On Thu, 7 Feb 2019 at 12:24, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
> >
> > Hi,
> >
> > here's an update for those who were not involved in the issue:
> > After a bit longer discussion we've come to the following
> > agreement:
> >
> >  * Introduce a new "results.yml" file which should contain
> >    results of individual tests in a defined format
> >
> >  * Ansible playbook should return non-zero status on
> >    infrastructure error and not on test failures
> >
> > This should allow the pipeline to clearly detect infrastructure
> > errors and report them accordingly. See the ticket for more
> > details and the list of proposed changes:
> >
> >  * https://pagure.io/fedora-ci/general/issue/18
> >
> > The solution is introducing new requirement for tests which are
> > not using Standard Test Roles, so please review this and let
> > us know if you have any concerns. Thanks.
> >
> > psss...
> >
> > On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com&gt; wrote:
> > >
> > > Hi!
> > >
> > > Currently the Standard Test Interface does not define how to
> > > distinguish test failure from an infrastructure error. When
> > > ansible playbook is run it usually does some environment
> > > preparation and only then executes tests. Upon a problem
> > > encountered during provisioning or configuration of the test
> > > environment ansible returns non-zero exit code but there is not
> > > an easy way how to detect such error from a successful test
> > > execution of test which failed (returns non-zero exit code as
> > > well).
> > >
> > > Let's make this part of the specification more clear so that
> > > tools can detect infrastructure errors and report them as such
> > > to the end user. This will help to direct problems (errors
> > > versus failures) to the right audience (infrastructure staff
> > > versus developer/tester).
> > >
> > > See the following ticket for proposed solution and share your
> > > thoughts there:
> > >
> > >     https://pagure.io/fedora-ci/general/issue/18
> > >
> > > Thanks.
> > >
> > > psss...