5:24 a.m.
Hi,
here's an update for those who were not involved in the issue:
After a bit longer discussion we've come to the following
agreement:
* Introduce a new "results.yml" file which should contain
results of individual tests in a defined format
* Ansible playbook should return non-zero status on
infrastructure error and not on test failures
This should allow the pipeline to clearly detect infrastructure
errors and report them accordingly. See the ticket for more
details and the list of proposed changes:
* https://pagure.io/fedora-ci/general/issue/18
The solution is introducing new requirement for tests which are
not using Standard Test Interface, so please review this and let
us know if you have any concerns. Thanks.
psss...
On Tue, 11 Dec 2018 at 14:05, Petr Šplíchal <psplicha(a)redhat.com> wrote:
>
> Hi!
>
> Currently the Standard Test Interface does not define how to
> distinguish test failure from an infrastructure error. When
> ansible playbook is run it usually does some environment
> preparation and only then executes tests. Upon a problem
> encountered during provisioning or configuration of the test
> environment ansible returns non-zero exit code but there is not
> an easy way how to detect such error from a successful test
> execution of test which failed (returns non-zero exit code as
> well).
>
> Let's make this part of the specification more clear so that
> tools can detect infrastructure errors and report them as such
> to the end user. This will help to direct problems (errors
> versus failures) to the right audience (infrastructure staff
> versus developer/tester).
>
> See the following ticket for proposed solution and share your
> thoughts there:
>
> https://pagure.io/fedora-ci/general/issue/18
>
> Thanks.
>
> psss...