Re: Why do gating tests start out 'failed'?
by Adam Williamson
On Thu, 2020-09-03 at 11:15 +0100, Andrew Price wrote:
> Hi,
>
> I've just enabled gating tests for gfs2-utils and after the new build
> finished I got a notification email from bodhi that said:
>
> "This update's test gating status has been changed to 'failed'."
>
> On the bodhi update page[0] the "fedora-ci.koji-build.tier0.functional"
> test was highlighted red and marked as failed, with no extra information
> available.
>
> After a while, the original 'failed' test turned green and showed
> successful, which was puzzling. Why was it marked failed in the first
> place? All's well that ends well, but I wasted a bit of time trying to
> work out why it had failed when it really hadn't...
Because the requirements aren't satisfied. The requirements are that
certain specific tests pass. When Bodhi asks Greenwave what the status
for a given update is, if there are passes for all of those tests, it
says "passed". If not, it says "failed". That's all the subtlety there
is. :)
I suggested a while back that we should be able to define requirements
in Greenwave as "if a result for this test exists, it must be pass"
(but if no result exists, that's OK too). It hasn't been done, though,
and some folks at least didn't think it was a good idea due to the
possibility of allowing broken things through due to test names
changing, issues in the test systems, etc. It also produces an obvious
loophole that everything passes CI right after it's submitted, unless
you start getting a bit hacky with minimum times to wait for results to
appear and stuff, which starts to get complicated and possibly fragile.
We could improve things somewhat here if we had an equivalent of
ResultsDB (the source of truth for test results for this whole system)
for test *execution*, but we don't. Any test system that wants into The
System has to submit results to ResultsDB, but we don't have an
equivalent test *execution* database that everything has to submit to.
We do have a standard for text execution *messaging* -
https://pagure.io/fedora-ci/messages - but not all test systems are
entirely compliant with it, and using the info would require work in
Greenwave and Bodhi and maybe elsewhere.
I think Greenwave may provide some extended info here that Bodhi could
use to be a bit more detailed (I *think* the 'extended' Greenwave
result includes info on which tests are *failed* and which tests are
*missing*); if it doesn't it could, and then Bodhi could actually draw
a visual distinction between "there are failed tests" and "there are no
failed tests but we don't have passes for all required tests yet". That
might be an improvement.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
3 years, 3 months
