11:02 p.m.
Hi folks! I thought this was worth sending out an announcement about so
as many packagers and testers as possible are aware of it.
With the rollout of Bodhi 2.6.0 to production today, openQA test
results for critpath updates now appear in the Bodhi webUI! Click the
'Automated Tests' tab on any critical path update (from the last two
months or so) and you should see, as well as the 'dist.*' tests you're
probably familiar with that are run by Taskotron, results for several
'update.*' tests. These are the openQA test results.
Clicking any result will take you to the openQA webUI page for that
job. If you're investigating a failure, look for thumbnails with a
*red* border. Usually, the first one of these will be the attempted
image match or console command that did not give the expected result.
If you can't understand a failure, please do come ask on test@ or in
#fedora-qa . Myself and garretraziel (Jan Sedlak) should be able to
explain.
This isn't a perfect test system, yet; there have been and will
continue to be 'false' failures, where something goes wrong in the test
process itself or the test just hits some transient bug that isn't
actually caused by the update. Because of this, we wrote an openQA
plugin that automatically retries all failed update tests one time, but
sometimes we get unlucky and a false failure happens again on the
retry.
This is quite common for Fedora 26 updates; there seem to be several of
these transient bugs in F26 at present, meaning sometimes the test VM
just doesn't boot successfully, sometimes GNOME crashes, and so on. If
you see a 'red' screenshot which is just a partially-completed
bootsplash screen, or the GDM login screen, this may be what happened.
Notably, *any* failure before the _advisory_update step cannot possibly
be a bug in the update, as nothing from the update is actually
installed until near the end of that step.
It's not currently possible for anyone but openQA admins to re-run
individual tests. You can cause *all* the openQA tests for your update
to be re-run by editing the update in any way (*any* edit event
triggers a re-run of the tests, not just changes to the update's
package manifest), but please be a bit sparing with this, as openQA
doesn't have unlimited capacity. For now you can, again, ask Jan or I
to re-run a single test if you'd like this. We will endeavour to set up
some kind of re-run request system in future.
Just like the taskotron results, at present these results are entirely
advisory. They do not have any effect on whether or when you can push
your update stable. But we set up this system to help out packagers, so
I hope you'll find it useful to keep an eye on the results and take a
look at any failures to see if they may indicate a bug in the update.
Once again, please do ask for any help you need in interpreting or
understanding the results. And please do send any suggestions, comments
or complaints our way!
And just to be clear, these tests are currently run only on *critical
path* updates. If your update does not include a critical path package,
the tests will not be run. I'm thinking of implementing some sort of
'whitelist' system for listing or otherwise marking non-critpath
packages for which we want to run some or all of the tests; for
instance, it would make a lot of sense to run the FreeIPA tests for any
package in the FreeIPA stack. But that's not implemented yet. We don't
just run the tests on *all* updates because we simply don't have the
capacity to do so at present.
I have written a blog post about this, with some more information,
including a brief explanation of what the current set of update tests
covers, here:
https://www.happyassassin.net/2017/04/24/automated-critical-path-update-f...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net