New automated test coverage: openQA tests of critical path updates

[Adam Williamson]

Hi folks!

I am currently rolling out some changes to the Fedora openQA deployment
which enable a new testing workflow. From now on, a subset of openQA
tests should be run automatically on every critpath update, both on
initial submission and on any edit of the update.

For the next little while, at least, this won't be incredibly visible.
openQA sends out fedmsgs for all tests, so you can sign up for FMN
notifications to learn about these results. They'll also be
discoverable from the openQA web UI - https://openqa.fedoraproject.org
. The results are also being forwarded to ResultsDB, so they'll be
visible via ResultsDB API queries and the ResultsDB web UI. But for
now, that's it...I think.

Our intent is to set up the necessary bits so that these results will
show up in the Bodhi web UI alongside the results for relevant
Taskotron tests. There's an outside possibility that Bodhi is actually
already set up to find these results in ResultsDB, in which case
they'll just suddenly start showing up in Bodhi - we should know about
that soon enough. :) But most likely Bodhi will need a bit of a tweak
to find them. This is probably a good thing, because we need to let the
tests run for a while to find out how reliable they are, and if there's
an unacceptable number of false negatives/positives. Once we have some
info on that and are happy that we can get things sufficiently reliable
for the results to be useful, we'll hook up the Bodhi integration.

The tests that are run are most of the tests that, on the 'compose
test' workflow, get run on the Server DVD and Workstation Live images
after installation. Between them they do a decent job of covering basic
system functionality. They also cover FreeIPA server and client setup,
and Workstation browser (Firefox) and terminal functionality. So
hopefully, if your critpath update completely breaks one of those basic
workflows, you'll find out about it before pushing it stable.

At present it looks like the Workstation tests may sometimes fail
simply because the base install gets stuck during boot for some reason;
I'm going to look into that this week. In testing so far the Server
tests seem fairly reliable, but I want to gather data from a few days
worth of test runs to see how those look. Once we start sending results
to Bodhi, I'll try and write up some basic instructions on how to
interpret and debug openQA test results; QA folks will also be
available in IRC and by email for help with this, of course.

You can see sample runs on Server:
https://openqa.stg.fedoraproject.org/tests/overview?groupid=1&build=F...
and Workstation:
https://openqa.stg.fedoraproject.org/tests/overview?version=25&distri...
the 'desktop_notifications_live' failure is a stale bit of data - that
test isn't actually run any more because obviously it makes no sense in
this context, but because it got run one time in early development,
openQA continues to show it for that update (it won't show for any
*other* update). The `desktop_update_graphical` fail is a good example
of the kind of issue I'll have to look into this week: it seems to have
failed because of an intermittent crasher bug in PackageKit, rather
than an issue in the update. We'll have to look at skipping known-
unreliable tests, or marking them somehow so you know the deal in
Bodhi, or automatically re-running them, or things along those lines.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Adam Williamson]

On Mon, 2017-02-27 at 10:22 -0800, Adam Williamson wrote:
>  Hi folks!
>  
>  I am currently rolling out some changes to the Fedora openQA deployment
>  which enable a new testing workflow. From now on, a subset of openQA
>  tests should be run automatically on every critpath update, both on
>  initial submission and on any edit of the update. 
Hi again folks! Just a quick update on progress here so far.

The deployment went pretty well, and the tests have been running now
for the last week or so. You can view all the results so far here:

https://openqa.fedoraproject.org/group_overview/2?limit_builds=400

One thing you might notice right away is the list sort order. openQA
currently sorts 'builds' (in this context, the update is the 'build')
on the assumption that they sort as dotted version strings, which
Fedora update IDs (the string we use as the 'build' value for these
tests) certainly don't. I've got a PR in progress upstream to allow us
to sort these differently, and that should get changed soon.

About half way through last week I implemented a change which means any
failed test is automatically retried; this cut down quite a lot on
false failures caused by transient bugs, mirror issues etc. There are
still occasional cases of this, though. For now you can force all the
tests to be re-run by editing the update in any way at all; in future
we'll probably try and set up some system which lets you request re-
runs of failed tests if the failures don't look like an actual bug in
the update.

This week I'm aiming to get the necessary changes made so that Bodhi
will find and display these results alongside the Taskotron results in
its web UI, which should make them much more visible.

There's another significant factor which I hadn't considered: today was
the Bodhi activation point for Fedora 26, meaning we now have Fedora 26
critpath updates we could test.

For now I've decided to go ahead and try and test Branched updates, and
just see how much of a mess it turns out to be. I suspect, though, that
we'll have problems with the tests failing due to underlying bugs far
more often (certainly several of the tests currently fail on Branched,
for instance), and also we'll have problems with the base disk images
much more often for pre-release branches. It may prove to be difficult
or impossible to provide useful feedback for Branched updates with this
system, and if so, we'll turn it off.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net