Hi folks!
I am currently rolling out some changes to the Fedora openQA deployment which enable a new testing workflow. From now on, a subset of openQA tests should be run automatically on every critpath update, both on initial submission and on any edit of the update.
For the next little while, at least, this won't be incredibly visible. openQA sends out fedmsgs for all tests, so you can sign up for FMN notifications to learn about these results. They'll also be discoverable from the openQA web UI - https://openqa.fedoraproject.org . The results are also being forwarded to ResultsDB, so they'll be visible via ResultsDB API queries and the ResultsDB web UI. But for now, that's it...I think.
Our intent is to set up the necessary bits so that these results will show up in the Bodhi web UI alongside the results for relevant Taskotron tests. There's an outside possibility that Bodhi is actually already set up to find these results in ResultsDB, in which case they'll just suddenly start showing up in Bodhi - we should know about that soon enough. :) But most likely Bodhi will need a bit of a tweak to find them. This is probably a good thing, because we need to let the tests run for a while to find out how reliable they are, and if there's an unacceptable number of false negatives/positives. Once we have some info on that and are happy that we can get things sufficiently reliable for the results to be useful, we'll hook up the Bodhi integration.
The tests that are run are most of the tests that, on the 'compose test' workflow, get run on the Server DVD and Workstation Live images after installation. Between them they do a decent job of covering basic system functionality. They also cover FreeIPA server and client setup, and Workstation browser (Firefox) and terminal functionality. So hopefully, if your critpath update completely breaks one of those basic workflows, you'll find out about it before pushing it stable.
At present it looks like the Workstation tests may sometimes fail simply because the base install gets stuck during boot for some reason; I'm going to look into that this week. In testing so far the Server tests seem fairly reliable, but I want to gather data from a few days worth of test runs to see how those look. Once we start sending results to Bodhi, I'll try and write up some basic instructions on how to interpret and debug openQA test results; QA folks will also be available in IRC and by email for help with this, of course.
You can see sample runs on Server: https://openqa.stg.fedoraproject.org/tests/overview?groupid=1&build=FEDO... and Workstation: https://openqa.stg.fedoraproject.org/tests/overview?version=25&distri=fe... the 'desktop_notifications_live' failure is a stale bit of data - that test isn't actually run any more because obviously it makes no sense in this context, but because it got run one time in early development, openQA continues to show it for that update (it won't show for any *other* update). The `desktop_update_graphical` fail is a good example of the kind of issue I'll have to look into this week: it seems to have failed because of an intermittent crasher bug in PackageKit, rather than an issue in the update. We'll have to look at skipping known- unreliable tests, or marking them somehow so you know the deal in Bodhi, or automatically re-running them, or things along those lines.
On Mon, 2017-02-27 at 10:22 -0800, Adam Williamson wrote:
Hi folks!
I am currently rolling out some changes to the Fedora openQA deployment which enable a new testing workflow. From now on, a subset of openQA tests should be run automatically on every critpath update, both on initial submission and on any edit of the update.
Hi again folks! Just a quick update on progress here so far.
The deployment went pretty well, and the tests have been running now for the last week or so. You can view all the results so far here:
https://openqa.fedoraproject.org/group_overview/2?limit_builds=400
One thing you might notice right away is the list sort order. openQA currently sorts 'builds' (in this context, the update is the 'build') on the assumption that they sort as dotted version strings, which Fedora update IDs (the string we use as the 'build' value for these tests) certainly don't. I've got a PR in progress upstream to allow us to sort these differently, and that should get changed soon.
About half way through last week I implemented a change which means any failed test is automatically retried; this cut down quite a lot on false failures caused by transient bugs, mirror issues etc. There are still occasional cases of this, though. For now you can force all the tests to be re-run by editing the update in any way at all; in future we'll probably try and set up some system which lets you request re- runs of failed tests if the failures don't look like an actual bug in the update.
This week I'm aiming to get the necessary changes made so that Bodhi will find and display these results alongside the Taskotron results in its web UI, which should make them much more visible.
There's another significant factor which I hadn't considered: today was the Bodhi activation point for Fedora 26, meaning we now have Fedora 26 critpath updates we could test.
For now I've decided to go ahead and try and test Branched updates, and just see how much of a mess it turns out to be. I suspect, though, that we'll have problems with the tests failing due to underlying bugs far more often (certainly several of the tests currently fail on Branched, for instance), and also we'll have problems with the base disk images much more often for pre-release branches. It may prove to be difficult or impossible to provide useful feedback for Branched updates with this system, and if so, we'll turn it off.
-
Adam Williamson