Hi folks! I thought this might be of interest to the Cockpit community, so I thought I'd write it up here in case anyone missed it elsewhere.
I work on the Fedora QA team, and we have been using the openQA automated test system (developed by our friends at SUSE) to run various functional tests on Fedora composes for the last couple of years.
As Cockpit is considered a critical part of Fedora Server, we run a few tests that exercise Cockpit. They're nowhere near as extensive as Cockpit's own test suite, but they run in a somewhat different context: they run on the Fedora packaged Cockpit, against the rest of a Fedora environment. They also test in a somewhat different way from Cockpit's own test suite: openQA tests a lot like a human, by booting up a virtual machine and then effectively "looking at the screen" and sending mouse and keyboard input events (via VNC). The test boots a full Fedora system, runs Firefox, and interacts with Cockpit like a human being would. For both of these reasons, it can sometimes catch problems Cockpit's own tests would not (it also sometimes goes wonky in ways that Selenium-based tests don't, but never mind, that's my job to deal with :>)
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
This means that for any Fedora update containing cockpit or any critical path package, Fedora's openQA cockpit tests should run, and you should see the results in the Fedora update system (Bodhi). You can see the results in Bodhi by clicking the Automated Updates tab for any update. For instance, here's a recent Firefox update for Fedora 25:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-31c64a0bbf
If you look at the Automated Tests tab, you can see passes for:
update.server_cockpit_basic update.server_cockpit_default update.realmd_join_cockpit
indicating that this Firefox update didn't cause any problems for Cockpit. Clicking on any test result will take you to the openQA page for the test, where you can diagnose failures and so on (explaining how to do this is a bit beyond the scope of this mail, please do ask me if you're interested!)
I hope this stuff will help us avoid shipping updates that break Cockpit (and other key components). If you have any questions, concerns, comments, or suggestions, please do ask!
To anticipate one question: you can cause *all* the tests for an update to be re-run by editing the update in any way (you don't have to change the package loadout, just changing a single character in the description or something will do). If you think just one test result is bogus and want it re-run, currently, you'll have to ask someone with the necessary power - either me or Jan Sedlak (garretraziel on IRC). I'm in North America and he's in Europe, so we should have most timezones covered between us. We're hoping to set up a better mechanism for this in future.
Note, if you're interested in the results for the nightly Fedora distribution composes, an email summary of the results for those is sent each time they're run to the Fedora test@ and devel@ lists, look for mails with "compose check report" in the subject. Any time any of the cockpit tests fails, the failure will be listed in the mail (passed tests are not specifically listed, just a count of them). I usually keep an eye on those results and analyze failures and file bugs, though.
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
On Tue, 2017-05-02 at 11:57 +0300, Marius Vollmer wrote:
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
For now no, you have to keep an eye on the 'Automated Tests' tab in Bodhi; I'd recommend taking a look at that for every update anyway, it's good to know the Taskotron results also.
I'm planning to look at whether we can hook up some FMN options for test results, but I haven't had time to start on that yet.
On 02.05.2017 17:47, Adam Williamson wrote:
On Tue, 2017-05-02 at 11:57 +0300, Marius Vollmer wrote:
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
For now no, you have to keep an eye on the 'Automated Tests' tab in Bodhi; I'd recommend taking a look at that for every update anyway, it's good to know the Taskotron results also.
I'm planning to look at whether we can hook up some FMN options for test results, but I haven't had time to start on that yet.
How can I affect the results I see? I see dist.rpmgrill and dist.rpmgrill.desktop-lint failures. Those failures are false-positives. Is there a rpmgrill configuration file I can place in our dist-git directory that will affect those results and remove the false positives?
Cheers,
Stef
On Wed, 2017-05-03 at 14:51 +0200, Stef Walter wrote:
On 02.05.2017 17:47, Adam Williamson wrote:
On Tue, 2017-05-02 at 11:57 +0300, Marius Vollmer wrote:
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
For now no, you have to keep an eye on the 'Automated Tests' tab in Bodhi; I'd recommend taking a look at that for every update anyway, it's good to know the Taskotron results also.
I'm planning to look at whether we can hook up some FMN options for test results, but I haven't had time to start on that yet.
How can I affect the results I see? I see dist.rpmgrill and dist.rpmgrill.desktop-lint failures. Those failures are false-positives. Is there a rpmgrill configuration file I can place in our dist-git directory that will affect those results and remove the false positives?
Those are actually the Taskotron tests - the tests called 'update.*' that are shown under the update ID are the openQA tests, the tests called 'dist.*' and shown under the package NVRs are the Taskotron tests. I'm not as familiar with those.
https://bitbucket.org/fedoraqa/task-rpmgrill is the code for that task; from the git logs it looks like kparal is the one who maintains it. Kamil, is there a way packagers can provide a local config as Stef asks?
On Wed, 2017-05-03 at 14:51 +0200, Stef Walter wrote:
On 02.05.2017 17:47, Adam Williamson wrote:
On Tue, 2017-05-02 at 11:57 +0300, Marius Vollmer wrote:
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
For now no, you have to keep an eye on the 'Automated Tests' tab in Bodhi; I'd recommend taking a look at that for every update anyway, it's good to know the Taskotron results also.
I'm planning to look at whether we can hook up some FMN options for test results, but I haven't had time to start on that yet.
How can I affect the results I see? I see dist.rpmgrill and dist.rpmgrill.desktop-lint failures. Those failures are false-positives. Is there a rpmgrill configuration file I can place in our dist-git directory that will affect those results and remove the false positives?
Kamil sends along a couple of notes:
1. You can set up notifications in FMN for each taskotron result on your package which is FAILED/NEEDS_INSPECTION (for example, or just for results for tasks starting with "dist.rpmgrill", etc). See some more in https://mkrizek.wordpress.com/2016/02/24/taskotron-results-notifications/
2. We don't support config files for rpmlint, rpmgrill and other generic tasks at the moment, unfortunately. But we know it needs to get done and is definitely in our plans.
To his point 1, I'll look at the implementation for that stuff in FMN and see how easy it'll be to extend it to the openQA results also.
On 05.05.2017 18:56, Adam Williamson wrote:
On Wed, 2017-05-03 at 14:51 +0200, Stef Walter wrote:
On 02.05.2017 17:47, Adam Williamson wrote:
On Tue, 2017-05-02 at 11:57 +0300, Marius Vollmer wrote:
Adam Williamson adamwill@fedoraproject.org writes:
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
Nice!
What happens when a test fails? Do we get a notification somehow?
For now no, you have to keep an eye on the 'Automated Tests' tab in Bodhi; I'd recommend taking a look at that for every update anyway, it's good to know the Taskotron results also.
I'm planning to look at whether we can hook up some FMN options for test results, but I haven't had time to start on that yet.
How can I affect the results I see? I see dist.rpmgrill and dist.rpmgrill.desktop-lint failures. Those failures are false-positives. Is there a rpmgrill configuration file I can place in our dist-git directory that will affect those results and remove the false positives?
Kamil sends along a couple of notes:
- You can set up notifications in FMN for each taskotron result on your
package which is FAILED/NEEDS_INSPECTION (for example, or just for results for tasks starting with "dist.rpmgrill", etc). See some more in https://mkrizek.wordpress.com/2016/02/24/taskotron-results-notifications/
- We don't support config files for rpmlint, rpmgrill and other generic
tasks at the moment, unfortunately. But we know it needs to get done and is definitely in our plans.
Cool. Looking forward to that. We'll start to pay attention to those tests once we as developers/packagers have a chance of making them green.
Cheers,
Stef
cockpit-devel@lists.fedorahosted.org
-
Adam Williamson -
Marius Vollmer -
Stef Walter