Hi folks! I thought this might be of interest to the Cockpit community, so I thought I'd write it up here in case anyone missed it elsewhere.
I work on the Fedora QA team, and we have been using the openQA automated test system (developed by our friends at SUSE) to run various functional tests on Fedora composes for the last couple of years.
As Cockpit is considered a critical part of Fedora Server, we run a few tests that exercise Cockpit. They're nowhere near as extensive as Cockpit's own test suite, but they run in a somewhat different context: they run on the Fedora packaged Cockpit, against the rest of a Fedora environment. They also test in a somewhat different way from Cockpit's own test suite: openQA tests a lot like a human, by booting up a virtual machine and then effectively "looking at the screen" and sending mouse and keyboard input events (via VNC). The test boots a full Fedora system, runs Firefox, and interacts with Cockpit like a human being would. For both of these reasons, it can sometimes catch problems Cockpit's own tests would not (it also sometimes goes wonky in ways that Selenium-based tests don't, but never mind, that's my job to deal with :>)
Until recently we ran these tests only on Fedora's nightly development release distribution composes. Recently, though, we deployed some enhancements to our openQA setup that let us run tests on Fedora distribution updates as well, and have the results made visible through the Fedora update system (Bodhi). I have just made it so that, from now on, these tests will run on any update that contains the 'cockpit' package. They also run on any critical path update.
This means that for any Fedora update containing cockpit or any critical path package, Fedora's openQA cockpit tests should run, and you should see the results in the Fedora update system (Bodhi). You can see the results in Bodhi by clicking the Automated Updates tab for any update. For instance, here's a recent Firefox update for Fedora 25:
https://bodhi.fedoraproject.org/updates/FEDORA-2017-31c64a0bbf
If you look at the Automated Tests tab, you can see passes for:
update.server_cockpit_basic update.server_cockpit_default update.realmd_join_cockpit
indicating that this Firefox update didn't cause any problems for Cockpit. Clicking on any test result will take you to the openQA page for the test, where you can diagnose failures and so on (explaining how to do this is a bit beyond the scope of this mail, please do ask me if you're interested!)
I hope this stuff will help us avoid shipping updates that break Cockpit (and other key components). If you have any questions, concerns, comments, or suggestions, please do ask!
To anticipate one question: you can cause *all* the tests for an update to be re-run by editing the update in any way (you don't have to change the package loadout, just changing a single character in the description or something will do). If you think just one test result is bogus and want it re-run, currently, you'll have to ask someone with the necessary power - either me or Jan Sedlak (garretraziel on IRC). I'm in North America and he's in Europe, so we should have most timezones covered between us. We're hoping to set up a better mechanism for this in future.
Note, if you're interested in the results for the nightly Fedora distribution composes, an email summary of the results for those is sent each time they're run to the Fedora test@ and devel@ lists, look for mails with "compose check report" in the subject. Any time any of the cockpit tests fails, the failure will be listed in the mail (passed tests are not specifically listed, just a count of them). I usually keep an eye on those results and analyze failures and file bugs, though.