Automated openQA testing of critpath and FreeIPA updates

[Adam Williamson]

Hi folks! You may have seen my recent post to test-announce and devel-
announce about automated update testing using openQA. If you didn't,
here's the executive summary:

* Since a few days ago, every time a critical path update is submitted
or edited, a subset of the openQA tests is run on it

* The results of the tests are submitted to ResultsDB

* Bodhi will show these results in the Automated Tests tab

As of today, we've now extended this system to also run the tests on
updates containing certain *non*-critical path packages as well.
Specifically - since I've been planning this all along, and also
because sgallagh asked for it - updates containing FreeIPA-adjacent
packages (also postgresql) will have the 'server' update tests (which
includes the FreeIPA tests) run on them. Here's an example of how this
looks in openQA:

https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=26...

And how it looks in Bodhi:

https://bodhi.fedoraproject.org/updates/FEDORA-2017-15e2a038b2 (click
on the 'Automated Tests' tab, it doesn't seem like Bodhi is capable of
forming a hyperlink to it...)

The key tests for FreeIPA are role_deploy_domain_controller (the test
that deploys a FreeIPA server and runs some basic checks on it),
realmd_join_sssd (enrols into the domain using realmd and runs some
checks that things seem to be working correctly - does a 'getent' for a
domain user, logs in as a domain user, that sort of thing) and
realmd_join_cockpit (enrols to the domain using Cockpit and does the
same tests as realmd_join_sssd; also tests the FreeIPA webUI and tests
changing a user's password both from the console and the webUI). If you
see any of those failing, there might be a problem with the update.

The package whitelist at present looks like this:

UPDATEWL = {
    # FreeIPA-related bits
    '389-ds': ('server',),
    '389-ds-base': ('server',),
    'bind': ('server',),
    'bind-dyndb-ldap': ('server',),
    'certmonger': ('server',),
    'ding-libs': ('server',),
    'freeipa': ('server',),
    'krb5-server': ('server',),
    'pki-core': ('server',),
    'sssd': ('server',),
    'tomcat': ('server',),
    # Cockpit-related bits
    'cockpit': ('server',),
    # PostgreSQL is a release-blocking server role
    'postgresql': ('server',),
}

It can be adjusted, but we can't just throw everything and the kitchen
sink on there, because openQA only has so much capacity (we can run 14
tests concurrently at any given time, and that's for both compose and
update testing), so we can't just go all the way down the dependency
stack from freeipa to kernel. Still, if you think something non-
critpath that's about as crucial to FreeIPA functioning as those
packages are is missing from the list, do let me know.

Hopefully this will help us make sure we don't break FreeIPA with
updates. Please do keep an eye on the results, if you're a packager or
a tester!

If you need help interpreting any of the results, or would like a test
re-run (they *do* occasionally fail for bogus reasons, though it's
fairly rare for the server tests), please just poke me or Jan Sedlak
(garretraziel on IRC).

These tests are run for relevant updates on all release branches, both
stable and Branched (so right now, 24, 25 and 26).

I hope folks find this useful!