Integration per-component tests in Fedora
hhorak at redhat.com
Mon Oct 20 21:16:08 UTC 2014
I've been thinking about how to implement integration testing in Fedora,
that would allow to execute per-package tests. This basically means to
solve the following questions:
1) Where to keep such tests:
I don't see any reason why not include them in Fedora dist-git, but
there are at least two ways to do so:
1a) Include the tests in git for component, say in 'tests/' directory.
With this approach it would not be easy to use some fragments from other
components (some general tests valid for set of components) -- well,
dist-git could use relative paths that would lead to different tasks'
directories or we might create some meta-tests repos that would be used
by more components, but it does not feels much like a clean solution.
1b) Another approach would be to package the test suites as separate
components using standard packaging process called by some convention
and which would be installed as RPMs. With this approach we'd be able to
prepare real integration tests related for more components, we would be
able to use RPM's dependencies to require depended tests, etc. It could
only be too big overhead for tests that has only two lines and just
check if a daemon can be started (which IMHO is a valid test worth
doing). But still, this would be my preferred option.
2) Which testing framework to use:
We might allow anything executable that will produce output in
standardized format, but if we agree on a shell library called beakerlib
, we could profit from having one platform for Fedora and RH internal
tests. But still, that could be a recommendation, but a must.
3) The most tricky part -- how to integrate the tests into the packaging
First, I thought we needed to wait until Taskotron implements a feature
to be able to run some tasks only for some components. But now I think
we can implement the "per-package" part in some general task, that will
be run for all components, just the decision which tests to fetch/run
would be done in the task, not in Taskotron.
4) How to run destructive tests:
Even a test that just starts a service can be destructive if a service
is broken, so we need to have some level of defence. Taskotron plans to
support destructive tests in the future, but we might be able to do so
in the task itself again. Not sure which technology would be the best,
there are several options from VM in VM, to docker or nspawn solutions,
but not sure if some of the solutions would be fine enough.
Anyway, this is how it might work in practice (using approach 2b):
* maintainer writes few lines of code that just install, start and stop
mariadb daemon using beakerlib
* this script and small config file that specifies how to run the script
is packaged as ci-mariadb RPM (using standard Fedora process)
* Taskotron has a task that is run after every build and which does the
- checks repo if there is some ci-mariadb package
- installs such a package
- reads the config file to see how to run the tests
- runs the tests in some safe environment (might be handled by task
itself of taskotron)
- stores the results
* maintainer is happy to catch mistakes soon
* user is happy to not get broken stuff
Oh, I'm a bit too chatty today, sorry for that... :)
Any ideas welcome!
More information about the env-and-stacks