Integration per-component tests in Fedora

Honza Horak hhorak at redhat.com
Mon Oct 20 21:16:08 UTC 2014


Hi guys,

I've been thinking about how to implement integration testing in Fedora, 
that would allow to execute per-package tests. This basically means to 
solve the following questions:

1) Where to keep such tests:
I don't see any reason why not include them in Fedora dist-git, but 
there are at least two ways to do so:

1a) Include the tests in git for component, say in 'tests/' directory. 
With this approach it would not be easy to use some fragments from other 
components (some general tests valid for set of components) -- well, 
dist-git could use relative paths that would lead to different tasks' 
directories or we might create some meta-tests repos that would be used 
by more components, but it does not feels much like a clean solution.

1b) Another approach would be to package the test suites as separate 
components using standard packaging process called by some convention 
and which would be installed as RPMs. With this approach we'd be able to 
prepare real integration tests related for more components, we would be 
able to use RPM's dependencies to require depended tests, etc. It could 
only be too big overhead for tests that has only two lines and just 
check if a daemon can be started (which IMHO is a valid test worth 
doing). But still, this would be my preferred option.

2) Which testing framework to use:
We might allow anything executable that will produce output in 
standardized format, but if we agree on a shell library called beakerlib 
[1], we could profit from having one platform for Fedora and RH internal 
tests. But still, that could be a recommendation, but a must.

3) The most tricky part -- how to integrate the tests into the packaging 
process:
First, I thought we needed to wait until Taskotron implements a feature 
to be able to run some tasks only for some components. But now I think 
we can implement the "per-package" part in some general task, that will 
be run for all components, just the decision which tests to fetch/run 
would be done in the task, not in Taskotron.

4) How to run destructive tests:
Even a test that just starts a service can be destructive if a service 
is broken, so we need to have some level of defence. Taskotron plans to 
support destructive tests in the future, but we might be able to do so 
in the task itself again. Not sure which technology would be the best, 
there are several options from VM in VM, to docker or nspawn solutions, 
but not sure if some of the solutions would be fine enough.

Anyway, this is how it might work in practice (using approach 2b):
* maintainer writes few lines of code that just install, start and stop 
mariadb daemon using beakerlib
* this script and small config file that specifies how to run the script 
is packaged as ci-mariadb RPM (using standard Fedora process)
* Taskotron has a task that is run after every build and which does the 
following:
   - checks repo if there is some ci-mariadb package
   - installs such a package
   - reads the config file to see how to run the tests
   - runs the tests in some safe environment (might be handled by task 
itself of taskotron)
   - stores the results
* maintainer is happy to catch mistakes soon
* user is happy to not get broken stuff

[1] https://fedorahosted.org/beakerlib/

Oh, I'm a bit too chatty today, sorry for that... :)

Any ideas welcome!
Honza


More information about the env-and-stacks mailing list