Integration per-component tests in Fedora

[John Dulaney]

> Date: Mon, 20 Oct 2014 23:16:08 +0200
> From: hhorak at redhat.com
> To: env-and-stacks at lists.fedoraproject.org
> Subject: Integration per-component tests in Fedora
>
> Hi guys,
>
> I've been thinking about how to implement integration testing in Fedora,
> that would allow to execute per-package tests. This basically means to
> solve the following questions:
>
> 1) Where to keep such tests:
> I don't see any reason why not include them in Fedora dist-git, but
> there are at least two ways to do so:
>
> 1a) Include the tests in git for component, say in 'tests/' directory.
> With this approach it would not be easy to use some fragments from other
> components (some general tests valid for set of components) -- well,
> dist-git could use relative paths that would lead to different tasks'
> directories or we might create some meta-tests repos that would be used
> by more components, but it does not feels much like a clean solution.
>
> 1b) Another approach would be to package the test suites as separate
> components using standard packaging process called by some convention
> and which would be installed as RPMs. With this approach we'd be able to
> prepare real integration tests related for more components, we would be
> able to use RPM's dependencies to require depended tests, etc. It could
> only be too big overhead for tests that has only two lines and just
> check if a daemon can be started (which IMHO is a valid test worth
> doing). But still, this would be my preferred option.
>
> 2) Which testing framework to use:
> We might allow anything executable that will produce output in
> standardized format, but if we agree on a shell library called beakerlib
> [1], we could profit from having one platform for Fedora and RH internal
> tests. But still, that could be a recommendation, but a must.
>
> 3) The most tricky part -- how to integrate the tests into the packaging
> process:
> First, I thought we needed to wait until Taskotron implements a feature
> to be able to run some tasks only for some components. But now I think
> we can implement the "per-package" part in some general task, that will
> be run for all components, just the decision which tests to fetch/run
> would be done in the task, not in Taskotron.
>
> 4) How to run destructive tests:
> Even a test that just starts a service can be destructive if a service
> is broken, so we need to have some level of defence. Taskotron plans to
> support destructive tests in the future, but we might be able to do so
> in the task itself again. Not sure which technology would be the best,
> there are several options from VM in VM, to docker or nspawn solutions,
> but not sure if some of the solutions would be fine enough.
>
> Anyway, this is how it might work in practice (using approach 2b):
> * maintainer writes few lines of code that just install, start and stop
> mariadb daemon using beakerlib
> * this script and small config file that specifies how to run the script
> is packaged as ci-mariadb RPM (using standard Fedora process)
> * Taskotron has a task that is run after every build and which does the
> following:
> - checks repo if there is some ci-mariadb package
> - installs such a package
> - reads the config file to see how to run the tests
> - runs the tests in some safe environment (might be handled by task
> itself of taskotron)
> - stores the results
> * maintainer is happy to catch mistakes soon
> * user is happy to not get broken stuff
>
> [1] https://fedorahosted.org/beakerlib/
>
> Oh, I'm a bit too chatty today, sorry for that... :)
>
> Any ideas welcome!
> Honza

So, a lot of what you want to accomplish is solved by Taskotron.  Rather than
work on a parallel setup, why not help out with Taskotron?  The next step
is with disposable test clients, so destructive testing can be done.

As you noted, Taskotron also handles running and scheduling of tests.  It also
provides a framework for reporting and storing results.

John.