Integration per-component tests in Fedora

[Honza Horak]

On 10/21/2014 12:23 AM, John Dulaney wrote:
>> Date: Mon, 20 Oct 2014 23:16:08 +0200
>> From: hhorak at redhat.com
>> To: env-and-stacks at lists.fedoraproject.org
>> Subject: Integration per-component tests in Fedora
>>
>> Hi guys,
>>
>> I've been thinking about how to implement integration testing in Fedora,
>> that would allow to execute per-package tests. This basically means to
>> solve the following questions:
>>
>> 1) Where to keep such tests:
>> I don't see any reason why not include them in Fedora dist-git, but
>> there are at least two ways to do so:
>>
>> 1a) Include the tests in git for component, say in 'tests/' directory.
>> With this approach it would not be easy to use some fragments from other
>> components (some general tests valid for set of components) -- well,
>> dist-git could use relative paths that would lead to different tasks'
>> directories or we might create some meta-tests repos that would be used
>> by more components, but it does not feels much like a clean solution.
>>
>> 1b) Another approach would be to package the test suites as separate
>> components using standard packaging process called by some convention
>> and which would be installed as RPMs. With this approach we'd be able to
>> prepare real integration tests related for more components, we would be
>> able to use RPM's dependencies to require depended tests, etc. It could
>> only be too big overhead for tests that has only two lines and just
>> check if a daemon can be started (which IMHO is a valid test worth
>> doing). But still, this would be my preferred option.
>>
>> 2) Which testing framework to use:
>> We might allow anything executable that will produce output in
>> standardized format, but if we agree on a shell library called beakerlib
>> [1], we could profit from having one platform for Fedora and RH internal
>> tests. But still, that could be a recommendation, but a must.
>>
>> 3) The most tricky part -- how to integrate the tests into the packaging
>> process:
>> First, I thought we needed to wait until Taskotron implements a feature
>> to be able to run some tasks only for some components. But now I think
>> we can implement the "per-package" part in some general task, that will
>> be run for all components, just the decision which tests to fetch/run
>> would be done in the task, not in Taskotron.
>>
>> 4) How to run destructive tests:
>> Even a test that just starts a service can be destructive if a service
>> is broken, so we need to have some level of defence. Taskotron plans to
>> support destructive tests in the future, but we might be able to do so
>> in the task itself again. Not sure which technology would be the best,
>> there are several options from VM in VM, to docker or nspawn solutions,
>> but not sure if some of the solutions would be fine enough.
>>
>> Anyway, this is how it might work in practice (using approach 2b):
>> * maintainer writes few lines of code that just install, start and stop
>> mariadb daemon using beakerlib
>> * this script and small config file that specifies how to run the script
>> is packaged as ci-mariadb RPM (using standard Fedora process)
>> * Taskotron has a task that is run after every build and which does the
>> following:
>> - checks repo if there is some ci-mariadb package
>> - installs such a package
>> - reads the config file to see how to run the tests
>> - runs the tests in some safe environment (might be handled by task
>> itself of taskotron)
>> - stores the results
>> * maintainer is happy to catch mistakes soon
>> * user is happy to not get broken stuff
>>
>> [1] https://fedorahosted.org/beakerlib/
>>
>> Oh, I'm a bit too chatty today, sorry for that... :)
>>
>> Any ideas welcome!
>> Honza
>
> So, a lot of what you want to accomplish is solved by Taskotron.  Rather than
> work on a parallel setup, why not help out with Taskotron?  The next step
> is with disposable test clients, so destructive testing can be done.

I've learned it from Kamil today, so yeah, the 4) would be much better 
to do in Taskotron, I just did not know it is the next big thing. But 
still we have the rest, which is not solved by Taskotron and maybe even 
should not be.

> As you noted, Taskotron also handles running and scheduling of tests.  It also
> provides a framework for reporting and storing results.

Right, this all should be done by Taskotron, sorry if it might be 
understood differently from my first proposal.

Honza

> John. 		 	   		
> _______________________________________________
> env-and-stacks mailing list
> env-and-stacks at lists.fedoraproject.org
> https://lists.fedoraproject.org/mailman/listinfo/env-and-stacks
>