Proposal for integration tests infrastructure

Honza Horak hhorak at redhat.com
Wed Oct 22 11:43:57 UTC 2014


Fedora lacks integration testing (unit testing done during build is not 
enough). Taskotron will be able to fill some gaps in the future, so 
maintainers will be able to set-up various tasks after their component 
is built. But even before this works we can benefit from having the 
tests already available (and run them manually if needed).

Hereby, I'd like to get ideas and figure out answers for how and where 
to keep the tests. A similar discussion already took place before, which 
I'd like to continue in:
https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html

And some short discussion already took place here as well:
https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570.html

Some high level requirements:
* tests will be written by maintainers or broader community, not a 
dedicated team
* tests will be easy to run on anybody's computer (but might be 
potentially destructive; some secure environment will not be part of tests)
* tests will be run automatically after related components get built 
(probably by Taskotron)

Where to keep tests?
a/ in current dist-git for related components (problem with sharing 
parts of code, problem where to keep tests related for more components)
b/ in separate git with similar functionality as dist-git (needs new 
infrastructure, components are not directly connected with tests, won't 
make mess in current dist-git)
c/ in current dist-git but as ordinary components (no new infrastructure 
needed but components are not directly connected with tests)

How to deliver tests?
a/ just use them directly from git (we need to keep some metadata for 
dependencies anyway)
b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will 
run only tests that have "Provides: ci-tests(mariadb)" after mariadb is 
built; we also might automate packaging tests to RPMs)

Structure for tests?
a/ similar to what components use (branches for Fedora versions)
b/ only one branch
Test maintainers should be allowed to behave the same as package 
maintainers do -- one likes keeping branches the same and uses "%if 
%fedora" macros, someone else likes specs clean and rather maintain more 
different branches) -- we won't find one structure that would fit all, 
so allowing both ways seems better.

Which framework to use?
People have no time to learn new things, so we should let them to write 
the tests in any language and just define some conventions how to run them.

Cheers,
Honza


More information about the devel mailing list