Proposal for integration tests infrastructure
hhorak at redhat.com
Wed Oct 22 11:43:57 UTC 2014
Fedora lacks integration testing (unit testing done during build is not
enough). Taskotron will be able to fill some gaps in the future, so
maintainers will be able to set-up various tasks after their component
is built. But even before this works we can benefit from having the
tests already available (and run them manually if needed).
Hereby, I'd like to get ideas and figure out answers for how and where
to keep the tests. A similar discussion already took place before, which
I'd like to continue in:
And some short discussion already took place here as well:
Some high level requirements:
* tests will be written by maintainers or broader community, not a
* tests will be easy to run on anybody's computer (but might be
potentially destructive; some secure environment will not be part of tests)
* tests will be run automatically after related components get built
(probably by Taskotron)
Where to keep tests?
a/ in current dist-git for related components (problem with sharing
parts of code, problem where to keep tests related for more components)
b/ in separate git with similar functionality as dist-git (needs new
infrastructure, components are not directly connected with tests, won't
make mess in current dist-git)
c/ in current dist-git but as ordinary components (no new infrastructure
needed but components are not directly connected with tests)
How to deliver tests?
a/ just use them directly from git (we need to keep some metadata for
b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will
run only tests that have "Provides: ci-tests(mariadb)" after mariadb is
built; we also might automate packaging tests to RPMs)
Structure for tests?
a/ similar to what components use (branches for Fedora versions)
b/ only one branch
Test maintainers should be allowed to behave the same as package
maintainers do -- one likes keeping branches the same and uses "%if
%fedora" macros, someone else likes specs clean and rather maintain more
different branches) -- we won't find one structure that would fit all,
so allowing both ways seems better.
Which framework to use?
People have no time to learn new things, so we should let them to write
the tests in any language and just define some conventions how to run them.
More information about the env-and-stacks