Proposal for integration tests infrastructure
jdulaney at fedoraproject.org
Fri Oct 24 17:18:30 UTC 2014
>> Where to keep tests? a/ in current dist-git for related components
>> (problem with sharing parts of code, problem where to keep tests
>> related for more components) b/ in separate git with similar
>> functionality as dist-git (needs new infrastructure, components are
>> not directly connected with tests, won't make mess in current
>> dist-git) c/ in current dist-git but as ordinary components (no new
>> infrastructure needed but components are not directly connected
>> with tests)
>> How to deliver tests? a/ just use them directly from git (we need
>> to keep some metadata for dependencies anyway) b/ package them as
>> RPMs (we can keep metadata there; e.g. Taskotron will run only
>> tests that have "Provides: ci-tests(mariadb)" after mariadb is
>> built; we also might automate packaging tests to RPMs)
To answer both of these, the plan is to keep taskotron tasks in their own
git repo; currently this is at (0).
To run the tasks, taskotron sets up a disposable task client and then git
clones the task to be run. This solves the issue of delivery and allows
a continuous integration-like solution.
>> Structure for tests? a/ similar to what components use (branches
>> for Fedora versions) b/ only one branch Test maintainers should be
>> allowed to behave the same as package maintainers do -- one likes
>> keeping branches the same and uses "%if %fedora" macros, someone
>> else likes specs clean and rather maintain more different branches)
>> -- we won't find one structure that would fit all, so allowing both
>> ways seems better.
This is something we'll need to figure out, but, I suspect git branches will
>> Which framework to use? People have no time to learn new things, so
>> we should let them to write the tests in any language and just
>> define some conventions how to run them.
You'll need to at least define the task in a yaml file, and output will need to
be TAP. The example task is at (1).
> TAP (Test Anything Protocol) FTW. It really makes sense when you're
> trying to combine tests from multiple different languages, testing
> frameworks, etc.
Indeed, which is why we settled on it.
More information about the env-and-stacks