Proposal for integration tests infrastructure

[John Dulaney]

Some thoughts:

>> Where to keep tests? a/ in current dist-git for related components
>> (problem with sharing parts of code, problem where to keep tests
>> related for more components) b/ in separate git with similar
>> functionality as dist-git (needs new infrastructure, components are
>> not directly connected with tests, won't make mess in current
>> dist-git) c/ in current dist-git but as ordinary components (no new
>> infrastructure needed but components are not directly connected
>> with tests)
>>
>> How to deliver tests? a/ just use them directly from git (we need
>> to keep some metadata for dependencies anyway) b/ package them as
>> RPMs (we can keep metadata there; e.g. Taskotron will run only
>> tests that have "Provides: ci-tests(mariadb)" after mariadb is
>> built; we also might automate packaging tests to RPMs)

To answer both of these, the plan is to keep taskotron tasks in their own
git repo; currently this is at (0).

To run the tasks, taskotron sets up a disposable task client and then git
clones the task to be run.  This solves the issue of delivery and allows
a continuous integration-like solution.

>> Structure for tests? a/ similar to what components use (branches
>> for Fedora versions) b/ only one branch Test maintainers should be
>> allowed to behave the same as package maintainers do -- one likes
>> keeping branches the same and uses "%if %fedora" macros, someone
>> else likes specs clean and rather maintain more different branches)
>> -- we won't find one structure that would fit all, so allowing both
>> ways seems better.

This is something we'll need to figure out, but, I suspect git branches will
be involved.

>> Which framework to use? People have no time to learn new things, so
>> we should let them to write the tests in any language and just
>> define some conventions how to run them.

You'll need to at least define the task in a yaml file, and output will need to
be TAP.  The example task is at (1).


> TAP (Test Anything Protocol) FTW. It really makes sense when you're
> trying to combine tests from multiple different languages, testing
> frameworks, etc.
>
> Stef
>

Indeed, which is why we settled on it.



John.



(0)  https://bitbucket.org/fedoraqa

(1)  https://bitbucket.org/fedoraqa/task-rpmlint