Some thoughts:
Where to keep tests? a/ in current dist-git for related components (problem with sharing parts of code, problem where to keep tests related for more components) b/ in separate git with similar functionality as dist-git (needs new infrastructure, components are not directly connected with tests, won't make mess in current dist-git) c/ in current dist-git but as ordinary components (no new infrastructure needed but components are not directly connected with tests)
How to deliver tests? a/ just use them directly from git (we need to keep some metadata for dependencies anyway) b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will run only tests that have "Provides: ci-tests(mariadb)" after mariadb is built; we also might automate packaging tests to RPMs)
To answer both of these, the plan is to keep taskotron tasks in their own git repo; currently this is at (0).
To run the tasks, taskotron sets up a disposable task client and then git clones the task to be run. This solves the issue of delivery and allows a continuous integration-like solution.
Structure for tests? a/ similar to what components use (branches for Fedora versions) b/ only one branch Test maintainers should be allowed to behave the same as package maintainers do -- one likes keeping branches the same and uses "%if %fedora" macros, someone else likes specs clean and rather maintain more different branches) -- we won't find one structure that would fit all, so allowing both ways seems better.
This is something we'll need to figure out, but, I suspect git branches will be involved.
Which framework to use? People have no time to learn new things, so we should let them to write the tests in any language and just define some conventions how to run them.
You'll need to at least define the task in a yaml file, and output will need to be TAP. The example task is at (1).
TAP (Test Anything Protocol) FTW. It really makes sense when you're trying to combine tests from multiple different languages, testing frameworks, etc.
Stef
Indeed, which is why we settled on it.
John.
qa-devel@lists.fedoraproject.org
-
John Dulaney