-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 22.10.2014 13:43, Honza Horak wrote:
Fedora lacks integration testing (unit testing done during build is not enough). Taskotron will be able to fill some gaps in the future, so maintainers will be able to set-up various tasks after their component is built. But even before this works we can benefit from having the tests already available (and run them manually if needed).
Hereby, I'd like to get ideas and figure out answers for how and where to keep the tests. A similar discussion already took place before, which I'd like to continue in: https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
And some short discussion already took place here as well:
https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570...
Some high level requirements: * tests will be written by
maintainers or broader community, not a dedicated team * tests will be easy to run on anybody's computer (but might be potentially destructive; some secure environment will not be part of tests) * tests will be run automatically after related components get built (probably by Taskotron)
Where to keep tests? a/ in current dist-git for related components (problem with sharing parts of code, problem where to keep tests related for more components) b/ in separate git with similar functionality as dist-git (needs new infrastructure, components are not directly connected with tests, won't make mess in current dist-git) c/ in current dist-git but as ordinary components (no new infrastructure needed but components are not directly connected with tests)
How to deliver tests? a/ just use them directly from git (we need to keep some metadata for dependencies anyway) b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will run only tests that have "Provides: ci-tests(mariadb)" after mariadb is built; we also might automate packaging tests to RPMs)
Structure for tests? a/ similar to what components use (branches for Fedora versions) b/ only one branch Test maintainers should be allowed to behave the same as package maintainers do -- one likes keeping branches the same and uses "%if %fedora" macros, someone else likes specs clean and rather maintain more different branches) -- we won't find one structure that would fit all, so allowing both ways seems better.
Which framework to use? People have no time to learn new things, so we should let them to write the tests in any language and just define some conventions how to run them.
TAP (Test Anything Protocol) FTW. It really makes sense when you're trying to combine tests from multiple different languages, testing frameworks, etc.
Stef