Proposal for integration tests infrastructure

[Stef Walter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22.10.2014 13:43, Honza Horak wrote:
> Fedora lacks integration testing (unit testing done during build
> is not enough). Taskotron will be able to fill some gaps in the 
> future, so maintainers will be able to set-up various tasks after 
> their component is built. But even before this works we can
> benefit from having the tests already available (and run them
> manually if needed).
> 
> Hereby, I'd like to get ideas and figure out answers for how and 
> where to keep the tests. A similar discussion already took place 
> before, which I'd like to continue in: 
> https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
>
>
> 
And some short discussion already took place here as well:
> https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570.html
>
>
>
>
> 
Some high level requirements: * tests will be written by
> maintainers or broader community, not a dedicated team * tests
> will be easy to run on anybody's computer (but might be
> potentially destructive; some secure environment will not be part
> of tests) * tests will be run automatically after related
> components get built (probably by Taskotron)
> 
> Where to keep tests? a/ in current dist-git for related components 
> (problem with sharing parts of code, problem where to keep tests 
> related for more components) b/ in separate git with similar 
> functionality as dist-git (needs new infrastructure, components
> are not directly connected with tests, won't make mess in current 
> dist-git) c/ in current dist-git but as ordinary components (no
> new infrastructure needed but components are not directly
> connected with tests)
> 
> How to deliver tests? a/ just use them directly from git (we need 
> to keep some metadata for dependencies anyway) b/ package them as 
> RPMs (we can keep metadata there; e.g. Taskotron will run only 
> tests that have "Provides: ci-tests(mariadb)" after mariadb is 
> built; we also might automate packaging tests to RPMs)
> 
> Structure for tests? a/ similar to what components use (branches 
> for Fedora versions) b/ only one branch Test maintainers should be 
> allowed to behave the same as package maintainers do -- one likes 
> keeping branches the same and uses "%if %fedora" macros, someone 
> else likes specs clean and rather maintain more different
> branches) -- we won't find one structure that would fit all, so
> allowing both ways seems better.
> 
> Which framework to use? People have no time to learn new things,
> so we should let them to write the tests in any language and just 
> define some conventions how to run them.

TAP (Test Anything Protocol) FTW. It really makes sense when you're
trying to combine tests from multiple different languages, testing
frameworks, etc.

Stef

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRJWOMACgkQe/sRCNknZa/ltQCfcTBBPOIl3fISqjm0j3YUw+TU
eSAAoIMo+NSOg/iWf27VQuq0J2rTebT/
=L9uQ
-----END PGP SIGNATURE-----