Proposal for integration tests infrastructure

[Stef Walter]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 22.10.2014 13:43, Honza Horak wrote:
> Fedora lacks integration testing (unit testing done during build is
> not enough). Taskotron will be able to fill some gaps in the
> future, so maintainers will be able to set-up various tasks after
> their component is built. But even before this works we can benefit
> from having the tests already available (and run them manually if
> needed).
> 
> Hereby, I'd like to get ideas and figure out answers for how and
> where to keep the tests. A similar discussion already took place
> before, which I'd like to continue in: 
> https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
>
>  And some short discussion already took place here as well: 
> https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570.html
>
> 
> 
> Some high level requirements: * tests will be written by
> maintainers or broader community, not a dedicated team * tests will
> be easy to run on anybody's computer (but might be potentially
> destructive; some secure environment will not be part of tests) *
> tests will be run automatically after related components get built 
> (probably by Taskotron)
> 
> Where to keep tests? a/ in current dist-git for related components
> (problem with sharing parts of code, problem where to keep tests
> related for more components) b/ in separate git with similar
> functionality as dist-git (needs new infrastructure, components are
> not directly connected with tests, won't make mess in current
> dist-git) c/ in current dist-git but as ordinary components (no new
> infrastructure needed but components are not directly connected
> with tests)
> 
> How to deliver tests? a/ just use them directly from git (we need
> to keep some metadata for dependencies anyway) b/ package them as
> RPMs (we can keep metadata there; e.g. Taskotron will run only
> tests that have "Provides: ci-tests(mariadb)" after mariadb is 
> built; we also might automate packaging tests to RPMs)
> 
> Structure for tests? a/ similar to what components use (branches
> for Fedora versions) b/ only one branch Test maintainers should be
> allowed to behave the same as package maintainers do -- one likes
> keeping branches the same and uses "%if %fedora" macros, someone
> else likes specs clean and rather maintain more different branches)
> -- we won't find one structure that would fit all, so allowing both
> ways seems better.
> 
> Which framework to use? People have no time to learn new things, so
> we should let them to write the tests in any language and just
> define some conventions how to run them.

TAP (Test Anything Protocol) FTW. It really makes sense when you're
trying to combine tests from multiple different languages, testing
frameworks, etc.

Stef

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlRJVmQACgkQe/sRCNknZa+eQgCg2OPFFbml+TBKLdRptyhg/m/T
NMgAnRqgksOGa/rOuwv+ng7xwYcrWRnb
=Tkp8
-----END PGP SIGNATURE-----