Re: Proposal for integration tests infrastructure

[Nick Coghlan]

On 10/22/2014 09:43 PM, Honza Horak wrote:
> Fedora lacks integration testing (unit testing done during build is not
> enough). Taskotron will be able to fill some gaps in the future, so
> maintainers will be able to set-up various tasks after their component
> is built. But even before this works we can benefit from having the
> tests already available (and run them manually if needed).
>
> Hereby, I'd like to get ideas and figure out answers for how and where
> to keep the tests. A similar discussion already took place before, which
> I'd like to continue in:
> https://lists.fedoraproject.org/pipermail/devel/2014-January/193498.html
>
> And some short discussion already took place here as well:
> https://lists.fedoraproject.org/pipermail/env-and-stacks/2014-October/000570...

It's worth clarifying your scope here, as "integration tests" means
different things to different people, and the complexity varies wildly
depending on *what* you're trying to test.

If you're just looking at tests of individual packages beyond what folks
have specified in their RPM %check macro, then this is exactly the case
that Taskotron is designed to cover.

If you're looking at more complex cases like multihost testing, bare
metal testing across multiple architectures, or installer integration
testing, then that's what Beaker was built to handle (and has already
been handling for RHEL for several years).

That level is where you start to cross the line into true system level
acceptance tests and you often *want* those maintained independently of
the individual components in order to catch regressions in behaviour
other services are relying on.
> Some high level requirements:
> tests will be written by maintainers or broader community, not a
>
> dedicated team
> tests will be easy to run on anybody's computer (but might be
>
> potentially destructive; some secure environment will not be part of tests)
> tests will be run automatically after related components get built
>
> (probably by Taskotron)
>
> Where to keep tests?
> a/ in current dist-git for related components (problem with sharing
> parts of code, problem where to keep tests related for more components)
> b/ in separate git with similar functionality as dist-git (needs new
> infrastructure, components are not directly connected with tests, won't
> make mess in current dist-git)
> c/ in current dist-git but as ordinary components (no new infrastructure
> needed but components are not directly connected with tests)

Note that any or all of the above may be appropriate, depending on the
exact nature of the specific tests.

For example, there are already some public Beaker installer tests at
https://bitbucket.org/fedoraqa/fedora-beaker-tests for execution on
http://beaker.fedoraproject.org/
> How to deliver tests?
> a/ just use them directly from git (we need to keep some metadata for
> dependencies anyway)
> b/ package them as RPMs (we can keep metadata there; e.g. Taskotron will
> run only tests that have "Provides: ci-tests(mariadb)" after mariadb is
> built; we also might automate packaging tests to RPMs)

Our experience with Beaker suggests that you want to support both -
running directly from Git tends to be better for test development, while
using RPMs tends to be better for dependency management and sharing test
infrastructure code.
> Which framework to use?
> People have no time to learn new things, so we should let them to write
> the tests in any language and just define some conventions how to run them.

Taskotron already covers this pretty well (even if invoking Beaker
tests, it would make more sense to do that via Taskotron rather than
directly).

Regards,
Nick.

Proposal for integration tests infrastructure

[Honza Horak]

On 10/28/2014 08:08 AM, Nick Coghlan wrote:
> Note that any or all of the above may be appropriate, depending on the
> exact nature of the specific tests.
>
> For example, there are already some public Beaker installer tests at
> https://bitbucket.org/fedoraqa/fedora-beaker-tests  for execution on
> http://beaker.fedoraproject.org/

I didn't know about this instance, interesting.

Can anybody shortly explain the plan with this instance, if there is 
any? It would make sense to me to use Taskotron to run Beaker tests at 
http://beaker.fedoraproject.org, but maybe there are some other plans?

Honza

Proposal for integration tests infrastructure

[Tim Flink]

On Mon, 03 Nov 2014 17:22:27 +0100
Honza Horak  wrote:
> On 10/28/2014 08:08 AM, Nick Coghlan wrote:
> > Note that any or all of the above may be appropriate, depending on
> > the exact nature of the specific tests.
> >
> > For example, there are already some public Beaker installer tests at
> > https://bitbucket.org/fedoraqa/fedora-beaker-tests  for execution on
> > http://beaker.fedoraproject.org/
>
> I didn't know about this instance, interesting.

It's currently a dev instance (read: not backed up, best-effort level
of support) mostly due to some issues in the instance. The beaker devs
are aware of the issue and hopefully there will be a fix before too
long.

Once that fix is released, we still need to make a full production
system and that'll take some doing - not impossible but not really
trivial. We're trying to get that all figured out but as a short
summary: a working, production beaker instance for fedora will take a
bit of time once beaker is fully compatible with how things are set up
in infra. How long depends on overall priority and if a couple details
work out the way we hope.

I'm happy to go into more details but will skip them for the sake of
brevity on a large distribution cross-posted thread.
> Can anybody shortly explain the plan with this instance, if there is 
> any? It would make sense to me to use Taskotron to run Beaker tests
> at http://beaker.fedoraproject.org, but maybe there are some other
> plans?

That's pretty much what we have in mind, yeah.

Tim