CTF - Containers (not only) Testing Framework

[Tomas Hozza]

Hi.

Petr Hracek asked me to share some brief introduction to CTF with the
Env&Stacks WG, since I'm one of the authors of the CTF project.

CTF [1] is a relatively simple framework built on top of Behave [1],
which is a BDD testing framework written in Python. Tests written for
Behave consist of Features and Steps (and environment).

Features
--------
You specify the Features and Scenarios you want to test as set of Steps
described in simple English. Those steps can be parametrized, so you can
pass strings or tables to the Step implementation. Steps are connected
together using keywords like "Given", "When", "Then", "And", "But", ...
. Nice thing about Features is that you can reuse the same Step in
multiple Scenarios, thus being able to write new test cases even without
the need to implement new test code, if reusing existing Steps. Also
when reading the Feature files, it is mostly obvious what the test is
testing, since you are not reading the code, but the "human-readable"
representation of the test case.

Steps
-----
Now the Steps used in Features to describe the test cases are
implemented in separate files as basic Python functions and decorated
with decorators provided by Behave. This is so that Behave can match the
Steps in Feature files to their implementation. As said before, you can
pass arguments like string, table (or even own one, if you implement
proper parser) to the step. Also a context file is passed between each
Step, so these can share some state.

Environment
-----------
There is also an environment.py file, where you can specify code that
will be run before/after each feature/scenario/step and you can even
filter the execution based on TAGs applied to the feature/scenario/step.

Now all of this still pure Behave specific. You can read more about it
on the Behave project page [1]. There are also some nice examples.


CTF
---
CTF is meant as a way to distribute and reuse existing test cases across
various projects with the aim on containers (although we want to make it
really general and not to limit ourselves only to containers).

The basic idea is that you can have multiple test cases (Features and
Steps) that are valid for multiple containers (e.g. any layered image
with database MariaDB/MongoDB/...) and you want to reuse them. Tests
(Features and Steps) are stored in remote git repositories.

CTF, when executed in the project repository, based on the
configuration, it clones all the remote test Features and Steps as git
submodules. It then creates a working directory for Behave, in which it
combines local tests (Features & Steps) with all the remote tests,
prepares environment.py and executes Behave in the working directory.

It is then possible to add/remove/update the cloned tests (git
submodules) if these were modified in the remote repo.

This enables developers to write e.g. set of tests that are common for
any Docker containers and then just execute these as part of testing a
specific container along with the specific local tests. It is also
possible to reuse already implemented remote Steps in local Features you
want to test. This means less code duplication, better tests sharing and
in the end hopefully higher test coverage.

There is pretty comprehensive description of how ctf-cli works, on the
project page on GitHub [3].

Feel free to reach out to the developers. The project is still in early
stage, but there is active development. It is also already used by some
teams to execute tests.


[1] http://pythonhosted.org/behave/
[2] https://github.com/Containers-Testing-Framework
[3] https://github.com/Containers-Testing-Framework/ctf-cli

Regards,
-- 
Tomas Hozza
Software Engineer - EMEA ENG Developer Experience

PGP: 1D9F3C2D
Red Hat Inc.                 http://cz.redhat.com