9:30 a.m.
Hi!
Current Fedora CI documentation suggests: "The tests to be
executed are stored in the dist-git repositories. The tests are
stored or wrapped along-side the spec files..." [0] While working
on porting tests to Fedora CI, several times we've noted concerns
from developers/maintainers about placing test code directly into
dist-git repositories. A common question is how to efficiently
maintain tests & minimize test code duplication. There are some
nice real-life examples available:
Examples
There are several shells which implement the POSIX specification:
bash, ksh, mksh, zsh, dash. All of them share a significant amount
of test coverage and it does not make sense to commit & maintain
identical tests in five different repositories (+ possible
branches). See the pull request [1] for a bit of context.
Another example is Ruby: With about 80 packages related to Ruby on
Rails it would be useful and efficient to have a single place for
integration tests which verify that the framework is correctly
working after updating any of these packages. Conversely,
maintaining those tests in 80 repos would be a tedious task.
Proposal: Share!
So this is where the idea of shared test repository comes from.
In general, tests define how the software works and the basic
functionality of many packages doesn’t change that often. We try
hard to keep the backward compatibility where possible. Thus it
seems natural that, for such components, tests guarding the spec
could change at a slower pace than the distribution branches.
Main goals
* Package source and tests are linked in a discoverable and
unambiguous way (dist-git)
* Prevent test code duplication (minimize test maintenance)
* Catch incompatibilities early
How it could work? A new namespace "tests" would be created in
Fedora as suggested in [2], test code would reside there and the
tests in the rpms dist git could directly reference a specific
version of the tests to be run, e.g. with a pre_task:
pre_tasks:
- name: Fetch the tests
git:
repo: https://src.fedoraproject.org/tests/shell.git
dest: tests
version: master
Also, standard-test-roles [3] should be enhanced to make it really
easy to fetch tests from a remote repo. Ansible git module is
quite simple but with standard-test-roles it could be even
shorter. The tests.yml file could look like this (we could support
fetching from multiple repositories):
- role: standard-test-beakerlib
tags:
- classic
- container
- atomic
tests:
- smoke
repo:
- https://src.fedoraproject.org/tests/shell.git
version: master
A consistent location for shared tests is a good source of
examples and helps drive automation in the future, e.g. trigger CI
jobs on test changes.
For stable packages providing specification-like functionality a
single master branch could be used, for those components which
change more, dist-git tests.yml could pick a proper branch or even
a specific commit (if having concerns about updated tests breaking
the continuous integration).
This seems to give a nice flexibility while at the same time
minimizing test code duplication. Referencing specific tests from
rpms dist git ties tests and code together very well. Storing the
test code in the tests namespace is only one option, any git repo
accessible from the CI pipeline would suffice.
Just One Place?
We could possibly go one more step further: Shall we aim at having
just a single test code for testing both Fedora and Red Hat
Enterprise Linux? At least for those cases where this is possible:
There might be some licensing or compatibility issues. For
example, speaking about Beaker: restraint harness can also be
instructed to fetch tests directly from git so the identical test
code could perhaps be used for running tests in Beaker as well.
A recent experience of a libselinux pull request [4] shows how
cumbersome it can be to keep both upstream and downstream version
of the test code in sync. Wouldn't it be great to have everything
at a single place and keep the test code up-to-date only there?
Instead of doing several rounds of "pull request - give feedback -
update internal tests - update the pull request - find another
issue - give feedback..." with many unnecessary steps and
reminders just directly collaborating on a single test repository?
When downstream testing is just a branch of upstream testing, we
can use our standard open source workflows. Just as in upstream,
there would likely be a downstream shared git repository that
branches from the upstream one.
Bonus
As a bonus we would keep the dist-git for such repositories a bit
more consistent as a place for metadata only: Build metadata (spec
file = how to build the package) and test metadata (tests.yml =
how to test the package).
Plus we would get a more streamlined test maintenance as QE and
volunteers willing to help with improving the test coverage could
be granted direct access to individual package tests repo. Not
every QE is a Fedora packager so contributing to dist-git is not
that straightforward.
Notes
There are cons of this approach as well. It isn’t possible to have
new feature code and tests submitted in a single commit or
triggering test execution for every change in tests (CI for
tests). There is a need for handling relevance of individual tests
for different environments. But the maintenance simplification
seems to be worth it, based on experience of the BaseOS QE team
which has been maintaining thousands of tests for many years and
using them for testing large number of supported versions of Red
Hat Enterprise Linux.
The overall benefits also depend on which test types we are
speaking about. Tests should always be as far upstream as
possible. Unit tests generally make more sense in upstream. For
integration tests it might make more sense to live in the shared
Fedora tests repository. Recommendations about which tests fit
well into this "tests" namespace will evolve, but it will be up to
each package to choose where to place its tests.
So?
What are your thoughts on this? Does it make sense to you?
psss...
[0] https://fedoraproject.org/wiki/CI#Tests_locations
[1] https://src.fedoraproject.org/rpms/bash/pull-request/1
[2] https://pagure.io/fedora-infrastructure/issue/6478
[3] https://pagure.io/standard-test-roles/
[4] https://src.fedoraproject.org/rpms/libselinux/pull-request/1