Announcing the Fedora MinGW Testsuite

[Erik van Pienbroek]

Hi all,

Lately we've had regressions in our toolchain which were discovered
quite late (too late in my opinion). Luckily these regressions are
solved now, but this got me thinking: Why don't we have a testsuite yet
which is able to catch these type of regressions? Especially now that
Fedora's AutoQA [1] is also gaining more and more attention these days.

So I decided to write a small tool in C using GLib's GTester framework
to create a testsuite which can be used to automate testing of the
various components in the Fedora MinGW toolchain.

This Fedora MinGW Testsuite tool can execute testcases of the following
types:
* Parse RPM code and verify the output (using rpmlib)
* Compile a .c / .cpp file containing source code and (optionally)
verify the output using wine
* Custom script where the exit code and output can be verified

Testcases can be easily created by writing a small .ini-style file which
describes what needs to be tested, how it needs to be tested and what
the expected output is.

The full details about this file format and all other documentation can
be found at [2].

At the moment the testsuite tool supports the MinGW toolchain as
currently available in Fedora, the MinGW toolchain as used in RHEL 6 /
CentOS 6 and the mingw-w64 toolchain which is currently residing in my
testing repo. While building the testsuite tool you'd have to indicate
which toolchain needs to be used for running the testcases.

There are currently 9 different testcases implemented (of which 3 are
mingw-w64 specific) but I hope to get this number up in the coming days.

This is also where you can help! As new testcases can be added really
easy (no programming experience required) and all areas of the toolchain
can be tested all Fedora MinGW package maintainers should be able to
come up with testcases for the packages they maintain.

Right now all code resides on a Subversion server of mine, but if
there's enough interest in continuing developing it I'm willing to
migrate it to Fedora Hosted so it can be more easily maintained.

For now you can send testcases to me (either direct or through this
mailing list) and I'll add them to the Subversion repository. Ideas for
testcases are also welcome.

Once the testsuite contains more testcases we can also integrate it in
AutoQA so the testsuite will automatically be executed when a Fedora
MinGW package maintainer tries to push an update so we can spot
regressions before the package hits the Fedora trees.

Kind regards,

Erik van Pienbroek


[1]: https://fedoraproject.org/wiki/AutoQA
[2]: https://fedoraproject.org/wiki/MinGW/Testsuite