New subject: AutoQA will be providing comments for Fedora updates really soon...

21 Mar 2011


      If you're a Fedora package maintainer, you will be surely interested in this topic.
AutoQA [1] is a framework for automatic test execution written for Fedora needs. It is developed by Fedora QA [2] team. We have finally reached the state where we can experimentally share some tests results with the wide public. They were already available in the autoqa-results [3] mailing list, but that's not really what we mean by public consumption. Let's improve that.
Now prepare the drums... ready?... we are going to post the results into Fedora Update System [4] (a.k.a. Bodhi)!
After you propose an update in Bodhi (either to -updates or to -updates-testing), you will start receiving AutoQA results as Bodhi comments.
Example comment: http://kparal.files.wordpress.com/2011/03/bodhi-comment.png
A few very important notes:
* The test results are purely *informative*. We won't block/reject your updates if they fail some test. But we hope that you (or RelEng team members) will have a look at the issue and it will help you/them to decide whether it is ok to push this update or not.
* The two tests which will be executed (described below) are designed to check for issues that don't offer too much speculation. We don't want new updates to break the dependencies, ever. We want new updates to keep upgradepath constraint, always. If the result is _failed_, there is either a problem in the update or a problem in our test. If you believe the latter is the case, please, report the problem to us - AutoQA [1], autoqa-devel [5]. Those tests are very complex and there is at least one bug in every program. We know that it will be a bumpy road until we have them perfect.
* It's very easy to disable Bodhi comment posting if something goes really wrong. We will probably do that if we discover some serious issue and re-enable it again after we fix it. So don't be afraid, we won't flood your inbox (the doomsday is not until 2012, you know), and also don't be surprised if the comments don't arrive sometimes.
There are two tests whose results will be posted to Bodhi:
* depcheck [6] - This test checks the dependencies (provides/requires/conflicts/obsoletes flags) of the proposed packages coming into -updates or -updates-testing. It tries to decide whether your update would break the repository (i.e. some packages would have unresolved dependencies in that case). There's a lot of very complex logic behind it.
* upgradepath [7] - This test checks whether the _upgradepath_ requirement is fulfilled. In an example with Fedora 13 and package foo-1.1-1.fc13 it means that every higher Fedora release (14, 15, 16) must contain same or higher NVR of the foo package than is the currently proposed one (and vice versa for lower Fedora releases). This constraint is currently checked only for -updates repository requests.
Next to every result you'll see an URL pointing to the full results log. You'll want to inspect that if your package was marked as failing some test. Example URL:
http://autoqa.fedoraproject.org/results/70273-autotest/qa03.c.fedoraproject....
The results directory is currently a little bit chaotic, but the important files are:
* <testname>/results/output.log where you'll see a shortened output of the test with important highlights as the first lines
* debug/client.DEBUG contains full debug log as created by the framework
In those files you should be able to find the reason why your package failed the test (search for your NVR).
The tests are executed fairly often (almost for every update request), which means that after you correct the problem (if there was any), you should quickly receive another comment saying that your package now passes the test. If your package fails again however, you won't receive another comment - we don't want to spam you with Bodhi notification emails too often. Currently we wait at least three days before inserting two subsequent _fail_ comments.
Feedback/comments welcome.
[1] https://fedoraproject.org/wiki/AutoQA
[2] http://fedoraproject.org/wiki/QA
[3] https://fedorahosted.org/pipermail/autoqa-results/2011-March/thread.html
[4] https://admin.fedoraproject.org/updates/
[5] https://fedorahosted.org/mailman/listinfo/autoqa-devel
[6] https://fedoraproject.org/wiki/QA:Depcheck_Test_Case
[7] https://fedoraproject.org/wiki/QA:Upgrade_Path_Test_Case