openSUSE announces release of openqa

Adam Williamson awilliam at redhat.com
Thu Oct 13 17:35:00 UTC 2011


On Thu, 2011-10-13 at 08:35 -0400, Neal Becker wrote:
> http://news.opensuse.org/2011/10/11/opensuse-announces-first-public-release-of-
> openqa/

saw it!

in fact, we've talked to one of the lead developers before.

it's got some cool design features, a reasonably nice results interface,
and it's somewhat more mature than AutoQA at present: these are good
points about it.

it relies on screenshot validation for pass/fail and it's written in
freakin' perl: these are bad points about it. =)

One problem with screenshot validation is that it's innately fragile: if
we used such an approach for anaconda testing, for instance, we'd have
to revise the 'pass' and 'fail' results for dozens of tests at least
once a cycle, most likely. The entire suite would be invalidated by a UI
rewrite like the one pending for F17. A consequence of the innate
fragility is that it's very difficult to make it truly end-to-end
automated; you can't just throw the raw results at developers because
they're very often going to be invalid or incomplete. Whenever a test
'fails' or returns 'unknown' you need human intervention to take a look
at it and verify what actually happened, then file an appropriate bug
report. If you look at the bug reports down the left hand side of the
openQA page, none of them was actually filed by an openQA bot, or
anything: they're all hand-filed reports that have been done by people
interpreting the openQA results. The AutoQA design is inherently more
capable of allowing tests which can spit out results that are directly
useful to developers without human intervention; in fact, depcheck
already does so, by giving the actual yum output describing the
dependency issues it finds.

The other problem with screenshot validation is that it's inherently
unsuited to some tests that are important to Fedora and that we have
more-or-less working *right now* in AutoQA - things like depcheck and
repoclosure - because all it can really tell you is 'does this results
screen look like a known pass case' - it's inherently pass/fail based,
which is no use if the output you want is "these five dependencies are
broken".

But hey, it's dumb but it works right now, which means it's not dumb: in
a sense, updating a screenshot hash once a week is still much better
than manually re-doing the test every day, and though the approach
doesn't work for _some_ tests, there are certainly tests that would be
useful to Fedora which we _could_ implement with screenshot validation.
The test format of openQA is nice and idiot-proof too, at least from the
quick look I took at it - each test can have as many known 'pass' and
'fail' screenshot hashes as you want, and adding one is literally a
one-line copy/paste/modify operation, so it would be very
contributable-to.

We're about to hit the F16 final crunch, at which point QA has about
zero time to look at anything but release validation, but once final is
done we could certainly try and find someone to take a quick look at
openQA and see if it would be viable to make it usable for at least a
bit of Fedora testing in the short term.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net



More information about the devel mailing list