openSUSE announces release of openqa

[Kevin Kofler]

Adam Williamson wrote:

> On Sat, 2011-10-15 at 10:03 +0200, Kevin Kofler wrote:
>> Adam Williamson wrote:
>> > The other problem with screenshot validation is that it's inherently
>> > unsuited to some tests that are important to Fedora and that we have
>> > more-or-less working *right now* in AutoQA - things like depcheck and
>> > repoclosure - because all it can really tell you is 'does this results
>> > screen look like a known pass case'
>> 
>> Well, the test could:
>> 1. bring up a known terminal emulator,
>> 2. type this into it:
>> su -c "yum -y install packagename >/dev/null 2>/dev/null; echo $?"
>> 3. have a PASS screenshot where the terminal emulator displays 0 and a
>> FAIL screenshot where the terminal emulator displays 1.
>> 
>> (But of course that takes a lot more work to set up than just checking
>> the dependencies directly.)
> 
> This is pretty much what it does - you get the screenshots as output
> from openQA (and a video, which is kinda awesome).
> 
> But it still doesn't know what the text *is*: it can't store it in a
> database or email it to you. You have to actually look at the picture.
> You can't search it or compare it or anything. It's just a picture of
> some words.
> 
> (If someone suggests putting an OCR in the loop I'm going to get my
> gun. :>)

The point of my test is that it'd get only 2 possible answers, 1 and 0. (If 
there are other exit codes, append an || false and it'll be only 1 and 0.) 
So you take a screenshot of the console output and it will be one of 2 
canned screenshots, where the one with 0 will be marked as known pass and 
the one with 1 as known fail. But of course, if the screenshot only shows 
"1", you have no idea what the error actually was.

        Kevin Kofler