Fedora 23 Branched 20151009 compose check report

Adam Williamson adamwill at fedoraproject.org
Fri Oct 9 20:05:10 UTC 2015


On Fri, 2015-10-09 at 21:41 +0200, Kalev Lember wrote:

> These failed test IDs are all nice to see, but they don't say much to
> the uninitiated. :)

Yeah, what I'd actually like to do is have links to the tests; the
reason we can't do that right now is that the openQA instance is behind
the RH firewall so only RHers can see it. We're trying to fix that - in
fact that's what I'd be doing today if I didn't have to retake a few
dozen screenshots instead :P

>  These tests are comparing screenshots pixel by
> pixel, right? Are the screenshots and other test results publicly
> available somewhere?

See above. And soooooorrt of, but not really. openQA matches on areas
within screenshots; an openQA 'needle' is a screenshot with an
accompany JSON file that identifies areas within it (and contains tags
used for actually doing matches). A needle 'matches' if all the marked
areas have matches within the screenshot being compared, after a bit of
image processing is applied (color reduction and sharpening), within a
degree of tolerance (~96% similarity according to some algorithm or
other).

> How are the screenshots rebased? Is it all manual work?

More or less. the openQA web UI has an interactive needle editor which
lets you basically run a test and pause after each failed match, then
you can fiddle with the needle and re-run the match. I haven't used it
much and never for doing a big screenshot rebase, so I can't say
whether it's more or less convenient than the way I did the last one,
which was to reproduce the tests by hand in a VM, take the screenshots
from virt-manager, copy the json file and make any necessary
adjustments to the match area with the 'crop.py' needle editor (which
is an alternative tool upstream provides in one of their repos).

> Is there any way to link to the failed screenshots from the status
> mail
> so that I could just follow links and see what's changed?

Once we get openQA deployed publicly, yeah. As I said, we're working on
that; https://phab.qadevel.cloud.fedoraproject.org/T623 for the gritty
details.

> Sorry for bombing you with all the questions. I think it's totally
> awesome that we have nightly testing, thanks for getting this set up!

No probs!

Ultimately this is an occupational hazard of how openQA works, it's
just that I was surprised that a font would suddenly change its
appearance quite noticeably (at least to a tool like openQA). But now I
got some references from you and Matthias I understand that it was to
fix clear rendering errors, which is pretty reasonable, I just hope
there isn't too much churn in future releases :)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net




More information about the devel mailing list