Fedora 23 Branched 20151009 compose check report
Adam Williamson
adamwill at fedoraproject.org
Fri Oct 9 20:05:10 UTC 2015
On Fri, 2015-10-09 at 21:41 +0200, Kalev Lember wrote:
> These failed test IDs are all nice to see, but they don't say much to
> the uninitiated. :)
Yeah, what I'd actually like to do is have links to the tests; the
reason we can't do that right now is that the openQA instance is behind
the RH firewall so only RHers can see it. We're trying to fix that - in
fact that's what I'd be doing today if I didn't have to retake a few
dozen screenshots instead :P
> These tests are comparing screenshots pixel by
> pixel, right? Are the screenshots and other test results publicly
> available somewhere?
See above. And soooooorrt of, but not really. openQA matches on areas
within screenshots; an openQA 'needle' is a screenshot with an
accompany JSON file that identifies areas within it (and contains tags
used for actually doing matches). A needle 'matches' if all the marked
areas have matches within the screenshot being compared, after a bit of
image processing is applied (color reduction and sharpening), within a
degree of tolerance (~96% similarity according to some algorithm or
other).
> How are the screenshots rebased? Is it all manual work?
More or less. the openQA web UI has an interactive needle editor which
lets you basically run a test and pause after each failed match, then
you can fiddle with the needle and re-run the match. I haven't used it
much and never for doing a big screenshot rebase, so I can't say
whether it's more or less convenient than the way I did the last one,
which was to reproduce the tests by hand in a VM, take the screenshots
from virt-manager, copy the json file and make any necessary
adjustments to the match area with the 'crop.py' needle editor (which
is an alternative tool upstream provides in one of their repos).
> Is there any way to link to the failed screenshots from the status
> mail
> so that I could just follow links and see what's changed?
Once we get openQA deployed publicly, yeah. As I said, we're working on
that; https://phab.qadevel.cloud.fedoraproject.org/T623 for the gritty
details.
> Sorry for bombing you with all the questions. I think it's totally
> awesome that we have nightly testing, thanks for getting this set up!
No probs!
Ultimately this is an occupational hazard of how openQA works, it's
just that I was surprised that a font would suddenly change its
appearance quite noticeably (at least to a tool like openQA). But now I
got some references from you and Matthias I understand that it was to
fix clear rendering errors, which is pretty reasonable, I just hope
there isn't too much churn in future releases :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
More information about the devel
mailing list