Hey, folks. Just to let everyone know, I started looking into why openQA tests are failing in Rawhide yesterday but got sidetracked trying to get rid of a bunch of rice weevils in my kitchen. (seriously, those little buggers are a pain in the rear end).
I'm not sure what's going on with the 'Begin Installation' button apparently not becoming available in boot.iso , but I know approximately what's wrong with the live install: the fonts have somehow changed on some (all?) pages. On the root and user creation screens at least, the text it's trying to match now looks sufficiently different (a pixel taller, for one thing) that the match fails. I've written a new needle for the root screen, but I wanted to try and see if I could find out what actually happened to cause the text to look different before we go and create new needles for every text match in the installer...
also I think the Boston box is on git master branch (of openqa_fedora and openqa_fedora_tools), not develop, and I think is missing some of the latest commits. I'm using openqa.happyassassin.net for my testing, you can see my jobs there.
On Tue, 2015-07-07 at 08:16 -0700, Adam Williamson wrote:
Hey, folks. Just to let everyone know, I started looking into why openQA tests are failing in Rawhide yesterday but got sidetracked trying to get rid of a bunch of rice weevils in my kitchen. (seriously, those little buggers are a pain in the rear end).
I'm not sure what's going on with the 'Begin Installation' button apparently not becoming available in boot.iso , but I know approximately what's wrong with the live install: the fonts have somehow changed on some (all?) pages. On the root and user creation screens at least, the text it's trying to match now looks sufficiently different (a pixel taller, for one thing) that the match fails. I've written a new needle for the root screen, but I wanted to try and see if I could find out what actually happened to cause the text to look different before we go and create new needles for every text match in the installer...
So I got a bit farther with this today. The difference actually seems to be not 22 vs. 23 but boot.iso vs. Workstation live. There's some kind of rather subtle difference in font rendering between the two. The easiest way to see it is to look at a Done button. On boot.iso (22 or Rawhide), the 'D' is 11 pixels tall. On Workstation live images, it's 10 pixels tall. There are other differences too, if you look closely: the capital 'R' and 'P' in 'Root Password' (on the root password screen) are 9 pixels high on Workstation live images, 10 pixels high on boot.iso.
Now the question is do we: i) try and figure out what the hell the difference between the two is - which could be a hell of a yak shave
ii) try and finesse the needles (match area, size, required similarity %age) until openQA is OK with the differences
iii) have two variants of affected needles?
In any case it's something we need to be aware of when using text needles in anaconda - please check your needles work with both boot.iso and live images...
I did try a *bit* of i). I know one obvious difference is that gnome -settings-daemon is active on the Workstation live, but not on boot.iso , and I know g-s-d does set various font options. The most obvious one that might differ is the hinting strength, but actually the default for that seems to be 'medium' both in the g-s-d dconf schema and in /etc/X11/Xresources (which I think sets the default if g-s-d isn't involved).
I can't reproduce the boot.iso 'look' by messing with the hinting and antialiasing settings in gnome-tweak-tool on a live image - I can certainly make the rendering look different, but no combination of settings seems to make it look exactly like boot.iso.
An interesting case I just discovered is that if I set GNOME's 'text scaling factor' to 1.01 on the Workstation live, it makes the glyph heights the same as in boot.iso, but there are still some very subtle differences in the rendering: for e.g. the kerning is a bit different (there's a one pixel gap between the 'D' and the 'o' of 'Done' in Workstation-at-1.01 while on boot.iso there's no gap).
a KDE live image has the same rendering as boot.iso, which is about what I'd expect (as g-s-d is not active there either).
On Tue, 2015-07-07 at 15:45 -0700, Adam Williamson wrote:
An interesting case I just discovered is that if I set GNOME's 'text scaling factor' to 1.01 on the Workstation live, it makes the glyph heights the same as in boot.iso, but there are still some very subtle differences in the rendering: for e.g. the kerning is a bit different (there's a one pixel gap between the 'D' and the 'o' of 'Done' in Workstation-at-1.01 while on boot.iso there's no gap).
Just to further update this, we've more or less pinned it down now. mclasen was extremely helpful in chasing it through various obscure bits of X11 and GTK+.
There's a neat GTK+ debugging thing called the GtkInspector which you can launch from any GTK+ 3 app with shift+ctrl+D. From that you can look at Properties then GtkSettings, and look at the gtk-xft-* settings. I noticed two significant differences between anaconda running from boot.iso or KDE live and anaconda running from Workstation live, both at 1024x768 in a VM. gtk-xft-hintstyle is 'medium' in Workstation but 'full' in boot.iso/KDE, and gtk-xft-dpi is 98304 in Workstation but 98401 in boot.iso/KDE. That value is a DPI multiplied by 1024; Workstation's is exactly 96 (98304/1024 = 96), boot.iso/KDE is...a little bit more than 96.
Basically we figured out that for some reason the defaults that should be set by /etc/X11/Xresources don't seem to actually be merged into X, in boot.iso and KDE live image at least. So GDK (part of GTK+) falls back on its internal defaults for hinting style and display resolution (DPI). Its default for hinting style, in the absence of any external indication, is 'hintfull', and its default for DPI is to actually calculate the 'correct' DPI for the display. It turns out that a KVM at 1024x768 reports its "monitor's" vertical size as 203mm; if you reproduce GDK's calculation with 768 pixels and 203mm:
768 (pixel height) * 25.4 (inches to MM) / 203 (height in MM) * 1024 (GDK's multiplication factor) ~= 98401 (actually a bit lower but it gets rounded to the nearest integer)
and that's where the weird 98401 value comes from.
From here there's a few angles of attack:
1) Why does libvirt/qemu/whatever report a display size which doesn't *quite* give a resolution of exactly 96dpi?
2) Why don't the settings in /etc/X11/Xresources actually get merged into X config? (/etc/X11/xinitrc-common looks like it ought to do this)
3) Should GDK just use the same defaults we've tried to put in to X (/etc/X11/Xresources) and GNOME (g-s-d defaults) - i.e. medium hinting and 96dpi?
which we'll look into tomorrow, cos I've got to go out for dinner now...:)
There's a neat GTK+ debugging thing called the GtkInspector which you can launch from any GTK+ 3 app with shift+ctrl+D. From that you can
Or Ctrl+Shift+I, if you find that easier to remember (I as inspector) :-)
look at Properties then GtkSettings, and look at the gtk-xft-* settings. I noticed two significant differences between anaconda running from boot.iso or KDE live and anaconda running from Workstation live, both at 1024x768 in a VM. gtk-xft-hintstyle is 'medium' in Workstation but 'full' in boot.iso/KDE, and gtk-xft-dpi is 98304 in Workstation but 98401 in boot.iso/KDE. That value is a DPI multiplied by 1024; Workstation's is exactly 96 (98304/1024 = 96), boot.iso/KDE is...a little bit more than 96.
Basically we figured out that for some reason the defaults that should be set by /etc/X11/Xresources don't seem to actually be merged into X, in boot.iso and KDE live image at least. So GDK (part of GTK+) falls back on its internal defaults for hinting style and display resolution (DPI). Its default for hinting style, in the absence of any external indication, is 'hintfull', and its default for DPI is to actually calculate the 'correct' DPI for the display. It turns out that a KVM at 1024x768 reports its "monitor's" vertical size as 203mm; if you reproduce GDK's calculation with 768 pixels and 203mm:
768 (pixel height) * 25.4 (inches to MM) / 203 (height in MM) * 1024 (GDK's multiplication factor) ~= 98401 (actually a bit lower but it gets rounded to the nearest integer)
and that's where the weird 98401 value comes from.
What, that's a one hell of a debug story. Awesome job.
also I think the Boston box is on git master branch (of openqa_fedora and openqa_fedora_tools), not develop, and I think is missing some of the latest commits
Just a side-note - yes, Boston machine is running on master. I thought of it as "production" machine. But it raises questions - should we push hot-fixes directly into master in cases like these? Or should we merge develop into master as quickly as possible? Or should we run Boston on develop until our tests get stable enough?
Jan
On Wed, 2015-07-08 at 09:09 -0400, Kamil Paral wrote:
What, that's a one hell of a debug story. Awesome job.
Oh, it goes deeper!
https://www.happyassassin.net/2015/07/09/of-dpis-desktops-and-toolkits/
On Tue, 2015-07-07 at 08:16 -0700, Adam Williamson wrote:
Hey, folks. Just to let everyone know, I started looking into why openQA tests are failing in Rawhide yesterday but got sidetracked trying to get rid of a bunch of rice weevils in my kitchen. (seriously, those little buggers are a pain in the rear end).
Found another thing that's causing problems, besides the font issue and the insensitive 'Begin Installation' button: the Software Selection screen's menus seem to be behaving a bit weirdly with GTK+ 3.17, which is screwing up the software selection step for all tests which do software selection. Filed on BGO: https://bugzilla.gnome.org/show_bug.cgi?id=752200
Great article! I'm little bit frustrated - things like this doesn't make it easy for us. On the other hand, if it wasn't for OpenQA, we perhaps wouldn't notice it at all.
----- Original Message ----- From: "Adam Williamson" adamwill@fedoraproject.org To: "Fedora QA Development" qa-devel@lists.fedoraproject.org Sent: Friday, July 10, 2015 12:44:07 AM Subject: Re: openQA vs. Rawhide
On Wed, 2015-07-08 at 09:09 -0400, Kamil Paral wrote:
What, that's a one hell of a debug story. Awesome job.
Oh, it goes deeper!
https://www.happyassassin.net/2015/07/09/of-dpis-desktops-and-toolkits/
Found another thing that's causing problems, besides the font issue and the insensitive 'Begin Installation' button: the Software Selection screen's menus seem to be behaving a bit weirdly with GTK+ 3.17, which is screwing up the software selection step for all tests which do software selection. Filed on BGO: https://bugzilla.gnome.org/show_bug.cgi?id=752200
You forgot to attach the video.
On Fri, 2015-07-10 at 05:24 -0400, Kamil Paral wrote:
Found another thing that's causing problems, besides the font issue and the insensitive 'Begin Installation' button: the Software Selection screen's menus seem to be behaving a bit weirdly with GTK+ 3.17, which is screwing up the software selection step for all tests which do software selection. Filed on BGO: https://bugzilla.gnome.org/show_bug.cgi?id=752200
You forgot to attach the video.
Huh - I was sure I did, but you're right, it's not there. Thanks.
qa-devel@lists.fedoraproject.org
-
Adam Williamson -
Jan Sedlak -
Kamil Paral