[Test-Announce] 2012-09 Graphics Test Week recap

Wed Oct 3 05:01:41 UTC 2012

Here's the recap of the Fedora 18 Graphics Test Week. Note that there
are no Fedora 17 Graphics Test Week numbers here because there wasn't
one! The last one was Fedora 16. So the 'outcome' numbers have the
results from the F16 test week added.

As expected, we got more tests and more bugs filed this time than we did
in F16; the F16 numbers were low because I didn't get the publicity and
organization done properly. We got it done better this time, with many
thanks to Martin Holec for covering some stuff I missed (and also for
running a ton of tests himself, which pumps the numbers up too :>)

Here's the numbers on tests conducted and bugs filed, with all the
historical numbers for comparison. I counted each non-example line in
the main results matrix as a 'test conducted'. I didn't count the runs
in the 'bonus' 3D matrix, in order to provide a more fair comparison
with previous releases. In future, once we have enough releases with the
bonus 3D tests included, I may start using those numbers. This might
affect the bugs-to-tests ratio somewhat.

f11 nouveau: 104 tests, 42 bugs - ratio 0.40
f12 nouveau: 53 tests, 34 bugs - ratio 0.64
f13 nouveau: 78 tests, 26 bugs - ratio 0.33
f14 nouveau: 39 tests, 8 bugs - ratio 0.21
f15 nouveau: 83 tests, 55 bugs - ratio 0.66
f16 nouveau: 16 tests, 8 bugs - ratio 0.50
f18 nouveau: 54 tests, 43 bugs - ratio 0.80

f11 radeon: 55 tests, 46 bugs - ratio 0.84
f12 radeon: 61 tests, 81 bugs - ratio 1.33
f13 radeon: 48 tests, 33 bugs - ratio 0.69
f14 radeon: 32 tests, 18 bugs - ratio 0.56
f15 radeon: 66 tests, 38 bugs - ratio 0.58
f16 radeon: 12 tests, 8 bugs - ratio 0.67
f18 radeon: 32 tests, 24 bugs - ratio 0.75

f11 intel: 23 tests, 21 bugs - ratio 0.91
f12 intel: 29 tests, 31 bugs - ratio 1.07
f13 intel: 38 tests, 38 bugs - ratio 1.00
f14 intel: 33 tests, 28 bugs - ratio 0.84
f15 intel: 37 tests, 25 bugs - ratio 0.68
f16 intel: 10 tests, 7 bugs - ratio 0.70
f18 intel: 27 tests, 7 bugs - ratio 0.26

Interesting that Intel is doing really well this cycle. The hardware
tested seems to be a reasonable range, so it's not that we were only
testing one small set of hardware that happens to work well. Radeon is
broadly in line with historical trends. nouveau is at its 'buggiest'
level ever, which is maybe worrying, though it's worth noting that quite
a lot of the bugs filed related to the advanced 3D tests, and two of
them were for a crash in extremetuxracer which is nothing to do with
nouveau. As for participation, we had 113 tests run this time, a huge
improvement on the 38 from F16, about even with F14, but still a bit
below the other releases from F11 to F15, so we can still make things
better for F18.

Here's the chart for how reported bugs are handled, adding the results
from the F16 events (this chart gets updated on a one-cycle delay,
because it's concerned with how the bugs filed are handled after a
reasonable amount of time for the developers to look at them, and as
noted above, there was no F17 event, so we're adding the F16 results
this time). A full explanation of what this tracks and how it's
calculated in the F13 recap at
https://lists.fedoraproject.org/pipermail/test/2010-April/090271.html
and the F16 recap at
https://lists.fedoraproject.org/pipermail/test/2011-November/104677.html .
Also, if you try to reproduce these numbers, note that you'll want to
remove the bugs 123456 and 234567 from the lists - they're used in the
'example' entries in the results matrix. The numbers for older releases
are copied/pasted, not re-generated each time. F16 may gain a slight
advantage here from having had a year for its bugs to be fixed, not six
months.

f11 nouveau: 42 bugs, 4 open, 8 closeddupe, 24 closedfixed, 6 closedunfixed - 70.59%
f12 nouveau: 34 bugs, 11 open, 8 closeddupe, 14 closedfixed, 1 closedunfixed - 53.85%
f13 nouveau: 27 bugs, 17 open, 6 closeddupe, 3 closedfixed, 1 closedunfixed - 14.29%
f14 nouveau: 8 bugs, 5 open, 3 closeddupe - 0% (small sample)
f15 nouveau: 58 bugs, 27 open, 13 closeddupe, 13 closedfixed, 5 closedunfixed - 28.89%
f16 nouveau: 8 bugs, 4 open, 3 closeddupe, 1 closedunfixed - 0% (small sample)

f11 radeon: 46 bugs, 14 open, 10 closeddupe, 19 closedfixed, 3 closedunfixed - 52.78%
f12 radeon: 81 bugs, 19 open, 32 closeddupe, 28 closedfixed, 2 closedunfixed - 57.14%
f13 radeon: 36 bugs, 28 open, 3 closeddupe, 5 closedfixed, 0 closedunfixed - 15.15%
f14 radeon: 18 bugs, 13 open, 0 closeddupe, 3 closedfixed, 2 closedunfixed - 16.67%
f15 radeon: 38 bugs, 17 open, 10 closeddupe, 9 closedfixed, 2 closedunfixed - 32.14%
f16 radeon: 8 bugs, 5 open*, 1 closeddupe, 1 closedfixed, 1 closedunfixed - 14.29%* (small sample)

f11 intel: 21 bugs, 7 open, 1 closeddupe, 12 closedfixed, 1 closedunfixed - 60%
f12 intel: 31 bugs, 7 open, 12 closeddupe, 12 closedfixed, 0 closedunfixed - 63.16%
f13 intel: 42 bugs, 26 open, 4 closeddupe, 11 closedfixed, 1 closedunfixed - 28.95%
f14 intel: 28 bugs, 21 open, 4 closeddupe, 1 closedfixed, 2 closedunfixed - 4.17%
f15 intel: 27 bugs, 8 open, 7 closeddupe, 12 closedfixed, 0 closedunfixed - 60%
f16 intel: 7 bugs, 5 open, 1 closeddupe, 1 closedfixed, 0 closedunfixed - 16.67% (small sample)

The low number of bugs makes these numbers pretty unreliable, really,
but obviously the results aren't great for any of the drivers, the
bottom line is exactly two bugs got fixed so far as a result of the F16
test days, which obviously isn't what we're aiming for. In the end
though it's probably a result of the overall neglect of the event in
that cycle, so we'll have to wait for the F18 numbers - where the bug
count is up again - to draw any conclusions. Re the asterisks on radeon
- one of the 'open' bugs is #698011, which is actually CLOSED UPSTREAM
but counted under 'open' because it's open upstream. It's actually under
active development right now, it looks like a patch that fixes it was
posted at the end of September. So probably, if I'd done this recap a
week later, that bug would be counted as closedfixed not open, and the
radeon numbers would look better.

Many thanks as always to all those who contributed tests and to our
plucky X developers and triagers, especially:

Adam Jackson
Dave Airlie
Jerome Glisse
Ben Skeggs
Matej Cepl
Martin Holec

The raw list of bugs filed at each of the F18 events follows.

Nouveau
-------

745202 NEW  - gnome-shell does not display correctly with NV3x adapters - multicolor corruption of panel, Shell-style menus and text [nvfx]
752613 NEW  - Nivida 100M on Lenovo W520 Wont boot if using Discrete Graphics
835648 NEW  - kernel freeze after [drm] Initialized drm 1.1.0 20060810 on T520 with Quadro NVS 4200M
848488 NEW  - [abrt] extremetuxracer-0.4-9.fc17: size: Process /usr/bin/etracer was killed by signal 11 (SIGSEGV)
855560 NEW  - F18 Live Alpha : nVidia GeForce 8600M GT, graphic driver (nouveau): very poor performance (vesa is fluid)
858503 NEW  - display shows garbage after suspend/resume on W520
860215 NEW  - [abrt]: WARNING: at kernel/power/suspend_test.c:53 suspend_test_finish+0x86/0x90()
860228 NEW  - Rendering / performance issues with polymol on GeForce 310 (GT218 / NVA8)
860248 NEW  - Fedora 18 AMD 64 Nouveau Desktop Suspend Fails
860256 NEW  - NV41 won't suspend
860259 NEW  - [abrt] extremetuxracer-0.4-10.fc18: pp::Font::advance: Process /usr/bin/etracer was killed by signal 11 (SIGSEGV)
860265 NEW  - NV44 Xorg segfault after suspend
860270 NEW  - pure virtual function called crash
860360 NEW  - Distorted text after system booted
860389 NEW  - NVC8 wont suspend
860405 NEW  - Video with nVidia Corporation G73 [GeForce 7300 GT] does not come back after suspend
860407 NEW  - system hangs while playing a video via totem and on pressing the windows key
860477 NEW  - nouveau Xorg crash following boot (GTX 580)
860584 NEW  - Incorrect video wake up with Nvidia G72M (Geforce Go 7300) and nouveau after suspend
860611 NEW  - Freezed and/or black Xorg output (with cursor) on G72M with nouveau on 2nd display configuration (multihead)
860624 NEW  - Gnome-shall hangs when requested activities
860628 NEW  - [abrt] mesa-demos-7.10-8.20101028.fc18: exit: Process /usr/lib64/mesa/fbo_firecube was killed by signal 6 (SIGABRT)
860637 NEW  - Gnome-shell is slow after suspend-resume
860660 NEW  - Transparency rendering problems on PyMOL with HD4890 and radeon driver
860956 NEW  - Resume from suspend fails on GeForce 7300 GT
861106 NEW  - X Test Day: Video breaks when secondary monitor is connected
861280 NEW  - KDE Crashes or Hangs After Moving Konqueror Window Quickly
861624 NEW  - NV43 (Geforce 6600GT) suspend artifact
861625 NEW  - NV43 (Geforce 6600GT) visual artefacts when running pymol
861628 NEW  - NV43 (Geforce 6600GT) tux racer doesn't restore default screen resolution when exiting
861646 NEW  - Sporadic GPU Lockups when Fast User Switching
861656 NEW  - Rendering / performance issues with polymol on GeForce 6150SE
861746 NEW  - GDM login screen artefacts and corruption with Geforce 210
861939 NEW  - [Nouveau] Missing some textures in fbo_firecube, teapot and reflect mesa demos
861946 NEW  - Slow rendering in OpenArena
861951 NEW  - Resume from suspend slowly freezes input.
861960 NEW  - Disabling primary display when secondary is enabled blanks both of them
861969 NEW  - Rendering glitch after resume from suspend
862026 NEW  - Rendering glitch after opening kickoff application launcher
857267 ON_QA  - SELinux is preventing /usr/bin/totem-video-thumbnailer from 'create' accesses on the directory gstreamer-1.0.
806491 ON_QA  - systemd-logind not tracking startx sessions
862363 CLOSED UPSTREAM - OpenGL rendering on Quadro 310 10de:107d is not supported
857662 CLOSED DUPLICATE - [mesa] libGL error: failed to open drm device

Radeon
------

790365 NEW  - Can't resume from suspend to RAM: Black screen and laptop frozen
847818 NEW  - Radeon RS690M/X1200 screen corruption beyond kernel-3.4.6-2.fc17.i686
848488 NEW  - [abrt] extremetuxracer-0.4-9.fc17: size: Process /usr/bin/etracer was killed by signal 11 (SIGSEGV)
849347 NEW  - radeon driver cannot handle restarting xorg server sometimes (fails with: *ERROR* Failed to parse relocation -35!)
860451 NEW  - Artifact between apply and rotated screen
860628 NEW  - [abrt] mesa-demos-7.10-8.20101028.fc18: exit: Process /usr/lib64/mesa/fbo_firecube was killed by signal 6 (SIGABRT)
860660 NEW  - Transparency rendering problems on PyMOL with HD4890 and radeon driver
860668 NEW  - artifacts on xrestart with rv710
860779 NEW  - Radeon HD6570 hangs on VT switching
860783 NEW  - Radeon HD6570 GLX test failed (radeon test day)
860789 NEW  - Multihead displays shows artifacts
860794 NEW  - openarena failed to start on Radeon HD6570 (radeon test day)
860798 NEW  - ExtremeTuxRacer crashes on HD6570 with a free driver
860843 NEW  - [abrt] extremetuxracer-0.4-10.fc18: radeon_bo_destroy: Process /usr/bin/etracer was killed by signal 11 (SIGSEGV)
861755 NEW  - X fails to start or display after reboot on ATI FirePro V4800
861772 NEW  - Xv video playback in Totem lags audio by several seconds on ATI FirePro V4800
861831 NEW  - GNOME 3 Activities panel takes several minutes to appear after login on ATI FirePro V4800
861946 NEW  - Slow rendering in OpenArena
862372 NEW  - etracer killed with SIGABRT on quit
860712 ASSIGNED  - [6550D SUMO] problems with second monitor on VGA, causing GPU lockups

Intel
-----

847479 NEW  - Kernel 3.5.0-2 and 3.5.1-1 nouveau driver causes Xorg to abrt after resume from RAM
848488 NEW  - [abrt] extremetuxracer-0.4-9.fc17: size: Process /usr/bin/etracer was killed by signal 11 (SIGSEGV)
860551 NEW  - black screen fedora 20120924-test_days-x86_64
861144 NEW  - Can't log in with fast user switch after use gstreamer
861219 NEW  - Switching to discret GPU render system unusable
862190 NEW  - X Test Week: warning: Failed to initialize plugin: '/usr/lib64/grilo-0.2/libgrlflickr.so'
861166 CLOSED DUPLICATE - [abrt] python-2.7.3-13.fc18: CShaderPrg_Enable: Process /usr/bin/python2.7 was killed by signal 11 (SIGSEGV)
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
test-announce mailing list
test-announce at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/test-announce