[Test-Announce] Very belated 2011-09 Graphics Test Week recap

Wed Nov 30 19:41:20 UTC 2011

I know this is horribly late, but I just worked my way around to it!
Sorry for that.

Here's the recap of the Fedora 16 Graphics Test Week. The overall
numbers of tests done and bugs filed are down significantly, because I
was very busy and did not do a great job of arranging and promoting the
events this cycle. I hope we'll be able to do a better job for the
Fedora 17 graphics test week.

Here's the numbers on tests conducted and bugs filed. I counted each
non-example line in the main results matrix as a 'test conducted'. I
didn't count the runs in the 'bonus' 3D matrix, in order to provide a
more fair comparison with previous releases. This might affect the
bugs-to-tests ratio somewhat.

f11 nouveau: 104 tests, 42 bugs - ratio 0.40
f12 nouveau: 53 tests, 34 bugs - ratio 0.64
f13 nouveau: 78 tests, 26 bugs - ratio 0.33
f14 nouveau: 39 tests, 8 bugs - ratio 0.21
f15 nouveau: 83 tests, 55 bugs - ratio 0.66
f16 nouveau: 16 tests, 8 bugs - ratio 0.50

f11 radeon: 55 tests, 46 bugs - ratio 0.84
f12 radeon: 61 tests, 81 bugs - ratio 1.33
f13 radeon: 48 tests, 33 bugs - ratio 0.69
f14 radeon: 32 tests, 18 bugs - ratio 0.56
f15 radeon: 66 tests, 38 bugs - ratio 0.58
f16 radeon: 12 tests, 8 bugs - ratio 0.67

f11 intel: 23 tests, 21 bugs - ratio 0.91
f12 intel: 29 tests, 31 bugs - ratio 1.07
f13 intel: 38 tests, 38 bugs - ratio 1.00
f14 intel: 33 tests, 28 bugs - ratio 0.84
f15 intel: 37 tests, 25 bugs - ratio 0.68
f16 intel: 10 tests, 7 bugs - ratio 0.70

It's hard to draw any conclusions from such small numbers, but really,
none of them are far out of line with previous events.

Here's the chart for how reported bugs are handled, adding the results
from the F15 events (this chart gets updated on a one-cycle delay,
because it's concerned with how the bugs filed are handled after a
reasonable amount of time for the developers to look at them). A full
explanation of what this tracks and how it's calculated in the F13 recap
at
https://lists.fedoraproject.org/pipermail/test/2010-April/090271.html .
Some resolutions showed up this time that didn't before. I counted
'WORKSFORME' in with 'closeddupe' - i.e. as part of the set of bugs that
should just be completed discarded from consideration. I handled
'UPSTREAM' bugs on a case-by-case basis, going in and look at whether
they'd actually been fixed upstream or what, and putting them in the
most appropriate group. Also, if you try to reproduce these numbers,
note that you'll want to remove the bugs 123456 and 234567 from the
lists - they're used in the 'example' entries in the results matrix. The
bug totals are slightly different from the above chart because I
copy/pasted the F15 numbers in the above chart from the last recap, but
the numbers in this chart are newly generated, and so a few bugs that
were filed and added to the Wiki pages *after* the last recap went out
have been added to the numbers in this chart.

f11 nouveau: 42 bugs, 4 open, 8 closeddupe, 24 closedfixed, 6 closedunfixed - 70.59%
f12 nouveau: 34 bugs, 11 open, 8 closeddupe, 14 closedfixed, 1 closedunfixed - 53.85%
f13 nouveau: 27 bugs, 17 open, 6 closeddupe, 3 closedfixed, 1 closedunfixed - 14.29%
f14 nouveau: 8 bugs, 5 open, 3 closeddupe - 0% (small sample)
f15 nouveau: 58 bugs, 27 open, 13 closeddupe, 13 closedfixed, 5 closedunfixed - 28.89%

f11 radeon: 46 bugs, 14 open, 10 closeddupe, 19 closedfixed, 3 closedunfixed - 52.78%
f12 radeon: 81 bugs, 19 open, 32 closeddupe, 28 closedfixed, 2 closedunfixed - 57.14%
f13 radeon: 36 bugs, 28 open, 3 closeddupe, 5 closedfixed, 0 closedunfixed - 15.15%
f14 radeon: 18 bugs, 13 open, 0 closeddupe, 3 closedfixed, 2 closedunfixed - 16.67%
f15 radeon: 38 bugs, 17 open, 10 closeddupe, 9 closedfixed, 2 closedunfixed - 32.14%

f11 intel: 21 bugs, 7 open, 1 closeddupe, 12 closedfixed, 1 closedunfixed - 60%
f12 intel: 31 bugs, 7 open, 12 closeddupe, 12 closedfixed, 0 closedunfixed - 63.16%
f13 intel: 42 bugs, 26 open, 4 closeddupe, 11 closedfixed, 1 closedunfixed - 28.95%
f14 intel: 28 bugs, 21 open, 4 closeddupe, 1 closedfixed, 2 closedunfixed - 4.17%
f15 intel: 27 bugs, 8 open, 7 closeddupe, 12 closedfixed, 0 closedunfixed - 60%

There's definitely good news here - as noted in the last recap, we had a
good crop of reports from the F15 event, so there's no reason we
shouldn't be able to get back up to this level with better organization
and promotion for F17. Also, the number of bugs actually getting *fixed*
- which is the ultimate goal of the event - was significantly higher
than F13 and F14, not quite back to F11/F12 levels, but a lot better. So
the trend of fewer bugs getting fixed seems to have been stopped during
F15 cycle. Intel showed a particularly marked improvement here. (It's
worth noting that some of the bugs filed and fixed were actually on
GNOME rather than the X drivers; I didn't compare the level of non-X
bugs in the results in other releases, so that is an untracked variable.
If anyone wants to look into that, please do!)

Many thanks as always to all those who contributed tests and to our
plucky X developers and triagers, especially:

Adam Jackson
Dave Airlie
Jerome Glisse
Ben Skeggs
Matej Cepl

The raw list of bugs filed at each of the F16 events follows.

Nouveau
-------

735703 NEW  - VT now working properly with the nouveau driver
692035 NEW  - [NV94] Dual-head problems: Nouveau on FC14 fails to drive
both outputs of Nvidia 9600GT correctly.
735702 NEW  - [NV67] fast switch user was not working
735893 NEW  - [NV42] Shadows on rendering
736323 NEW  - startx starts fallback mode
736837 CLOSED WORKSFORME - Rotate display fails on GTX 560
737850 CLOSED DUPLICATE - nouveau testday: focus issues while glx
testcase (NV43)
735770 CLOSED NOTABUG - [NVa8] External monitor doesn't work after upgrade to F16

Radeon
------

735703 NEW  - VT now working properly with the nouveau driver
698711 NEW  - [Crestline] Generic video multihead test failure
735702 NEW  - [NV67] fast switch user was not working
733857 NEW  - No display with AMD A6-3650 APU (ATI HD 6530D)
736498 NEW  - [RV635] Testcase radeon xv failed
736562 NEW  - [RS740] Gnome 3 dualhead fallback mode
739078 CLOSED ERRATA - Booting to runlevel 3 failed
698011 CLOSED UPSTREAM - [RS690] LiveCD doesn't boot into Gnome3. Only get background, no panel. Mouse clicks ignored.

Intel
-----

736859 NEW  - dual-head: re-arranging screen leaves one screen off
736358 NEW  - Gtk-WARNING **: Unknown property: GtkDialog.has-separator
735702 NEW  - [NV67] fast switch user was not working
734265 NEW  - [abrt] gnome-shell-3.1.4-2.gite7b9933.fc16: __GI_raise: Process /usr/bin/gnome-shell was killed by signal 6 (SIGABRT)
736323 NEW  - startx starts fallback mode
736330 NEW  - [Cantiga] Internal display turned off when playing game in dual monitor setup
736846 CLOSED CURRENTRELEASE - user switching with color manager enabled causes segfaults and gnome "Oh no..."
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
test-announce mailing list
test-announce at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/test-announce