[Fedora QA] #145: Request to join ProvenTester
by fedora-badges
#145: Request to join ProvenTester
-----------------------------------------+----------------------------------
Reporter: dbhole | Owner:
Type: proventester request | Status: new
Priority: major | Milestone:
Component: Proventester Mentor Request | Version:
Keywords: |
-----------------------------------------+----------------------------------
OpenJDK is a widely used package, and has quarterly security updates (at
the very least). While we (devel and qe) team test it, there is no way for
us to set a +1 karma as a proven-tester.
To that end, I would like to apply for proven tester status. In addition
to OpenJDK, this will also allow me to test and set the proper Karma for
other packages our team owns.
I'd be happy to follow and training guidelines laid out by a Proventester
Mentor.
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/145>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
12 years, 4 months
[Fedora QA] #98: Request for a Mentor
by fedora-badges
#98: Request for a Mentor
-----------------------------------------+----------------------------------
Reporter: jdulaney | Owner:
Type: proventester request | Status: new
Priority: major | Milestone:
Component: Proventester Mentor Request | Version:
Keywords: |
-----------------------------------------+----------------------------------
I would like to now formally request a Proven Tester Mentor. I have been
running Fedora since FC1, generally updating every other release (I
skipped 9 and went straight to 10). I have several boxes so as to get
different hardware involved.
Thanks,
John H. Dulaney
https://fedoraproject.org/wiki/User:Jdulaney
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/98>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
12 years, 4 months
[Fedora QA] #60: Devise a better location for critpath.txt
by fedora-badges
#60: Devise a better location for critpath.txt
-------------------------+--------------------------------------------------
Reporter: kparal | Owner:
Type: enhancement | Status: new
Priority: major | Milestone:
Component: Wiki | Version:
Keywords: |
-------------------------+--------------------------------------------------
critpath.txt lists critical path packages [1]. Currently it is referenced
as
{{{
http://kojipkgs.fedoraproject.org/mash/rawhide-<<CURRENT
DATE>>/logs/critpath.txt
}}}
or
{{{
http://kojipkgs.fedoraproject.org/mash/rawhide-<<YESTERDAY
DATE>>/logs/critpath.txt
}}}
depending on which page. Similarly for branched releases.
This is not optimal. Very often the url just doesn't work. For example at
the time of writing critpath.txt on F13 Alpha Release Criteria page [2]
references rawhide-20100407/ directory, where critpath.txt is not
available. Nor it is available in rawhide-20100406/ directory. Only in
rawhide-20100405/ it is finally available.
The very page Critical Path Packages [1] references critpath.txt for
Branched release, which is available ATM, but also references critpath.txt
for Rawhide, which is again not available.
Let's imagine other use case - some automated tool must decide whether
package in the critical path. It can't rely on path which are constantly
changing and sometimes are not available.
critpath.txt is created dynamically, so we probably must have the contents
created every day or so. But we should devise such mechanism that the
location is stable and the file is always available (if new contents fails
to create, the old contents is still available). That will also allow
third-party tools to rely on it.
[1] https://fedoraproject.org/wiki/Critical_Path_Packages [[BR]]
[2] https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/60>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
12 years, 4 months
[Fedora QA] #236: No qa test for making sure that the boot.iso is hybridised
by fedora-badges
#236: No qa test for making sure that the boot.iso is hybridised
--------------------+-------------------------------------------------------
Reporter: ausil | Owner: adamw
Type: defect | Status: new
Priority: major | Milestone:
Component: Trac | Version:
Keywords: |
--------------------+-------------------------------------------------------
= bug description =
F-16 alpha shipped witha boot.iso that did not have isohybrid run on it,
this means you can not just dd it to a usb drive and boot it.
= bug analysis =
slipped though as there was no test case
= fix recommendation =
lorax has been fixed, but we should make sure we cover it in testing.
--
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/236>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance
12 years, 4 months
Is there a need for more Xen test cases for Fedora?
by Adam Williamson
Hey, folks. I'm working through the f16 QA retrospective, and one of the
suggestions is:
"might need improved / more Xen test cases"
I wanted to reach out particularly to those who've been involved with
the Xen support and testing, but also the group in general, and see if
this is correct - should we have more or improved Xen test cases to
ensure the Xen support is good for F17 and future releases?
What we have ATM is
https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Xen_Para_Virt -
I note that Konrad modified this a few weeks back, and it looks better
than it did previously. Is this enough, or should we enhance this test
case, add more, replace it...? Thanks!
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
12 years, 4 months
A Glorious Vision of Our Shared Update Feedback Future (bodhi, karma, and proventesters, oh my)
by Adam Williamson
Hey, folks.
So in the recent proven tester discussion, and in various other threads,
I've oft stated that the limits of the current Bodhi karma system are a
significant problem, and the planned Bodhi 2.0 karma system has to
potentially to significantly improve our update testing process. But it
occurred to me that I haven't really laid out why in much detail, and
those who aren't as involved with the process as we in QA are might not
have a really clear vision of why this is so important.
So, I thought I'd lay it out in the form of a glorious vision of the
future! Note: I have zero UI design skills. This UI as described would
suck. But the idea is that there would be a decent UI which *represents
all the described choices*.
In the Great Bodhi Of The Future, for any given update, a tester will
not simply have a comment box and a drop-down for -1, 0 or +1. They will
see:
* The list of test cases associated with the package, with a PASS / FAIL
choice for each
* Checkboxes for 'This update does / does not break critical path
functionality' (with a link out to the critpath definition)
* A checkbox for 'I installed this update and continued to use my system
as normal and did not note any regressions'
* Any custom choices the package maintainer opts to provide, via some
kind of interface to Bodhi
This is the kind of flexibility that would make karma massively more
useful. The tighter definition of what feedback actually *means*
provides far more information to the maintainer, and enables us to
automate certain outcomes much more aggressively.
For me, one the principal benefits of such a system would be that we
could make the 'This update breaks critical path functionality' checkbox
an absolutely red flashing light, wailing siren emergency button. I
mean, you hit that thing and trucks roll from Fedora HQ, metaphorically
speaking. It would have a confirmation page which clearly described the
impact of asserting that an update broke critpath, so we could be
confident that it didn't get falsely triggered very often. I'd suggest
that:
1. Any update that is marked as 'critpath breaking' by a FAS-registered
tester would be blocked from going any further in the update process
without manual intervention (no autopushes at all)
2. Any update marked as 'critpath breaking' by a proven tester would be
blocked from being pushed stable at all - automatically or manually -
until the PT modified the feedback or it was overridden by someone with
appropriately godlike powers (TBD, but probably not just the maintainer)
3. Any update marked as 'critpath breaking' should probably get
announced on at least test-announce and/or devel-announce
4. Any update marked as 'critpath breaking' *after it has already been
pushed* would similarly trigger a major response: notify maintainer very
hard, notify lists, generally do stuff to make sure it gets immediate
attention
We would also obviously be able to offer maintainers more nuanced
options for autopushing updates - require a certain number of passes on
a certain set of test cases, for instance.
To go a bit into the theory, the really nasty limitation of the current
system is we simply have very little easily consumable indication of
what the karma provided on an update *means*. A +1 can mean anything
from 'I booted and nothing exploded' to 'I tested every line of this
code, then did it backwards standing on my head'. A -1 can mean 'I
booted this update and then my cat got sick, I think there's a
connection!', 'this update breaks $REALLY OBSCURE FEATURE Z', or 'this
update exploded my monitor'. We just _don't know_. You can get the
information from comments, usually, but that's obviously a complete
non-starter for any kind of programmatized or automated action based on
the feedback. In particular, we currently can't do anything dramatic
based on negative feedback because of the uncertainty around exactly
what it means. We do have a few policies about when to file negative
feedback, but even this only very slightly mitigates the problem - we
can't say 'only file negative feedback if the update breaks critpath',
that's just not feasible. So we can't identify the 'really really
negative' feedback and respond appropriate drastically to it without
causing lots of pain through false positives (or rather, false
negatives...)
With a more advanced feedback system we can identify the really big
problems and be much more aggressive about handling them, without
over-reacting to smaller problems. We could, correspondingly, be a bit
less strict about how much *positive* feedback you need to push an
update, if we can be a bit more confident that we'll definitely identify
the really bad problems through negative feedback.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
12 years, 4 months
I/O Bound Procs Clobbering Responsiveness
by Chuck Forsberg WA7KGX
This a subjective comment, but it seems that I/O bound procs
such as disk copy are seriously impacting GUI response more
than before. For example, switching windows can take 30 seconds
or more. Indeed it appears the system has halted except that the
clock seconds keep ticking away.
Setup: R6550 8GB Fedora 16 64 bit accessed via TigerVNC over
a gigabit LAN.
--
Chuck Forsberg WA7KGX N2469R caf(a)omen.com www.omen.com
Developer of Industrial ZMODEM(Tm) for Embedded Applications
Omen Technology Inc "The High Reliability Software"
10255 NW Old Cornelius Pass Portland OR 97231 503-614-0430
12 years, 4 months
[Test-Announce] Very belated 2011-09 Graphics Test Week recap
by Adam Williamson
I know this is horribly late, but I just worked my way around to it!
Sorry for that.
Here's the recap of the Fedora 16 Graphics Test Week. The overall
numbers of tests done and bugs filed are down significantly, because I
was very busy and did not do a great job of arranging and promoting the
events this cycle. I hope we'll be able to do a better job for the
Fedora 17 graphics test week.
Here's the numbers on tests conducted and bugs filed. I counted each
non-example line in the main results matrix as a 'test conducted'. I
didn't count the runs in the 'bonus' 3D matrix, in order to provide a
more fair comparison with previous releases. This might affect the
bugs-to-tests ratio somewhat.
f11 nouveau: 104 tests, 42 bugs - ratio 0.40
f12 nouveau: 53 tests, 34 bugs - ratio 0.64
f13 nouveau: 78 tests, 26 bugs - ratio 0.33
f14 nouveau: 39 tests, 8 bugs - ratio 0.21
f15 nouveau: 83 tests, 55 bugs - ratio 0.66
f16 nouveau: 16 tests, 8 bugs - ratio 0.50
f11 radeon: 55 tests, 46 bugs - ratio 0.84
f12 radeon: 61 tests, 81 bugs - ratio 1.33
f13 radeon: 48 tests, 33 bugs - ratio 0.69
f14 radeon: 32 tests, 18 bugs - ratio 0.56
f15 radeon: 66 tests, 38 bugs - ratio 0.58
f16 radeon: 12 tests, 8 bugs - ratio 0.67
f11 intel: 23 tests, 21 bugs - ratio 0.91
f12 intel: 29 tests, 31 bugs - ratio 1.07
f13 intel: 38 tests, 38 bugs - ratio 1.00
f14 intel: 33 tests, 28 bugs - ratio 0.84
f15 intel: 37 tests, 25 bugs - ratio 0.68
f16 intel: 10 tests, 7 bugs - ratio 0.70
It's hard to draw any conclusions from such small numbers, but really,
none of them are far out of line with previous events.
Here's the chart for how reported bugs are handled, adding the results
from the F15 events (this chart gets updated on a one-cycle delay,
because it's concerned with how the bugs filed are handled after a
reasonable amount of time for the developers to look at them). A full
explanation of what this tracks and how it's calculated in the F13 recap
at
https://lists.fedoraproject.org/pipermail/test/2010-April/090271.html .
Some resolutions showed up this time that didn't before. I counted
'WORKSFORME' in with 'closeddupe' - i.e. as part of the set of bugs that
should just be completed discarded from consideration. I handled
'UPSTREAM' bugs on a case-by-case basis, going in and look at whether
they'd actually been fixed upstream or what, and putting them in the
most appropriate group. Also, if you try to reproduce these numbers,
note that you'll want to remove the bugs 123456 and 234567 from the
lists - they're used in the 'example' entries in the results matrix. The
bug totals are slightly different from the above chart because I
copy/pasted the F15 numbers in the above chart from the last recap, but
the numbers in this chart are newly generated, and so a few bugs that
were filed and added to the Wiki pages *after* the last recap went out
have been added to the numbers in this chart.
f11 nouveau: 42 bugs, 4 open, 8 closeddupe, 24 closedfixed, 6 closedunfixed - 70.59%
f12 nouveau: 34 bugs, 11 open, 8 closeddupe, 14 closedfixed, 1 closedunfixed - 53.85%
f13 nouveau: 27 bugs, 17 open, 6 closeddupe, 3 closedfixed, 1 closedunfixed - 14.29%
f14 nouveau: 8 bugs, 5 open, 3 closeddupe - 0% (small sample)
f15 nouveau: 58 bugs, 27 open, 13 closeddupe, 13 closedfixed, 5 closedunfixed - 28.89%
f11 radeon: 46 bugs, 14 open, 10 closeddupe, 19 closedfixed, 3 closedunfixed - 52.78%
f12 radeon: 81 bugs, 19 open, 32 closeddupe, 28 closedfixed, 2 closedunfixed - 57.14%
f13 radeon: 36 bugs, 28 open, 3 closeddupe, 5 closedfixed, 0 closedunfixed - 15.15%
f14 radeon: 18 bugs, 13 open, 0 closeddupe, 3 closedfixed, 2 closedunfixed - 16.67%
f15 radeon: 38 bugs, 17 open, 10 closeddupe, 9 closedfixed, 2 closedunfixed - 32.14%
f11 intel: 21 bugs, 7 open, 1 closeddupe, 12 closedfixed, 1 closedunfixed - 60%
f12 intel: 31 bugs, 7 open, 12 closeddupe, 12 closedfixed, 0 closedunfixed - 63.16%
f13 intel: 42 bugs, 26 open, 4 closeddupe, 11 closedfixed, 1 closedunfixed - 28.95%
f14 intel: 28 bugs, 21 open, 4 closeddupe, 1 closedfixed, 2 closedunfixed - 4.17%
f15 intel: 27 bugs, 8 open, 7 closeddupe, 12 closedfixed, 0 closedunfixed - 60%
There's definitely good news here - as noted in the last recap, we had a
good crop of reports from the F15 event, so there's no reason we
shouldn't be able to get back up to this level with better organization
and promotion for F17. Also, the number of bugs actually getting *fixed*
- which is the ultimate goal of the event - was significantly higher
than F13 and F14, not quite back to F11/F12 levels, but a lot better. So
the trend of fewer bugs getting fixed seems to have been stopped during
F15 cycle. Intel showed a particularly marked improvement here. (It's
worth noting that some of the bugs filed and fixed were actually on
GNOME rather than the X drivers; I didn't compare the level of non-X
bugs in the results in other releases, so that is an untracked variable.
If anyone wants to look into that, please do!)
Many thanks as always to all those who contributed tests and to our
plucky X developers and triagers, especially:
Adam Jackson
Dave Airlie
Jerome Glisse
Ben Skeggs
Matej Cepl
The raw list of bugs filed at each of the F16 events follows.
Nouveau
-------
735703 NEW - VT now working properly with the nouveau driver
692035 NEW - [NV94] Dual-head problems: Nouveau on FC14 fails to drive
both outputs of Nvidia 9600GT correctly.
735702 NEW - [NV67] fast switch user was not working
735893 NEW - [NV42] Shadows on rendering
736323 NEW - startx starts fallback mode
736837 CLOSED WORKSFORME - Rotate display fails on GTX 560
737850 CLOSED DUPLICATE - nouveau testday: focus issues while glx
testcase (NV43)
735770 CLOSED NOTABUG - [NVa8] External monitor doesn't work after upgrade to F16
Radeon
------
735703 NEW - VT now working properly with the nouveau driver
698711 NEW - [Crestline] Generic video multihead test failure
735702 NEW - [NV67] fast switch user was not working
733857 NEW - No display with AMD A6-3650 APU (ATI HD 6530D)
736498 NEW - [RV635] Testcase radeon xv failed
736562 NEW - [RS740] Gnome 3 dualhead fallback mode
739078 CLOSED ERRATA - Booting to runlevel 3 failed
698011 CLOSED UPSTREAM - [RS690] LiveCD doesn't boot into Gnome3. Only get background, no panel. Mouse clicks ignored.
Intel
-----
736859 NEW - dual-head: re-arranging screen leaves one screen off
736358 NEW - Gtk-WARNING **: Unknown property: GtkDialog.has-separator
735702 NEW - [NV67] fast switch user was not working
734265 NEW - [abrt] gnome-shell-3.1.4-2.gite7b9933.fc16: __GI_raise: Process /usr/bin/gnome-shell was killed by signal 6 (SIGABRT)
736323 NEW - startx starts fallback mode
736330 NEW - [Cantiga] Internal display turned off when playing game in dual monitor setup
736846 CLOSED CURRENTRELEASE - user switching with color manager enabled causes segfaults and gnome "Oh no..."
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
_______________________________________________
test-announce mailing list
test-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/test-announce
12 years, 5 months
