<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<title></title>
</head>
<body text="#000000" bgcolor="#ffffff">
Thanks for the stats and information. <br>
<br>
How big is the gap in testing. <br>
<br>
Is there a significant amount of package releases etc walked back
because after they passed minimum time in QA and were published it
turned out they were broken ? Ie. percent wise or some other
metric or in the absence of that a gut assessment.<br>
<br>
Are certain areas in more need of focus than others due to
criticality and lack of testing/testers ? If so what areas are
those ?<br>
<br>
Wanting to get a handle on things around here so I can understand
where I can be most effective in helping out. I want to look into
the automated qa testing via this link<a
href="https://fedoraproject.org/wiki/AutoQA">
https://fedoraproject.org/wiki/AutoQA</a> <br>
<br>
Thanks again.<br>
<br>
<br>
<br>
On 02/15/2012 10:16 PM, Adam Williamson wrote:
<blockquote cite="mid:1329362190.3482.11.camel@adam" type="cite">
<pre wrap="">On Wed, 2012-02-15 at 21:35 -0500, Vincent L. wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 02/13/2012 09:30 PM, Bruno Wolff III wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Note that statistics are still gathered and that future changes might depend
on whether or not proventesters do a better job than average of correctly
tagging builds as good or bad.
</pre>
</blockquote>
<pre wrap="">
Probably stating the obvious, and I am new around here, but the biggest
challenge I see is that testing is not well defined. Certainly for the
core items standard regressions or checklists of what items should be
validated etc do not seem to be present [ or at least i can't find any
]. This naturally leads to inconsistent approaches to testing from
tester to tester.
There are a lot of packages, and likely a lack of staffing/volunteers to
develop and maintain testplans. However as in most commercial release
management having these things would help ensure each tester validated
things in a similar fashion and ensure better release quality.
</pre>
</blockquote>
<pre wrap="">
Yes, this is broadly the problem.
We have a system in place that allows you to create a test plan for a
package and have it show up in the update request. See it in action at
<a class="moz-txt-link-freetext" href="https://admin.fedoraproject.org/updates/FEDORA-2012-1766/dracut-016-1.fc17">https://admin.fedoraproject.org/updates/FEDORA-2012-1766/dracut-016-1.fc17</a> - note the links to test cases - and details on how to actually set up the test cases to make this work are at <a class="moz-txt-link-freetext" href="https://fedoraproject.org/wiki/QA:SOP_package_test_plan_creation">https://fedoraproject.org/wiki/QA:SOP_package_test_plan_creation</a> . We don't have test plans for many packages, really because of the resource issue. Jon Stanley did suggest he might work on this as his 'board advocacy' task.
</pre>
<blockquote type="cite">
<pre wrap="">May I ask how many "proventesters" there are ballpark -vs- how many
approximate testers of standard status participate at any given time ?
</pre>
</blockquote>
<pre wrap="">
We can say with precision how many proven testers there are, because
there's an associated FAS group - there are 90 members of the
'proventesters' group in FAS. Active non-proven testers is a bit harder
to count, but Luke can generate Bodhi statistics. There's one fairly
'famous' set from 2010 here:
<a class="moz-txt-link-freetext" href="https://lists.fedoraproject.org/pipermail/devel/2010-June/137413.html">https://lists.fedoraproject.org/pipermail/devel/2010-June/137413.html</a>
There's a less famous report from March 2011 here:
<a class="moz-txt-link-freetext" href="http://lmacken.fedorapeople.org/bodhi-metrics-20110330">http://lmacken.fedorapeople.org/bodhi-metrics-20110330</a>
you can get some numbers from. At the time of the 2011 report it seems
like there was a roughly 1:10 proventester/regular tester ratio for F15
and F14, but it does seem to be slightly unclear.
</pre>
</blockquote>
<br>
</body>
</html>