<br><br><div class="gmail_quote">On Fri, Jan 21, 2011 at 11:40 AM, He Rui <span dir="ltr"><<a href="mailto:rhe@redhat.com">rhe@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
> > Also: It might be useful to add an "unclear" testcase result, similar<br>
<div class="im">> > to how Mozilla's Litmus system does it (<a href="https://litmus.mozilla.org" target="_blank">https://litmus.mozilla.org</a>).<br>
><br>
</div>> I do like litmus! It's a nice evolution from testopia for upstream<br>
<div class="im">> mozilla. We don't currently have an 'unclear' test result. I'm not<br>
> opposed to it, but would need better understand how that field is used,<br>
> and the process around it, in litmus.<br>
<br>
</div>Agree with James.<br></blockquote></div><br>What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to "Passed", "Failed", and any other common states (Blocked, In Progress, etc.) you have an "Unclear" result state.<br>
<br>This forces the result (including a comment section I presume is forced for all non-passing items) to be evaluated during the release cycle to see if it was an actual failure, or if the test case needs to be updated and/or clarified.<br>
<br>See the "recently unclear" and "testcases most frequently marked as unclear" links at the bottom of pretty much every page of <a href="http://litmus.mozilla.org">litmus.mozilla.org</a> for examples. On the former, you can click on the dialog icon beneath each result ID to see the comment without switching pages.<br>
<br>Personally I like this approach because management of many projects (in my experience) rarely budget time for test cases to be updated. They prefer that you start testing the next release instead. Projects I have been on have required me to use test cases which were last notably updated a few years ago if not 5-10+ years ago by simply acting as if the inaccurate information is not there. By treating potentially incorrect test cases as a form of failure it forces them to be ideally updated and rerun promptly.<br>
<br>The downside is if a tester is unfamiliar with an area or needs hand-holding, they might mark a lot of results unclear which actually are rather straightforward. Or alternatively, no one looks at them, and you have a muddy result at the end of the release. These would put additional effort on the core QA team to remedy the situation.<br>