[Test-Announce] Call for reviewing TCMS use cases and comparison!

[He Rui]

On Mon, 2011-01-24 at 23:05 -0800, Adam Williamson wrote:
> On Fri, 2011-01-21 at 19:02 +0000, Samuel Greenfeld wrote:
> 
> >         > I do like litmus!  It's a nice evolution from testopia for
> >         upstream
> >         > mozilla.  We don't currently have an 'unclear' test result.
> >          I'm not
> >         > opposed to it, but would need better understand how that
> >         field is used,
> >         > and the process around it, in litmus.
> >         
> >         
> >         Agree with James.
> > 
> > What I believe Mozilla is doing (since I have not had a chance to work
> > with their QA team yet) is flagging test cases with a form of soft
> > failure in that the result of a testcase neither clearly passed, nor
> > clearly failed.  So in addition to "Passed", "Failed", and any other
> > common states (Blocked, In Progress, etc.) you have an "Unclear"
> > result state.

I didn't receive the mail replied by Samuel. Weird. 

> Hurry is somewhat wrong to say we don't currently have an 'unclear'
> result; we do have the 'warn' result, which is in some ways similar. We
> usually use it to indicate when a test turns up some kind of anomalous
> behaviour which isn't exactly a failure.

It depends on what 'unclear' means here and how it reflects the results.
If you mean a soft failure or an issue that doesn't block the case run,
the 'warn' result is similar to it, then calling it 'unclear' is
confusing and not accurate in my opinion.

In nitrate system, it has 'blocked', 'failed' and 'error' result status
to reflect a problem. User guide suggests 'error' is used for test
environment that has problems that prevent Test Case execution. I think
we can modify 'error' status to include all soft failures. 

I've added this to the requirements for further evaluation.  

Thanks,
Hurry
-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: 86-010-62608141
IRC nick: rhe #fedora-qa #fedora-zh