[Fedora QA] #152: Test Cases Management

[James Laska]

Just a few extra thoughts on the subject ...

On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
> On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
> > Hi,
> > yes, you are right there are tests, but in my opinion they are in few 
> > different places under different  categories.
> 
> That wasn't what I meant: I meant we already use the Wiki for the
> purposes you identified as an advantage of a TCMS (listing the tests
> that need to be performed in relation to some specific process, and
> whether they have already been performed, by whom, and with which
> result).
> 
> > I think we could organise them better -i.e create test category and put 
> > all of them instead of many places.
> 
> We sure could, but we don't necessarily need a TCMS to do this. :) Note
> that we do try to keep them all within one Wiki namespace and we do use
> Wiki categories to organize some test cases.
> 
> > Moreover, i just have taken a look briefly and i see there are round 100 
> > test cases in total (please correct me if i am wrong).I think that for 
> > such project/system it is not enough at all.
> > 
> > We have big community, let`s assume everyone from QA create one test, we 
> > will have quite huge number of tests and obviously more faults detected 
> > before main release, less corrections after=better stability,usability-> 
> > better overall opinion.
> 
> Sure, we can always do with more test cases.

More test cases/plans would certainly change the conversation a bit.  I
think we all want to increase the value that the Fedora QA team can
offer to the project.  One way to increase our value is by improving our
test coverage by way of test documentation (procedures, plans and
cases).  There are plenty of other ways ... but we can save those for
other threads.

I've always been hesitant to add tests for the sake of adding tests.
Test plans/cases are just like software.  If the tests aren't addressing
a priority issue, they won't be used as much, and like unused software,
will suffer from bit rot.  The best test cases/plans are the ones
frequently used, referenced and have maintainer buy-in.  Meaning, if the
tests fail, the maintainer cares.  I want to grow the library of tests
we maintain and run, but hopefully grow in a manner and pace that we, as
a community, can sustain.

With the test plans that Adam points to, I'm pretty confident in our
ability to develop, discuss/debate and execute desktop and installation
tests as a community.  We've ironed out the kinks in the workflow,
increased community engagement and developed good test plans as a
result.  My impression is we are ready for additional test areas.

That's what's exciting to me about the proventesters effort.  As you can
tell from recent (and old) devel@ list threads, testing proposed updates
is important work that's needed, requested by package maintainers and
well under-documentated.  I don't worry as much that tests written for
frequent proventester use will go stale given it's been a long-standing
exposure in the project.  Also, given the huge number of components in
Fedora, there is room for just about every contributor to participate
and carve out a niche.  But which tests do we prioritize first, where do
we write the tests, where to review+discuss them, how to run them etc...
(more on this later).

For me, these are two separate (but related) efforts.  TCMS is tool
designed to address specific workflow/tracking needs.  We also need to
determine how to best to sustainably expand the test coverage we can
offer to the project.  We have a wiki-based "TCMS" now.  It has met our
needs for the current set of organized test efforts.  It's not perfect,
but the return on the investment has been huge.  The questions I'd like
to see answered in ticket#152, is (1) whether the wiki can continue to
scale as our test management needs grow, and (2) what aspects of our
wiki-based TCMS are good/bad?

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20101203/7617fd96/attachment.bin