[Fedora QA] #152: Test Cases Management

[Karol Cieśla]

Hi,
i agree with both of you ,so that rather than just sending e-mails i 
would like initiate something which could improve the current situation.

So the steps I see now look as follows:
*I stage*
_1)_ someone could sketch out/outline/describe the functionality of 
Wiki/TCMS we would like to have for the test cases (if possible also 
provide graphical form) -i.e table, page, buttons
_2)_ discuss with the whole community such form -pros,cons
_3)_ reach out to people maintaining the Wiki and ask them if they can 
create such form wit help of Wiki/TCMS and can sustain bigger number of 
test cases (i.e up to 300)

*II stage*

To ponder as the whole community the process of 
adding/creating/deleting/marking/prioritizing  test cases.

P.S -i can try to prepare for the middle of the next week some draft to 
point 1 -it will be something simple but reflecting the idea.


Regards,
Karol



W dniu 2010-12-03 17:36, James Laska pisze:
> Just a few extra thoughts on the subject ...
>
> On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
>> On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
>>> Hi,
>>> yes, you are right there are tests, but in my opinion they are in few
>>> different places under different  categories.
>> That wasn't what I meant: I meant we already use the Wiki for the
>> purposes you identified as an advantage of a TCMS (listing the tests
>> that need to be performed in relation to some specific process, and
>> whether they have already been performed, by whom, and with which
>> result).
>>
>>> I think we could organise them better -i.e create test category and put
>>> all of them instead of many places.
>> We sure could, but we don't necessarily need a TCMS to do this. :) Note
>> that we do try to keep them all within one Wiki namespace and we do use
>> Wiki categories to organize some test cases.
>>
>>> Moreover, i just have taken a look briefly and i see there are round 100
>>> test cases in total (please correct me if i am wrong).I think that for
>>> such project/system it is not enough at all.
>>>
>>> We have big community, let`s assume everyone from QA create one test, we
>>> will have quite huge number of tests and obviously more faults detected
>>> before main release, less corrections after=better stability,usability->
>>> better overall opinion.
>> Sure, we can always do with more test cases.
> More test cases/plans would certainly change the conversation a bit.  I
> think we all want to increase the value that the Fedora QA team can
> offer to the project.  One way to increase our value is by improving our
> test coverage by way of test documentation (procedures, plans and
> cases).  There are plenty of other ways ... but we can save those for
> other threads.
>
> I've always been hesitant to add tests for the sake of adding tests.
> Test plans/cases are just like software.  If the tests aren't addressing
> a priority issue, they won't be used as much, and like unused software,
> will suffer from bit rot.  The best test cases/plans are the ones
> frequently used, referenced and have maintainer buy-in.  Meaning, if the
> tests fail, the maintainer cares.  I want to grow the library of tests
> we maintain and run, but hopefully grow in a manner and pace that we, as
> a community, can sustain.
>
> With the test plans that Adam points to, I'm pretty confident in our
> ability to develop, discuss/debate and execute desktop and installation
> tests as a community.  We've ironed out the kinks in the workflow,
> increased community engagement and developed good test plans as a
> result.  My impression is we are ready for additional test areas.
>
> That's what's exciting to me about the proventesters effort.  As you can
> tell from recent (and old) devel@ list threads, testing proposed updates
> is important work that's needed, requested by package maintainers and
> well under-documentated.  I don't worry as much that tests written for
> frequent proventester use will go stale given it's been a long-standing
> exposure in the project.  Also, given the huge number of components in
> Fedora, there is room for just about every contributor to participate
> and carve out a niche.  But which tests do we prioritize first, where do
> we write the tests, where to review+discuss them, how to run them etc...
> (more on this later).
>
> For me, these are two separate (but related) efforts.  TCMS is tool
> designed to address specific workflow/tracking needs.  We also need to
> determine how to best to sustainably expand the test coverage we can
> offer to the project.  We have a wiki-based "TCMS" now.  It has met our
> needs for the current set of organized test efforts.  It's not perfect,
> but the return on the investment has been huge.  The questions I'd like
> to see answered in ticket#152, is (1) whether the wiki can continue to
> scale as our test management needs grow, and (2) what aspects of our
> wiki-based TCMS are good/bad?
>
> Thanks,
> James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.fedoraproject.org/pipermail/test/attachments/20101203/aada567b/attachment.html