On Fri, 2010-12-03 at 23:33 +0100, Karol Cieśla wrote:
Hi,
i agree with both of you ,so that rather than just sending e-mails i
would like initiate something which could improve the current
situation.
So the steps I see now look as follows:
I stage
1) someone could sketch out/outline/describe the functionality of
Wiki/TCMS we would like to have for the test cases (if possible also
provide graphical form) -i.e table, page, buttons
Hurry and I discussed this topic yesterday on IRC. I suspect she'll
take a similar approach, but the general idea will be to first identify
what our QA community needs are when it comes to test
documentation/results. For example, what works well, what isn't working
well, where do we want to improve when it comes to our current test
management.
2) discuss with the whole community such form -pros,cons
As always, I expect the process will be transparent and there will be
plenty of opportunity for anyone interested to get involved and help
move things forward.
3) reach out to people maintaining the Wiki and ask them if they can
create such form wit help of Wiki/TCMS and can sustain bigger number
of test cases (i.e up to 300)
If the analysis shows that investing in wiki-based solution is the most
sustainable option forward, there are several solutions available
(
http://www.mediawiki.org/wiki/Extension:Semantic_Forms and possibly
http://www.mediawiki.org/wiki/Extension:StructuredInput). My impression
with Semantic was that while it may scratch the wiki form-input itch for
us, it's certainly not without a startup cost and I'm still unclear
whether that's the right tool for the job. It's a fun experiment, but
without knowing what where our current gaps are, and where we want to
improve, it's hard to make a decision to invest in semantic-mediawiki
over another actively maintained upstream project.
II stage
To ponder as the whole community the process of
adding/creating/deleting/marking/prioritizing test cases.
While it would be an interesting discussion, I don't know if we'd be
able to move forward with actionable work after a general community
ponder session. I'd fear that would revolve too much around ponies [1].
I'm inclined to put emphasis on reviewing what we're doing now,
understand why we're doing it, document the pro's con's and prioritize
the MUSTHAVE features.
P.S -i can try to prepare for the middle of the next week some draft
to point 1 -it will be something simple but reflecting the idea.
Always welcome additional hands to help move the discussion forward!
Thanks,
James
W dniu 2010-12-03 17:36, James Laska pisze:
> Just a few extra thoughts on the subject ...
>
> On Fri, 2010-12-03 at 07:30 -0800, Adam Williamson wrote:
> > On Fri, 2010-12-03 at 16:06 +0100, xcieja wrote:
> > > Hi,
> > > yes, you are right there are tests, but in my opinion they are in few
> > > different places under different categories.
> > That wasn't what I meant: I meant we already use the Wiki for the
> > purposes you identified as an advantage of a TCMS (listing the tests
> > that need to be performed in relation to some specific process, and
> > whether they have already been performed, by whom, and with which
> > result).
> >
> > > I think we could organise them better -i.e create test category and put
> > > all of them instead of many places.
> > We sure could, but we don't necessarily need a TCMS to do this. :) Note
> > that we do try to keep them all within one Wiki namespace and we do use
> > Wiki categories to organize some test cases.
> >
> > > Moreover, i just have taken a look briefly and i see there are round 100
> > > test cases in total (please correct me if i am wrong).I think that for
> > > such project/system it is not enough at all.
> > >
> > > We have big community, let`s assume everyone from QA create one test, we
> > > will have quite huge number of tests and obviously more faults detected
> > > before main release, less corrections after=better
stability,usability->
> > > better overall opinion.
> > Sure, we can always do with more test cases.
> More test cases/plans would certainly change the conversation a bit. I
> think we all want to increase the value that the Fedora QA team can
> offer to the project. One way to increase our value is by improving our
> test coverage by way of test documentation (procedures, plans and
> cases). There are plenty of other ways ... but we can save those for
> other threads.
>
> I've always been hesitant to add tests for the sake of adding tests.
> Test plans/cases are just like software. If the tests aren't addressing
> a priority issue, they won't be used as much, and like unused software,
> will suffer from bit rot. The best test cases/plans are the ones
> frequently used, referenced and have maintainer buy-in. Meaning, if the
> tests fail, the maintainer cares. I want to grow the library of tests
> we maintain and run, but hopefully grow in a manner and pace that we, as
> a community, can sustain.
>
> With the test plans that Adam points to, I'm pretty confident in our
> ability to develop, discuss/debate and execute desktop and installation
> tests as a community. We've ironed out the kinks in the workflow,
> increased community engagement and developed good test plans as a
> result. My impression is we are ready for additional test areas.
>
> That's what's exciting to me about the proventesters effort. As you can
> tell from recent (and old) devel@ list threads, testing proposed updates
> is important work that's needed, requested by package maintainers and
> well under-documentated. I don't worry as much that tests written for
> frequent proventester use will go stale given it's been a long-standing
> exposure in the project. Also, given the huge number of components in
> Fedora, there is room for just about every contributor to participate
> and carve out a niche. But which tests do we prioritize first, where do
> we write the tests, where to review+discuss them, how to run them etc...
> (more on this later).
>
> For me, these are two separate (but related) efforts. TCMS is tool
> designed to address specific workflow/tracking needs. We also need to
> determine how to best to sustainably expand the test coverage we can
> offer to the project. We have a wiki-based "TCMS" now. It has met our
> needs for the current set of organized test efforts. It's not perfect,
> but the return on the investment has been huge. The questions I'd like
> to see answered in ticket#152, is (1) whether the wiki can continue to
> scale as our test management needs grow, and (2) what aspects of our
> wiki-based TCMS are good/bad?
>
> Thanks,
> James
[1]
http://i-want-a-pony.com/