[Test-Announce] Call for reviewing TCMS use cases and comparison!

[James Laska]

On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
+AD4 Unfortunately while I am familiar with a few test case management
+AD4 systems, I have not been involved with the Fedora project long enough
+AD4 to know its workflows.  And a quick search online is not turning up
+AD4 much about Nitrate, with its changelog+AFs-1+AF0 implying it was first made
+AD4 open source software in July 2010.
+AD4 
+AD4 I might be able to provide some general comments though if I knew more
+AD4 about the product.  So for those of us unfamiliar with the history of
+AD4 Nitrate, could you please answer the following:

Really good list of questions Samuel, thanks for jumping in.  I know
Hurry can provide feedback on the questions.  However, I'll add my
thoughts as well.

+AD4      1. What is the history of Nitrate and the Fedora Project?  What
+AD4         does the Fedora project expect to gain from using it?

Hurry can touch on the goals for using nitrate.  As for history, I can
add my experiences ...

This goes back to an eval we did using testopia in Fedora many releases
agove.  Unfortunately, the effort was canceled due to license
incompatibility between Fedora and testopia.  At that point, we invested
in leveraging the wiki to best of our ability.

  https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
  https://bugzilla.redhat.com/show+AF8-bug.cgi?id+AD0-450013

After this, several folks got together and decided they would implement
a new front-end on top of the testopia db schema.  This would resolve
the original license incompatibility and address usability issues that
were raised with the testopia UI.  The project started development
internally, and was open-sourced in 2010.

+AD4      1. Is a sample play/sandbox test instance with more-or-less full
+AD4         access available online? 

Not yet, I believe that's in plan for sometime during Fedora 15.  Hurry
and I were discussing the requirements for such an instance earlier this
week.  If you're interested in helping here, just let us know :)

+AD4      1. How does Nitrate compare to other open +ACY closed sourced TCMS
+AD4         solutions?  Why was it written as opposed to using an existing
+AD4         solution, and what are its straights +ACY weaknesses?

See history comments above.  Also, maybe the nitrate developers +AFs-1+AF0 can
offer more insight on how it compares to other open-source solutions?  I
+ACo-think+ACo that comparison work has been done in the past, I'm just not
sure where to find the results.

+AFs-1+AF0 https://fedorahosted.org/mailman/listinfo/nitrate-devel

+AD4      1. Is test case +ACY test plan import and export (to XML, etc.)
+AD4         support available?  If so is this compatible with any other
+AD4         TCMS's import/export system?

I believe import/export is supported using the testopia.dtd format.

http://git.fedorahosted.org/git/?p+AD0-nitrate.git+ADs-a+AD0-blob+ADs-f+AD0-trunk/nitrate/docs/ChangeLog
- Fixed +ACM-564258 - +AFs-REF+AF0 Ability to export/print specified cases
- Fixed +ACM-612803 - add an export feature for test case runs, can export +ICY

+AD4      1. Are nested test plan and/or test case hierarchies supported?

I don't recall.  Perhaps Hurry or the nitrate developers know?

I know this feature has been discussed a +ACo-lot+ACo with nitrate, and other
solutions we've used in the past.  I don't think support for nested test
plans is something we've had a tremendous need for now, so I don't
anticipate this being a MUSTHAVE feature in the near term.

+AD4      1. Can multiple projects share test cases, and even reference
+AD4         older versions of test cases if they are lagging behind the
+AD4         current rawhide/Fedora release?  Will Spins be able to make
+AD4         their own (simultaneously running) test plans?

This is the hope.  It's not really useful if we can only use it for
release validation.  I don't think we've fully explored how best to
model other spins/projects, but I don't foresee big problems there.
That will be fun to explore on the sandbox/staging instance.

With regards to referencing older versions of a test case, I believe
that support is there, although I'm not certain it's right for our
needs.  Keeping test documentation (plans and cases) updated is a pretty
sizable maintenance challenge.  I've seen many instances where support
for versioned test cases allows test plans to suffer over time as they
were linked against old and inaccurate test cases.

Much like how the wiki is used now, we have support for linking against
older versions of tests (wiki history), but we rarely ever use that
feature.  I expect that trend would continue in the short-term.

+AD4      1. How long will historical test case results be made available?

I suspect the limiting factor here is database size.  I'm not aware of
any rules or process that would require removal/archival of old results.
However, at some point that could certainly be an issue we'd need to
plan for.

+AD4      1. Is there any plan to tie this into Bodhi and other tools to
+AD4         detect updated packages that may imply test cases need
+AD4         re-running and/or updating?

That is certainly possible, but there are currently no detailed plans to
integrate bodhi/f-e-k with nitrate specifically.  Nitrate offers a
robust XMLRPC interface to allow other tools, such as bodhi, to query
for applicable tests/results/runs etc...  So I wouldn't think it would
be a difficult task.

That said, Adam Williamson is exploring this linkage using our current
wiki-based test management system.  While the two implementations will
be completely different, the end result is the same ... improved
integration between our update tools (bodhi/f-e-k) and our test
infrastructure.

+AD4      1. Is this going to be available as a Fedorahosted service like
+AD4         Trac is?  If so will all the instances be able to share test
+AD4         cases?

Hmm, I don't envision multiple hosted instances of nitrate (or any other
test case management) in Fedora.  I expect we would have a single test
management system available for all to use, much like we have a build
system, updates and bug reporting.

+AD4      1. Is there any concern that changing test tracking systems may
+AD4         encourage/discourage existing testers to participate?

Yes+ACE  That's part of why Hurry is investing a lot of time researching
our current workflows and comparing features between our current
wiki-based implementation.

+AD4 Some of this information may be useful to post on the Trac main page
+AD4 and/or in the Fedora wiki.
+AD4 
+AD4 Thanks in advance for your time.

Thank you for your input, good comments/questions+ACE

Thanks,
James
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application+AC8-pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : http:+AC8ALw-lists.fedoraproject.org+AC8-pipermail+AC8-test+AC8-attachments+AC8-20110120+AC8-61af5d08+AC8-attachment.bin