[Test-Announce] Call for reviewing TCMS use cases and comparison!

[He Rui]

Greetings,

Many thanks for Samuel's comments, I found out many valuable things to
think, identify and add to use cases/feature comparison.

The questions also remind me that many testers so far may be still not
familiar with Nitrate, therefore can't give any suggestion upon it. So
I'll introduce Nitrate a bit here. 

Nitrate is a new test case management system which is open-sourced
recently. It has advantages on managing, tracking and querying
cases,results and is similar to testopia. The introduction pages along
with some screenshots can be found at:

https://fedorahosted.org/nitrate/
https://fedoraproject.org/wiki/Nitrate


One can set up a local nitrate system with the steps in the link, we are
also planing to build a testing instance during F-15.   


On Thu, 2011-01-20 at 15:02 -0500, James Laska wrote:
> On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote:
> > A few more inline comments on a subset of the questions, and two more thoughts:
> > 
> > On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jlaska at redhat.com> wrote:
> > > On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
> > > >      1. What is the history of Nitrate and the Fedora Project?  What
> > > >         does the Fedora project expect to gain from using it?
> > >
> > > This goes back to an eval we did using testopia in Fedora many releases
> > > agove.  Unfortunately, the effort was canceled due to license
> > > incompatibility between Fedora and testopia.  At that point, we invested
> > > in leveraging the wiki to best of our ability.
> > >
> > >  https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
> > >  https://bugzilla.redhat.com/show_bug.cgi?id=450013
> > 
> > >From the bug I'm guessing this is because Testopia used Ext-JS, and
> > Ext-JS kept changing licenses.  Hopefully there will only be licensing
> > restrictions for Nitrate on the software itself, and not the created
> > work of what one does while actively using it.
> 
> That was a question I had opened with Hurry.  I have no idea how we'd
> license the content in the system.  If anything, I'd hope/expect that to
> match that of our current wiki.

Oh, licensing and license issues. I've listed both of them in feature
comparison as requirements. For test days and release validation test
events use cases, I identified the steps with different permissions by
QA, event host and tester(including anonymous user)[#], and hope this
can help to license these instances in Nitrate. Of course, I need to
extend more to cover other instances. 

[#]
https://fedoraproject.org/wiki/Rhe/tcms_use_cases#Test_Events_Use_Cases

> 
> > > >      1. How does Nitrate compare to other open & closed sourced TCMS
> > > >         solutions?  Why was it written as opposed to using an existing
> > > >         solution, and what are its strengths & weaknesses?
> > >
> > > See history comments above.  Also, maybe the nitrate developers [1] can
> > > offer more insight on how it compares to other open-source solutions?  I
> > > *think* that comparison work has been done in the past, I'm just not
> > > sure where to find the results.
> > >
> > > [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel
> > 
> > Are the developers actively monitoring/using this list?  I looked at
> > it before, and all I saw were three test posts from July in the
> > archives.
> 
> I believe they do, but don't yet have a strong upstream presence since
> there hasn't been a lot of code/progress to share until recently.  This
> will be something I'm sure they'll want to improve as community interest
> increases.

Add it into requirements. But I can confirm that all the nitrate
developers are monitoring the list now. When community interest
increases, it will become active. :)  

> 
> > > >      1. Can multiple projects share test cases, and even reference
> > > >         older versions of test cases if they are lagging behind the
> > > >         current rawhide/Fedora release?  Will Spins be able to make
> > > >         their own (simultaneously running) test plans?
> > >
> > > This is the hope.  It's not really useful if we can only use it for
> > > release validation.  I don't think we've fully explored how best to
> > > model other spins/projects, but I don't foresee big problems there.
> > > That will be fun to explore on the sandbox/staging instance.
> > >
> > > With regards to referencing older versions of a test case, I believe
> > > that support is there, although I'm not certain it's right for our
> > > needs.  Keeping test documentation (plans and cases) updated is a pretty
> > > sizable maintenance challenge.  I've seen many instances where support
> > > for versioned test cases allows test plans to suffer over time as they
> > > were linked against old and inaccurate test cases.
> > >
> > > Much like how the wiki is used now, we have support for linking against
> > > older versions of tests (wiki history), but we rarely ever use that
> > > feature.  I expect that trend would continue in the short-term.
> > 
> > The reason I bring this up is because OLPC's Spin releases tend to lag
> > behind the official Fedora release.  For instance, we just released
> > our hopefully final Fedora 11-based release this past month, and are
> > in the early stages of the Fedora 14-based one.  At least some Fedora
> > ARM development work may still be going on with Fedora 13 as well.
> > 
> > While decently written test cases will survive somewhat over time,
> > major changes in GUI look & feel or other areas can break their
> > backwards compatibility.
> > 
> > Ideally Nitrate will default to using the current version of a
> > testcase when making a new plan, preferably following the updates of
> > said testcase until a result is committed which forces the test case
> > version to be needed.
> 
> I hope so, yes! :)  I'm fairly certain support exists for linking
> against versioned cases, that is, I know it was present in testopia.
> Hopefully Hurry, or the nitrate folks can help here.  Sounds like we
> need to add this to our list of requirements.

In Nitrate, the text part of a test case is versioned, and one can view
different text versions of the case but can't reference old versions to
test run. testopia can, but nitrate abandoned it. I've listed it in the
requirements for valuating. 

> 
> > > >      1. How long will historical test case results be made available?
> > >
> > > I suspect the limiting factor here is database size.  I'm not aware of
> > > any rules or process that would require removal/archival of old results.
> > > However, at some point that could certainly be an issue we'd need to
> > > plan for.
> > 
> > Again; this would be to help lagging Spins and similar.  Right now it
> > looks like Bodhi at least publicly hides information for Fedora
> > versions which have reached end-of-life.  (Either that, or I don't
> > know how to find it.)
> 
> That or garbage collected.  
> 
> Sounds like we'll need to do some further research on this front.  I
> personally see no reason to purge results for EOL'd releases.  I've
> often referenced old test results to understand when the last time
> something was tested.  Test results aren't disk-hogs like builds
> (*cough* oo.org *cough*) can be, so I don't expect this to be as much of
> an issue.  Nonetheless, you're point is valid.

Add it to the requirement list.
> 
> > I presume that Nitrate has already been determined to be scalable;
> > otherwise it is a big risk to incorporate it into Fedora.
> 
> I'll let the nitrate folks speak to scalability issues.  That's
> definitely something we'll want to explore during a pilot.  Remember
> too, I don't expect we'll throw away the wiki, and switch to nitrate
> without a good bake-in period to identify bugs/gaps etc...
> 
> > Also: It might be useful to add an "unclear" testcase result, similar
> > to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
> 
> I do like litmus!  It's a nice evolution from testopia for upstream
> mozilla.  We don't currently have an 'unclear' test result.  I'm not
> opposed to it, but would need better understand how that field is used,
> and the process around it, in litmus.

Agree with James. 

> 
> Thanks,
> James
> -- 
> test mailing list
> test at lists.fedoraproject.org
> To unsubscribe: 
> https://admin.fedoraproject.org/mailman/listinfo/test

In addition, I've also added 'Integration between our update tools
(bodhi/f-e-k) and our test infrastructure' to requirements for
evaluation.

Feel free to tell if there's any concern.


Thanks,
Hurry

-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: 86-010-62608141
IRC nick: rhe #fedora-qa #fedora-zh