[Test-Announce] Call for reviewing TCMS use cases and comparison!

[Samuel Greenfeld]

A few more inline comments on a subset of the questions, and two more thoughts:

On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jlaska at redhat.com> wrote:
> On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
> >      1. What is the history of Nitrate and the Fedora Project?  What
> >         does the Fedora project expect to gain from using it?
>
> This goes back to an eval we did using testopia in Fedora many releases
> agove.  Unfortunately, the effort was canceled due to license
> incompatibility between Fedora and testopia.  At that point, we invested
> in leveraging the wiki to best of our ability.
>
>  https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
>  https://bugzilla.redhat.com/show_bug.cgi?id=450013

>From the bug I'm guessing this is because Testopia used Ext-JS, and
Ext-JS kept changing licenses.  Hopefully there will only be licensing
restrictions for Nitrate on the software itself, and not the created
work of what one does while actively using it.


> >      1. How does Nitrate compare to other open & closed sourced TCMS
> >         solutions?  Why was it written as opposed to using an existing
> >         solution, and what are its strengths & weaknesses?
>
> See history comments above.  Also, maybe the nitrate developers [1] can
> offer more insight on how it compares to other open-source solutions?  I
> *think* that comparison work has been done in the past, I'm just not
> sure where to find the results.
>
> [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel

Are the developers actively monitoring/using this list?  I looked at
it before, and all I saw were three test posts from July in the
archives.


> >      1. Can multiple projects share test cases, and even reference
> >         older versions of test cases if they are lagging behind the
> >         current rawhide/Fedora release?  Will Spins be able to make
> >         their own (simultaneously running) test plans?
>
> This is the hope.  It's not really useful if we can only use it for
> release validation.  I don't think we've fully explored how best to
> model other spins/projects, but I don't foresee big problems there.
> That will be fun to explore on the sandbox/staging instance.
>
> With regards to referencing older versions of a test case, I believe
> that support is there, although I'm not certain it's right for our
> needs.  Keeping test documentation (plans and cases) updated is a pretty
> sizable maintenance challenge.  I've seen many instances where support
> for versioned test cases allows test plans to suffer over time as they
> were linked against old and inaccurate test cases.
>
> Much like how the wiki is used now, we have support for linking against
> older versions of tests (wiki history), but we rarely ever use that
> feature.  I expect that trend would continue in the short-term.

The reason I bring this up is because OLPC's Spin releases tend to lag
behind the official Fedora release.  For instance, we just released
our hopefully final Fedora 11-based release this past month, and are
in the early stages of the Fedora 14-based one.  At least some Fedora
ARM development work may still be going on with Fedora 13 as well.

While decently written test cases will survive somewhat over time,
major changes in GUI look & feel or other areas can break their
backwards compatibility.

Ideally Nitrate will default to using the current version of a
testcase when making a new plan, preferably following the updates of
said testcase until a result is committed which forces the test case
version to be needed.


> >      1. How long will historical test case results be made available?
>
> I suspect the limiting factor here is database size.  I'm not aware of
> any rules or process that would require removal/archival of old results.
> However, at some point that could certainly be an issue we'd need to
> plan for.

Again; this would be to help lagging Spins and similar.  Right now it
looks like Bodhi at least publicly hides information for Fedora
versions which have reached end-of-life.  (Either that, or I don't
know how to find it.)


I presume that Nitrate has already been determined to be scalable;
otherwise it is a big risk to incorporate it into Fedora.

Also: It might be useful to add an "unclear" testcase result, similar
to how Mozilla's Litmus system does it (https://litmus.mozilla.org).