[Test-Announce] Call for reviewing TCMS use cases and comparison!

[He Rui]

Greetings!

Wiki workflow/use cases are reorganized and grouped by general and main
test events(runs) use cases. The general cases cover the basic uses of
wiki, and the events cases covers the detailed certain steps for
organizing the events.

Please review it:

https://fedoraproject.org/wiki/Rhe/tcms_use_cases

    
Meanwhile, Wiki and Nitrate comparison is also listed and grouped by
above use cases. I've listed as many features as I can think out for
comparison. Please have a look and see if any features are missed. You
can review it by the use case you're familiar with if reviewing all of
them is tough, but be aware that features compared in former use cases
are not listed in later cases again to avoid overlaps:

https://fedoraproject.org/wiki/Rhe/tcms_Comparison

 
Ticket#152[1] is currently set up for tracking this event. Feel free to
add comments or discuss it here. 

Note: the feedback deadline is Jan 21, the end of this week. After that
date, I'll move forward to the next phase. 

Waiting for your feedback!

Cheers,
Hurry

[1] https://fedorahosted.org/fedora-qa/ticket/152

-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh

_______________________________________________
test-announce mailing list
test-announce(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/test-announce

[James Laska]

On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote:
>  Greetings!
>  
>  Wiki workflow/use cases are reorganized and grouped by general and main
>  test events(runs) use cases. The general cases cover the basic uses of
>  wiki, and the events cases covers the detailed certain steps for
>  organizing the events.
>  
>  Please review it:
>  
>  https://fedoraproject.org/wiki/Rhe/tcms_use_cases
>  
>      
>  Meanwhile, Wiki and Nitrate comparison is also listed and grouped by
>  above use cases. I've listed as many features as I can think out for
>  comparison. Please have a look and see if any features are missed. You
>  can review it by the use case you're familiar with if reviewing all of
>  them is tough, but be aware that features compared in former use cases
>  are not listed in later cases again to avoid overlaps:
>  
>  https://fedoraproject.org/wiki/Rhe/tcms_Comparison
>  
>   
>  Ticket#152[1] is currently set up for tracking this event. Feel free to
>  add comments or discuss it here. 
>  
>  Note: the feedback deadline is Jan 21, the end of this week. After that
>  date, I'll move forward to the next phase. 
>  
>  Waiting for your feedback! 
Woah, I like the fancy table layout used for comparing use cases between
the wiki and nitrate!  Thank you for taking the time to identify our
current test workflows and offer a comparison between nitrate and the
wiki.  You've got a really comprehensive list of use cases that will
certainly help us evaluate nitrate.

While looking through the comparison wiki, I've noted a few
comments/questions below.  Hope this helps!

I'm unclear what the use case 'Creating A Common Page' is, can you
elaborate?  Is that when we create a category page to group all test
results for a milestone (e.g.
https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)?

Under 'creating a common page', the row titled 'Submitting format'.
Would that be better named as 'Supports data entry using a form'?

Under 'creating a common page', there is a row titled 'Subpage using'.
I don't think I understand that use case, can you explain?

Under section 'Creating A Test Case', there are some red cells for the
wiki for (these are also under 'Creating a Test Plan').  What does the
'red' mean.  Does it mean that they are not possible, or that the
solution provided is not optimal?
      * Renaming a case - move to another page
      * Case draft status - manually add draft category/note
      * Case review status - approved in ticket or somewhere else

Also under section 'Creating A Test Case', the row titled 'Test case
re-use (write once, link anywhere)' talks about nitrate support for
cloning tests.  Does this create a new copy of the original, or does it
re-use the existing test?  If any changes are made to one, are the
propagated to the other clone?

Under 'Creating A Test Result Page Template', the row titled 'Group
cases (by media)', I imagine nitrate supports other methods of
grouping/organizing tests in a test run?  Oh wait ... this is grouping,
not sorting.  How would this be done with nitrate, would a test run be
created for each media type?  Or would we sort tests of different media
types differently?

Under 'Creating A Test Result Page', the row titled 'Redirecting
links' ... is this when a test run has expired, or been replaced by a
newer test run?  Like if we get RC2, we'll go and adjust RC1 so people
know it's no longer active?  Not sure what to call this feature, retire
a test run?

Under 'Posting Test Results', the row titled 'Opening current links' ...
I'm trying to think of a more generic name for that feature.  Maybe
'Quick access to active test runs' instead?

Under 'Generating A Test Summary', the row titled 'Result summary/report
generation'.  Haha, you list that as *RED* ... very accurate! :)

Under 'Administering', the row titled 'License', is this the license for
nitrate and mediawiki, or the license for the content in the cases,
plans and results?

Under 'Release Validation Test Event', the row titled 'Roadmap for each
run' ... can you explain, or link to, what is meant by this?

Thanks again!
James

[He Rui]

Hi James,

Wow, firstly, many thanks for your careful review of all the items.:)

On Wed, 2011-01-19 at 16:08 -0500, James Laska wrote:
>  On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote:
>  > Greetings!
>  > 
>  > Wiki workflow/use cases are reorganized and grouped by general and main
>  > test events(runs) use cases. The general cases cover the basic uses of
>  > wiki, and the events cases covers the detailed certain steps for
>  > organizing the events.
>  > 
>  > Please review it:
>  > 
>  > https://fedoraproject.org/wiki/Rhe/tcms_use_cases
>  > 
>  >     
>  > Meanwhile, Wiki and Nitrate comparison is also listed and grouped by
>  > above use cases. I've listed as many features as I can think out for
>  > comparison. Please have a look and see if any features are missed. You
>  > can review it by the use case you're familiar with if reviewing all of
>  > them is tough, but be aware that features compared in former use cases
>  > are not listed in later cases again to avoid overlaps:
>  > 
>  > https://fedoraproject.org/wiki/Rhe/tcms_Comparison
>  > 
>  >  
>  > Ticket#152[1] is currently set up for tracking this event. Feel free to
>  > add comments or discuss it here. 
>  > 
>  > Note: the feedback deadline is Jan 21, the end of this week. After that
>  > date, I'll move forward to the next phase. 
>  > 
>  > Waiting for your feedback!
>  
>  Woah, I like the fancy table layout used for comparing use cases between
>  the wiki and nitrate!  Thank you for taking the time to identify our
>  current test workflows and offer a comparison between nitrate and the
>  wiki.  You've got a really comprehensive list of use cases that will
>  certainly help us evaluate nitrate.
>  
>  While looking through the comparison wiki, I've noted a few
>  comments/questions below.  Hope this helps!
>  
>  I'm unclear what the use case 'Creating A Common Page' is, can you
>  elaborate?  Is that when we create a category page to group all test
>  results for a milestone (e.g.
>  https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)? 
No, it means creating a general new page on wiki, like a documentation
page etc, and test cases/plans/categories/results pages are particular
cases based on creating a normal page like this. It lists what a user
may do when creating a new page. 

>  Under 'creating a common page', the row titled
> 'Submitting format'.
>  Would that be better named as 'Supports data entry using a form'? 
Ah great, I've changed it.
>  
>  Under 'creating a common page', there is a row titled 'Subpage using'.
>  I don't think I understand that use case, can you explain? 
Okay, for example on wiki, Test_Days is under the path of QA like:
https://fedoraproject.org/wiki/QA/Test_Days

So I tried to compare if nitrate can also organize sub-page under its
main page. 

>  Under section 'Creating A Test Case', there are some red
> cells for the
>  wiki for (these are also under 'Creating a Test Plan').  What does the
>  'red' mean.  Does it mean that they are not possible, or that the
>  solution provided is not optimal? 
Oh the first two below are not optimal, the last one is not possible on
wiki. 

>        * Renaming a case - move to another page 
I think moving to another page is not a good way to rename a case, there
will be two link addresses for this case.   

>        * Case draft status - manually add draft category/note 
It's better to have a case status called draft, or proposed like nitrate
does.

>        * Case review status - approved in ticket or somewhere else
It's better to have a case status called 'confirmed', or 'need
update'
or 'disabled' like nitrate. I remember we used to track the cases status
in tickets and after approved in tickets or reviewed in test@,   


>  Also under section 'Creating A Test Case', the row titled
> 'Test case
>  re-use (write once, link anywhere)' talks about nitrate support for
>  cloning tests.  Does this create a new copy of the original, or does it
>  re-use the existing test?  If any changes are made to one, are the
>  propagated to the other clone?
>  
>  Under 'Creating A Test Result Page Template', the row titled 'Group
>  cases (by media)', I imagine nitrate supports other methods of
>  grouping/organizing tests in a test run?  Oh wait ... this is grouping,
>  not sorting.  How would this be done with nitrate, would a test run be
>  created for each media type?  Or would we sort tests of different media
>  types differently?
>  
>  Under 'Creating A Test Result Page', the row titled 'Redirecting
>  links' ... is this when a test run has expired, or been replaced by a
>  newer test run?  Like if we get RC2, we'll go and adjust RC1 so people
>  know it's no longer active?  Not sure what to call this feature, retire
>  a test run?
>  
>  Under 'Posting Test Results', the row titled 'Opening current links' ...
>  I'm trying to think of a more generic name for that feature.  Maybe
>  'Quick access to active test runs' instead?
>  
>  Under 'Generating A Test Summary', the row titled 'Result summary/report
>  generation'.  Haha, you list that as *RED* ... very accurate! :)
>  
>  Under 'Administering', the row titled 'License', is this the license for
>  nitrate and mediawiki, or the license for the content in the cases,
>  plans and results?
>  
>  Under 'Release Validation Test Event', the row titled 'Roadmap for each
>  run' ... can you explain, or link to, what is meant by this?
>  
>  Thanks again!
>  James
>  -- 
>  test mailing list
>  test(a)lists.fedoraproject.org
>  To unsubscribe: 
>  https://admin.fedoraproject.org/mailman/listinfo/test 
-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh

[Samuel Greenfeld]

Unfortunately while I am familiar with a few test case management systems, I
have not been involved with the Fedora project long enough to know its
workflows.  And a quick search online is not turning up much about Nitrate,
with its changelog[1] implying it was first made open source software in
July 2010.

I might be able to provide some general comments though if I knew more about
the product.  So for those of us unfamiliar with the history of Nitrate,
could you please answer the following:


   1. What is the history of Nitrate and the Fedora Project?  What does the
   Fedora project expect to gain from using it?

   2. Is a sample play/sandbox test instance with more-or-less full access
   available online?

   3. How does Nitrate compare to other open & closed sourced TCMS
   solutions?  Why was it written as opposed to using an existing solution, and
   what are its straights & weaknesses?

   4. Is test case & test plan import and export (to XML, etc.) support
   available?  If so is this compatible with any other TCMS's import/export
   system?

   5. Are nested test plan and/or test case hierarchies supported?

   6. Can multiple projects share test cases, and even reference older
   versions of test cases if they are lagging behind the current rawhide/Fedora
   release?  Will Spins be able to make their own (simultaneously running) test
   plans?

   7. How long will historical test case results be made available?

   8. Is there any plan to tie this into Bodhi and other tools to detect
   updated packages that may imply test cases need re-running and/or updating?

   9. Is this going to be available as a Fedorahosted service like Trac is?
   If so will all the instances be able to share test cases?

   10. Is there any concern that changing test tracking systems may
   encourage/discourage existing testers to participate?


Some of this information may be useful to post on the Trac main page and/or
in the Fedora wiki.

Thanks in advance for your time.

---
SJG

[1] https://fedorahosted.org/nitrate/browser/trunk/nitrate/docs/ChangeLog



On Tue, Jan 18, 2011 at 6:36 AM, He Rui <rhe(a)redhat.com&gt; wrote:

>  Greetings!
>
>  Wiki workflow/use cases are reorganized and grouped by general and main
>  test events(runs) use cases. The general cases cover the basic uses of
>  wiki, and the events cases covers the detailed certain steps for
>  organizing the events.
>
>  Please review it:
>
>  https://fedoraproject.org/wiki/Rhe/tcms_use_cases
>
>
>  Meanwhile, Wiki and Nitrate comparison is also listed and grouped by
>  above use cases. I've listed as many features as I can think out for
>  comparison. Please have a look and see if any features are missed. You
>  can review it by the use case you're familiar with if reviewing all of
>  them is tough, but be aware that features compared in former use cases
>  are not listed in later cases again to avoid overlaps:
>
>  https://fedoraproject.org/wiki/Rhe/tcms_Comparison
>
>
>  Ticket#152[1] is currently set up for tracking this event. Feel free to
>  add comments or discuss it here.
>
>  Note: the feedback deadline is Jan 21, the end of this week. After that
>  date, I'll move forward to the next phase.
>
>  Waiting for your feedback!
>
>  Cheers,
>  Hurry
>
>  [1] https://fedorahosted.org/fedora-qa/ticket/152
>
>  --
>  Contacts
>
>  Hurry
>  FAS Name: Rhe
>  Timezone: UTC+8
>  TEL: [phone]
>  IRC nick: rhe #fedora-qa #fedora-zh
>
>  _______________________________________________
>  test-announce mailing list
>  test-announce(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/test-announce
>

[James Laska]

On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
>  Unfortunately while I am familiar with a few test case management
>  systems, I have not been involved with the Fedora project long enough
>  to know its workflows.  And a quick search online is not turning up
>  much about Nitrate, with its changelog[1] implying it was first made
>  open source software in July 2010.
>  
>  I might be able to provide some general comments though if I knew more
>  about the product.  So for those of us unfamiliar with the history of
>  Nitrate, could you please answer the following: 
Really good list of questions Samuel, thanks for jumping in.  I know
Hurry can provide feedback on the questions.  However, I'll add my
thoughts as well.

>       1. What is the history of Nitrate and the Fedora Project?  What
>          does the Fedora project expect to gain from using it? 
Hurry can touch on the goals for using nitrate.  As for history, I can
add my experiences ...

This goes back to an eval we did using testopia in Fedora many releases
agove.  Unfortunately, the effort was canceled due to license
incompatibility between Fedora and testopia.  At that point, we invested
in leveraging the wiki to best of our ability.

  https://fedoraproject.org/wiki/QA/Testopia_Evaluation
  https://bugzilla.redhat.com/show_bug.cgi?id=450013

After this, several folks got together and decided they would implement
a new front-end on top of the testopia db schema.  This would resolve
the original license incompatibility and address usability issues that
were raised with the testopia UI.  The project started development
internally, and was open-sourced in 2010.

>       1. Is a sample play/sandbox test instance with more-or-less
> full
>          access available online?  
Not yet, I believe that's in plan for sometime during Fedora 15.  Hurry
and I were discussing the requirements for such an instance earlier this
week.  If you're interested in helping here, just let us know :)

>       1. How does Nitrate compare to other open & closed sourced
> TCMS
>          solutions?  Why was it written as opposed to using an existing
>          solution, and what are its straights & weaknesses? 
See history comments above.  Also, maybe the nitrate developers [1] can
offer more insight on how it compares to other open-source solutions?  I
*think* that comparison work has been done in the past, I'm just not
sure where to find the results.

[1] https://fedorahosted.org/mailman/listinfo/nitrate-devel

>       1. Is test case & test plan import and export (to XML,
> etc.)
>          support available?  If so is this compatible with any other
>          TCMS's import/export system? 
I believe import/export is supported using the testopia.dtd format.

http://git.fedorahosted.org/git/?p=nitrate.git;a=blob;f=trunk/nitrate/doc...
- Fixed #564258 - [REF] Ability to export/print specified cases
- Fixed #612803 - add an export feature for test case runs, can export …

>       1. Are nested test plan and/or test case hierarchies supported?
I don't recall.  Perhaps Hurry or the nitrate developers know?

I know this feature has been discussed a *lot* with nitrate, and other
solutions we've used in the past.  I don't think support for nested test
plans is something we've had a tremendous need for now, so I don't
anticipate this being a MUSTHAVE feature in the near term.

>       1. Can multiple projects share test cases, and even reference
>          older versions of test cases if they are lagging behind the
>          current rawhide/Fedora release?  Will Spins be able to make
>          their own (simultaneously running) test plans? 
This is the hope.  It's not really useful if we can only use it for
release validation.  I don't think we've fully explored how best to
model other spins/projects, but I don't foresee big problems there.
That will be fun to explore on the sandbox/staging instance.

With regards to referencing older versions of a test case, I believe
that support is there, although I'm not certain it's right for our
needs.  Keeping test documentation (plans and cases) updated is a pretty
sizable maintenance challenge.  I've seen many instances where support
for versioned test cases allows test plans to suffer over time as they
were linked against old and inaccurate test cases.

Much like how the wiki is used now, we have support for linking against
older versions of tests (wiki history), but we rarely ever use that
feature.  I expect that trend would continue in the short-term.

>       1. How long will historical test case results be made available?
I suspect the limiting factor here is database size.  I'm not aware of
any rules or process that would require removal/archival of old results.
However, at some point that could certainly be an issue we'd need to
plan for.

>       1. Is there any plan to tie this into Bodhi and other tools to
>          detect updated packages that may imply test cases need
>          re-running and/or updating? 
That is certainly possible, but there are currently no detailed plans to
integrate bodhi/f-e-k with nitrate specifically.  Nitrate offers a
robust XMLRPC interface to allow other tools, such as bodhi, to query
for applicable tests/results/runs etc...  So I wouldn't think it would
be a difficult task.

That said, Adam Williamson is exploring this linkage using our current
wiki-based test management system.  While the two implementations will
be completely different, the end result is the same ... improved
integration between our update tools (bodhi/f-e-k) and our test
infrastructure.

>       1. Is this going to be available as a Fedorahosted service like
>          Trac is?  If so will all the instances be able to share test
>          cases? 
Hmm, I don't envision multiple hosted instances of nitrate (or any other
test case management) in Fedora.  I expect we would have a single test
management system available for all to use, much like we have a build
system, updates and bug reporting.

>       1. Is there any concern that changing test tracking systems may
>          encourage/discourage existing testers to participate? 
Yes!  That's part of why Hurry is investing a lot of time researching
our current workflows and comparing features between our current
wiki-based implementation.

>  Some of this information may be useful to post on the Trac main page
>  and/or in the Fedora wiki.
>  
>  Thanks in advance for your time. 
Thank you for your input, good comments/questions!

Thanks,
James

[Samuel Greenfeld]

A few more inline comments on a subset of the questions, and two more thoughts:

On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jlaska(a)redhat.com&gt; wrote:
>  On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
>  >      1. What is the history of Nitrate and the Fedora Project?  What
>  >         does the Fedora project expect to gain from using it?
>
>  This goes back to an eval we did using testopia in Fedora many releases
>  agove.  Unfortunately, the effort was canceled due to license
>  incompatibility between Fedora and testopia.  At that point, we invested
>  in leveraging the wiki to best of our ability.
>
>   https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
>   https://bugzilla.redhat.com/show_bug.cgi?id=450013 
> From the bug I'm guessing this is because Testopia used Ext-JS,
> and 
Ext-JS kept changing licenses.  Hopefully there will only be licensing
restrictions for Nitrate on the software itself, and not the created
work of what one does while actively using it.


>  >      1. How does Nitrate compare to other open & closed
> sourced TCMS
>  >         solutions?  Why was it written as opposed to using an existing
>  >         solution, and what are its strengths & weaknesses?
>
>  See history comments above.  Also, maybe the nitrate developers [1] can
>  offer more insight on how it compares to other open-source solutions?  I
>  *think* that comparison work has been done in the past, I'm just not
>  sure where to find the results.
>
>  [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel 
Are the developers actively monitoring/using this list?  I looked at
it before, and all I saw were three test posts from July in the
archives.


>  >      1. Can multiple projects share test cases, and even
> reference
>  >         older versions of test cases if they are lagging behind the
>  >         current rawhide/Fedora release?  Will Spins be able to make
>  >         their own (simultaneously running) test plans?
>
>  This is the hope.  It's not really useful if we can only use it for
>  release validation.  I don't think we've fully explored how best to
>  model other spins/projects, but I don't foresee big problems there.
>  That will be fun to explore on the sandbox/staging instance.
>
>  With regards to referencing older versions of a test case, I believe
>  that support is there, although I'm not certain it's right for our
>  needs.  Keeping test documentation (plans and cases) updated is a pretty
>  sizable maintenance challenge.  I've seen many instances where support
>  for versioned test cases allows test plans to suffer over time as they
>  were linked against old and inaccurate test cases.
>
>  Much like how the wiki is used now, we have support for linking against
>  older versions of tests (wiki history), but we rarely ever use that
>  feature.  I expect that trend would continue in the short-term. 
The reason I bring this up is because OLPC's Spin releases tend to lag
behind the official Fedora release.  For instance, we just released
our hopefully final Fedora 11-based release this past month, and are
in the early stages of the Fedora 14-based one.  At least some Fedora
ARM development work may still be going on with Fedora 13 as well.

While decently written test cases will survive somewhat over time,
major changes in GUI look & feel or other areas can break their
backwards compatibility.

Ideally Nitrate will default to using the current version of a
testcase when making a new plan, preferably following the updates of
said testcase until a result is committed which forces the test case
version to be needed.


>  >      1. How long will historical test case results be made
> available?
>
>  I suspect the limiting factor here is database size.  I'm not aware of
>  any rules or process that would require removal/archival of old results.
>  However, at some point that could certainly be an issue we'd need to
>  plan for. 
Again; this would be to help lagging Spins and similar.  Right now it
looks like Bodhi at least publicly hides information for Fedora
versions which have reached end-of-life.  (Either that, or I don't
know how to find it.)


I presume that Nitrate has already been determined to be scalable;
otherwise it is a big risk to incorporate it into Fedora.

Also: It might be useful to add an "unclear" testcase result, similar
to how Mozilla's Litmus system does it (https://litmus.mozilla.org).

[James Laska]

On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote:
>  A few more inline comments on a subset of the questions, and two more
> thoughts:
>  
>  On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jlaska(a)redhat.com&gt; wrote:
>  > On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
>  > >      1. What is the history of Nitrate and the Fedora Project?  What
>  > >         does the Fedora project expect to gain from using it?
>  >
>  > This goes back to an eval we did using testopia in Fedora many releases
>  > agove.  Unfortunately, the effort was canceled due to license
>  > incompatibility between Fedora and testopia.  At that point, we invested
>  > in leveraging the wiki to best of our ability.
>  >
>  >  https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
>  >  https://bugzilla.redhat.com/show_bug.cgi?id=450013
>  
>  >From the bug I'm guessing this is because Testopia used Ext-JS, and
>  Ext-JS kept changing licenses.  Hopefully there will only be licensing
>  restrictions for Nitrate on the software itself, and not the created
>  work of what one does while actively using it. 
That was a question I had opened with Hurry.  I have no idea how we'd
license the content in the system.  If anything, I'd hope/expect that to
match that of our current wiki.

>  > >      1. How does Nitrate compare to other open & closed
> sourced TCMS
>  > >         solutions?  Why was it written as opposed to using an existing
>  > >         solution, and what are its strengths & weaknesses?
>  >
>  > See history comments above.  Also, maybe the nitrate developers [1] can
>  > offer more insight on how it compares to other open-source solutions?  I
>  > *think* that comparison work has been done in the past, I'm just not
>  > sure where to find the results.
>  >
>  > [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel
>  
>  Are the developers actively monitoring/using this list?  I looked at
>  it before, and all I saw were three test posts from July in the
>  archives. 
I believe they do, but don't yet have a strong upstream presence since
there hasn't been a lot of code/progress to share until recently.  This
will be something I'm sure they'll want to improve as community interest
increases.

>  > >      1. Can multiple projects share test cases, and even
> reference
>  > >         older versions of test cases if they are lagging behind the
>  > >         current rawhide/Fedora release?  Will Spins be able to make
>  > >         their own (simultaneously running) test plans?
>  >
>  > This is the hope.  It's not really useful if we can only use it for
>  > release validation.  I don't think we've fully explored how best to
>  > model other spins/projects, but I don't foresee big problems there.
>  > That will be fun to explore on the sandbox/staging instance.
>  >
>  > With regards to referencing older versions of a test case, I believe
>  > that support is there, although I'm not certain it's right for our
>  > needs.  Keeping test documentation (plans and cases) updated is a pretty
>  > sizable maintenance challenge.  I've seen many instances where support
>  > for versioned test cases allows test plans to suffer over time as they
>  > were linked against old and inaccurate test cases.
>  >
>  > Much like how the wiki is used now, we have support for linking against
>  > older versions of tests (wiki history), but we rarely ever use that
>  > feature.  I expect that trend would continue in the short-term.
>  
>  The reason I bring this up is because OLPC's Spin releases tend to lag
>  behind the official Fedora release.  For instance, we just released
>  our hopefully final Fedora 11-based release this past month, and are
>  in the early stages of the Fedora 14-based one.  At least some Fedora
>  ARM development work may still be going on with Fedora 13 as well.
>  
>  While decently written test cases will survive somewhat over time,
>  major changes in GUI look & feel or other areas can break their
>  backwards compatibility.
>  
>  Ideally Nitrate will default to using the current version of a
>  testcase when making a new plan, preferably following the updates of
>  said testcase until a result is committed which forces the test case
>  version to be needed. 
I hope so, yes! :)  I'm fairly certain support exists for linking
against versioned cases, that is, I know it was present in testopia.
Hopefully Hurry, or the nitrate folks can help here.  Sounds like we
need to add this to our list of requirements.

>  > >      1. How long will historical test case results be made
> available?
>  >
>  > I suspect the limiting factor here is database size.  I'm not aware of
>  > any rules or process that would require removal/archival of old results.
>  > However, at some point that could certainly be an issue we'd need to
>  > plan for.
>  
>  Again; this would be to help lagging Spins and similar.  Right now it
>  looks like Bodhi at least publicly hides information for Fedora
>  versions which have reached end-of-life.  (Either that, or I don't
>  know how to find it.) 
That or garbage collected.  

Sounds like we'll need to do some further research on this front.  I
personally see no reason to purge results for EOL'd releases.  I've
often referenced old test results to understand when the last time
something was tested.  Test results aren't disk-hogs like builds
(*cough* oo.org *cough*) can be, so I don't expect this to be as much of
an issue.  Nonetheless, you're point is valid.

>  I presume that Nitrate has already been determined to be scalable;
>  otherwise it is a big risk to incorporate it into Fedora. 
I'll let the nitrate folks speak to scalability issues.  That's
definitely something we'll want to explore during a pilot.  Remember
too, I don't expect we'll throw away the wiki, and switch to nitrate
without a good bake-in period to identify bugs/gaps etc...

>  Also: It might be useful to add an "unclear" testcase
> result, similar
>  to how Mozilla's Litmus system does it (https://litmus.mozilla.org). 
I do like litmus!  It's a nice evolution from testopia for upstream
mozilla.  We don't currently have an 'unclear' test result.  I'm not
opposed to it, but would need better understand how that field is used,
and the process around it, in litmus.

Thanks,
James

[He Rui]

Greetings,

Many thanks for Samuel's comments, I found out many valuable things to
think, identify and add to use cases/feature comparison.

The questions also remind me that many testers so far may be still not
familiar with Nitrate, therefore can't give any suggestion upon it. So
I'll introduce Nitrate a bit here. 

Nitrate is a new test case management system which is open-sourced
recently. It has advantages on managing, tracking and querying
cases,results and is similar to testopia. The introduction pages along
with some screenshots can be found at:

https://fedorahosted.org/nitrate/
https://fedoraproject.org/wiki/Nitrate


One can set up a local nitrate system with the steps in the link, we are
also planing to build a testing instance during F-15.   


On Thu, 2011-01-20 at 15:02 -0500, James Laska wrote:
>  On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote:
>  > A few more inline comments on a subset of the questions, and two more thoughts:
>  > 
>  > On Thu, Jan 20, 2011 at 11:54 AM, James Laska <jlaska(a)redhat.com&gt; wrote:
>  > > On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
>  > > >      1. What is the history of Nitrate and the Fedora Project?  What
>  > > >         does the Fedora project expect to gain from using it?
>  > >
>  > > This goes back to an eval we did using testopia in Fedora many releases
>  > > agove.  Unfortunately, the effort was canceled due to license
>  > > incompatibility between Fedora and testopia.  At that point, we invested
>  > > in leveraging the wiki to best of our ability.
>  > >
>  > >  https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation
>  > >  https://bugzilla.redhat.com/show_bug.cgi?id=450013
>  > 
>  > >From the bug I'm guessing this is because Testopia used Ext-JS, and
>  > Ext-JS kept changing licenses.  Hopefully there will only be licensing
>  > restrictions for Nitrate on the software itself, and not the created
>  > work of what one does while actively using it.
>  
>  That was a question I had opened with Hurry.  I have no idea how we'd
>  license the content in the system.  If anything, I'd hope/expect that to
>  match that of our current wiki. 
Oh, licensing and license issues. I've listed both of them in feature
comparison as requirements. For test days and release validation test
events use cases, I identified the steps with different permissions by
QA, event host and tester(including anonymous user)[#], and hope this
can help to license these instances in Nitrate. Of course, I need to
extend more to cover other instances. 

[#]
https://fedoraproject.org/wiki/Rhe/tcms_use_cases#Test_Events_Use_Cases

>  
>  > > >      1. How does Nitrate compare to other open & closed sourced TCMS
>  > > >         solutions?  Why was it written as opposed to using an existing
>  > > >         solution, and what are its strengths & weaknesses?
>  > >
>  > > See history comments above.  Also, maybe the nitrate developers [1] can
>  > > offer more insight on how it compares to other open-source solutions?  I
>  > > *think* that comparison work has been done in the past, I'm just not
>  > > sure where to find the results.
>  > >
>  > > [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel
>  > 
>  > Are the developers actively monitoring/using this list?  I looked at
>  > it before, and all I saw were three test posts from July in the
>  > archives.
>  
>  I believe they do, but don't yet have a strong upstream presence since
>  there hasn't been a lot of code/progress to share until recently.  This
>  will be something I'm sure they'll want to improve as community interest
>  increases. 
Add it into requirements. But I can confirm that all the nitrate
developers are monitoring the list now. When community interest
increases, it will become active. :)  

>  
>  > > >      1. Can multiple projects share test cases, and even reference
>  > > >         older versions of test cases if they are lagging behind the
>  > > >         current rawhide/Fedora release?  Will Spins be able to make
>  > > >         their own (simultaneously running) test plans?
>  > >
>  > > This is the hope.  It's not really useful if we can only use it for
>  > > release validation.  I don't think we've fully explored how best to
>  > > model other spins/projects, but I don't foresee big problems there.
>  > > That will be fun to explore on the sandbox/staging instance.
>  > >
>  > > With regards to referencing older versions of a test case, I believe
>  > > that support is there, although I'm not certain it's right for our
>  > > needs.  Keeping test documentation (plans and cases) updated is a pretty
>  > > sizable maintenance challenge.  I've seen many instances where support
>  > > for versioned test cases allows test plans to suffer over time as they
>  > > were linked against old and inaccurate test cases.
>  > >
>  > > Much like how the wiki is used now, we have support for linking against
>  > > older versions of tests (wiki history), but we rarely ever use that
>  > > feature.  I expect that trend would continue in the short-term.
>  > 
>  > The reason I bring this up is because OLPC's Spin releases tend to lag
>  > behind the official Fedora release.  For instance, we just released
>  > our hopefully final Fedora 11-based release this past month, and are
>  > in the early stages of the Fedora 14-based one.  At least some Fedora
>  > ARM development work may still be going on with Fedora 13 as well.
>  > 
>  > While decently written test cases will survive somewhat over time,
>  > major changes in GUI look & feel or other areas can break their
>  > backwards compatibility.
>  > 
>  > Ideally Nitrate will default to using the current version of a
>  > testcase when making a new plan, preferably following the updates of
>  > said testcase until a result is committed which forces the test case
>  > version to be needed.
>  
>  I hope so, yes! :)  I'm fairly certain support exists for linking
>  against versioned cases, that is, I know it was present in testopia.
>  Hopefully Hurry, or the nitrate folks can help here.  Sounds like we
>  need to add this to our list of requirements. 
In Nitrate, the text part of a test case is versioned, and one can view
different text versions of the case but can't reference old versions to
test run. testopia can, but nitrate abandoned it. I've listed it in the
requirements for valuating. 

>  
>  > > >      1. How long will historical test case results be made available?
>  > >
>  > > I suspect the limiting factor here is database size.  I'm not aware of
>  > > any rules or process that would require removal/archival of old results.
>  > > However, at some point that could certainly be an issue we'd need to
>  > > plan for.
>  > 
>  > Again; this would be to help lagging Spins and similar.  Right now it
>  > looks like Bodhi at least publicly hides information for Fedora
>  > versions which have reached end-of-life.  (Either that, or I don't
>  > know how to find it.)
>  
>  That or garbage collected.  
>  
>  Sounds like we'll need to do some further research on this front.  I
>  personally see no reason to purge results for EOL'd releases.  I've
>  often referenced old test results to understand when the last time
>  something was tested.  Test results aren't disk-hogs like builds
>  (*cough* oo.org *cough*) can be, so I don't expect this to be as much of
>  an issue.  Nonetheless, you're point is valid. 
Add it to the requirement list.
>  
>  > I presume that Nitrate has already been determined to be scalable;
>  > otherwise it is a big risk to incorporate it into Fedora.
>  
>  I'll let the nitrate folks speak to scalability issues.  That's
>  definitely something we'll want to explore during a pilot.  Remember
>  too, I don't expect we'll throw away the wiki, and switch to nitrate
>  without a good bake-in period to identify bugs/gaps etc...
>  
>  > Also: It might be useful to add an "unclear" testcase result, similar
>  > to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
>  
>  I do like litmus!  It's a nice evolution from testopia for upstream
>  mozilla.  We don't currently have an 'unclear' test result.  I'm not
>  opposed to it, but would need better understand how that field is used,
>  and the process around it, in litmus. 
Agree with James. 

>  
>  Thanks,
>  James
>  -- 
>  test mailing list
>  test(a)lists.fedoraproject.org
>  To unsubscribe: 
>  https://admin.fedoraproject.org/mailman/listinfo/test 
In addition, I've also added 'Integration between our update tools
(bodhi/f-e-k) and our test infrastructure' to requirements for
evaluation.

Feel free to tell if there's any concern.


Thanks,
Hurry

-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh

[Samuel Greenfeld]

On Fri, Jan 21, 2011 at 11:40 AM, He Rui <rhe(a)redhat.com&gt; wrote:

>  > > Also: It might be useful to add an "unclear"
> testcase result, similar
>  > > to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
>  >
>  > I do like litmus!  It's a nice evolution from testopia for upstream
>  > mozilla.  We don't currently have an 'unclear' test result.  I'm
> not
>  > opposed to it, but would need better understand how that field is used,
>  > and the process around it, in litmus.
>
>  Agree with James.
>  
What I believe Mozilla is doing (since I have not had a chance to work with
their QA team yet) is flagging test cases with a form of soft failure in
that the result of a testcase neither clearly passed, nor clearly failed.
So in addition to "Passed", "Failed", and any other common states
(Blocked,
In Progress, etc.) you have an "Unclear" result state.

This forces the result (including a comment section I presume is forced for
all non-passing items) to be evaluated during the release cycle to see if it
was an actual failure, or if the test case needs to be updated and/or
clarified.

See the "recently unclear" and "testcases most frequently marked as
unclear"
links at the bottom of pretty much every page of litmus.mozilla.org for
examples.  On the former, you can click on the dialog icon beneath each
result ID to see the comment without switching pages.

Personally I like this approach because management of many projects (in my
experience) rarely budget time for test cases to be updated.  They prefer
that you start testing the next release instead.  Projects I have been on
have required me to use test cases which were last notably updated a few
years ago if not 5-10+ years ago by simply acting as if the inaccurate
information is not there.  By treating potentially incorrect test cases as a
form of failure it forces them to be ideally updated and rerun promptly.

The downside is if a tester is unfamiliar with an area or needs
hand-holding, they might mark a lot of results unclear which actually are
rather straightforward.  Or alternatively, no one looks at them, and you
have a muddy result at the end of the release.  These would put additional
effort on the core QA team to remedy the situation.

[Adam Williamson]

On Fri, 2011-01-21 at 19:02 +0000, Samuel Greenfeld wrote:

>          > I do like litmus!  It's a nice evolution from
> testopia for
>          upstream
>          > mozilla.  We don't currently have an 'unclear' test result.
>           I'm not
>          > opposed to it, but would need better understand how that
>          field is used,
>          > and the process around it, in litmus.
>          
>          
>          Agree with James.
>  
>  What I believe Mozilla is doing (since I have not had a chance to work
>  with their QA team yet) is flagging test cases with a form of soft
>  failure in that the result of a testcase neither clearly passed, nor
>  clearly failed.  So in addition to "Passed", "Failed", and any other
>  common states (Blocked, In Progress, etc.) you have an "Unclear"
>  result state. 
Hurry is somewhat wrong to say we don't currently have an 'unclear'
result; we do have the 'warn' result, which is in some ways similar. We
usually use it to indicate when a test turns up some kind of anomalous
behaviour which isn't exactly a failure.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net

[He Rui]

On Mon, 2011-01-24 at 23:05 -0800, Adam Williamson wrote:
>  On Fri, 2011-01-21 at 19:02 +0000, Samuel Greenfeld wrote:
>  
>  >         > I do like litmus!  It's a nice evolution from testopia for
>  >         upstream
>  >         > mozilla.  We don't currently have an 'unclear' test
> result.
>  >          I'm not
>  >         > opposed to it, but would need better understand how that
>  >         field is used,
>  >         > and the process around it, in litmus.
>  >         
>  >         
>  >         Agree with James.
>  > 
>  > What I believe Mozilla is doing (since I have not had a chance to work
>  > with their QA team yet) is flagging test cases with a form of soft
>  > failure in that the result of a testcase neither clearly passed, nor
>  > clearly failed.  So in addition to "Passed", "Failed", and any
> other
>  > common states (Blocked, In Progress, etc.) you have an "Unclear"
>  > result state. 
I didn't receive the mail replied by Samuel. Weird. 

>  Hurry is somewhat wrong to say we don't currently have an
> 'unclear'
>  result; we do have the 'warn' result, which is in some ways similar. We
>  usually use it to indicate when a test turns up some kind of anomalous
>  behaviour which isn't exactly a failure. 
It depends on what 'unclear' means here and how it reflects the results.
If you mean a soft failure or an issue that doesn't block the case run,
the 'warn' result is similar to it, then calling it 'unclear' is
confusing and not accurate in my opinion.

In nitrate system, it has 'blocked', 'failed' and 'error' result
status
to reflect a problem. User guide suggests 'error' is used for test
environment that has problems that prevent Test Case execution. I think
we can modify 'error' status to include all soft failures. 

I've added this to the requirements for further evaluation.  

Thanks,
Hurry
-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh