Re: Proposed must-have Features for Nitrate compared with Wiki

[He Rui]

On Wed, 2011-03-09 at 14:43 -0500, James Laska wrote:
>  On Wed, 2011-03-09 at 18:06 +0800, He Rui wrote:
>  > Greetings,
>  > 
>  > Now let's continue the discussion[1] about TCMS management. After the
>  > feature comparison between wiki and nitrate, the features which are
>  > supported in current wiki, but not in nitrate have been identified[2].
>  > Among these, I priorized some of them as Must-have features, which were
>  > summarized as below[3]:
>  > 
>  > 1. History rollback -
>  > User can view history versions and undo changes.
>  
>  Good, I do use this more often than I realize.
>  
>  > 2. History comparison -
>  > Different versions can compare with each other.
>  > 
>  > 3. Description part in test case -
>  > Currently there's no 'Description' part for test case format on
> Nitrate,
>  > should we add this part or use 'Setup' part instead?
>  
>  No, I don't think 'setup' is the right place since we also use
> 'setup'
>  in our wiki test template.  Does nitrate support a summary or any other
>  field to describe the intent of the test in human readable terms? 
Oh, it has "Notes" area, maybe we can use that.

>  
>  > 4. Grouping cases (by media) -
>  > Need a way to separate the cases to different groups in one test run.
>  > How can we achieve this in Nitrate?
>  
>  Probably different test runs of the same plan.  One run would be for the
>  DVD ISO, one for the netinst.iso etc...  Perhaps this will need further
>  exploration. 
Yeah, use different test runs or sorting the cases by order. 

>  > 5. Documents in test result page(Run) - 
>  > There's only a little space called 'Note' for documentation on Nitrate
>  > and it doesn't support syntax. should we ask for more spaces with syntax
>  > supported for test runs like test days?
>  
>  Can you explain in more detail?  Are you proposing we move the test day
>  wiki content into nitrate, so it's a one-stop-shop for test information
>  for that event? 
I meant in nitrate, there's only a 'Notes' field to support a simple
summary of each test run. For a test day which has many contents[1] such
as "What to test", "Prerequisite for Test Day", "How to
test" etc, it is
hard to put all of these into the 'Notes' field. This field doesn't
support TinyMCE or any other syntax editing.   

[1] https://fedoraproject.org/wiki/QA/Test_Days/Template
>  
>  > 6. Moving test results -
>  > Moving previous results from a test run to another. AFAIK, there's no
>  > way to copy some of one test run results to another.
>  
>  Good catch.  I think with nitrate, you can selectively include test
>  cases when creating a new test run, right?  Meaning, we wouldn't need to
>  include *all* tests in the plan when creating a new test run? 
Yeah. but then we need to make very sure that certain cases results
won't be changed in new run. The summary of the new run results is also
not intact. Anyway, this is a workaround.  

>  This brings up another question ... permissions.  With the wiki,
> anyone
>  has permission to add or remove tests from a test run.  How will that be
>  handled in nitrate? 
Since it's internally use only, the permissions only have three: 1.
anonymous - read. 2. tester - read and write any case/run/plan. 3. Admin
- manage the tcms system. 
>  
>  > 7. Result type -
>  > Nitrate doesn't have 'warn' but has 'error' result status.
> Modify it to
>  > 'warn'?
>  
>  What is the significance of 'error' in nitrate?  Does that mean there
>  was a test case error, not necessarily a software error? 
It presents an environment error, the user guide suggesting the use of
result status is: 

Idle - Default value. The Test Case has not been examined.
Passed - Test Case met all the expected results.
Failed - Test Case did not meet all the expected results, or produced an
unhandled exception.
Running - Test Case is in progress.
Paused - This status is used to denote a problem with the test case
itself that prevents the test from being completed.
Blocked - Test Case has a dependency that has failed.
Error - Test environment has problems that prevent Test Case execution. 

But I guess such 'error' status will be seldom used.

>  > 8. Multiple contributions for each case -
>  > Support multiple test results for one case run.
>  
>  Agreed and good catch!  I think it's very important to allow multiple
>  testers to provide feedback on a single test given that the environment
>  under test (hardware) can vary for tests. 
Yeah, so far, nitrate supports the view of editing logs but only display
the result changed at last time.  

>  > 9. Authorities for pages -
>  > Pages on wiki with different namespaces can have different permissions,
>  > so hopefully pages on Nitrate can also have diff permissions.
>  
>  I was confused by this a little earlier as well.  I don't understand the
>  concept of pages within nitrate.  Is this to replace the use of our wiki
>  test plan document and validation pages (e.g.
>  http://fedoraproject.org/wiki/QA:Installation_validation_testing)? 
I meant for example if use certain namespace like 'Test_Results:' or
'Test_Day:', these test runs allow anonymous editing, while test runs
without such names don't. So nitrate should also provide a way to give
different permissions for test runs.   

>  > 10. Supporting anonymous user read-write access -
>  > Anonymous have read-write access for specified test runs.
>  
>  +1
>  
>  > 11. Page protection -
>  > Protect certain plans/cases
>  
>  Oh, I think I understand now.  Is the analogy here that a valid FAS
>  login would be required, like we currently have with our wiki test cases
>  and test plans? 
Yeah, it's similar to No.9, a valid login is required to change certain
test cases/plans. 

>  > 12. License the content -
>  > License the contents the same with wiki
>  
>  Or at least listed in the list of "GOOD" licenses -
>  https://fedoraproject.org/wiki/Licensing#Good_Licenses_3
>  
>  > 13. Upstream project community -
>  > Monitoring it and actively discussing topics in nitrate-devel@, as well
>  > as file/share bugs/requirements in bugzilla
>  
>  +1 
>  
>  > 14. Test day page(run) creation - 
>  > In Nitrate each test run are built based on its test plan. Therefore,
>  > test day plan and its cases are needed before creating a test day run.
>  > Should we change our test day process(adding test plan and its cases) or
>  > should we change Nitrate to support creating test runs without test
>  > plan? 
>  
>  Perhaps something we need to experiment/research with a pilot instance?  
Okay, then in a pilot instance, I'll create a test day plan and add
cases to it first.

>  > 15. Test cases priority -
>  > Nitrate now has P1, P2, P3..., modify them to Alpha, Beta, and Final?
>  
>  Alpha/Beta/Final are appropriate for our two test plans; desktop and
>  installation.  I don't know if they apply for all test plans folks might
>  contribute.  But it seems like an easy future adjustment, so for our
>  immediate needs, P3=Alpha, P2=Beta, P1=Final etc...
>  
>  In general, for all nitrate metadata fields (priority, status,
>  resolution etc...), whomever administers the tool in Fedora should have
>  control over re-defining the values to something appropriate for Fedora.
>  Perhaps that might be a more generic way to phrase the requirement? 
Ah, I see. Agree with it!

>  > 16. Each case with different platforms -
>  > Better support two platform results(i386 and x86_64) submitting for each
>  > case run 
>  
>  Sure, a test run can only be for a single architecture, right?  So we'll
>  have two test runs for a compose.
>  
>       1. RC1 - Installation (i386)
>       2. RC1 - Installation (x86_64) 
Yeah, more runs can achieve this as well as grouping cases etc, but I'm
afraid this way runs of each event will be too many for management. 

>  > Welcome to discuss the above features and provide
> feedback/suggestion.
>  > If there's any important feature missed, or this reminds you the
>  > advantages of other tcms tools etc, feel free to talk.
>  
>  I know you mention anonymous login support above, can you also list FAS
>  authentication?  Apologies if I missed it already. 
Do you mean integration with FAS? 

>  
>  Thank you for continuing to move forward with this initiative.  I'll
>  keep thinking of important features and reply here if I have anything
>  worth sharing :)
>  
>  Thanks,
>  James 
Thanks!
Hurry
-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh

Proposed must-have Features for Nitrate compared with Wiki

[Danqing Li]

Hi ,thank you very much for this great feature requirement list.  

On Mar 10, 2011, at 6:01 PM, He Rui wrote:

>  On Wed, 2011-03-09 at 14:43 -0500, James Laska wrote:
> > On Wed, 2011-03-09 at 18:06 +0800, He Rui wrote:
> >> Greetings,
> >> 
> >> Now let's continue the discussion[1] about TCMS management. After the
> >> feature comparison between wiki and nitrate, the features which are
> >> supported in current wiki, but not in nitrate have been identified[2].
> >> Among these, I priorized some of them as Must-have features, which were
> >> summarized as below[3]:
> >> 
> >> 1. History rollback -
> >> User can view history versions and undo changes.
> > 
> > Good, I do use this more often than I realize.
> > 
> >> 2. History comparison -
> >> Different versions can compare with each other. 
Yes , after talk with Hurry I understand this is very important for community cooperation
testing.  But the implementation of roll back of test case is really not easy . 

In development of 3.4 and 3.5 we plan to improve history function of test plan/case/run,
show the edit history of every field. And in fact  there has change log for test case. 


> >> 
> >> 3. Description part in test case -
> >> Currently there's no 'Description' part for test case format on
> Nitrate,
> >> should we add this part or use 'Setup' part instead?
> > 
> > No, I don't think 'setup' is the right place since we also use
> 'setup'
> > in our wiki test template.  Does nitrate support a summary or any other
> > field to describe the intent of the test in human readable terms?
>  
>  Oh, it has "Notes" area, maybe we can use that. 
Ok , let me know if you have other requirement about this 'Notes' field.

>  
> > 
> >> 4. Grouping cases (by media) -
> >> Need a way to separate the cases to different groups in one test run.
> >> How can we achieve this in Nitrate?
> > 
> > Probably different test runs of the same plan.  One run would be for the
> > DVD ISO, one for the netinst.iso etc...  Perhaps this will need further
> > exploration.
>  
>  Yeah, use different test runs or sorting the cases by order.  


>  
> >> 5. Documents in test result page(Run) - 
> >> There's only a little space called 'Note' for documentation on
> Nitrate
> >> and it doesn't support syntax. should we ask for more spaces with syntax
> >> supported for test runs like test days?
> > 
> > Can you explain in more detail?  Are you proposing we move the test day
> > wiki content into nitrate, so it's a one-stop-shop for test information
> > for that event?
>  
>  I meant in nitrate, there's only a 'Notes' field to support a simple
>  summary of each test run. For a test day which has many contents[1] such
>  as "What to test", "Prerequisite for Test Day", "How to
> test" etc, it is
>  hard to put all of these into the 'Notes' field. This field doesn't
>  support TinyMCE or any other syntax editing.   
>  
>  [1] https://fedoraproject.org/wiki/QA/Test_Days/Template 
I see that Test Day is a very good cooperation testing activity.  I think you can use Test
Plan to describe the Test Day Document instead of Test Run.

The organizer write all information above in Test Plan document.  And put test cases into
this plan. 
On that day ,every tester could make a new run for himself/ herself . 

I user Nitrate made a test plan to simulate  fedora test day on Feb25, please see the
attached screenshot. In this way , information could be stored in plan documents, and test
result in every test runs.

But now Nitrate can't supply the matrix test result show all user's result like
the wikipage[1]. 

We need talk about  the test result report feature in future development.

> > 
> >> 6. Moving test results -
> >> Moving previous results from a test run to another. AFAIK, there's no
> >> way to copy some of one test run results to another.
> > 
> > Good catch.  I think with nitrate, you can selectively include test
> > cases when creating a new test run, right?  Meaning, we wouldn't need to
> > include *all* tests in the plan when creating a new test run?
>  
>  Yeah. but then we need to make very sure that certain cases results
>  won't be changed in new run. The summary of the new run results is also
>  not intact. Anyway, this is a workaround.  
>  
> > This brings up another question ... permissions.  With the wiki, anyone
> > has permission to add or remove tests from a test run.  How will that be
> > handled in nitrate?
>  
>  Since it's internally use only, the permissions only have three: 1.
>  anonymous - read. 2. tester - read and write any case/run/plan. 3. Admin
>  - manage the tcms system.  
Sorry ,I am not clear about the necessity of import result form another test run,is it
mean those test run very similar, so why make a new run?Do you explain some more detail
about this feature's scenario ? 


> > 
> >> 7. Result type -
> >> Nitrate doesn't have 'warn' but has 'error' result status.
> Modify it to
> >> 'warn'?
> > 
> > What is the significance of 'error' in nitrate?  Does that mean there
> > was a test case error, not necessarily a software error?
>  
>  It presents an environment error, the user guide suggesting the use of
>  result status is: 
>  
>  Idle - Default value. The Test Case has not been examined.
>  Passed - Test Case met all the expected results.
>  Failed - Test Case did not meet all the expected results, or produced an
>  unhandled exception.
>  Running - Test Case is in progress.
>  Paused - This status is used to denote a problem with the test case
>  itself that prevents the test from being completed.
>  Blocked - Test Case has a dependency that has failed.
>  Error - Test environment has problems that prevent Test Case execution. 
>  
>  But I guess such 'error' status will be seldom used. 
So what mean do you think 'warn' with test case result ? 

>  
> >> 8. Multiple contributions for each case -
> >> Support multiple test results for one case run.
> > 
> > Agreed and good catch!  I think it's very important to allow multiple
> > testers to provide feedback on a single test given that the environment
> > under test (hardware) can vary for tests.
>  
>  Yeah, so far, nitrate supports the view of editing logs but only display
>  the result changed at last time.   

Similar with feature 5. 
I  suggest every contributor use different test run . So Nitrate could save all results. 
What we need to do is show all result at a glance. 

>  
> >> 9. Authorities for pages -
> >> Pages on wiki with different namespaces can have different permissions,
> >> so hopefully pages on Nitrate can also have diff permissions.
> > 
> > I was confused by this a little earlier as well.  I don't understand the
> > concept of pages within nitrate.  Is this to replace the use of our wiki
> > test plan document and validation pages (e.g.
> > http://fedoraproject.org/wiki/QA:Installation_validation_testing)?
>  
>  I meant for example if use certain namespace like 'Test_Results:' or
>  'Test_Day:', these test runs allow anonymous editing, while test runs
>  without such names don't. So nitrate should also provide a way to give
>  different permissions for test runs.    
First , wiki use 'Page' to store and display it's content , and these pages
are all same type, so it use namespace to control permission. Nitrate is more professional
for test case management, use 'plan','case','run' to manage whole
testing process. these could been seen as  three types.
Now Nitrate can control permission of create/ edit/ delete of these three type of
contents. But can't set permission for specific one run / case.  
And now every editing action need account authentication, anonymous only can browse and
view. 
We need some time  to discuss about permission control I think.

>  
> >> 10. Supporting anonymous user read-write access -
> >> Anonymous have read-write access for specified test runs.
> > 
> > +1
> > 
> >> 11. Page protection -
> >> Protect certain plans/cases
> > 
> > Oh, I think I understand now.  Is the analogy here that a valid FAS
> > login would be required, like we currently have with our wiki test cases
> > and test plans?
>  
>  Yeah, it's similar to No.9, a valid login is required to change certain
>  test cases/plans. 
>  
> >> 12. License the content -
> >> License the contents the same with wiki
> > 
> > Or at least listed in the list of "GOOD" licenses -
> > https://fedoraproject.org/wiki/Licensing#Good_Licenses_3
> > 
> >> 13. Upstream project community -
> >> Monitoring it and actively discussing topics in nitrate-devel@, as well
> >> as file/share bugs/requirements in bugzilla
> > 
> > +1 
> > 
> >> 14. Test day page(run) creation - 
> >> In Nitrate each test run are built based on its test plan. Therefore,
> >> test day plan and its cases are needed before creating a test day run.
> >> Should we change our test day process(adding test plan and its cases) or
> >> should we change Nitrate to support creating test runs without test
> >> plan? 
> > 
> > Perhaps something we need to experiment/research with a pilot instance? 
>  
>  Okay, then in a pilot instance, I'll create a test day plan and add
>  cases to it first.
>  
> >> 15. Test cases priority -
> >> Nitrate now has P1, P2, P3..., modify them to Alpha, Beta, and Final?
> > 
> > Alpha/Beta/Final are appropriate for our two test plans; desktop and
> > installation.  I don't know if they apply for all test plans folks might
> > contribute.  But it seems like an easy future adjustment, so for our
> > immediate needs, P3=Alpha, P2=Beta, P1=Final etc...
> > 
> > In general, for all nitrate metadata fields (priority, status,
> > resolution etc...), whomever administers the tool in Fedora should have
> > control over re-defining the values to something appropriate for Fedora.
> > Perhaps that might be a more generic way to phrase the requirement?
>  
>  Ah, I see. Agree with it!
>  
> >> 16. Each case with different platforms -
> >> Better support two platform results(i386 and x86_64) submitting for each
> >> case run 
> > 
> > Sure, a test run can only be for a single architecture, right?  So we'll
> > have two test runs for a compose.
> > 
> >     1. RC1 - Installation (i386)
> >     2. RC1 - Installation (x86_64)
>  
>  Yeah, more runs can achieve this as well as grouping cases etc, but I'm
>  afraid this way runs of each event will be too many for management. 
>  
> >> Welcome to discuss the above features and provide feedback/suggestion.
> >> If there's any important feature missed, or this reminds you the
> >> advantages of other tcms tools etc, feel free to talk.
> > 
> > I know you mention anonymous login support above, can you also list FAS
> > authentication?  Apologies if I missed it already.
>  
>  Do you mean integration with FAS? 
>  
> > 
> > Thank you for continuing to move forward with this initiative.  I'll
> > keep thinking of important features and reply here if I have anything
> > worth sharing :)
> > 
> > Thanks,
> > James
>  
>  Thanks!
>  Hurry
>  -- 
>  Contacts
>  
>  Hurry
>  FAS Name: Rhe 
>  Timezone: UTC+8
>  TEL: [phone]
>  IRC nick: rhe #fedora-qa #fedora-zh
>  
>  _______________________________________________
>  nitrate-devel mailing list
>  nitrate-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/nitrate-devel 

Thanks again and very appreciate for your creative idea about improvement this system.
Please go on feel free to talk about it. 

Best Regards.

Danqing Li

Tel: [phone]
IRC : dli (#tcms, #eng-china)
Red Hat Software (Beijing) Co., Ltd
Room 907, Tower North, Raycom InfoTech Park,
No. 2 Kexueyuan South Road,100080, Beijing

Proposed must-have Features for Nitrate compared with Wiki

[He Rui]

Hi Danqing,

Many thanks for replying this mail.

On Fri, 2011-03-11 at 16:59 +0800, Danqing Li wrote:
>  Hi ,thank you very much for this great feature requirement list.  
>  
>  On Mar 10, 2011, at 6:01 PM, He Rui wrote:
>  
>  > On Wed, 2011-03-09 at 14:43 -0500, James Laska wrote:
>  > > On Wed, 2011-03-09 at 18:06 +0800, He Rui wrote:
>  > > > Greetings,
>  > > > 
>  > > > Now let's continue the discussion[1] about TCMS management.
>  > > > After the
>  > > > feature comparison between wiki and nitrate, the features which
>  > > > are
>  > > > supported in current wiki, but not in nitrate have been
>  > > > identified[2].
>  > > > Among these, I priorized some of them as Must-have features,
>  > > > which were
>  > > > summarized as below[3]:
>  > > > 
>  > > > 1. History rollback -
>  > > > User can view history versions and undo changes.
>  > > 
>  > > Good, I do use this more often than I realize.
>  > > 
>  > > > 2. History comparison -
>  > > > Different versions can compare with each other.
>  
>  
>  Yes , after talk with Hurry I understand this is very important
>  for community cooperation testing.  But the implementation of roll
>  back of test case is really not easy . 
>  
>  
>  In development of 3.4 and 3.5 we plan to improve history function of
>  test plan/case/run, show the edit history of every field. And in fact
>   there has change log for test case. 
>   
I see, if this is a difficult task, IMO we can achieve this step by
step.
>  
>  > > > 
>  > > > 3. Description part in test case -
>  > > > Currently there's no 'Description' part for test case format
> on
>  > > > Nitrate,
>  > > > should we add this part or use 'Setup' part instead?
>  > > 
>  > > No, I don't think 'setup' is the right place since we also use
>  > > 'setup'
>  > > in our wiki test template.  Does nitrate support a summary or any
>  > > other
>  > > field to describe the intent of the test in human readable terms?
>  > 
>  > Oh, it has "Notes" area, maybe we can use that.
>  > 
>  
>  
>  Ok , let me know if you have other requirement about this 'Notes'
>  field. 
Thanks for supporting, at a first thought, we may want the tinyMCE
syntax for it.


>  > 
>  > > 
>  > > > 4. Grouping cases (by media) -
>  > > > Need a way to separate the cases to different groups in one test
>  > > > run.
>  > > > How can we achieve this in Nitrate?
>  > > 
>  > > Probably different test runs of the same plan.  One run would be
>  > > for the
>  > > DVD ISO, one for the netinst.iso etc...  Perhaps this will need
>  > > further
>  > > exploration.
>  > 
>  > Yeah, use different test runs or sorting the cases by order. 
>  > 
>  
>  
>  
>  
>  
>  > 
>  > > > 5. Documents in test result page(Run) - 
>  > > > There's only a little space called 'Note' for documentation
> on
>  > > > Nitrate
>  > > > and it doesn't support syntax. should we ask for more spaces
>  > > > with syntax
>  > > > supported for test runs like test days?
>  > > 
>  > > Can you explain in more detail?  Are you proposing we move the
>  > > test day
>  > > wiki content into nitrate, so it's a one-stop-shop for test
>  > > information
>  > > for that event?
>  > 
>  > I meant in nitrate, there's only a 'Notes' field to support a simple
>  > summary of each test run. For a test day which has many contents[1]
>  > such
>  > as "What to test", "Prerequisite for Test Day", "How to
> test" etc,
>  > it is
>  > hard to put all of these into the 'Notes' field. This field doesn't
>  > support TinyMCE or any other syntax editing.   
>  > 
>  > [1] https://fedoraproject.org/wiki/QA/Test_Days/Template
>  > 
>  
>  
>  I see that Test Day is a very good cooperation testing activity.  I
>  think you can use Test Plan to describe the Test Day Document instead
>  of Test Run.
>  
>  
>  The organizer write all information above in Test Plan document.  And
>  put test cases into this plan. 
>  On that day ,every tester could make a new run for himself/ herself . 
>  
>  
>  I user Nitrate made a test plan to simulate  fedora test day on Feb25,
>  please see the attached screenshot. In this way , information could be
>  stored in plan documents, and test result in every test runs. 
Wow, good idea. Make a plan for each test day first, then include test
cases to the plan and create a run. I will try this when setting up a
test instance later.:)
>  
>  But now Nitrate can't supply the matrix test result show all user's
>  result like the wikipage[1]. 
>  
>  We need talk about  the test result report feature in future
>  development. 

>  > > > 6. Moving test results -
>  > > > Moving previous results from a test run to another. AFAIK,
>  > > > there's no
>  > > > way to copy some of one test run results to another.
>  > > 
>  > > Good catch.  I think with nitrate, you can selectively include
>  > > test
>  > > cases when creating a new test run, right?  Meaning, we wouldn't
>  > > need to
>  > > include *all* tests in the plan when creating a new test run?
>  > 
>  > Yeah. but then we need to make very sure that certain cases results
>  > won't be changed in new run. The summary of the new run results is
>  > also
>  > not intact. Anyway, this is a workaround.  
>  > 
>  > > This brings up another question ... permissions.  With the wiki,
>  > > anyone
>  > > has permission to add or remove tests from a test run.  How will
>  > > that be
>  > > handled in nitrate?
>  > 
>  > Since it's internally use only, the permissions only have three: 1.
>  > anonymous - read. 2. tester - read and write any case/run/plan. 3.
>  > Admin
>  > - manage the tcms system. 
>  > 
>  
>  
>  Sorry ,I am not clear about the necessity of import result form
>  another test run,is it mean those test run very similar, so why make a
>  new run?Do you explain some more detail about this feature's
>  scenario ? 
>   
Okay, take F15 Alpha candidate for example. When RC1 is available, its
test run is created for testing, if it has some blocker bugs which don't
meet release criteria, RC2 with the fixes would be posted for testing. A
new test run would be created and the test cases are the same as RC1
run. But for some cases that we assume the fixes won't affect, on wiki,
we moved their RC1 results which are supposed to be unchanged directly
to the RC2 run, see the example pages[!]. This way, for RC2 run, we have
both new tested and moved results, and the cases are intact for each
run. 


[!]
https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC1_Install
https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install


>  
>  > > 
>  > > > 7. Result type -
>  > > > Nitrate doesn't have 'warn' but has 'error' result
> status.
>  > > > Modify it to
>  > > > 'warn'?
>  > > 
>  > > What is the significance of 'error' in nitrate?  Does that mean
>  > > there
>  > > was a test case error, not necessarily a software error?
>  > 
>  > It presents an environment error, the user guide suggesting the use
>  > of
>  > result status is: 
>  > 
>  > Idle - Default value. The Test Case has not been examined.
>  > Passed - Test Case met all the expected results.
>  > Failed - Test Case did not meet all the expected results, or
>  > produced an
>  > unhandled exception.
>  > Running - Test Case is in progress.
>  > Paused - This status is used to denote a problem with the test case
>  > itself that prevents the test from being completed.
>  > Blocked - Test Case has a dependency that has failed.
>  > Error - Test environment has problems that prevent Test Case
>  > execution. 
>  > 
>  > But I guess such 'error' status will be seldom used.
>  > 
>  
>  
>  So what mean do you think 'warn' with test case result ?  
As James noted,  warn means the test passed, but bugs were discovered
during execution of the test, but did not impact the outcome of the
test.
>  
>  > 
>  > > > 8. Multiple contributions for each case -
>  > > > Support multiple test results for one case run.
>  > > 
>  > > Agreed and good catch!  I think it's very important to allow
>  > > multiple
>  > > testers to provide feedback on a single test given that the
>  > > environment
>  > > under test (hardware) can vary for tests.
>  > 
>  > Yeah, so far, nitrate supports the view of editing logs but only
>  > display
>  > the result changed at last time.  
>  > 
>  
>  
>  
>  
>  Similar with feature 5. 
>  I  suggest every contributor use different test run . So Nitrate could
>  save all results. 
>  What we need to do is show all result at a glance.  
Sorry, what do you mean by using different test run? Do we need to
create more? For each run with some cases, we prefer each case run can
support multiple contributed results.    


>  > 
>  > > > 9. Authorities for pages -
>  > > > Pages on wiki with different namespaces can have different
>  > > > permissions,
>  > > > so hopefully pages on Nitrate can also have diff permissions.
>  > > 
>  > > I was confused by this a little earlier as well.  I don't
>  > > understand the
>  > > concept of pages within nitrate.  Is this to replace the use of
>  > > our wiki
>  > > test plan document and validation pages (e.g.
>  > > http://fedoraproject.org/wiki/QA:Installation_validation_testing)?
>  > 
>  > I meant for example if use certain namespace like 'Test_Results:' or
>  > 'Test_Day:', these test runs allow anonymous editing, while test
>  > runs
>  > without such names don't. So nitrate should also provide a way to
>  > give
>  > different permissions for test runs.   
>  > 
>  
>  
>  First , wiki use 'Page' to store and display it's content , and these
>  pages are all same type, so it use namespace to control permission.
>  Nitrate is more professional for test case management, use
>  'plan','case','run' to manage whole testing process. these could
> been
>  seen as  three types.
>  Now Nitrate can control permission of create/ edit/ delete of these
>  three type of contents. But can't set permission for specific one
>  run / case.  
>  And now every editing action need account authentication, anonymous
>  only can browse and view. 
>  We need some time  to discuss about permission control I think.
>   
Yeah, this can be discussed with FAS integration together. By
controlling the permissions for different user groups, the test
runs/plans/cases/ can be managed separately by different group admins,
this requirement can be achieved then IMO.


>  <skip>
>  
>  Thanks again and very appreciate for your creative idea about
>  improvement this system. Please go on feel free to talk about it.  
I will.:)
>  
>  Best Regards.
>  
>  
>  Danqing Li
>  
>  Tel: [phone]
>  IRC : dli (#tcms, #eng-china)
>  Red Hat Software (Beijing) Co., Ltd
>  Room 907, Tower North, Raycom InfoTech Park,
>  No. 2 Kexueyuan South Road,100080, Beijing
>  
>  
>  
>  
>  _______________________________________________
>  nitrate-devel mailing list
>  nitrate-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/nitrate-devel 
Thanks,
Hurry

-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh

Proposed must-have Features for Nitrate compared with Wiki

[Andrew Ross]

>  > Sorry ,I am not clear about the necessity of import result form
>  > another test run,is it mean those test run very similar, so why make
>  > a
>  > new run?Do you explain some more detail about this feature's
>  > scenario ?
>  >
>  
>  Okay, take F15 Alpha candidate for example. When RC1 is available, its
>  test run is created for testing, if it has some blocker bugs which
>  don't
>  meet release criteria, RC2 with the fixes would be posted for testing.
>  A
>  new test run would be created and the test cases are the same as RC1
>  run. But for some cases that we assume the fixes won't affect, on
>  wiki,
>  we moved their RC1 results which are supposed to be unchanged directly
>  to the RC2 run, see the example pages[!]. This way, for RC2 run, we
>  have
>  both new tested and moved results, and the cases are intact for each
>  run.
>  
>  
>  [!]
>  https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC1_Install
>  https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install
>   
Hi Hurry,

In this use case are you assuming that there would be no regression bugs between RC1 and
RC2? 

Cheers,

Andrew
--

Proposed must-have Features for Nitrate compared with Wiki

[He Rui]

On Mon, 2011-03-14 at 16:23 -0400, Andrew Ross wrote:
>  > > Sorry ,I am not clear about the necessity of import result
> form
>  > > another test run,is it mean those test run very similar, so why make
>  > > a
>  > > new run?Do you explain some more detail about this feature's
>  > > scenario ?
>  > >
>  > 
>  > Okay, take F15 Alpha candidate for example. When RC1 is available, its
>  > test run is created for testing, if it has some blocker bugs which
>  > don't
>  > meet release criteria, RC2 with the fixes would be posted for testing.
>  > A
>  > new test run would be created and the test cases are the same as RC1
>  > run. But for some cases that we assume the fixes won't affect, on
>  > wiki,
>  > we moved their RC1 results which are supposed to be unchanged directly
>  > to the RC2 run, see the example pages[!]. This way, for RC2 run, we
>  > have
>  > both new tested and moved results, and the cases are intact for each
>  > run.
>  > 
>  > 
>  > [!]
>  > https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC1_Install
>  > https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install
>  > 
>  
>  Hi Hurry,
>  
>  In this use case are you assuming that there would be no regression bugs between RC1 and
> RC2?  
For the test cases with moved results, I'm assuming there would be no
regression bugs. But since it's community testing, these tests in each
new run(eg RC2) are still open for testing to encourage contributions to
cover this area.    

>  Cheers,
>  
>  Andrew 
Thanks,
Hurry
-- 
Contacts

Hurry
FAS Name: Rhe 
Timezone: UTC+8
TEL: [phone]
IRC nick: rhe #fedora-qa #fedora-zh