[Fedora QA] #175: Improve transfer of previous test results in the installation matrix

[Fedora QA]

#175: Improve transfer of previous test results in the installation matrix
--------------------------+-------------------------------------------------
  Reporter:  kparal       |       Owner:  rhe      
      Type:  enhancement  |      Status:  new      
  Priority:  major        |   Milestone:  Fedora 15
 Component:  Wiki         |     Version:           
Resolution:               |    Keywords:           
--------------------------+-------------------------------------------------
Comment (by kparal):

 Replying to [comment:2 rhe]:
 >
 > Thanks for raising this topic, it was originally discussed at:
 >  - https://fedorahosted.org/fedora-qa/ticket/84

 Ah, that ticket slipped my attention :-)

 > {{result|fail|previous rc1 run|12345}} is good for me. Why do we need a
 link here? The result has already been transferred to the page, I don't
 think every previous result should have a link to previous page.

 I just supposed that [[User:previous run]] could explain a little how to
 result transfer works. So noone thinks "previous run" is a real user name
 :-) But you're right, that's not necessary. And "previous rc1 run" is even
 more explanatory. No need for links.

 > Ah yeah, when one issue affected all cases in the same area, I used to
 only assign a bug id to one case, use {{result|fail}} to the others. It's
 not due to the transfer, I will assign the bug number to all cases to
 reduce the confusion.

 Thanks, that will help. Maybe we can share the reference number somehow
 (so the bug is not duplicated many times)?

 > The transfer of results depend on the change of anaconda/certain
 components, not the results themselves in my opinion. It's not necessary
 to retest many cases just because they passed in previous run. For
 example, if the change of a new build won't affect the partitioning and
 recovery parts, which were all passed in previous runs, do we need to test
 them again and again in every new run? Certainly I can't guarantee the
 passed case still pass in new run, that's why these results should be
 distinguished with new tested ones.

 We can never be sure where we can find some regressions. Even comments
 change can break up stuff :-) OTOH I understand saving our resources. I
 have split opinions on this. But if you selectively transfer only
 "probably safe" passes, I am absolutely fine with that.

 > >  5. We should document all of this in the "Test Results Format" table
 and also advise people how to interact with these transfered results.
 > I can see what you mean, but I'm afraid that rule will make posting
 results more complicated, and also different environments could have
 different results. Sometimes I can't reproduce a issue even though there's
 not updates fixing it. If I know it's not fixed, I will keep previous
 result, if I don't know, I will just remove it. So my general idea is: if
 you test the case carefully and step by step according to the case, you
 can replace the previous result with yours. I assume most testers will pay
 attention on the previous results and carefully remove them. Besides, we
 have history rollback and bugzilla for tracking, do we really need such
 strict rule?

 I see. So what about this - no rules at all. Let the tester just add their
 results, don't recommend them any replacing of the old ones. At the end of
 the day we will look over it and decide where we believe the old one
 should be kept (or transferred to the next run) and where it should be
 erased. I think it's better to count on our judgement than on arbitrary
 tester's judgement. But I have no hard opinions in here.

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/175#comment:3>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance