[Fedora QA] #175: Improve transfer of previous test results in the installation matrix

[Fedora QA]

#175: Improve transfer of previous test results in the installation matrix
--------------------------+-------------------------------------------------
  Reporter:  kparal       |       Owner:  rhe      
      Type:  enhancement  |      Status:  new      
  Priority:  major        |   Milestone:  Fedora 15
 Component:  Wiki         |     Version:           
Resolution:               |    Keywords:           
--------------------------+-------------------------------------------------
Comment (by rhe):

 Replying to [comment:3 kparal]:
 > Replying to [comment:2 rhe]:
 > >
 > > Thanks for raising this topic, it was originally discussed at:
 > >  - https://fedorahosted.org/fedora-qa/ticket/84
 >
 > Ah, that ticket slipped my attention :-)
 >
 > > {{result|fail|previous rc1 run|12345}} is good for me. Why do we need
 a link here? The result has already been transferred to the page, I don't
 think every previous result should have a link to previous page.
 >
 > I just supposed that [[User:previous run]] could explain a little how to
 result transfer works. So noone thinks "previous run" is a real user name
 :-) But you're right, that's not necessary. And "previous rc1 run" is even
 more explanatory. No need for links.

 Okay, then I'll begin using the format as {{result|status|previous <build>
 run}} from the next text event.

 > > Ah yeah, when one issue affected all cases in the same area, I used to
 only assign a bug id to one case, use {{result|fail}} to the others. It's
 not due to the transfer, I will assign the bug number to all cases to
 reduce the confusion.
 >
 > Thanks, that will help. Maybe we can share the reference number somehow
 (so the bug is not duplicated many times)?

 Yes, how do you think the reference No.29 in
 [https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test#References
 current install test]?

 > > The transfer of results depend on the change of anaconda/certain
 components, not the results themselves in my opinion. It's not necessary
 to retest many cases just because they passed in previous run. For
 example, if the change of a new build won't affect the partitioning and
 recovery parts, which were all passed in previous runs, do we need to test
 them again and again in every new run? Certainly I can't guarantee the
 passed case still pass in new run, that's why these results should be
 distinguished with new tested ones.
 >
 > We can never be sure where we can find some regressions. Even comments
 change can break up stuff :-) OTOH I understand saving our resources. I
 have split opinions on this. But if you selectively transfer only
 "probably safe" passes, I am absolutely fine with that.

 Right, transfer task need be very careful. So when all tests in a test
 area seem safe passes, I would still keep at least one test with no
 previous result so that every test area is tested in each run.

 > > >  5. We should document all of this in the "Test Results Format"
 table and also advise people how to interact with these transfered
 results.
 > > I can see what you mean, but I'm afraid that rule will make posting
 results more complicated, and also different environments could have
 different results. Sometimes I can't reproduce a issue even though there's
 not updates fixing it. If I know it's not fixed, I will keep previous
 result, if I don't know, I will just remove it. So my general idea is: if
 you test the case carefully and step by step according to the case, you
 can replace the previous result with yours. I assume most testers will pay
 attention on the previous results and carefully remove them. Besides, we
 have history rollback and bugzilla for tracking, do we really need such
 strict rule?
 >
 > I see. So what about this - no rules at all. Let the tester just add
 their results, don't recommend them any replacing of the old ones. At the
 end of the day we will look over it and decide where we believe the old
 one should be kept (or transferred to the next run) and where it should be
 erased. I think it's better to count on our judgement than on arbitrary
 tester's judgement. But I have no hard opinions in here.

 Good idea. Agree with it. :)

 I've modified the key section to:

  *
 https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key

 Do we need to add more instructions? Feel free to adjust it.

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/175#comment:4>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance