[Fedora QA] #175: Improve transfer of previous test results in the installation matrix

[Fedora QA]

#175: Improve transfer of previous test results in the installation matrix
--------------------------+-------------------------------------------------
  Reporter:  kparal       |       Owner:  rhe      
      Type:  enhancement  |      Status:  new      
  Priority:  major        |   Milestone:  Fedora 15
 Component:  Wiki         |     Version:           
Resolution:               |    Keywords:           
--------------------------+-------------------------------------------------
Comment (by rhe):

 Replying to [ticket:175 kparal]:
 > Currently in our installation matrix there are some test results which
 are from "anonymous" source (no name filled in). Example:
 >
 > https://fedoraproject.org/wiki/Test_Results:Fedora_15_Alpha_RC2_Install
 >
 > I was little confused by that, but rhe explained me that those are the
 test results that were transfered from the previous test result (i.e. from
 F15 Alpha RC1 in this case). We already agreed if would be better if we
 added <ref> with explanation to each such result, so other testers
 understand the meaning of it (I suppose many of them might be confused
 same as I was). Still, I think there are some improvements that could be
 made. I'd like to discuss them in this ticket.

 Thanks for raising this topic, it was originally discussed at:
  - https://fedorahosted.org/fedora-qa/ticket/84

 I supposed testers would look up the unknown results icons at
 [https://fedoraproject.org/wiki/QA:Fedora_15_Install_Results_Template#Key
 key section] where I explained. But it seems this part could be ignored, I
 agree to have a more visible solution.

 > My proposal:
 >  1. Transferring old results to the new matrices is a great idea, I
 strongly support that.
 >  2. In order to avoid confusion about "anonymous" results, we should
 clearly separate new results from transfered results. Using a <ref>
 comment is possible, but I'd like to do something more visible. We could
 do something like {{result|fail|rc1|12345}} and then link rc1 to correct
 wiki page (if {{result}} template is not suitable for it, we could create
 {{oldresult}} template or similar). But even though it could be scripted,
 it seems like too much work. Easier solution is to do
 {{result|fail|previous run|12345}}, and if "previous run" is linked to
 [[User:previous run]], we could use that wiki page for short explanation.

 {{result|fail|previous rc1 run|12345}} is good for me. Why do we need a
 link here? The result has already been transferred to the page, I don't
 think every previous result should have a link to previous page.

 >  3. We should transfer only fails and warnings, not passes. The
 rationale is that the pass result makes you (me, many people) think: "Hey,
 there's a pass, let's work on something else". But pass from previous run
 doesn't mean pass in current run (as I experienced today, when I changed
 two previous passes to fails). Therefore warnings and fails are good to
 know (we can check whether they are fixed or not, and we won't forget
 about them), but passes are counter-productive - let's leave empty fields
 instead.

 The transfer of results depend on the change of anaconda/certain
 components, not the results themselves in my opinion. It's not necessary
 to retest many cases just because they passed in previous run. For
 example, if the change of a new build won't affect the partitioning and
 recovery parts, which were all passed in previous runs, do we need to test
 them again and again in every new run? Certainly I can't guarantee the
 passed case still pass in new run, that's why these results should be
 distinguished with new tested ones.

 >  4. All transfered results must have bugzilla number assigned. Otherwise
 there's no information value. If I don't know what was broken, I can't
 check whether it's been fixed or not.

 Ah yeah, when one issue affected all cases in the same area, I used to
 only assign a bug id to one case, use {{result|fail}} to the others. It's
 not due to the transfer, I will assign the bug number to all cases to
 reduce the confusion.

 >  5. We should document all of this in the "Test Results Format" table
 and also advise people how to interact with these transfered results. My
 idea: If there was a transfered warn/fail, and you checked that the
 problem is fixed, remove it from the table (and put your result in there
 instead). If you checked that the problem is still present, again remove
 it from the table and put your result in there instead (referencing the
 same bug number). If you didn't check the problem, put your result in the
 table and leave the transfered result intact.

 I can see what you mean, but I'm afraid that rule will make posting
 results more complicated, and also different environments could have
 different results. Sometimes I can't reproduce a issue even though there's
 not updates fixing it. If I know it's not fixed, I will keep previous
 result, if I don't know, I will just remove it. So my general idea is: if
 you test the case carefully and step by step according to the case, you
 can replace the previous result with yours. I assume most testers will pay
 attention on the previous results and carefully remove them. Besides, we
 have history rollback and bugzilla for tracking, do we really need such
 strict rule?

-- 
Ticket URL: <https://fedorahosted.org/fedora-qa/ticket/175#comment:2>
Fedora QA <http://fedorahosted.org/fedora-qa>
Fedora Quality Assurance