[AutoQA] #298: test.py - split postprocess_iteration reporting into standalone methods

[fedora-badges]

#298: test.py - split postprocess_iteration reporting into standalone methods
----------------------+-----------------------------------------------------
 Reporter:  jskladan  |       Owner:       
     Type:  task      |      Status:  new  
 Priority:  minor     |   Milestone:  0.5.0
Component:  core      |    Keywords:       
----------------------+-----------------------------------------------------
 At the moment, postprocess_iteration() handles the whole reporting/sending
 results. When the run_once ends, this method takes the content of
 self.{result, summary, highlights, outputs} and automagically sends an
 email to the mailing list, creates output.log file, etc.

 This approach was very reasonable for tests which actually test just one
 thing (update/build/etc), because just a single result is to be sent.

 With the new tests like Depcheck and Upgradepath, we'd love to be able to
 force-send several results as the test proceeds - e.g. to be able to send
 bodhi comment to every update, but send just one overall email...

 So what I'd like to have is:

 1) Take postprocess_iteration, and split it into standalone methods
 according to the 'destination' of the report (e.g. send_email(),
 create_output_log(), ...). These will be able to take {result, summary,
 highlights, outputs} parameters, which will override the 'automagicall'
 self.{result, summary, highlights, outputs}. I.e. if I set result
 parameter, summary, highlights and outputs will be filled using the 'self'
 variables, and so on.

 2) Take the bodhi-reporting method (as used in depcheck and upgradepath),
 and move it to the AutoQATest class.

 3) Add a method report_results(), again with the same parameters ({result,
 summary, highlights, outputs}) and one more (e.g. a dictionary) to control
 which reporting methods to call (i.e. to be able to say "send email, store
 in resultsdb, but do not send bodhi comment"). By default, this will call
 all the reporting routines.

 4) In postprocess_iteration(), call the report_results(), so the
 'automagical reporting' behaviour is not changed.

 5) Add a variable, which will turn on/off the call of report_results()
 (True as default) in postprocess_iteration() (needs to be some attribute
 of AutoQATest class, since postprocess_iteration() does not take
 arguments).
 This is for the wrapper-writer, to be able to control if results are to be
 sent in the postprocess_iteration() or not (imagine Upgradepath - we will
 report all the results in the run_once(), and do not need postprocess
 iteration to actually send anything).

 -------------------------------

 This is also preparation for resultsdb, because it solves the problem with
 reporting multiple results from one test. Also turning the resultsdb
 reporting on is a matter of adding one method, and calling it from
 report_results() (and adding more types of reporting [I'm looking at you
 fedora message bus!] in the future will be also this simple.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/298>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#298: test.py - split postprocess_iteration reporting into standalone methods
----------------------+-----------------------------------------------------
 Reporter:  jskladan  |        Owner:  kparal  
     Type:  task      |       Status:  assigned
 Priority:  major     |    Milestone:  0.5.0   
Component:  core      |   Resolution:          
 Keywords:            |  
----------------------+-----------------------------------------------------
Changes (by kparal):

  * owner:  => kparal
  * status:  new => assigned

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/298#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#298: test.py - split postprocess_iteration reporting into standalone methods
---------------------------+------------------------------------------------
 Reporter:  jskladan       |        Owner:  kparal  
     Type:  task           |       Status:  assigned
 Priority:  major          |    Milestone:  0.5.0   
Component:  documentation  |   Resolution:          
 Keywords:                 |  
---------------------------+------------------------------------------------
Changes (by kparal):

  * component:  core => documentation

Comment:

 Pushed into master as 8a2b9ce2410009da8a4715ea0e33342ef46d4b2f .

 This will definitely need some documentation adjustments.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/298#comment:3>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#298: test.py - split postprocess_iteration reporting into standalone methods
----------------------+-----------------------------------------------------
 Reporter:  jskladan  |        Owner:  kparal
     Type:  task      |       Status:  closed
 Priority:  major     |    Milestone:  0.5.0 
Component:  core      |   Resolution:  fixed 
 Keywords:            |  
----------------------+-----------------------------------------------------
Changes (by kparal):

  * status:  assigned => closed
  * resolution:  => fixed
  * component:  documentation => core

Comment:

 I have changed the wiki docs here:


https://fedoraproject.org/w/index.php?title=Writing_AutoQA_Tests&acti...

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/298#comment:4>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project