[AutoQA] #204: add depcheck to post-bodhi-update hook

[fedora-badges]

#204: add depcheck to post-bodhi-update hook
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |       Owner:                                                
     Type:  task    |      Status:  new                                           
 Priority:  major   |   Milestone:  Package Update Acceptance Test Plan - depcheck
Component:  tests   |     Version:  1.0                                           
 Keywords:          |  
--------------------+-------------------------------------------------------
 Actually get the depcheck test running on post-bodhi-update.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#204: add depcheck to post-bodhi-update hook
---------------------+------------------------------------------------------
  Reporter:  wwoods  |       Owner:                                                
      Type:  task    |      Status:  new                                           
  Priority:  major   |   Milestone:  Package Update Acceptance Test Plan - depcheck
 Component:  tests   |     Version:  1.0                                           
Resolution:          |    Keywords:                                                
---------------------+------------------------------------------------------
Comment (by wwoods):

 The key bit here is putting some code in the test wrapper that provides
 depcheck with a ''full'' list of ''all'' the packages
that have requested
 (but not been granted) the given kojitag. This is necessary for correct
 testing - we can't test updates individually, since updates can be
 interdependent.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:1>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#204: add depcheck to post-bodhi-update hook
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:                                                
     Type:  task    |       Status:  new                                           
 Priority:  major   |    Milestone:  Package Update Acceptance Test Plan - depcheck
Component:  tests   |   Resolution:                                                
 Keywords:          |  
--------------------+-------------------------------------------------------
Comment (by kparal):

 The test wrapper can't help in this case. If you run "watch-bodhi-
 requests.py --dryrun" you will see that for every new update in Bodhi a
 new autoqa instance is called with different arguments:
 {{{
 autoqa post-bodhi-update --title libvmime07-0.7.1-4.fc14 --updateid
 FEDORA-2010-17068 --target-tag dist-f14-updates --arch i686 --arch x86_64
 --arch noarch libvmime07-0.7.1-4.fc14
 autoqa post-bodhi-update --title libisofs-0.6.38-1.fc14 --updateid
 FEDORA-2010-17030 --target-tag dist-f14-updates --arch i686 --arch x86_64
 --arch noarch libisofs-0.6.38-1.fc14
 autoqa post-bodhi-update --title openmpi-1.4.3-7.fc14 --updateid
 FEDORA-2010-16765 --target-tag dist-f14-updates --arch i686 --arch x86_64
 --arch noarch openmpi-1.4.3-7.fc14
 ...
 }}}

 The test wrapper will receive only arguments from each of this call,
 separately. It can't group it together.

 We will need to do some ugly hack (like creating new watcher tailored to
 depcheck) or change our workflow logic quite a bit. Probably the best way
 would be to adjust watchers to pass over all information at once. And then
 the hook or the particular test's control.autoqa file can adjust the
 handling as needed.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#204: add depcheck to post-bodhi-update hook
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:       
     Type:  task    |       Status:  new  
 Priority:  major   |    Milestone:  0.4.4
Component:  tests   |   Resolution:       
 Keywords:          |  
--------------------+-------------------------------------------------------
Changes (by kparal):

  * milestone:  Package Update Acceptance Test Plan - depcheck => 0.4.4

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:3>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[AutoQA] #204: Adjust watchers to pass over all information at once (required for depcheck)

[fedora-badges]

#204: Adjust watchers to pass over all information at once (required for
depcheck)
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:  jskladan
     Type:  task    |       Status:  assigned
 Priority:  major   |    Milestone:  0.4.4   
Component:  tests   |   Resolution:          
 Keywords:          |  
--------------------+-------------------------------------------------------
Changes (by jskladan):

  * owner:  => jskladan
  * status:  new => assigned
  * summary:  add depcheck to post-bodhi-update hook => Adjust watchers to
              pass over all information at once (required for
              depcheck)

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:4>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[AutoQA] #204: Adjust watchers to pass over all information at once (required for depcheck)

[fedora-badges]

#204: Adjust watchers to pass over all information at once (required for
depcheck)
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:  jskladan
     Type:  task    |       Status:  assigned
 Priority:  major   |    Milestone:  0.4.4   
Component:  tests   |   Resolution:          
 Keywords:          |  
--------------------+-------------------------------------------------------
Comment (by jlaska):

 I spoke to wwoods about some of the remaining depcheck tasks, and he
 discussed a depcheck task dependency on this ticket.  He clarified some of
 my confusion on this ticket ... I'll try to summarize below.

 As Kamil notes in comment#2, post-bodhi-update currently will launch
 'autoqa' for every new update discovered.  Wwoods indicated that this is
 the desired outcome for the watcher.  I didn't understand why until he
 mentioned that we want our watchers to match the behavior expected in the
 'glorious future'.  That is, when the message bus is in action, AutoQA
 would theoretically receive notification about *each* bodhi update.  It
 wouldn't receive a single notification for when the pool of bodhi updates
 changes (that could certainly be another event to monitor, but it sounds
 like it was different from the original intent for post-bodhi-update).
 Additionally, there may be tests that we want to run against a single
 bodhi-update (rpmlint/rpmguard/initscripts by the looks of
 http://fedoraproject.org/wiki/QA:Package_Update_Acceptance_Test_Plan)?
 Will felt we should keep the post-bodhi-update watcher behavior as it is
 now, but provide some shared code that can grab a list of bodhi-updates
 needed for testing.  This code would be used in the wrapper for tests that
 need to operate over the entire set, such as depcheck, future-conflicts-
 test and upgradepath.

 Of course, all that is a bit of guess work since we don't have a message
 bus in action.  But it at least describes the original idea for the bodhi
 event notification.  In general though, for tests, like depcheck, that
 operate over the entire update set ... it's an odd workflow that we
 haven't yet fully documented.  For example, some questions remain for me
 ...
  1. Should the test provide feedback for all packages in the set that
 PASS, or only the package it was asked to test?  This seems like we would
 have a lot of duplicate testing going on if we only provide feedback for
 the requested update.
  2. If providing feedback against all -pending updates, shouldn't the
 wrapper get a list of *untested* -pending updates, and only test that set?
 If no untested -pending updates ... no testing needed?  Otherwise, again
 we'd have repeated testing.
  3. Then there is the whole timing issue kparal and wwoods were discussing
 regarding multiple depcheck tests running at the same time, each trying to
 move items out of the -pending set, and into the ACCEPTED -pending set.

 Will expects to have another blog post published shortly to attempt to
 answer some questions about the flow.  Does this make sense?  Are there
 additional flows we need to consider?

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:5>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[AutoQA] #204: Adjust watchers to pass over all information at once (required for depcheck)

[fedora-badges]

#204: Adjust watchers to pass over all information at once (required for
depcheck)
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:  jskladan
     Type:  task    |       Status:  assigned
 Priority:  major   |    Milestone:  0.4.4   
Component:  tests   |   Resolution:          
 Keywords:          |  
--------------------+-------------------------------------------------------
Comment (by wwoods):

>   [It] seems like we would have a lot of duplicate testing going on if
> we
>  
 only provide feedback for the requested update.

 Agreed. It seems to make sense for the test to provide feedback for
 ''all'' packages that pass, not just the package that triggered the
test.

>   2. If providing feedback against all -pending updates, shouldn't
> the
>  
 wrapper get a list of *untested* -pending updates, and only test that set?
 If no untested -pending updates ... no testing needed?  Otherwise, again
 we'd have repeated testing.

 No. Repeated testing is ''required'' for packages that haven't been
 accepted. We have to keep testing them until they get accepted, or they
 get pulled out of the 'pending' list (by being obsoleted or forcibly
 removed).

 To illustrate the point: say you have an update for package A, which
 requires package B. The build for B isn't done yet, though, so A gets
 tested and fails. When the update for B arrives, we need to test A again -
 and then it will pass, and be accepted.

 So the wrapper needs:
  1. A way to get the pending package list (let's call it
 {{{get_pending()}}})
  1. A way to get the accepted package list (which is a subset of the
 pending list). Let's call it {{{get_accepted()}}}.

 We can do {{{get_pending()}}} just by listing the contents of the -pending
 tag. {{{get_accepted()}}} is not as straightforward, but we can achieve
 that by checking our previous test results (as reported in Bodhi or
 wherever) to see which pending packages have previously passed. Something
 like:
 {{{
 accepted = []
 for p in get_pending():
     if check_accepted(p):
         accepted.append(p)
 return accepted
 }}}

>   3. Then there is the whole timing issue kparal and wwoods were
>  
 discussing regarding multiple depcheck tests running at the same time,
 each trying to move items out of the -pending set, and into the ACCEPTED
 -pending set.

 Yes - this is ticket #248. And Kamil is correct: if the pending/accepted
 sets change during the test, our results become invalid. So when this
 happens, the test wrapper needs to retry the test with the new data.

 We can optimize this though. If any new packages have been added to the
 'pending' set, that means that a new test run has already been scheduled
 for those new packages - so we can just cancel our run and let the other
 one handle it.

 See ticket #248 for details.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:6>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[AutoQA] #204: Adjust watchers to pass over all information at once (required for depcheck)

[fedora-badges]

#204: Adjust watchers to pass over all information at once (required for
depcheck)
--------------------+-------------------------------------------------------
 Reporter:  wwoods  |        Owner:  jskladan
     Type:  task    |       Status:  closed  
 Priority:  major   |    Milestone:  0.4.4   
Component:  tests   |   Resolution:  fixed   
 Keywords:          |  
--------------------+-------------------------------------------------------
Changes (by jskladan):

  * status:  assigned => closed
  * resolution:  => fixed

Comment:

 The ability to batch-schedule is present in the new_koji_watcher branch.
 Closing.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/204#comment:7>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project