[AutoQA] #330: upgradepath: improve behavior when pushing update to multiple releases

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |       Owner:       
     Type:  enhancement  |      Status:  new  
 Priority:  major        |   Milestone:  0.5.0
Component:  tests        |    Keywords:       
-------------------------+--------------------------------------------------
 This ticket stems from my talk to mcepl and our following discussion:

 https://fedorahosted.org/pipermail/autoqa-devel/2011-May/002215.html

 Package maintainers usually push fixes to many Fedora releases at once.
 Upgradepath usually fails for all but the highest one. We should improve
 the output.

  1. If upgradepath passes, no changes.
  2. If upgradepath fails, check update-pending repos for the failed
 sections and determine whether the same proposed update is available
 there. If it is, check upgradepath again and determine whether test would
 pass if all those proposed updates for all releases were pushed at the
 same time. Inform the user about it.
  3. If the condition in 2) is valid, change FAIL result to
 NEEDS_INSPECTION.

 I'll gladly take this ticket when I finish my current work.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:       
     Type:  enhancement  |       Status:  new  
 Priority:  major        |    Milestone:  0.6.0
Component:  tests        |   Resolution:       
 Keywords:               |  
-------------------------+--------------------------------------------------
Changes (by kparal):

  * milestone:  0.5.0 => 0.6.0

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:1>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Changes (by kparal):

  * owner:  => kparal
  * status:  new => assigned

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by kparal):

 The patch is in RB:

 https://fedorahosted.org/reviewboard/r/173/

 or in Git as origin/ticket330.

 Please review.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:3>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Changes (by jlaska):

 * cc: sgallagh (added)

Comment:

 On #fedora-devel, sgallagh requested this very feature for the next autoqa
 release.  He was happy to see the change was already in development.  I
 directed sgallagh to the mock-up kparal generated
 (http://kparal.fedorapeople.org/autoqa/t330.html).  I'm passing along
 sgallagh's feedback into this ticket.

 The only suggested sgallagh had was to change the individual test result
 from FAIL to INFO (or WARN).  The rationale being that FAIL implies there
 is something the maintainer can do to resolve the problem.  In the case of
 pushing updates, the only course of action a maintainer has is to ensure
 they push the updates at the same time (so that the nightly repos are
 updated at the same time).

 I've included the chat log below.
 {{{
 07:36  sgallagh: jlaska: I've got an autoqa bug for you :)
 07:37  sgallagh: jlaska: Disregard that. I misread.
 07:48  jlaska: sgallagh: okay
 07:48  sgallagh: jlaska: One enhancement request, though :)
 07:48  jlaska: sgallagh: let's have it!
 07:48  sgallagh: jlaska: Autoqa should also check updates-testing-pending
 for Fedora N+1
 07:49  jlaska: this is for the upgradepath test?
 07:49  sgallagh: jlaska: Because if I push sssd-1.5.11-2.fc15 and then
 immediately push sssd-1.5.11-2.fc14 as well, autoqa complains that
 sssd-1.5.11-2.  fc14 breaks upgrades
 07:49  jlaska: yeah
 07:49  sgallagh: Even though they're both pending and will hit stable in
 the same push
 07:50  jlaska: iirc, a fix for that will be landing in the next version
 ... checking ...
 07:50 --> mmorsi_ [~mmorsi(a)cpe-24-58-175-240.twcny.res.rr.com] has joined
 #fedora-devel
 07:51  jlaska: sgallagh: https://fedorahosted.org/autoqa/ticket/330 I
 believe kparal may be working on that already
 07:51  jlaska: here's a sample of how that will look ...
 http://kparal.fedorapeople.org/autoqa/t330.html
 07:52  sgallagh: Looks good to me :) Glad you guys are on top of this
 07:52  jlaska: sgallagh: if you have other suggestions for how to
 present/visualize the result for this ... don't hesitate to drop a line in
 the ticket   (or autoqa-devel(a)fedorahosted.org)
 07:55  sgallagh: WIll do
 07:55  sgallagh: jlaska: I'd suggest that this shouldn't be FAIl but INFO,
 though
 07:56  sgallagh: Because if the package is in updates-pending, then it's
 pretty much guaranteed to be pushed at the same time.
 07:56  jlaska: I think the problem we've had is it wasn't guarrunteed (or
 at least wasn't 100%)
 07:57  sgallagh: jlaska: Well, it's vulnerable to interrupted pushes I
 guess
 07:57  jlaska: and due to any number of repo/mirror/sync issues, one
 createrepo could fail, while the other releases could be fine
 07:57  jlaska: yeah
 07:57  sgallagh: But since there's nothing the maintainer can do at that
 point, I think it's disingenuous to mark it as a failure
 07:57  jlaska: okay
 07:57  sgallagh: Because that to me says "You need to fix this!"
 07:57  jlaska: so something like WARN || INFO would be better?
 07:57  jlaska: ah
 07:58  jlaska: I guess the only control a maintainer has at that point is
 to ensure they push them all at the same time?
 07:58  sgallagh: jlaska: Well, I think if I push them in the correct
 order, (f15, then f14) I've already done everything I'm equipped to
 07:59  sgallagh: As a maintainer, I can't have any control after that as
 to whether they actually hit the repos at the same time.
 07:59  sgallagh: So there's always going to be a slight risk that there's
 a window of bad upgrade.
 08:00  jlaska: right
 08:00  sgallagh: But I don't think it's fair to require that f14 pushes
 wait until f15 is actually IN stable
 08:00  jlaska: yeah, that's out of autoqa's domain anyway
 08:00  jlaska: more policy I mean
 08:01  sgallagh: jlaska: Right, but I think that's a policy no one would
 be willing to follow anyway
 08:01  sgallagh: And it wouldn't work for security updates, etc. very well
 08:01  jlaska: it'd sure make a lot of folks cranky :)
 08:01  jlaska: sgallagh: I'll add your feedback to the autoqa ticket where
 kparal is tracking this change ... and cc you in case I misstate anything
 08:02  jlaska: sound good?
 08:02  sgallagh: Sure
 }}}

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:4>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by kparal):

 Thanks, James and sgallagh, for providing that feedback.

 I couldn't understand where do you see FAIL result in the provided example
 when I marked the result as NEEDS_INSPECTION (in my understanding that
 should serve the same purpose as WARN/INFO).

 But then I saw it - you mean the
 {{{
 [FAIL] dist-f15 + dist-f15-updates
 }}}
 line. Do you think that using WARN keyword would look better?

 We don't have WARN keyword defined at the moment. But we can discuss
 adding it.

 Apart from that, I find these sentences intriguing:
 {{{
 But since there's nothing the maintainer can do at that point, I think
 it's disingenuous to mark it as a failure
 Well, I think if I push them in the correct order, (f15, then f14) I've
 already done everything I'm equipped to
 }}}

 That is a great question. Do we create these tests for package maintainers
 or release engineers? When we have proper architecture, I see this test as
 targeted for releng. But in the meantime I understand the package
 maintainers as the target audience. And if there is really nothing they
 can do once they pushed the update to all releases, should we "punish"
 them with WARN and NEEDS_INSPECTION results? Or could we see it as a full-
 fledged PASS?

 Of course this would apply only if the upgrade path constraint is
 perfectly satisfied for all the -pending repos. If there is just a single
 problem, it would still be FAIL (e.g. update pushed to F{13,14}-pending,
 but not to F15-pending).

 Yes, with this approach we loose some corner-cases. Like pushing update to
 F{14,15}-pending and then unpushing only the F15 one. But do we really
 have it covered at the present time with the super-correct approach? Most
 probably not, because noone waits with pushing the update until
 upgradepath turns PASS. In sgallagh's words:
 {{{
 But I don't think it's fair to require that f14 pushes wait until f15 is
 actually IN stable
 I think that's a policy no one would be willing to follow anyway
 }}}

 Currently I see a benefit loosing up a little our requirements (and
 allowing some corner cases), because we're targeting the wrong audience
 with the failures. What do you think?

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:5>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by sgallagh):

 Replying to [comment:5 kparal]:

>  That is a great question. Do we create these tests for package
>  
 maintainers or release engineers? When we have proper architecture, I see
 this test as targeted for releng. But in the meantime I understand the
 package maintainers as the target audience. And if there is really nothing
 they can do once they pushed the update to all releases, should we
 "punish" them with WARN and NEEDS_INSPECTION results? Or could we see it
 as a full-fledged PASS?
>  
 I think if we're displaying an error message here, all we're doing is
 training our package maintainers to ignore error messages (probably not
 the goal!).

>  Of course this would apply only if the upgrade path constraint is
>  
 perfectly satisfied for all the -pending repos. If there is just a single
 problem, it would still be FAIL (e.g. update pushed to F{13,14}-pending,
 but not to F15-pending).
>  
 Yes, exactly. If it's satisfied for all updates-pending, then it should be
 a PASS, I think.

>  Yes, with this approach we loose some corner-cases. Like pushing
> update
>  
 to F{14,15}-pending and then unpushing only the F15 one. But do we really
 have it covered at the present time with the super-correct approach? Most
 probably not, because noone waits with pushing the update until
 upgradepath turns PASS. In sgallagh's words:
>  {{{
>  But I don't think it's fair to require that f14 pushes wait until f15 is
>  
 actually IN stable
 > I think that's a policy no one would be willing to follow anyway
 > }}}
>  
 > Currently I see a benefit loosing up a little our requirements (and
 allowing some corner cases), because we're targeting the wrong audience
 with the failures. What do you think?

 This sounds like a good plan to me.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:6>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by kparal):

 I have created a mockup how the report could look like (compare to the
 previous one in comment 4):

 http://kparal.fedorapeople.org/autoqa/t330_2.html

 Josef agrees this is probably a good way to go.

 James, what's your take on that?

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:7>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by jlaska):

 Replying to [comment:7 kparal]:
>  James, what's your take on that?
>  
 I like the mock-up.  Only suggestion I have is around the wording of the
 INFO result.  Does the following make sense?

    "Upgradepath may fail if the discovered pending package is not pushed
 before, or at the same time as, the tested package"

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:8>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by kparal):

 OK, I'll discard the current patch from the reviewboard and create another
 one using the discussed approach.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:9>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by kparal):

 The new patch is in:

 https://fedorahosted.org/reviewboard/r/180/

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:10>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal  
     Type:  enhancement  |       Status:  assigned
 Priority:  major        |    Milestone:  0.6.0   
Component:  tests        |   Resolution:          
 Keywords:               |  
-------------------------+--------------------------------------------------
Comment (by jskladan):

 reviewed

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:11>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
---------------------------+------------------------------------------------
 Reporter:  kparal         |        Owner:  kparal  
     Type:  enhancement    |       Status:  assigned
 Priority:  major          |    Milestone:  0.6.0   
Component:  documentation  |   Resolution:          
 Keywords:                 |  
---------------------------+------------------------------------------------
Changes (by kparal):

  * component:  tests => documentation

Comment:

 Pushed as a935d329e857c86320ba9bfe25438f45ca2452d0 .

 This will certainly require documentation changes, re-assigning to
 documentation component.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:12>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
---------------------------+------------------------------------------------
 Reporter:  kparal         |        Owner:  kparal  
     Type:  enhancement    |       Status:  assigned
 Priority:  major          |    Milestone:  0.6.0   
Component:  documentation  |   Resolution:          
 Keywords:                 |  
---------------------------+------------------------------------------------
Comment (by kparal):

 I found one more issue with upgradepath, fixed it as another commit
 3c3a5b9b7d4d4cd96d98cca2972d542a94b9c894 .

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:13>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#330: upgradepath: improve behavior when pushing update to multiple releases
-------------------------+--------------------------------------------------
 Reporter:  kparal       |        Owner:  kparal
     Type:  enhancement  |       Status:  closed
 Priority:  major        |    Milestone:  0.6.0 
Component:  tests        |   Resolution:  fixed 
 Keywords:               |  
-------------------------+--------------------------------------------------
Changes (by kparal):

  * status:  assigned => closed
  * resolution:  => fixed
  * component:  documentation => tests

Comment:

 I have updated the documentation:


https://fedoraproject.org/w/index.php?title=AutoQA_tests%2FUpgradepath&am...

 That means we have to release soon now :-)

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/330#comment:14>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project