#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: Type: enhancement | Status: new Priority: major | Milestone: 0.5.0 Component: tests | Keywords: -------------------------+-------------------------------------------------- This ticket stems from my talk to mcepl and our following discussion:
https://fedorahosted.org/pipermail/autoqa-devel/2011-May/002215.html
Package maintainers usually push fixes to many Fedora releases at once. Upgradepath usually fails for all but the highest one. We should improve the output.
1. If upgradepath passes, no changes. 2. If upgradepath fails, check update-pending repos for the failed sections and determine whether the same proposed update is available there. If it is, check upgradepath again and determine whether test would pass if all those proposed updates for all releases were pushed at the same time. Inform the user about it. 3. If the condition in 2) is valid, change FAIL result to NEEDS_INSPECTION.
I'll gladly take this ticket when I finish my current work.
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: Type: enhancement | Status: new Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Changes (by kparal):
* milestone: 0.5.0 => 0.6.0
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Changes (by kparal):
* owner: => kparal * status: new => assigned
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by kparal):
The patch is in RB:
https://fedorahosted.org/reviewboard/r/173/
or in Git as origin/ticket330.
Please review.
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Changes (by jlaska):
* cc: sgallagh (added)
Comment:
On #fedora-devel, sgallagh requested this very feature for the next autoqa release. He was happy to see the change was already in development. I directed sgallagh to the mock-up kparal generated (http://kparal.fedorapeople.org/autoqa/t330.html). I'm passing along sgallagh's feedback into this ticket.
The only suggested sgallagh had was to change the individual test result from FAIL to INFO (or WARN). The rationale being that FAIL implies there is something the maintainer can do to resolve the problem. In the case of pushing updates, the only course of action a maintainer has is to ensure they push the updates at the same time (so that the nightly repos are updated at the same time).
I've included the chat log below. {{{ 07:36 sgallagh: jlaska: I've got an autoqa bug for you :) 07:37 sgallagh: jlaska: Disregard that. I misread. 07:48 jlaska: sgallagh: okay 07:48 sgallagh: jlaska: One enhancement request, though :) 07:48 jlaska: sgallagh: let's have it! 07:48 sgallagh: jlaska: Autoqa should also check updates-testing-pending for Fedora N+1 07:49 jlaska: this is for the upgradepath test? 07:49 sgallagh: jlaska: Because if I push sssd-1.5.11-2.fc15 and then immediately push sssd-1.5.11-2.fc14 as well, autoqa complains that sssd-1.5.11-2. fc14 breaks upgrades 07:49 jlaska: yeah 07:49 sgallagh: Even though they're both pending and will hit stable in the same push 07:50 jlaska: iirc, a fix for that will be landing in the next version ... checking ... 07:50 --> mmorsi_ [~mmorsi@cpe-24-58-175-240.twcny.res.rr.com] has joined #fedora-devel 07:51 jlaska: sgallagh: https://fedorahosted.org/autoqa/ticket/330 I believe kparal may be working on that already 07:51 jlaska: here's a sample of how that will look ... http://kparal.fedorapeople.org/autoqa/t330.html 07:52 sgallagh: Looks good to me :) Glad you guys are on top of this 07:52 jlaska: sgallagh: if you have other suggestions for how to present/visualize the result for this ... don't hesitate to drop a line in the ticket (or autoqa-devel@fedorahosted.org) 07:55 sgallagh: WIll do 07:55 sgallagh: jlaska: I'd suggest that this shouldn't be FAIl but INFO, though 07:56 sgallagh: Because if the package is in updates-pending, then it's pretty much guaranteed to be pushed at the same time. 07:56 jlaska: I think the problem we've had is it wasn't guarrunteed (or at least wasn't 100%) 07:57 sgallagh: jlaska: Well, it's vulnerable to interrupted pushes I guess 07:57 jlaska: and due to any number of repo/mirror/sync issues, one createrepo could fail, while the other releases could be fine 07:57 jlaska: yeah 07:57 sgallagh: But since there's nothing the maintainer can do at that point, I think it's disingenuous to mark it as a failure 07:57 jlaska: okay 07:57 sgallagh: Because that to me says "You need to fix this!" 07:57 jlaska: so something like WARN || INFO would be better? 07:57 jlaska: ah 07:58 jlaska: I guess the only control a maintainer has at that point is to ensure they push them all at the same time? 07:58 sgallagh: jlaska: Well, I think if I push them in the correct order, (f15, then f14) I've already done everything I'm equipped to 07:59 sgallagh: As a maintainer, I can't have any control after that as to whether they actually hit the repos at the same time. 07:59 sgallagh: So there's always going to be a slight risk that there's a window of bad upgrade. 08:00 jlaska: right 08:00 sgallagh: But I don't think it's fair to require that f14 pushes wait until f15 is actually IN stable 08:00 jlaska: yeah, that's out of autoqa's domain anyway 08:00 jlaska: more policy I mean 08:01 sgallagh: jlaska: Right, but I think that's a policy no one would be willing to follow anyway 08:01 sgallagh: And it wouldn't work for security updates, etc. very well 08:01 jlaska: it'd sure make a lot of folks cranky :) 08:01 jlaska: sgallagh: I'll add your feedback to the autoqa ticket where kparal is tracking this change ... and cc you in case I misstate anything 08:02 jlaska: sound good? 08:02 sgallagh: Sure }}}
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by kparal):
Thanks, James and sgallagh, for providing that feedback.
I couldn't understand where do you see FAIL result in the provided example when I marked the result as NEEDS_INSPECTION (in my understanding that should serve the same purpose as WARN/INFO).
But then I saw it - you mean the {{{ [FAIL] dist-f15 + dist-f15-updates }}} line. Do you think that using WARN keyword would look better?
We don't have WARN keyword defined at the moment. But we can discuss adding it.
Apart from that, I find these sentences intriguing: {{{ But since there's nothing the maintainer can do at that point, I think it's disingenuous to mark it as a failure Well, I think if I push them in the correct order, (f15, then f14) I've already done everything I'm equipped to }}}
That is a great question. Do we create these tests for package maintainers or release engineers? When we have proper architecture, I see this test as targeted for releng. But in the meantime I understand the package maintainers as the target audience. And if there is really nothing they can do once they pushed the update to all releases, should we "punish" them with WARN and NEEDS_INSPECTION results? Or could we see it as a full- fledged PASS?
Of course this would apply only if the upgrade path constraint is perfectly satisfied for all the -pending repos. If there is just a single problem, it would still be FAIL (e.g. update pushed to F{13,14}-pending, but not to F15-pending).
Yes, with this approach we loose some corner-cases. Like pushing update to F{14,15}-pending and then unpushing only the F15 one. But do we really have it covered at the present time with the super-correct approach? Most probably not, because noone waits with pushing the update until upgradepath turns PASS. In sgallagh's words: {{{ But I don't think it's fair to require that f14 pushes wait until f15 is actually IN stable I think that's a policy no one would be willing to follow anyway }}}
Currently I see a benefit loosing up a little our requirements (and allowing some corner cases), because we're targeting the wrong audience with the failures. What do you think?
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by sgallagh):
Replying to [comment:5 kparal]:
That is a great question. Do we create these tests for package
maintainers or release engineers? When we have proper architecture, I see this test as targeted for releng. But in the meantime I understand the package maintainers as the target audience. And if there is really nothing they can do once they pushed the update to all releases, should we "punish" them with WARN and NEEDS_INSPECTION results? Or could we see it as a full-fledged PASS?
I think if we're displaying an error message here, all we're doing is training our package maintainers to ignore error messages (probably not the goal!).
Of course this would apply only if the upgrade path constraint is
perfectly satisfied for all the -pending repos. If there is just a single problem, it would still be FAIL (e.g. update pushed to F{13,14}-pending, but not to F15-pending).
Yes, exactly. If it's satisfied for all updates-pending, then it should be a PASS, I think.
Yes, with this approach we loose some corner-cases. Like pushing update
to F{14,15}-pending and then unpushing only the F15 one. But do we really have it covered at the present time with the super-correct approach? Most probably not, because noone waits with pushing the update until upgradepath turns PASS. In sgallagh's words:
{{{ But I don't think it's fair to require that f14 pushes wait until f15 is
actually IN stable
I think that's a policy no one would be willing to follow anyway }}}
Currently I see a benefit loosing up a little our requirements (and
allowing some corner cases), because we're targeting the wrong audience with the failures. What do you think?
This sounds like a good plan to me.
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by kparal):
I have created a mockup how the report could look like (compare to the previous one in comment 4):
http://kparal.fedorapeople.org/autoqa/t330_2.html
Josef agrees this is probably a good way to go.
James, what's your take on that?
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by jlaska):
Replying to [comment:7 kparal]:
James, what's your take on that?
I like the mock-up. Only suggestion I have is around the wording of the INFO result. Does the following make sense?
"Upgradepath may fail if the discovered pending package is not pushed before, or at the same time as, the tested package"
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by kparal):
OK, I'll discard the current patch from the reviewboard and create another one using the discussed approach.
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by kparal):
The new patch is in:
https://fedorahosted.org/reviewboard/r/180/
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: tests | Resolution: Keywords: | -------------------------+-------------------------------------------------- Comment (by jskladan):
reviewed
#330: upgradepath: improve behavior when pushing update to multiple releases ---------------------------+------------------------------------------------ Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: documentation | Resolution: Keywords: | ---------------------------+------------------------------------------------ Changes (by kparal):
* component: tests => documentation
Comment:
Pushed as a935d329e857c86320ba9bfe25438f45ca2452d0 .
This will certainly require documentation changes, re-assigning to documentation component.
#330: upgradepath: improve behavior when pushing update to multiple releases ---------------------------+------------------------------------------------ Reporter: kparal | Owner: kparal Type: enhancement | Status: assigned Priority: major | Milestone: 0.6.0 Component: documentation | Resolution: Keywords: | ---------------------------+------------------------------------------------ Comment (by kparal):
I found one more issue with upgradepath, fixed it as another commit 3c3a5b9b7d4d4cd96d98cca2972d542a94b9c894 .
#330: upgradepath: improve behavior when pushing update to multiple releases -------------------------+-------------------------------------------------- Reporter: kparal | Owner: kparal Type: enhancement | Status: closed Priority: major | Milestone: 0.6.0 Component: tests | Resolution: fixed Keywords: | -------------------------+-------------------------------------------------- Changes (by kparal):
* status: assigned => closed * resolution: => fixed * component: documentation => tests
Comment:
I have updated the documentation:
https://fedoraproject.org/w/index.php?title=AutoQA_tests%2FUpgradepath&a...
That means we have to release soon now :-)
autoqa-devel@lists.fedorahosted.org
-
fedora-badges