Fedora 13 has released Release Candidate stage. We have reached a state where the known blockers were fixed and were able to make a release candidate. This happened last Thursday, and almost immediately we found a need to spin a second release candidate. From this point on, only items critical to the release will be tagged into the branched Fedora 13 repo.
I'm currently doing the first push of Fedora 13 stable updates. These will be what we call "0-day" updates, that is updates available at release time. Things in the bodhi update system for Fedora 13 can now be pushed "stable" to this updates repo. This allows our maintainers to continue to improve Fedora 13 as release engineering and QA finalize the release bits.
If you have a bug that you think is critical to the release of Fedora 13 and must be fixed on the media, please make your bug block "F13Blocker". We will be reviewing the contents of this blocker bug frequently throughout the days as we near our go / no go decision.
Thank you for your hard work in making Fedora 13 an awesome release!
Jesse Keating said the following on 05/10/2010 04:08 PM Pacific Time:
Fedora 13 has released Release Candidate stage. We have reached a state where the known blockers were fixed and were able to make a release candidate. This happened last Thursday, and almost immediately we found a need to spin a second release candidate. From this point on, only items critical to the release will be tagged into the branched Fedora 13 repo.
I'm currently doing the first push of Fedora 13 stable updates. These will be what we call "0-day" updates, that is updates available at release time. Things in the bodhi update system for Fedora 13 can now be pushed "stable" to this updates repo. This allows our maintainers to continue to improve Fedora 13 as release engineering and QA finalize the release bits.
If you have a bug that you think is critical to the release of Fedora 13 and must be fixed on the media, please make your bug block "F13Blocker". We will be reviewing the contents of this blocker bug frequently throughout the days as we near our go / no go decision.
I've noticed some discussion on #fedora-devel about "taking in nice-to-haves since the release is slipping." If these new packages are not blockers or critical to the release when/where did we decide to deviate from what is stated above?
Who decides what gets in and what is the criteria? Better yet, is this written down anywhere?
John
On Wed, 2010-05-12 at 11:14 -0700, John Poelstra wrote:
I've noticed some discussion on #fedora-devel about "taking in nice-to-haves since the release is slipping." If these new packages are not blockers or critical to the release when/where did we decide to deviate from what is stated above?
Who decides what gets in and what is the criteria? Better yet, is this written down anywhere?
These were high value issues that were either discussed at the various blocker meetings as "we'd take this if we slipped, but wouldn't slip because of it", or made such a decision today while looking at tickets filed in releng requesting the builds.
Judgment calls made by releng, qa, and engineering.
On 05/13/2010 12:36 AM, Jesse Keating wrote:
These were high value issues that were either discussed at the various blocker meetings as "we'd take this if we slipped, but wouldn't slip because of it", or made such a decision today while looking at tickets filed in releng requesting the builds.
Judgment calls made by releng, qa, and engineering.
Since a couple of people complained, have you considered taking in the OpenArena and Wesnoth updates? How about the Pino update? I have a ticket in trac for it.
Rahul
On Thu, 2010-05-13 at 01:00 +0530, Rahul Sundaram wrote:
Since a couple of people complained, have you considered taking in the OpenArena and Wesnoth updates? How about the Pino update? I have a ticket in trac for it.
We took pino, we did not take the games.
On 05/13/2010 01:07 AM, Jesse Keating wrote:
On Thu, 2010-05-13 at 01:00 +0530, Rahul Sundaram wrote:
Since a couple of people complained, have you considered taking in the OpenArena and Wesnoth updates? How about the Pino update? I have a ticket in trac for it.
We took pino, we did not take the games.
I would like to hear some more thoughts on that. IMO, either the game update should getting pulled in or people should just accept that the size of the games are large and updates are going to be big as well and focus on the updates for the default applications instead. It seems a waste of time to create more and more threads on frequent intervals without forming some consensus and guidelines on the right approach. I am happy to follow any guidelines set forward but not as happy to be singled out for an update that does affect except those who deliberately choose to install it.
Rahul
On Thu, May 13, 2010 at 01:25:11 +0530, Rahul Sundaram metherid@gmail.com wrote:
I would like to hear some more thoughts on that. IMO, either the game update should getting pulled in or people should just accept that the size of the games are large and updates are going to be big as well and focus on the updates for the default applications instead. It seems a waste of time to create more and more threads on frequent intervals without forming some consensus and guidelines on the right approach. I am happy to follow any guidelines set forward but not as happy to be singled out for an update that does affect except those who deliberately choose to install it.
I don't see that pulling in the games is a good idea. The release process is that only blockers should be pulled in right now, though that is being bent a little. There should be some clarification done in that regard for the next release, but there is no way these game updates are in that category. While the risk that they would break other things seems very low, I don't think starting out with a smaller updates repository at this point is worth taking the risk. I might be convinced otherwise for a large package that was on the default install as that could result in significant bandwidth savings. But I don't think that applies to wesnoth or openarena.
On 05/13/2010 02:22 AM, Bruno Wolff III wrote:
I don't see that pulling in the games is a good idea. The release process is that only blockers should be pulled in right now, though that is being bent a little. There should be some clarification done in that regard for the next release, but there is no way these game updates are in that category. While the risk that they would break other things seems very low, I don't think starting out with a smaller updates repository at this point is worth taking the risk. I might be convinced otherwise for a large package that was on the default install as that could result in significant bandwidth savings. But I don't think that applies to wesnoth or openarena.
Well then, why do I hear people complain about it?
Rahul
On Wed, May 12, 2010 at 10:53 PM, Rahul Sundaram metherid@gmail.com wrote:
On 05/13/2010 02:22 AM, Bruno Wolff III wrote:
I don't see that pulling in the games is a good idea. The release process is that only blockers should be pulled in right now, though that is being bent a little. There should be some clarification done in that regard for the next release, but there is no way these game updates are in that category. While the risk that they would break other things seems very low, I don't think starting out with a smaller updates repository at this point is worth taking the risk. I might be convinced otherwise for a large package that was on the default install as that could result in significant bandwidth savings. But I don't think that applies to wesnoth or openarena.
Well then, why do I hear people complain about it?
Because they want to ;)
On Thu, May 13, 2010 at 02:23:38 +0530, Rahul Sundaram metherid@gmail.com wrote:
On 05/13/2010 02:22 AM, Bruno Wolff III wrote:
I don't see that pulling in the games is a good idea. The release process is that only blockers should be pulled in right now, though that is being bent a little. There should be some clarification done in that regard for the next release, but there is no way these game updates are in that category. While the risk that they would break other things seems very low, I don't think starting out with a smaller updates repository at this point is worth taking the risk. I might be convinced otherwise for a large package that was on the default install as that could result in significant bandwidth savings. But I don't think that applies to wesnoth or openarena.
Well then, why do I hear people complain about it?
I think because they see the size of updates as important and the risk of something breaking as being very low. Also the rules for including new packages are being bent for some packages, which makes one think they can be bent for other packages.
I don't think the complaints are unreasonable. I think the answer should still be no. The process should be explained as well as at least a general rational for other exceptions granted this go around. And for the future the process should be more closely followed and the process documentation updated if there are reasons we will take updates for packages other than to fix blockers after the RC process has started.
El Wed, 12-05-2010 a las 15:59 -0500, Bruno Wolff III escribió:
On Thu, May 13, 2010 at 02:23:38 +0530, Rahul Sundaram metherid@gmail.com wrote:
On 05/13/2010 02:22 AM, Bruno Wolff III wrote:
I don't see that pulling in the games is a good idea. The release process is that only blockers should be pulled in right now, though that is being bent a little. There should be some clarification done in that regard for the next release, but there is no way these game updates are in that category. While the risk that they would break other things seems very low, I don't think starting out with a smaller updates repository at this point is worth taking the risk. I might be convinced otherwise for a large package that was on the default install as that could result in significant bandwidth savings. But I don't think that applies to wesnoth or openarena.
Well then, why do I hear people complain about it?
I think because they see the size of updates as important and the risk of something breaking as being very low. Also the rules for including new packages are being bent for some packages, which makes one think they can be bent for other packages.
Ironically, the OpenArena update *does* indeed break something:
bernie@giskard:~$ openarena [...] Sound initialization successful. -------------------------------- Loading vm file vm/ui.qvm... ...which has vmMagic VM_MAGIC_VER2 Loading 1550 jump table targets total 0, hsize 1021, zero 1021, min 0, max 0 Segmentation fault (core dumped)
I gave a -1 to this update a few days ago, but it's been ignored:
https://admin.fedoraproject.org/updates/openarena-0.8.5-1.fc13
Whether a broken update comes before, after or during a freeze does not significantly change the overall distro quality perceived by users.
What we really need, imho, is a better QC process between packagers and stable updates. Bodhi was supposed to implement such process, but in fact it's mostly useless because there's no incentive for testers to go there and report about their experience with a package installed from updates-testing.
One reason why I myself neglect to give karma points to updates is that it's hard to remember which packages were installed from updates-testing. Perhaps yum and gpk-application could remind you to do it. Even better, the abrt applet could popup after one day from installing an update to let you file a comment in Bodhi easily without going through the web interface.
I don't think the complaints are unreasonable. I think the answer should still be no. The process should be explained as well as at least a general rational for other exceptions granted this go around. And for the future the process should be more closely followed and the process documentation updated if there are reasons we will take updates for packages other than to fix blockers after the RC process has started.
I think we could take any package at any time, after a sufficient amount of testing has been done on it. If we want to raise the quality bar between RC and release, we may simply require more karma points to push an update to stable.
On Thu, 13 May 2010, Bernie Innocenti wrote:
What we really need, imho, is a better QC process between packagers and stable updates. Bodhi was supposed to implement such process, but in fact it's mostly useless because there's no incentive for testers to go there and report about their experience with a package installed from updates-testing.
One reason why I myself neglect to give karma points to updates is that it's hard to remember which packages were installed from updates-testing. Perhaps yum and gpk-application could remind you to do it. Even better, the abrt applet could popup after one day from installing an update to let you file a comment in Bodhi easily without going through the web interface.
good idea!
sudo yum install fedora-easy-karma fedora-easy-karma
follow the prompts (if any)
-sv
El Thu, 13-05-2010 a las 09:57 -0400, Seth Vidal escribió:
sudo yum install fedora-easy-karma fedora-easy-karma
follow the prompts (if any)
Works fantastically, thanks!
El Thu, 13-05-2010 a las 09:53 -0400, Bernie Innocenti escribió:
I gave a -1 to this update a few days ago, but it's been ignored:
https://admin.fedoraproject.org/updates/openarena-0.8.5-1.fc13
Analyzing the event log in Bodhi exposes where our quality process ultimately fails: the update got my -1, then it was pushed to stable regardless of its negative karma.
sundaram - 2010-04-27 15:36:35 This update has been submitted for testing.
bodhi - 2010-04-28 03:08:08 This update has been pushed to testing
sundaram - 2010-05-07 22:21:43 This update has been submitted for stable.
bernie - 2010-05-08 15:28:43 Core dumps after initializing sound. (-1)
bodhi - 2010-05-10 23:44:08 This update has been pushed to stable
Is the last action automated or are humans overseeing it?
On Thu, 2010-05-13 at 10:03 -0400, Bernie Innocenti wrote:
El Thu, 13-05-2010 a las 09:53 -0400, Bernie Innocenti escribió:
I gave a -1 to this update a few days ago, but it's been ignored:
https://admin.fedoraproject.org/updates/openarena-0.8.5-1.fc13
Analyzing the event log in Bodhi exposes where our quality process ultimately fails: the update got my -1, then it was pushed to stable regardless of its negative karma.
sundaram - 2010-04-27 15:36:35 This update has been submitted for testing.
bodhi - 2010-04-28 03:08:08 This update has been pushed to testing
sundaram - 2010-05-07 22:21:43 This update has been submitted for stable.
bernie - 2010-05-08 15:28:43 Core dumps after initializing sound. (-1)
bodhi - 2010-05-10 23:44:08 This update has been pushed to stable
Is the last action automated or are humans overseeing it?
-- // Bernie Innocenti - http://codewiz.org/ \X/ Sugar Labs - http://sugarlabs.org/
hey,
It works normally here. No breakage at all
....
Available Devices: PulseAudio Software ALSA Software Sound initialization successful. -------------------------------- Loading vm file vm/ui.qvm... ...which has vmMagic VM_MAGIC_VER2 Loading 1550 jump table targets total 0, hsize 1021, zero 1021, min 0, max 0 total 10042, hsize 1021, zero 2, min 0, max 30 VM file ui compiled to 3203472 bytes of code (0x7f97e563e000 - 0x7f97e594c190) compilation took 2.008561 seconds ui loaded in 1450816 bytes on the hunk 51 arenas parsed 1 arenas ignored to make count divisible by 4 24 bots parsed ^3WARNING: unknown blend mode 'gl_one_minus_dst_color' in shader 'menuback_blueish', substituting GL_ONE --- Common Initialization Complete ---
I hadn't realized it was in testing so hadn't given it karma. Did now ;)
Ankur
El Thu, 13-05-2010 a las 19:39 +0530, Ankur Sinha escribió:
It works normally here. No breakage at all
I figured out that something in my config file was making it crash:
http://people.sugarlabs.org/bernie/q3config.cfg
I had no time to bisect it against a pristine configuration file, so I'm not sure if it happens due to some strange setting of mine or with any configuration saved by OpenArena 0.8.1.
On Thu, 2010-05-13 at 10:03 -0400, Bernie Innocenti wrote:
El Thu, 13-05-2010 a las 09:53 -0400, Bernie Innocenti escribió:
I gave a -1 to this update a few days ago, but it's been ignored:
https://admin.fedoraproject.org/updates/openarena-0.8.5-1.fc13
Analyzing the event log in Bodhi exposes where our quality process ultimately fails: the update got my -1, then it was pushed to stable regardless of its negative karma.
sundaram - 2010-04-27 15:36:35 This update has been submitted for testing.
bodhi - 2010-04-28 03:08:08 This update has been pushed to testing
sundaram - 2010-05-07 22:21:43 This update has been submitted for stable.
bernie - 2010-05-08 15:28:43 Core dumps after initializing sound. (-1)
bodhi - 2010-05-10 23:44:08 This update has been pushed to stable
Is the last action automated or are humans overseeing it?
It's a little of both. once the update has been requested for stable, the maintainer could rescind that request before releng does the push. However there are generally hundreds of updates across the releases that get pushed at one time, and releng does not have the time or man power to click through each one looking for negative karma. We rely on the maintainer to do the right thing.
El Thu, 13-05-2010 a las 09:22 -0700, Jesse Keating escribió:
It's a little of both. once the update has been requested for stable, the maintainer could rescind that request before releng does the push. However there are generally hundreds of updates across the releases that get pushed at one time, and releng does not have the time or man power to click through each one looking for negative karma. We rely on the maintainer to do the right thing.
In this case, the maintainer did the right^W usual thing we do in Fedora:
1) push the update for testing
2) after two weeks with no feedback in bodhi, request pushing to stable
3) my -1 karma came at this point
4) releng approves and pushes to stable
There were just 2 days between (3) and (4). If the maintainer was supposed to notice and cancel the push, we're prone to race-conditions like this one :-)
Perhaps the (web?) UI used by releng could be enhanced to display the current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
On Thu, May 13, 2010 at 01:19:49PM -0400, Bernie Innocenti wrote:
El Thu, 13-05-2010 a las 09:22 -0700, Jesse Keating escribió:
It's a little of both. once the update has been requested for stable, the maintainer could rescind that request before releng does the push. However there are generally hundreds of updates across the releases that get pushed at one time, and releng does not have the time or man power to click through each one looking for negative karma. We rely on the maintainer to do the right thing.
In this case, the maintainer did the right^W usual thing we do in Fedora:
push the update for testing
after two weeks with no feedback in bodhi, request pushing to stable
my -1 karma came at this point
releng approves and pushes to stable
There were just 2 days between (3) and (4). If the maintainer was supposed to notice and cancel the push, we're prone to race-conditions like this one :-)
Perhaps the (web?) UI used by releng could be enhanced to display the
The web UI does display it. However, using the web UI to submit the pushes really sucks for other unrelated reasons, so we don't use it.
current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
I can look at doing this on the client side for pushes. That's a pretty good idea. Could you file a ticket against bodhi and assign it to me so I don't forget?
josh
On Thu, 2010-05-13 at 13:45 -0400, Josh Boyer wrote:
current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
I can look at doing this on the client side for pushes. That's a pretty good idea. Could you file a ticket against bodhi and assign it to me so I don't forget?
we've pretty much established already that numerical karma is a bad concept; I think it'd be better for to automatically rescind the request (or warn) if any negative feedback is posted after the push request but before the push, never mind what the overall numerical value is.
On Thu, May 13, 2010 at 07:31:32PM +0100, Adam Williamson wrote:
On Thu, 2010-05-13 at 13:45 -0400, Josh Boyer wrote:
current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
I can look at doing this on the client side for pushes. That's a pretty good idea. Could you file a ticket against bodhi and assign it to me so I don't forget?
we've pretty much established already that numerical karma is a bad concept; I think it'd be better for to automatically rescind the request (or warn) if any negative feedback is posted after the push request but before the push, never mind what the overall numerical value is.
If we combine that with requiring valid bug numbers before negative karma can be applied, then I'd be ok with that. Until we require that, there are way to many corner cases where something like that isn't going to work well.
josh
On Thu, 2010-05-13 at 15:53 -0400, Josh Boyer wrote:
If we combine that with requiring valid bug numbers before negative karma can be applied, then I'd be ok with that. Until we require that, there are way to many corner cases where something like that isn't going to work well.
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
I don't really mind requiring bug numbers for negative karma (though, if anything, I reckon that'd have *more* problematic corner cases in itself). But I'm not sure it's really necessary for this.
On Thu, May 13, 2010 at 11:23:10PM +0100, Adam Williamson wrote:
On Thu, 2010-05-13 at 15:53 -0400, Josh Boyer wrote:
If we combine that with requiring valid bug numbers before negative karma can be applied, then I'd be ok with that. Until we require that, there are way to many corner cases where something like that isn't going to work well.
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
The three biggest areas where I see auto-unsubmit being problematic are:
1) Negative karma for problems we can't fix. E.g. "OMG MY DISPLAY GOES BLANK WHEN I USE THE BINARY NVIDIA MODULE WITH THIS KERNEL/XORG/FORTUNE!" We don't care, we can't fix it, and it's just a hassle to make the maintainer resubmit for crap like that. The same is true for odd conglomerations of releases, custom packages, and third party repos.
2) Packages like the kernel, where it works fine on 300 machines but somebody has a crappy ass old Pentium 4 with the original BIOS and RDRAM and it causes some weird oops. In a perfect world, we'd hold up the kernel update and get that fixed. I'd also be able to crap gold bricks and pay my mortgage in smiles. We aren't in a perfect world, and the greater good of fixing a larger number of machines seems to warrant pushing that update out. Again, more maintainer hassle.
3) People just being jerks. This is the rarest worry, but it would be rather easy to intentionally cause another maintainer problems and never let them get their updates out if we auto-unsubmitted on negative karma.
Now, if we make the auto-unsubmit value tunable per update, or a warning as you also suggested, that might be more doable. In the meantime, I can still work on getting the client side push stuff to allow rel-eng to filter on karma level. It's probably pretty trivial to do that.
josh
On 14 May 2010 06:42, Josh Boyer wrote:
On Thu, May 13, 2010 at 11:23:10PM +0100, Adam Williamson wrote:
On Thu, 2010-05-13 at 15:53 -0400, Josh Boyer wrote:
If we combine that with requiring valid bug numbers before negative karma can be applied, then I'd be ok with that. Until we require that, there are way to many corner cases where something like that isn't going to work well.
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
I don't know about real statistics of these kinds of reports, but In case it is really a big number I would suggest to increase the gap between submitting so that maintainer gets a week or few more days to decide (reach to his mail and take a decision, whether to un-submit the push).
On Fri, 2010-05-14 at 09:31 +0530, Rakesh Pandit wrote:
On 14 May 2010 06:42, Josh Boyer wrote:
On Thu, May 13, 2010 at 11:23:10PM +0100, Adam Williamson wrote:
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
I don't know about real statistics of these kinds of reports, but In case it is really a big number I would suggest to increase the gap between submitting so that maintainer gets a week or few more days to decide (reach to his mail and take a decision, whether to un-submit the push).
The maintainer can already decide when to submit for stable based on the total amount of testing he/she thinks the update needs before it is pushed. Arbitrarily lengthening the push delay would just make the process less efficient. If what you are after is a minimum time between the testing push and stable push, that is a policy question and should be considered as such.
The issue Bernie raised was simply that if a negative report that merits stopping the update happens to come in while we still have the option to cancel the push, we might as well cancel it.
On 14 May 2010 09:50, Matt McCutchen wrote:
On Fri, 2010-05-14 at 09:31 +0530, Rakesh Pandit wrote:
On 14 May 2010 06:42, Josh Boyer wrote:
On Thu, May 13, 2010 at 11:23:10PM +0100, Adam Williamson wrote:
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
I don't know about real statistics of these kinds of reports, but In case it is really a big number I would suggest to increase the gap between submitting so that maintainer gets a week or few more days to decide (reach to his mail and take a decision, whether to un-submit the push).
The maintainer can already decide when to submit for stable based on the total amount of testing he/she thinks the update needs before it is pushed. Arbitrarily lengthening the push delay would just make the process less efficient. If what you are after is a minimum time between the testing push and stable push, that is a policy question and should be considered as such.
The issue Bernie raised was simply that if a negative report that merits stopping the update happens to come in while we still have the option to cancel the push, we might as well cancel it.
What I meant was (suggestion):
No change in normal process. Just 2-3 days extra between this particular case in which a nack (-ve karma) is received between maintainer requesting a push for stable and rel-eng submitting it. It will prevent `race condition` where say maintainer wants to pull it back but before he does so it is pushed.
Rakesh Pandit wrote:
No change in normal process. Just 2-3 days extra between this particular case in which a nack (-ve karma) is received between maintainer requesting a push for stable and rel-eng submitting it. It will prevent `race condition` where say maintainer wants to pull it back but before he does so it is pushed.
It'll also cause important updates to be delayed because of an invalid or even malicious -1 vote.
Kevin Kofler
On Fri, May 14, 2010 at 3:12 AM, Josh Boyer jwboyer@gmail.com wrote:
On Thu, May 13, 2010 at 11:23:10PM +0100, Adam Williamson wrote:
On Thu, 2010-05-13 at 15:53 -0400, Josh Boyer wrote:
If we combine that with requiring valid bug numbers before negative karma can be applied, then I'd be ok with that. Until we require that, there are way to many corner cases where something like that isn't going to work well.
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore. And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
The three biggest areas where I see auto-unsubmit being problematic are:
- Negative karma for problems we can't fix. E.g. "OMG MY DISPLAY GOES BLANK
WHEN I USE THE BINARY NVIDIA MODULE WITH THIS KERNEL/XORG/FORTUNE!" We don't care, we can't fix it, and it's just a hassle to make the maintainer resubmit for crap like that. The same is true for odd conglomerations of releases, custom packages, and third party repos.
- Packages like the kernel, where it works fine on 300 machines but somebody
has a crappy ass old Pentium 4 with the original BIOS and RDRAM and it causes some weird oops. In a perfect world, we'd hold up the kernel update and get that fixed. I'd also be able to crap gold bricks and pay my mortgage in smiles. We aren't in a perfect world, and the greater good of fixing a larger number of machines seems to warrant pushing that update out. Again, more maintainer hassle.
- People just being jerks. This is the rarest worry, but it would be rather
easy to intentionally cause another maintainer problems and never let them get their updates out if we auto-unsubmitted on negative karma.
4) People adding negative karma because "unrelated bug that has been present in the older version is still not fixed"
On 14 May 2010 14:22, drago01 drago01@gmail.com wrote:
- People adding negative karma because "unrelated bug that has been
present in the older version is still not fixed"
I get this all the time. It would be nice to be able to have a "discount this karma" button for maintainers, rather than having to add an additional +1 just to cancel the misguided -1.
Richard.
On Monday, May 17, 2010, 7:24:14 AM, Richard Hughes wrote:
On 14 May 2010 14:22, drago01 drago01@gmail.com wrote:
- People adding negative karma because "unrelated bug that has been
present in the older version is still not fixed"
I get this all the time. It would be nice to be able to have a "discount this karma" button for maintainers, rather than having to add an additional +1 just to cancel the misguided -1. Richard.
Sounds good as long as the maintainer is required to append a reason. Al
On Mon, 17 May 2010 12:24:14 +0100, Richard wrote:
- People adding negative karma because "unrelated bug that has been
present in the older version is still not fixed"
I get this all the time. It would be nice to be able to have a "discount this karma" button for maintainers, rather than having to add an additional +1 just to cancel the misguided -1.
Would you really want to spend time on "moderating" karma/comments in bodhi? Sounds like a waste of time to me.
Just choose to disable karma automatism when you create the ticket in bodhi, and be done with it.
On Tue, 2010-05-18 at 10:58 +0200, Michael Schwendt wrote:
On Mon, 17 May 2010 12:24:14 +0100, Richard wrote:
- People adding negative karma because "unrelated bug that has been
present in the older version is still not fixed"
I get this all the time. It would be nice to be able to have a "discount this karma" button for maintainers, rather than having to add an additional +1 just to cancel the misguided -1.
Would you really want to spend time on "moderating" karma/comments in bodhi? Sounds like a waste of time to me.
Just choose to disable karma automatism when you create the ticket in bodhi, and be done with it.
In the Glorious Future, when Bodhi has different feedback types rather than a simple 'positive/negative' sum, this problem should more or less go away. We can create a special category for 'this update doesn't fix my already-existing bug, wah wah!' so people can file such feedback, and everyone else can cheerfully ignore it. :)
Adam Williamson wrote:
Really? I don't think there's *that* many cases where a negative piece of karma is filed between the submission and the push which you'd want to ignore.
I think there are actually very many. We get a lot of invalid -1 votes for KDE updates (issues which have been there for ages, issues which have been caused by a completely unrelated update which happened to hit testing or stable at the same time) etc.
It'd also open the doors to effectively DoS updates.
And even in the rare cases when that happens, if we warn or even unsubmit the update, it's not like you can't do anything about it. If we make it a warning...ignore the warning. If we make it withdraw the update...just submit it again. I'm having a hard time seeing that fall apart.
It means that a well-timed -1 can cause the update to miss the push (which is already a forced delay and thus a form of DoS), then it can be done again at the next push, ad infinitum, instant DoS.
I don't really mind requiring bug numbers for negative karma (though, if anything, I reckon that'd have *more* problematic corner cases in itself). But I'm not sure it's really necessary for this.
I think it actually won't solve the problem at hand. The bug pointed to might not actually be caused by the update (see the first paragraph), or it could even be a dummy bug filed by a malicious DoSer.
Kevin Kofler
Adam Williamson wrote:
On Thu, 2010-05-13 at 13:45 -0400, Josh Boyer wrote:
current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
I can look at doing this on the client side for pushes. That's a pretty good idea. Could you file a ticket against bodhi and assign it to me so I don't forget?
we've pretty much established already that numerical karma is a bad concept; I think it'd be better for to automatically rescind the request (or warn) if any negative feedback is posted after the push request but before the push, never mind what the overall numerical value is.
Jumping in late here... obviously there are some concerns /cancelling/ the push. What about improving the tools so that such updates don't get pushed in the mass pushes but simply flagged as needing review? That way rel-eng could know to actually look at these instead of blindly trusting the maintainer, that maybe hasn't had time to react. That way if it is something silly, rel-eng can still push it anyway, but there is more chance to catch real problems.
This assumes that the % of packages that get negative karma between request and push is low, of course, but I would think that is the case.
On Fri, 2010-05-14 at 19:42 -0500, Matthew Woehlke wrote:
Adam Williamson wrote:
On Thu, 2010-05-13 at 13:45 -0400, Josh Boyer wrote:
current karma next to each push request? Or maybe Bodhi could be configured to automatically cancel stable requests when the karma drops below 0?
I can look at doing this on the client side for pushes. That's a pretty good idea. Could you file a ticket against bodhi and assign it to me so I don't forget?
we've pretty much established already that numerical karma is a bad concept; I think it'd be better for to automatically rescind the request (or warn) if any negative feedback is posted after the push request but before the push, never mind what the overall numerical value is.
Jumping in late here... obviously there are some concerns /cancelling/ the push. What about improving the tools so that such updates don't get pushed in the mass pushes but simply flagged as needing review? That way rel-eng could know to actually look at these instead of blindly trusting the maintainer, that maybe hasn't had time to react. That way if it is something silly, rel-eng can still push it anyway, but there is more chance to catch real problems.
This assumes that the % of packages that get negative karma between request and push is low, of course, but I would think that is the case.
-- Matthew Please do not quote my e-mail address unobfuscated in message bodies. -- "You ended that sentence in a preposition!" -- Jack O'Neill (from "The Other Guys", Richard Dean Anderson, Stargate: SG1)
What is releng supposed to do here though? We can't be experts in every package. How are we to know that the negative karma is really appropriately negative, or bad negative, or just misfiled or confused users? That's what the maintainer is for.
On Fri, 14 May 2010 20:27:51 -0700, Jesse wrote:
What is releng supposed to do here though?
It's a hard problem related to tools *and* people.
The longer it takes to push packages into a repo, the longer the window that creates the race condition. It could be that the push has completed 98% of the stuff that needs to be done, and a tester would vote -1 late because of a show-stopper bug in one package.
If aborting a push is too expensive, that's not an option to do it frequently. Especially not if some of the packages to be pushed in the same transaction are considered urgent.
On the other hand, it's better to delay updates than to publish bad updates.
What you could do is to sort updates by priority and process them in separate transactions. You could use tools that lock and depcheck the package set that will be pushed in one transaction. Then you could implement a "grace time" that would make it possible to abort a push transaction _once_ based on late votes, withdraw packages, and restart a final push that would not be aborted again (with releng team being free to override the automation, of course). The maintainer could opt-in to "negative vote will abort push to stable". Testers/voters may lose the ability to veto a push to stable if they repeatedly use -1 inappropriately. How often a push will be aborted remains to be seen. Packages that lead to aborting a push often could be sanctioned (e.g. by increasing the time they must sit in updates-testing).
Bodhi would add a comment to an update ticket when beginning to push a package, and it would lock the package so it cannot be edited. So far, the maintainers cannot know whether a package is being pushed already.
We can't be experts in every package. How are we to know that the negative karma is really appropriately negative, or bad negative, or just misfiled or confused users? That's what the maintainer is for.
Same applies to positive karma. Is the +1 the result of substantial testing or just a +1 to get the new "adventurous" stuff, which makes Fedora less boring?
On Sat, 2010-05-15 at 11:06 +0200, Michael Schwendt wrote:
Same applies to positive karma. Is the +1 the result of substantial testing or just a +1 to get the new "adventurous" stuff, which makes Fedora less boring?
Yes, a standard for +1 karma would be helpful. But even before that, we need a standard (or at least an understanding shared by maintainers) for how much total testing an update needs before being pushed to stable.
I have been using fedora-easy-karma for a few weeks now, and I typically give a +1 after successfully performing my most common workflow(s) using the package. The fact that the package works at all in my environment is valuable information, but it's still important to ensure that the package gets the desired total amount of testing.
On a tangent, one thing that should probably be tested is that each bug claimed to be fixed is in fact fixed. That test should be done by someone who was able to reproduce the bug in the first place, otherwise it is meaningless. For an example where an update was pushed without any attempt at such testing and turned out not to fix the bug, see:
https://admin.fedoraproject.org/updates/rsync-3.0.7-2.fc12
On Sat, 15 May 2010 05:40:27 -0400, Matt wrote:
Is the +1 the result of substantial testing or just a +1 to get the new "adventurous" stuff, which makes Fedora less boring?
Yes, a standard for +1 karma would be helpful. But even before that, we need a standard (or at least an understanding shared by maintainers) for how much total testing an update needs before being pushed to stable.
It should be up to the maintainer to decide that.
_Unless_ there is more "global interest" in the package. From the distributor's point of view (for example, the CRITPATH package set). Or from the community's point of view. Then it's up to those extra people to contribute the testing if they want to apply special rules to a package. For the community that means it's time to build a team that takes care of the package and put tools to good use -- and as soon as there is a team, they can decide on minimum karma or mandatory tests.
As long as nobody steps up and says "I want to influence the quality of this package and its updates", it's only the package maintainer to run the whole show ... including mistakes (such as not noticing a problem upstream hasn't noticed either).
And there are bug-fix updates, which don't need any special testing, because the maintainer knows that the applied fix is fine. How likely is it that only the x86_64 builder damaged a byte in the build that makes the software crash in a single function which used to work before? ;)
I have been using fedora-easy-karma for a few weeks now, and I typically give a +1 after successfully performing my most common workflow(s) using the package. The fact that the package works at all in my environment is valuable information, but it's still important to ensure that the package gets the desired total amount of testing.
I've given up using it and only run it sporadically to see what test updates are still installed. Too many of the update tickets refer to specific problems I haven't run into.
Disappointing to see that. :-/
Michael Schwendt wrote:
The longer it takes to push packages into a repo, the longer the window that creates the race condition. It could be that the push has completed 98% of the stuff that needs to be done, and a tester would vote -1 late because of a show-stopper bug in one package.
If aborting a push is too expensive, that's not an option to do it frequently. Especially not if some of the packages to be pushed in the same transaction are considered urgent.
The main race is actually not for updates -1ed DURING a push (though that can also be seen as a concern), but for updates -1ed between the submission and the push, which is a much longer window (e.g. it can be the entire weekend, because pushes are rarely done on weekends).
Kevin Kofler
On Fri, 2010-05-14 at 20:27 -0700, Jesse Keating wrote:
What is releng supposed to do here though? We can't be experts in every package. How are we to know that the negative karma is really appropriately negative, or bad negative, or just misfiled or confused users? That's what the maintainer is for.
Let's be honest, though, practically speaking - in most cases - you wouldn't need to be. All the examples of 'incorrect' negative karma that Josh gave as the 'biggest' are ones that a generalist can usually recognize quite well. And if it's one you can't confidently decide about - get the maintainer in. It's better to be right than to be fast, isn't it?
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
Rahul
On 05/12/2010 04:12 PM, Rahul Sundaram wrote:
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
Rahul
My understanding was that we would still open a rel-eng ticket for a freeze exception. Which I didn't do for Wesnoth. Because the outcry for it was underwhelming.
-J
On Wed, May 12, 2010 at 16:22:13 -0500, Jon Ciesla limb@jcomserv.net wrote:
My understanding was that we would still open a rel-eng ticket for a freeze exception. Which I didn't do for Wesnoth. Because the outcry for it was underwhelming.
And likely another rebuild will be needed shortly. I still need to do a test, but it looks like wesnoth will need to get built against the new boost-devel (after a boost update gets pushed), because the bug causing the problem for x86_64 is in a header using during the wesnoth build.
I can prep for the test tonight, but it's a pain to do the final test remotely. So that will wait until tomorrow.
On 05/12/2010 11:11 PM, Bruno Wolff III wrote:
On Wed, May 12, 2010 at 16:22:13 -0500, Jon Cieslalimb@jcomserv.net wrote:
My understanding was that we would still open a rel-eng ticket for a freeze exception. Which I didn't do for Wesnoth. Because the outcry for it was underwhelming.
And likely another rebuild will be needed shortly. I still need to do a test, but it looks like wesnoth will need to get built against the new boost-devel (after a boost update gets pushed), because the bug causing the problem for x86_64 is in a header using during the wesnoth build.
I can prep for the test tonight, but it's a pain to do the final test remotely. So that will wait until tomorrow.
Email me as soon as you want this done, and I'll do it ASAP.
-J
On Thu, May 13, 2010 at 09:06:39 -0500, Jon Ciesla limb@jcomserv.net wrote:
I can prep for the test tonight, but it's a pain to do the final test remotely. So that will wait until tomorrow.
Email me as soon as you want this done, and I'll do it ASAP.
Two people have confirmed that a combination of installing the scratch build and rebuilding and then installing wesnoth solves the x86_64 connect to server issue.
To do this quickly, it will require chainbuilding both packages so that both can get tested at the same time. Otherwise the boost update needs to go through testing and get to stable before a wesnoth rebuild would do any good.
On Thu, 2010-05-13 at 02:42 +0530, Rahul Sundaram wrote:
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
Rahul
I was trying to do away with the tickets for freeze exceptions, as I was trying to do away with freeze exceptions outside of blocker bugs. However there are a few bugs which we would classify as "not blockers" but things which we "would take a fix for". These generally start out by being proposed blockers.
The process isn't perfect, nor all that documented, as we're exploring as we go. This is the first time we've done a RC phase with the no frozen rawhide style of development. I suspect we'll be better about process and documentation next time around.
Jesse Keating said the following on 05/12/2010 03:11 PM Pacific Time:
On Thu, 2010-05-13 at 02:42 +0530, Rahul Sundaram wrote:
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
Rahul
I was trying to do away with the tickets for freeze exceptions, as I was trying to do away with freeze exceptions outside of blocker bugs. However there are a few bugs which we would classify as "not blockers" but things which we "would take a fix for". These generally start out by being proposed blockers.
I'd like to see these "would take a fix for" bugs added or kept on the blocker list with a short comment explaining why they are being taken in since they don't meet the regular definition of "blocker bug".
This provides a very clear list of bugs that should get extra verification attention when the RC comes out. It also provides a clear trail of the decisions made so others can understand our process and reasoning.
The process isn't perfect, nor all that documented, as we're exploring as we go. This is the first time we've done a RC phase with the no frozen rawhide style of development. I suspect we'll be better about process and documentation next time around.
I'm looking forward to continuing to help expand the documentation about our release processes. This is important because then everyone is working with the same information and can approach the process in the same way.
John
Am Donnerstag, den 13.05.2010, 06:22 -0700 schrieb John Poelstra:
I'd like to see these "would take a fix for" bugs added or kept on the blocker list with a short comment explaining why they are being taken in since they don't meet the regular definition of "blocker bug".
Isn't this exactly the purpose of the "F13-Target" tracker bug? I think all these "nice to have" bugs should block it, so we have a better overview than we currently have where some bugs have tickets in trac, others are in bugzilla or get discussed on IRC.
Regards, Christoph
On Fri, 2010-05-14 at 14:09 +0200, Christoph Wickert wrote:
Am Donnerstag, den 13.05.2010, 06:22 -0700 schrieb John Poelstra:
I'd like to see these "would take a fix for" bugs added or kept on the blocker list with a short comment explaining why they are being taken in since they don't meet the regular definition of "blocker bug".
Isn't this exactly the purpose of the "F13-Target" tracker bug? I think all these "nice to have" bugs should block it, so we have a better overview than we currently have where some bugs have tickets in trac, others are in bugzilla or get discussed on IRC.
Regards, Christoph
That tracker bug is not managed, and thus we can't say with any kind of certainty that we'd take issues on the tracker.
On Thu, 2010-05-13 at 02:42 +0530, Rahul Sundaram wrote:
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
The problem is there aren't really criteria, it's a very squishy thing. We usually take at least one not-strictly-a-blocker fix during the RC phase of a release, based on a sort of instinctive judgement call that gets kicked around between QA and rel-eng.
It's really really hard to codify this, because there's just so many parameters. We did a decent job, I think, of codifying the issues that can really block a release, but the risk vs. benefit calculation involved in 'should we take this fix for this spin' is a massively more difficult thing to codify :/
if someone wants to try that would be awesome, but honestly I wouldn't know where to start.
Adam Williamson said the following on 05/13/2010 11:26 AM Pacific Time:
On Thu, 2010-05-13 at 02:42 +0530, Rahul Sundaram wrote:
On 05/13/2010 02:37 AM, Bill Nottingham wrote:
There was an open ticket requesting Pino. There was not anything from the maintainers requesting the games.
I did mention this on IRC but what is the criteria for pulling in the updates? If I knew what would be reasonable to request, it would help for future releases. I am also looking for guidelines on, what updates are considered a good thing. Is the game update a problem?
The problem is there aren't really criteria, it's a very squishy thing. We usually take at least one not-strictly-a-blocker fix during the RC phase of a release, based on a sort of instinctive judgement call that gets kicked around between QA and rel-eng.
It's really really hard to codify this, because there's just so many parameters. We did a decent job, I think, of codifying the issues that can really block a release, but the risk vs. benefit calculation involved in 'should we take this fix for this spin' is a massively more difficult thing to codify :/
if someone wants to try that would be awesome, but honestly I wouldn't know where to start.
I'm up for the challenge.... previously having been told it wasn't possible for release criteria and blocker bugs ;-)
John
On Fri, 2010-05-14 at 06:52 -0700, John Poelstra wrote:
I'm up for the challenge.... previously having been told it wasn't possible for release criteria and blocker bugs ;-)
And we're still making judgment calls there, because it is very very difficult to codify.
On Fri, May 14, 2010 at 08:58:04 -0700, Jesse Keating jkeating@redhat.com wrote:
On Fri, 2010-05-14 at 06:52 -0700, John Poelstra wrote:
I'm up for the challenge.... previously having been told it wasn't possible for release criteria and blocker bugs ;-)
And we're still making judgment calls there, because it is very very difficult to codify.
I think it would help to make notes about why exceptions were granted and collect them. That may help with writing policy and modifying it later where needed. As well as possibly helping speed decisions when similar cases come up again.
Jesse Keating said the following on 05/14/2010 08:58 AM Pacific Time:
On Fri, 2010-05-14 at 06:52 -0700, John Poelstra wrote:
I'm up for the challenge.... previously having been told it wasn't possible for release criteria and blocker bugs ;-)
And we're still making judgment calls there, because it is very very difficult to codify.
I fully realize that too :)
"Most good processes have room for exceptions. There are definitely times when judgment calls must be made by certain teams or people to get the release out on time. After completing almost 40 releases of Fedora (test and final releases), we should be fairly familiar with what those situations are and who needs to make them. I plan to propose a light weight process for transparently documenting those situations and which teams and people act on the release’s behalf." [1]
John
[1] http://poelcat.wordpress.com/2010/05/14/fedora-13-end-game-part-2/
On Wed, 2010-05-12 at 11:14 -0700, John Poelstra wrote:
Jesse Keating said the following on 05/10/2010 04:08 PM Pacific Time:
Fedora 13 has released Release Candidate stage. We have reached a state where the known blockers were fixed and were able to make a release candidate. This happened last Thursday, and almost immediately we found a need to spin a second release candidate. From this point on, only items critical to the release will be tagged into the branched Fedora 13 repo.
I'm currently doing the first push of Fedora 13 stable updates. These will be what we call "0-day" updates, that is updates available at release time. Things in the bodhi update system for Fedora 13 can now be pushed "stable" to this updates repo. This allows our maintainers to continue to improve Fedora 13 as release engineering and QA finalize the release bits.
If you have a bug that you think is critical to the release of Fedora 13 and must be fixed on the media, please make your bug block "F13Blocker". We will be reviewing the contents of this blocker bug frequently throughout the days as we near our go / no go decision.
I've noticed some discussion on #fedora-devel about "taking in nice-to-haves since the release is slipping." If these new packages are not blockers or critical to the release when/where did we decide to deviate from what is stated above?
Just to clarify this: the nice-to-have I have been asking about is a newer release of Shotwell. The only change in the release is the inclusion of several new translations.
I would not have asked for it to be 'sneaked in', normally, but
a) the shotwell guys went out of their way to produce a new tarball for me in time for F13, because I pointed them at a Fedora bug where the lack of Russian translations in our shotwell package was bemoaned.
b) shotwell is on the live cd, so leaving this as an update would increase the zero-day download size for everybody.
Matthias