Hi,
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the packages to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>. Do I have to wait until the packages for f15 are pushed to repush the packages for f14 and then wait again for pushing into f13? Or is there some automatism that reevaluates and repushes packages w/o any further intervention from the packagers?
I guess this is also slowing down the people that grant the package push. Previously they could evaluate the package updates in one sweep, now they will see and approve the "same" package in three cycles.
On Sat, 16 Apr 2011 22:52:32 +0300, AT wrote:
Hi,
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the packages to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>. Do I have to wait until the packages for f15 are pushed to repush the packages for f14 and then wait again for pushing into f13? Or is there some automatism that reevaluates and repushes packages w/o any further intervention from the packagers?
I guess this is also slowing down the people that grant the package push. Previously they could evaluate the package updates in one sweep, now they will see and approve the "same" package in three cycles.
AutoQA's comments in bodhi so far are just informative. They don't block any update [yet].
On Sat, 2011-04-16 at 23:03 +0200, Michael Schwendt wrote:
On Sat, 16 Apr 2011 22:52:32 +0300, AT wrote:
Hi,
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the packages to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>. Do I have to wait until the packages for f15 are pushed to repush the packages for f14 and then wait again for pushing into f13? Or is there some automatism that reevaluates and repushes packages w/o any further intervention from the packagers?
I guess this is also slowing down the people that grant the package push. Previously they could evaluate the package updates in one sweep, now they will see and approve the "same" package in three cycles.
AutoQA's comments in bodhi so far are just informative. They don't block any update [yet].
Are you sure? I requested a push of the packages to stable (some were in testing for a week, others were security updates) and the message was that it doesn't pass AutoQA, so it converted to request to push only to testing and indeed bodhi has marked the request as to testing only.
Also the comments are not in bodhi at all. All I get is the message that AutoQA blocked the packages and sometimes an email with the results of the AutoQA run.
Axel Thimm wrote:
Are you sure? I requested a push of the packages to stable (some were in testing for a week, others were security updates) and the message was that it doesn't pass AutoQA, so it converted to request to push only to testing and indeed bodhi has marked the request as to testing only.
It did that because you changed it.
If you had kept the request to stable, it would have been pushed to stable.
AutoQA is still in testing phase, there is no enforcement yet.
Kevin Kofler
On Sun, 17 Apr 2011 08:01:55 +0200, KK wrote:
Axel Thimm wrote:
Are you sure? I requested a push of the packages to stable (some were in testing for a week, others were security updates) and the message was that it doesn't pass AutoQA, so it converted to request to push only to testing and indeed bodhi has marked the request as to testing only.
It did that because you changed it.
If you had kept the request to stable, it would have been pushed to stable.
AutoQA is still in testing phase, there is no enforcement yet.
Kevin Kofler
Probably there's a misunderstanding of the AutoQA notifications. They are added to bodhi tickets, see e.g. the "mediawiki" tickets here,
https://admin.fedoraproject.org/updates/user/athimm
and those comments are forwarded to the update submitter (and ticket subscribers). Also, several AutoQA check results have been mailed privately (rpmlint, rpmguard).
On Sun, 2011-04-17 at 08:01 +0200, Kevin Kofler wrote:
Axel Thimm wrote:
Are you sure? I requested a push of the packages to stable (some were in testing for a week, others were security updates) and the message was that it doesn't pass AutoQA, so it converted to request to push only to testing and indeed bodhi has marked the request as to testing only.
It did that because you changed it.
If you had kept the request to stable, it would have been pushed to stable.
AutoQA is still in testing phase, there is no enforcement yet.
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.
And the conversion to testing was mentioned when I tried to push the package, not later, when a human could process the request.
Currently the autoqa process is very muddy to me and probably other packagers as well:
* There are contradictory statements on whether autoqa will be enforced or informative only. bodhi claims it is in place, Tim claims it will be in place by F16 and other claim it will always be informative only. For me it looks like bodhi rejects my packages based on autoqa partly without any autoqa run output available to me (read below). * Currently bodhi behaves differently than before. For example the two package sets I'm worried about right now are fail2ban and mediawiki. ATM all packages are in rawhide and testing (f2b for f15 is even stable), so I would expect at least the next-highest Fedora packages to pass the autoqa. Still, trying to push any of these packages from testing to stable I get: * "This update has not yet met the minimum testing requirements defined in the Package Update Acceptance Criteria" * The package request field in bodhi is empty, e.g. for he packager at least it looks like no push attempt was ever made, I don't know how it looks on the push granting side, but I assume it looks the same. * In some cases I don't get any indication why the autoqa failed. For example for the fail2ban packages I either received positive autoqa results or none at all (for f14 I never received any autoqa). Still the fail2ban packages are rejected (all but f15). * I also never received any autoqa info on mediawiki-1.16.4-57.fc15 even though bodhi/autoqa rejects it. * Most of the autoqa bits are accessible only through my mail folder. Given that some autoqa never made it there (the assumed fail2ban autoqa failures), nor the bodhi comments, I am currently at a loss what to do with the fail2ban packages.
I like the general idea of an automated test suite, but currently autoqa is blocking/rejecting package pushes on a.f.o/updates, or at least bodhi tells me so. And the reasons for doing so are non transparent.
Focusing on fail2ban, which is the more traditional update from the two (mediawiki was fasttracked by another update, so let's keep this example simpler by ignoring it):
* The packages were submitted to testing, accepted and stayed there for a while. * packages pushed to stable are being rejected for f13/f14 w/o the packager understanding why (there is no (f14) or just positive (f13/f15) autoqa feedback), especially when f15 did get pushed through. * My push request seem to get currently dropped, so my packages remain in a non-requesting testing-updates limbo, and probably no one notices.
Please help with these packages - it takes me much longer to get them through bodhi than the actual packaging/updating/testing of the packages.
Also please consider the following for autoqa:
* make a firm decision on whether, when and how autoqa will enforce its results. It isn't just informative ATM. * place ALL autoqa results in bodhi, even repeated ones. Hunting the autoqa output though mails and bodhi isn't helping. * Make autoqa always informative. If autoqa fails allow the packager to add a comment to the update for the pusher to evaluate whether the packages should be pushed regardless of the autoqa failures. Please don't automatically reset push requests!
Thanks!
Axel Thimm wrote:
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.
The criteria which are being enforced do not include AutoQA results at this time. They do, however, include minimum testing (time and/or karma) requirements, which probably explains why your stable request was rejected. (And I've been fighting against those requirements since they were first proposed, because I strongly believe this decision should really be up to the maintainer, but I lost that battle.)
Kevin Kofler
On Tue, 2011-04-19 at 10:50 +0200, Kevin Kofler wrote:
Axel Thimm wrote:
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.The criteria which are being enforced do not include AutoQA results at this time. They do, however, include minimum testing (time and/or karma) requirements, which probably explains why your stable request was rejected. (And I've been fighting against those requirements since they were first proposed, because I strongly believe this decision should really be up to the maintainer, but I lost that battle.)
Well, two questions:
a) Weren't updates marked as security updates handled specially? E.g. the packages to get tagged as push-requested with the final decision being a pusher's review of the request?
b) In the past if karma/time requirements were not met one could still mark the request and the request would show up. Possibly not granted/processed until the requirements were met (unless the package was security related or fixing a too nasty bug), but not immediately cleared as if it never happened (which is the current state).
On Tue, 19 Apr 2011 11:30:44 +0300, AT wrote:
AutoQA is still in testing phase, there is no enforcement yet.
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.
This links to the Wiki and explains what is being enforced. It is much to read, but in short,
https://fedoraproject.org/wiki/Package_update_acceptance_criteria#All_other_...
your packages (F-14, F-13) either need to sit for one week in testing or reach the karma threshold you configured. Mediawiki updates have been submitted on 04-16, so the one week is not over yet. For F-15 it's three days only currently. Bodhi adds a comment to a ticket when the acceptance criteria are satisfied and when the minimum time in testing has been reached. (It adds that comment even if the ticket has reached a negative karma value meanwhile, so be warned.)
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
And the conversion to testing was mentioned when I tried to push the package, not later, when a human could process the request.
You've tried to select "stable" as the target already when submitting the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki, why didn't you choose "security" instead of "stable"?
Currently the autoqa process is very muddy to me and probably other packagers as well:
The AutoQA stuff that's been running lately does not add to the Package Update Acceptance criteria yet. It would need to either spend karma points to inform bodhi or even receive special veto privileges to influence bodhi's decision on whether an update may enter the stable repo.
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
On Tue, 19 Apr 2011 11:30:44 +0300, AT wrote:
AutoQA is still in testing phase, there is no enforcement yet.
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.This links to the Wiki and explains what is being enforced. It is much to read, but in short,
https://fedoraproject.org/wiki/Package_update_acceptance_criteria#All_other_...
your packages (F-14, F-13) either need to sit for one week in testing or reach the karma threshold you configured. Mediawiki updates have been submitted on 04-16, so the one week is not over yet. For F-15 it's three days only currently. Bodhi adds a comment to a ticket when the acceptance criteria are satisfied and when the minimum time in testing has been reached. (It adds that comment even if the ticket has reached a negative karma value meanwhile, so be warned.)
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
And the conversion to testing was mentioned when I tried to push the package, not later, when a human could process the request.
You've tried to select "stable" as the target already when submitting the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki, why didn't you choose "security" instead of "stable"?
Currently the autoqa process is very muddy to me and probably other packagers as well:
The AutoQA stuff that's been running lately does not add to the Package Update Acceptance criteria yet. It would need to either spend karma points to inform bodhi or even receive special veto privileges to influence bodhi's decision on whether an update may enter the stable repo.
Right, once we iron out any kinks with bodhi tagging and autoqa tests +scheduling, we'll revisit better ways to convey automated test results in a non-irritating manner.
To reiterate again, AutoQA simply adds a comment to the bodhi update informing the maintainer about automated test results for packages in that update. It does not add karma (positive or negative) in any way at this time. If the results don't make sense or are wrong, please stop by autoqa-devel@lists.fedorahosted.org and let folks know.
Thanks, James
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
Not...really. The update submitter's own vote should count as 0 (I can't remember if this has landed in current Bodhi yet). Setting the karma threshold to 1 cannot circumvent the 'proventesters +1 and any +1' requirement for stable releases; just try it, it doesn't work. The correct requirements are enforced whatever you set the autopush threshold to.
On Tue, 19 Apr 2011 08:37:25 -0700 Adam Williamson wrote:
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
Not...really. The update submitter's own vote should count as 0 (I can't remember if this has landed in current Bodhi yet). Setting the karma threshold to 1 cannot circumvent the 'proventesters +1 and any +1' requirement for stable releases; just try it, it doesn't work. The correct requirements are enforced whatever you set the autopush threshold to.
I just tried it (ok after the threshold of 3 days in f15) and it worked: https://admin.fedoraproject.org/updates/ipython-0.10.2-1.fc15
Thomas
On Tue, 2011-04-19 at 18:30 +0200, Thomas Spura wrote:
On Tue, 19 Apr 2011 08:37:25 -0700 Adam Williamson wrote:
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
Not...really. The update submitter's own vote should count as 0 (I can't remember if this has landed in current Bodhi yet). Setting the karma threshold to 1 cannot circumvent the 'proventesters +1 and any +1' requirement for stable releases; just try it, it doesn't work. The correct requirements are enforced whatever you set the autopush threshold to.
I just tried it (ok after the threshold of 3 days in f15) and it worked: https://admin.fedoraproject.org/updates/ipython-0.10.2-1.fc15
F15 is a pre-release and does not have that requirement. Its requirement is +1 from anyone - so you're not circumventing anything. :)
On Tue, 19 Apr 2011 10:40:22 -0700 Adam Williamson wrote:
On Tue, 2011-04-19 at 18:30 +0200, Thomas Spura wrote:
On Tue, 19 Apr 2011 08:37:25 -0700 Adam Williamson wrote:
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
Not...really. The update submitter's own vote should count as 0 (I can't remember if this has landed in current Bodhi yet). Setting the karma threshold to 1 cannot circumvent the 'proventesters +1 and any +1' requirement for stable releases; just try it, it doesn't work. The correct requirements are enforced whatever you set the autopush threshold to.
I just tried it (ok after the threshold of 3 days in f15) and it worked: https://admin.fedoraproject.org/updates/ipython-0.10.2-1.fc15
F15 is a pre-release and does not have that requirement. Its requirement is +1 from anyone - so you're not circumventing anything. :)
Hmm, I still think own votes should never count or always. Either you can trust the packager to not "just +1" it to push it and properly test it, or you should never trust the owner of the update. Isn't it?
Thomas
On Tue, 19 Apr 2011 10:40:22 -0700, AW wrote:
On Tue, 2011-04-19 at 18:30 +0200, Thomas Spura wrote:
On Tue, 19 Apr 2011 08:37:25 -0700 Adam Williamson wrote:
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
Some packagers have been observed circumventing the system by configuring a karma threshold of 1, so their own +1 vote or the first one from an arbitrary tester make it possible to mark the update stable.
Not...really. The update submitter's own vote should count as 0 (I can't remember if this has landed in current Bodhi yet). Setting the karma threshold to 1 cannot circumvent the 'proventesters +1 and any +1' requirement for stable releases; just try it, it doesn't work. The correct requirements are enforced whatever you set the autopush threshold to.
I just tried it (ok after the threshold of 3 days in f15) and it worked: https://admin.fedoraproject.org/updates/ipython-0.10.2-1.fc15
F15 is a pre-release and does not have that requirement. Its requirement is +1 from anyone - so you're not circumventing anything. :)
We've always had the rule of thumb that packagers should not vote for their own updates. It is assumed that the packagers test their own updates and don't need to be explicit about their confidence in the update with karma points in bodhi.
One goal of the update acceptance criteria for pre-releases is also that _anyone_ gets an opportunity to try out a test-update before it is marked stable.
Anyway, I think I've seen counted "self-votes" also for older dists, but not done by many packagers.
On 04/19/2011 01:33 PM, Michael Schwendt wrote:
On Tue, 19 Apr 2011 10:40:22 -0700, AW wrote: We've always had the rule of thumb that packagers should not vote for their own updates. It is assumed that the packagers test their own updates and don't need to be explicit about their confidence in the update with karma points in bodhi.
One goal of the update acceptance criteria for pre-releases is also that _anyone_ gets an opportunity to try out a test-update before it is marked stable.
Anyway, I think I've seen counted "self-votes" also for older dists, but not done by many packagers.
I may have added karma to an update or two of mine. Usually with packages that can break more critical services than regular packages. The reason being is that I can test on my local workstation and in a VM, however it isn't the same environment as my live servers. So I test locally, push to updates-testing and consume it on my production machines. If good, then I usually add karma. I think its fairly rare for me, and I can only think of one package where I do that...
Michael Schwendt mschwendt@gmail.com wrote:
We've always had the rule of thumb that packagers should not vote for their own updates. It is assumed that the packagers test their own updates and don't need to be explicit about their confidence in the update with karma points in bodhi.
One goal of the update acceptance criteria for pre-releases is also that _anyone_ gets an opportunity to try out a test-update before it is marked stable.
Anyway, I think I've seen counted "self-votes" also for older dists, but not done by many packagers.
What about to counteract "misplaced" karma? Example:
- Bug exists in version X.Y - Update filed for X.Y+1 - User reports that bug still exists with -1 karma - Maintainer replies with +1 karma that bug is not expected to be fixed
--Ben
On Tue, 2011-04-19 at 22:47 +0000, Ben Boeckel wrote:
What about to counteract "misplaced" karma? Example:
- Bug exists in version X.Y
- Update filed for X.Y+1
- User reports that bug still exists with -1 karma
- Maintainer replies with +1 karma that bug is not expected to be fixed
The 'correct' thing to do is to contact the user and ask them to correct the feedback: if a user who previously gave a -1 gives a +1, this is counted as a total of +1, not 0.
Adam Williamson <awilliam <at> redhat.com> writes:
On Tue, 2011-04-19 at 22:47 +0000, Ben Boeckel wrote:
What about to counteract "misplaced" karma? Example:
- Bug exists in version X.Y
- Update filed for X.Y+1
- User reports that bug still exists with -1 karma
- Maintainer replies with +1 karma that bug is not expected to be fixed
The 'correct' thing to do is to contact the user and ask them to correct the feedback: if a user who previously gave a -1 gives a +1, this is counted as a total of +1, not 0.
But if the user just wants to revoke their karma by changing their +/-1 to a 0 (say they realize they're not able to test properly, which happened to me), that's still not possible (see https://fedorahosted.org/bodhi/ticket/296 ). So it's necessary to get someone else to cancel it out, and it doesn't matter who.
Andre Robatino <robatino <at> fedoraproject.org> writes:
But if the user just wants to revoke their karma by changing their +/-1 to a 0 (say they realize they're not able to test properly, which happened to me), that's still not possible (see https://fedorahosted.org/bodhi/ticket/296 ). So it's necessary to get someone else to cancel it out, and it doesn't matter who.
Sorry to respond to myself, but there's another relevant bug here: due to https://bugzilla.redhat.com/show_bug.cgi?id=612328 , bodhi email notification only works for users with working fp.o email aliases, which requires being in at least one non-CLA group. A user might simply be unresponsive, or due to this bug, may not even be contactable to ask them to change their karma.
On Wed, 20 Apr 2011 06:15:43 +0000 (UTC), AR wrote:
Andre Robatino <robatino <at> fedoraproject.org> writes:
But if the user just wants to revoke their karma by changing their +/-1 to a 0 (say they realize they're not able to test properly, which happened to me), that's still not possible (see https://fedorahosted.org/bodhi/ticket/296 ). So it's necessary to get someone else to cancel it out, and it doesn't matter who.
Sorry to respond to myself, but there's another relevant bug here: due to https://bugzilla.redhat.com/show_bug.cgi?id=612328 , bodhi email notification only works for users with working fp.o email aliases, which requires being in at least one non-CLA group. A user might simply be unresponsive, or due to this bug, may not even be contactable to ask them to change their karma.
Anonymous users' votes don't add to the karma level. Their votes are just informative.
Michael Schwendt <mschwendt <at> gmail.com> writes:
Anonymous users' votes don't add to the karma level. Their votes are just informative.
Not being in a non-CLA group != anonymous. Anyone can create a bodhi account and log in, but unless they're in a non-CLA group, their fp.o email doesn't work and they get no bodhi email (at least that was true last time I checked).
(Sorry for the excessive pruning, have to make gmane happy.)
On 04/20/2011 02:54 PM, Andre Robatino wrote:
(Sorry for the excessive pruning, have to make gmane happy.)
Pardon me for pontificating, but your pruning was perfect.
Too many posts technically use interleaved quoting, but render it useless by failing to trim the quoted material. The purpose of quoting is to provide context---so it requires some care in editing the quotes.
People who don't trim quotes might as well top-post---there's no benefit in bottom posting if it simply follows the entire quoted message, and top posting makes it actually easier to read the new material.
</old fart grumbling>
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
You've tried to select "stable" as the target already when submitting the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki, why didn't you choose "security" instead of "stable"?
But I did. All packages are marked as "security updates" in their "type". As a target ("request") you only have the choice "testing" or "stable" (and "none"). There isn't any from that mentions "security" and "stable".
E.g. the packages are marked as security updates and whatever the cause, autoqa, missing karma, missing time, for some reason (partly undisclosed as mentioned in my post yesterday) bodhi rejects them. IMO if the packager marks the package as as security update bodhi should stay out of the way and allow a human to decide on pushing the update or not. ATM bodhi cuts me off the pushers.
On Wed, 20 Apr 2011 11:02:12 +0300 Axel Thimm Axel.Thimm@ATrpms.net wrote:
On Tue, 2011-04-19 at 11:05 +0200, Michael Schwendt wrote:
You've tried to select "stable" as the target already when submitting the updates, and bodhi rejected that. With the CVEs mentioned for Mediawiki, why didn't you choose "security" instead of "stable"?
But I did. All packages are marked as "security updates" in their "type". As a target ("request") you only have the choice "testing" or "stable" (and "none"). There isn't any from that mentions "security" and "stable".
Right. Security updates aren't allowed to just go direct to stable either.
E.g. the packages are marked as security updates and whatever the cause, autoqa, missing karma, missing time, for some reason (partly undisclosed as mentioned in my post yesterday) bodhi rejects them. IMO if the packager marks the package as as security update bodhi should stay out of the way and allow a human to decide on pushing the update or not. ATM bodhi cuts me off the pushers.
Sadly, this is not practical.
Several points to note:
The various update streams flow differently. For a normal day, EPEL4/5/6 might have about 2-20 updates. It might be practical to look at all these for a quick glance. f14 (updates and testing) has around 30-50ish. f13 has around 5-20, and f15 has too many to even count. ;) It's just not at all practical to have the people signing the updates look at each one for critera.
We have had security updates that caused considerable problems. If the update is an important one, enlist users of that software to help test and +1 it.
kevin
Kevin Fenzi wrote:
On Wed, 20 Apr 2011 11:02:12 +0300 Axel Thimm Axel.Thimm@ATrpms.net wrote:
E.g. the packages are marked as security updates and whatever the cause, autoqa, missing karma, missing time, for some reason (partly undisclosed as mentioned in my post yesterday) bodhi rejects them. IMO if the packager marks the package as as security update bodhi should stay out of the way and allow a human to decide on pushing the update or not. ATM bodhi cuts me off the pushers.
Sadly, this is not practical.
Several points to note:
The various update streams flow differently. For a normal day, EPEL4/5/6 might have about 2-20 updates. It might be practical to look at all these for a quick glance. f14 (updates and testing) has around 30-50ish. f13 has around 5-20, and f15 has too many to even count. ;) It's just not at all practical to have the people signing the updates look at each one for critera.
The human making the decision should be the maintainer. That's what the maintainer is for.
We have had security updates that caused considerable problems.
We've had one such instance in years of Fedora.
Kevin Kofler
On Wed, 2011-04-20 at 12:26 -0600, Kevin Fenzi wrote:
The various update streams flow differently. For a normal day, EPEL4/5/6 might have about 2-20 updates. It might be practical to look at all these for a quick glance. f14 (updates and testing) has around 30-50ish. f13 has around 5-20, and f15 has too many to even count. ;) It's just not at all practical to have the people signing the updates look at each one for critera.
Are all these security updates? I'm only arguing in favour of a fastlane method for security updates.
The package in question may not be used by many people, but may have severe security implications. If the user count is low you will not find many or any users to karma it up, or even a proventester, OTOH the users that do have this package in operation will be exposed until the package sits off its time in testing - where probably no one will have given it a go anyway. You may also not want to advertise the security issues too loudly: You don't only attract testers that way, but also exploiters.
On Thu, 21 Apr 2011 10:51:16 +0300 Axel Thimm Axel.Thimm@ATrpms.net wrote:
On Wed, 2011-04-20 at 12:26 -0600, Kevin Fenzi wrote:
The various update streams flow differently. For a normal day, EPEL4/5/6 might have about 2-20 updates. It might be practical to look at all these for a quick glance. f14 (updates and testing) has around 30-50ish. f13 has around 5-20, and f15 has too many to even count. ;) It's just not at all practical to have the people signing the updates look at each one for critera.
Are all these security updates? I'm only arguing in favour of a fastlane method for security updates.
No, but the bodhi interface doesn't seperate them. You can push testing or stable for each release, and it basically gives you a long list of packages that are pending for those states. Then you sign them and push them out. To review security ones we would have to have it seperate them out, print out a url for each and have to review each one.
The package in question may not be used by many people, but may have severe security implications. If the user count is low you will not find many or any users to karma it up, or even a proventester, OTOH the users that do have this package in operation will be exposed until the package sits off its time in testing - where probably no one will have given it a go anyway. You may also not want to advertise the security issues too loudly: You don't only attract testers that way, but also exploiters.
Sure, but it's always an issue with projects like Fedora. You commit a fix to a security issue, someone watching commits can see it right then, before the package is even built much less pushed to stable.
It might be nice if we had a group of testers specifically testing security updates. That way they could check the CVE and commit and test the package out to get them moved as quickly as possible. Not sure how to create such a group however.
kevin
On Tue, 2011-04-19 at 11:30 +0300, Axel Thimm wrote:
On Sun, 2011-04-17 at 08:01 +0200, Kevin Kofler wrote:
Axel Thimm wrote:
Are you sure? I requested a push of the packages to stable (some were in testing for a week, others were security updates) and the message was that it doesn't pass AutoQA, so it converted to request to push only to testing and indeed bodhi has marked the request as to testing only.
It did that because you changed it.
If you had kept the request to stable, it would have been pushed to stable.
AutoQA is still in testing phase, there is no enforcement yet.
Maybe the bodhi messages confused me. When I logon to a.f.o/updates it prominently displays:
Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases.
One thing to clarify in case it's not sufficiently clear from all the other replies: this message is *not new* and has nothing to do with AutoQA at all, and specifically nothing to do with the recent implementation of advisory AutoQA upgrade path checking. This message has been there for several months, ever since we started enforcing karma requirements, and is nothing to do with AutoQA. Note that the message doesn't *say* anything about AutoQA.
On 04/16/2011 01:52 PM, Axel Thimm wrote:
Hi,
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the packages to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>. Do I have to wait until the packages for f15 are pushed to repush the packages for f14 and then wait again for pushing into f13? Or is there some automatism that reevaluates and repushes packages w/o any further intervention from the packagers?
I guess this is also slowing down the people that grant the package push. Previously they could evaluate the package updates in one sweep, now they will see and approve the "same" package in three cycles.
Others have already commented on this, but I wanted to re-state that AutoQA is only in an informative mode right now. Any bodhi comments made by AutoQA have no karma and are not used for anything other than information. This will change sometime in the future, but not until at least the F16 timeframe.
I took a look at your packages in bodhi [1] and was only able to find one update with FAILED comments from AutoQA. Are there updates that I'm missing here?
Looking at the depcheck output for mediawiki-1.16.4-57.fc13, however, I see:
mediawiki-nomath-1.16.4-57.fc13.x86_64 from pending has depsolving problems --> mediawiki-nomath conflicts with php-common
This stems from the addition of the "Conflicts: php-common = 5.3.1" that was added to the spec file [2] for 1.16.2-56.
I assume that this isn't the only package with a "Conflicts:" in the spec and have filed a ticket against autoqa to better handle these issues [3].
Tim
PS - you will generally get a faster response to AutoQA related questions by posting to autoqa-devel@lists.fedorahosted.org instead of just devel@
[1] https://admin.fedoraproject.org/updates/user/athimm [2] http://pkgs.fedoraproject.org/gitweb/?p=mediawiki.git;a=blob;f=mediawiki.spe... [3] https://fedorahosted.org/autoqa/ticket/308
On Sun, 2011-04-17 at 21:19 -0600, Tim Flink wrote:
On 04/16/2011 01:52 PM, Axel Thimm wrote: Others have already commented on this, but I wanted to re-state that AutoQA is only in an informative mode right now. Any bodhi comments made by AutoQA have no karma and are not used for anything other than information. This will change sometime in the future, but not until at least the F16 timeframe.
See my post from a few minutes ago: 5 packages are currently rejected on a "mark as stable" submission leaving the request field empty (as if I hadn't tried to push) with the comment that the packages do not pass autoqa.
I took a look at your packages in bodhi [1] and was only able to find one update with FAILED comments from AutoQA. Are there updates that I'm missing here?
This is probably part of the problem, I have been trying to push all 5 packages that are now in testing with bodhi rejecting due to autoqa. Even packages that do have a positive autoqa tag on them like fail2ban-0.8.4-27.fc13.
Looking at the depcheck output for mediawiki-1.16.4-57.fc13, however, I see:
mediawiki-nomath-1.16.4-57.fc13.x86_64 from pending has depsolving problems --> mediawiki-nomath conflicts with php-common
This stems from the addition of the "Conflicts: php-common = 5.3.1" that was added to the spec file [2] for 1.16.2-56.
What about f14 and f15? At least for f15 autoqa should have greenlit the package, yet it never even tested the package, but bodhi calims it fails autoqa. f14 should had never passed before f15, but it still did, but is blocked anyway (???).
PS - you will generally get a faster response to AutoQA related questions by posting to autoqa-devel@lists.fedorahosted.org instead of just devel@
Thanks, but I just want to get the packages through, I'm on several dozen fedora lists already and cannot process the information stream in time. I also consider autoqa issues - if they are stalling packages as I experience - important for anyone in devel. Alone the fact that everyone here thinks differently about how autoqa works or should work shows that the discussion is helpful on this list.
On Tue, Apr 19, 2011 at 11:44:03AM +0300, Axel Thimm wrote:
This is probably part of the problem, I have been trying to push all 5 packages that are now in testing with bodhi rejecting due to autoqa. Even packages that do have a positive autoqa tag on them like fail2ban-0.8.4-27.fc13.
According to the Bodhi-Status-Site, you unpushed the update on 2011-04-17 21:24:29, then submitted it again a few seconds later.
It has been pushed to testing on 2011-04-17 21:24:29 and it now needs to stay in testing for a week (or until it has reached sufficient karma including proventester feedback) until it can be pushed to stable.
This is what bodhi refers to with "Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases." - that text also links to https://fedoraproject.org/wiki/Package_update_acceptance_criteria
On Tue, 2011-04-19 at 10:54 +0200, Sven Lankes wrote:
On Tue, Apr 19, 2011 at 11:44:03AM +0300, Axel Thimm wrote:
This is probably part of the problem, I have been trying to push all 5 packages that are now in testing with bodhi rejecting due to autoqa. Even packages that do have a positive autoqa tag on them like fail2ban-0.8.4-27.fc13.
According to the Bodhi-Status-Site, you unpushed the update on 2011-04-17 21:24:29, then submitted it again a few seconds later.
Yes, this was after I had desperately tried to push it from testing to stable and at the end tried to resubmit directly to stable. As said previously security updates at least did get their requests noted.
It has been pushed to testing on 2011-04-17 21:24:29 and it now needs to stay in testing for a week (or until it has reached sufficient karma including proventester feedback) until it can be pushed to stable.
Oh well, it had been in testing for almost a week and I even felt bad about not stable-pushing it earlier as it was a security update.
This is what bodhi refers to with "Bodhi is now enforcing the Package Update Acceptance Criteria across all Fedora releases." - that text also links to https://fedoraproject.org/wiki/Package_update_acceptance_criteria
This mentions that critical packages and security updates need at least 2 karma points and a proventester, that's even more than setting karma on threshold 1 (???). Is that really the current policy for _security updates_?
On Wed, 2011-04-20 at 11:30 +0300, Axel Thimm wrote:
This mentions that critical packages and security updates need at least 2 karma points and a proventester, that's even more than setting karma on threshold 1 (???). Is that really the current policy for _security updates_?
2 karma points *including* a proventester. That is:
1 proventester + 1 non-pt
OR
2 proventester
are sufficient. And yes, that is the current policy.
Adam Williamson wrote:
2 karma points *including* a proventester. That is:
1 proventester + 1 non-pt
OR
2 proventester
are sufficient. And yes, that is the current policy.
Uh, that's the current policy for critical path packages, not for non- critical packages which require only the autokarma set by the maintainer (which can be as low as 1) to be reached.
Kevin Kofler
On Wed, 2011-04-20 at 23:52 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
2 karma points *including* a proventester. That is:
1 proventester + 1 non-pt
OR
2 proventester
are sufficient. And yes, that is the current policy.
Uh, that's the current policy for critical path packages, not for non- critical packages which require only the autokarma set by the maintainer (which can be as low as 1) to be reached.
Unfortunately security updates are explicitly listed as to be handled like critical path packages, so Adam is correct in the current context, at least if the wiki page is not incorrect.
There is a contradiction here: As some people noticed in the thread it is easy to fool the system for conventional updates by setting the karma threshold to a low value as you suggest. For security updates that need to be pushed rather earlier than later these requirements result in them being harder to be pushed. As a consequence if I as a packager were eager to fix a security issue I would indeed have to *not* tag it as a security update to reduce the ETA.
This cannot be right and in general the discussion moved to how to fool the system with adjusting parameters etc. and to my surprise there aren't many that object to doing so!
I think when it becomes normal to circumvent the system then perhaps we need to rethink it and implement it in a sane way?
On Thu, 21 Apr 2011 10:42:04 +0300, AT wrote:
I think when it becomes normal to circumvent the system then perhaps we need to rethink it and implement it in a sane way?
I wouldn't say "it becomes normal", but certainly there's a tendency towards wanting updates be pushed into the stable repo _as soon as possible_. Especially those updates where the package maintainer knows what he's doing. Also those, where a few guys try to bump the karma level even before an update has been distributed widely enough by the mirror system. The "minimum time to spend in updates-testing" is not put to good use.
Add to that all those updates, which need to get marked stable by their submitter without anyone having spent karma points on them (and with bug reporters in bugzilla not commenting on them either).
For those the Update Acceptance Criteria are just a hindrance.
[...]
And with all the testing requirements, major breakage slips through nevertheless, because we're missing testing instructions for nearly all updates.
<rant> Some weeks ago someone rushed out a mesa-dri-drivers upgrade for F-14 that made projectM stop working for ATI/Radeon. The corresponding assignee at Red Hat doesn't answer in bugzilla. Yesterday evening, an incoming crash of Audacious on F-14 made me think "WTF?", and this time it was a bad projectM update that has not been tested painstakingly despite me having raised a few essential questions in the bodhi ticket.</rant>
On Thu, 2011-04-21 at 12:17 +0200, Michael Schwendt wrote:
And with all the testing requirements, major breakage slips through nevertheless, because we're missing testing instructions for nearly all updates.
To reiterate this again (thanks, Babe!) we've never claimed the process would catch all breakage, or expected it to. The point is to catch as much as we can, and the amount of issues we've caught with updates-testing is a lot more than 0.
On Thu, 21 Apr 2011 11:50:15 -0700, AW wrote:
To reiterate this again (thanks, Babe!) we've never claimed the process would catch all breakage, or expected it to. The point is to catch as much as we can,
In my point of view, "as much as we can" is not what is being done so far at Fedora.
What is implemented is a race between people, who are involved in rushing out updates ("a day without risky upgrades is a boring day"), and other parts of the Fedora user community, who may need a little bit more time to take notice of new Test Updates [and possibly even more time to actually run some tests or start evaluating the new packages].
The update acceptance criteria are half-hearted for some updates and a hindrance for other updates.
and the amount of issues we've caught with updates-testing is a lot more than 0.
That doesn't say anything about whether this would have been much different if package maintainers were free to decide on how long to keep their updates in updates-testing and how long to wait for pos/neg feedback from testers.
Axel Thimm wrote:
Unfortunately security updates are explicitly listed as to be handled like critical path packages, so Adam is correct in the current context, at least if the wiki page is not incorrect.
I think you're misunderstanding the wiki page. The "including security updates" phrase there means that security updates for critical path packages are to be treated like any other critpath updates. It doesn't say anything about security updates for other packages.
Kevin Kofler
On Thu, 2011-04-21 at 19:05 +0200, Kevin Kofler wrote:
Axel Thimm wrote:
Unfortunately security updates are explicitly listed as to be handled like critical path packages, so Adam is correct in the current context, at least if the wiki page is not incorrect.
I think you're misunderstanding the wiki page. The "including security updates" phrase there means that security updates for critical path packages are to be treated like any other critpath updates. It doesn't say anything about security updates for other packages.
Right. It's pretty simple - at present, whether an update is a security update simply doesn't matter to the acceptance process. It's not taken into account at all. What's taken into account is whether the update is for a stable release or for a pre-release, and whether it's critical path. That's all.
On Wed, 2011-04-20 at 23:52 +0200, Kevin Kofler wrote:
Uh, that's the current policy for critical path packages, not for non- critical packages which require only the autokarma set by the maintainer (which can be as low as 1) to be reached.
No, they always just require +1 from anyone (or an X day wait). This happens to be equivalent to the lowest possible autokarma setting, but that's just a coincidence, it doesn't *mean* anything. If you set autokarma higher, you can still push manually once you have +1 from anyone.
Adam Williamson wrote:
No, they always just require +1 from anyone (or an X day wait). This happens to be equivalent to the lowest possible autokarma setting, but that's just a coincidence, it doesn't *mean* anything. If you set autokarma higher, you can still push manually once you have +1 from anyone.
Actually, that doesn't work. It's one of the many things Bodhi doesn't implement as specified. You have to edit the autokarma down to 1; only then, Bodhi will let you push the update.
Kevin Kofler
On Thu, 2011-04-21 at 21:11 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
No, they always just require +1 from anyone (or an X day wait). This happens to be equivalent to the lowest possible autokarma setting, but that's just a coincidence, it doesn't *mean* anything. If you set autokarma higher, you can still push manually once you have +1 from anyone.
Actually, that doesn't work. It's one of the many things Bodhi doesn't implement as specified. You have to edit the autokarma down to 1; only then, Bodhi will let you push the update.
Erf, I thought that got fixed ages ago. It's certainly not intended to be that way - it's supposed to work as I described.
On 04/21/2011 12:11 PM, Kevin Kofler wrote:
Adam Williamson wrote:
No, they always just require +1 from anyone (or an X day wait). This happens to be equivalent to the lowest possible autokarma setting, but that's just a coincidence, it doesn't *mean* anything. If you set autokarma higher, you can still push manually once you have +1 from anyone.
Actually, that doesn't work. It's one of the many things Bodhi doesn't implement as specified. You have to edit the autokarma down to 1; only then, Bodhi will let you push the update.
No. Case in point, I just pushed https://admin.fedoraproject.org/updates/FEDORA-2011-5635 which has autokarma of 3, and only 1 karma point.
Christopher Aillon wrote:
No. Case in point, I just pushed https://admin.fedoraproject.org/updates/FEDORA-2011-5635 which has autokarma of 3, and only 1 karma point.
Oh, great that this has been fixed! It must have been fixed fairly recently. Last time I tried, it didn't work. That was one of the things which always annoyed me about Bodhi.
Kevin Kofler
Christopher Aillon wrote:
On 04/21/2011 12:11 PM, Kevin Kofler wrote:
Actually, that doesn't work. It's one of the many things Bodhi doesn't implement as specified. You have to edit the autokarma down to 1; only then, Bodhi will let you push the update.
No. Case in point, I just pushed https://admin.fedoraproject.org/updates/FEDORA-2011-5635 which has autokarma of 3, and only 1 karma point.
It's still broken for at least F14 updates with autokarma disabled: I tried pushing https://admin.fedoraproject.org/updates/git-cola-1.4.3.2-1.fc14 to stable, which has a karma of +1, and it says "This update has not yet met the minimum testing requirements defined in the Package Update Acceptance Criteria" (which is very stupid because I can just set the autokarma to 1 and push it then).
Kevin Kofler
Hi,
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the packages to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>.
I suppose you're talking about upgradepath test. Yes, that's exactly its behavior.
Do I have to wait until the packages for f15 are pushed to repush the packages for f14 and then wait again for pushing into f13? Or is there some automatism that reevaluates and repushes packages w/o any further intervention from the packagers?
No need to do that, upgradepath re-evaluates all proposed updates again and again (every time a new update is proposed into bodhi). If your update failed upgradepath, a new comment will appear once it passes (or every 3 days if it still fails).
I guess this is also slowing down the people that grant the package push. Previously they could evaluate the package updates in one sweep, now they will see and approve the "same" package in three cycles.
This should be improved in the future.
On Mon, 2011-04-18 at 04:35 -0400, Kamil Paral wrote:
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the
packages
to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>.
I suppose you're talking about upgradepath test. Yes, that's exactly its behavior.
This seems wrong to me. If I as a packager create an upgrade in bodhi, autoqa should consider what is present in f<N+1> not only as stable package but also as proposed upgrade. It should then impose a constraint that the upgrade for f<N> can only be pushed to stable if f<N+1> is pushed to stable at the same time (or obviously independently before).
Seeing a failed upgradepath test comment at least gives me the wrong feeling on what is going on.
- Andreas
On Mon, 18 Apr 2011 12:57:55 +0200, AB wrote:
On Mon, 2011-04-18 at 04:35 -0400, Kamil Paral wrote:
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the
packages
to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>.
I suppose you're talking about upgradepath test. Yes, that's exactly its behavior.
This seems wrong to me. If I as a packager create an upgrade in bodhi, autoqa should consider what is present in f<N+1> not only as stable package but also as proposed upgrade. It should then impose a constraint that the upgrade for f<N> can only be pushed to stable if f<N+1> is pushed to stable at the same time (or obviously independently before).
Seeing a failed upgradepath test comment at least gives me the wrong feeling on what is going on.
That is also going too far.
The upgradepath check ought to stay informative and should not impose any constraints on the packagers. Consider it an early-warning system plus a way to educate packagers. Once the packager has learned about the violated upgradepath, it's up to the packager to decide whether to delay the upgrade of f<N> and work on an upgrade of f<N+1> first. It should be possible to update f<N> before f<N+1>, possibly via karma automatism or for a security-fix that's ready for f<N> but needs a revised build for f<N+1>. Let's not mess too much with a packager's work-flow.
On Mon, 2011-04-18 at 04:35 -0400, Kamil Paral wrote:
I built two sets of security updates for f13/f14/f15 and autoqa rejected the f13/f14 packages. It looks like autoqa is waiting for the
packages
to be properly pushed to f<N+1> stable before green-lighting the matching package for f<N>.
I suppose you're talking about upgradepath test. Yes, that's exactly its behavior.
This seems wrong to me. If I as a packager create an upgrade in bodhi, autoqa should consider what is present in f<N+1> not only as stable package but also as proposed upgrade. It should then impose a constraint that the upgrade for f<N> can only be pushed to stable if f<N+1> is pushed to stable at the same time (or obviously independently before).
It is just a best effort for now. Once we are able to create those "safe to push" update sets, we will be able to consider not only packages in stable, but also packages in other currently proposed updates.
We want to reach the state you're talking about (while still staying informative). It's just not a few hours work :)
I've created a short blogpost about this issue:
http://kparal.wordpress.com/2011/04/18/autoqa-upgradepath-vs-updates-to-mult...
-
Adam Williamson -
Andre Robatino -
Andreas Bierfert -
Axel Thimm -
Ben Boeckel -
Christopher Aillon -
James Laska -
Kamil Paral -
Kevin Fenzi -
Kevin Kofler -
Michael Schwendt -
Nathanael D. Noblet -
Przemek Klosowski -
Sven Lankes -
Thomas Spura -
Tim Flink