Comments intended for my clarification on the role of depcheck and differences between builds and updates.
On Fri, 2011-04-15 at 05:40 -0400, Kamil Paral wrote:
I wonder how hard it would be to change the way depcheck and bodhi work a little bit.
Instead of just scraping comments, another option would be to have a text field in bodhi that held the list of passing tests. When a test passes, it adds its name to the field. When an update is changed, that field is reset and the tests are re-run.
That would be a nice solution if we didn't have ResultsDB in our plan.
We might want to get in touch with lmacken, I think there are some rough plans for that type of karma feedback.
It might make more sense to wait for resultsdb on this one, I'm not sure. Either one but might be more accurate and faster than trying to interpret comments.
Sending bodhi comments was just a quick solution for letting maintainers know. Instead of spending time on further temporary hacks (like hacking into Bodhi something that won't be needed soon) I see as a better solution to spend the time on the proper solution - ResultDB. It's not hard, it just needs focus.
Do we know how often this is happening?
Sometimes. Maintainers change their updates every now and then. Level 1 solution (re-test just the changed update) is extremely easy to implement and eliminates a lot of problems. So it's reasonable to spend a few hours and have it done.
=== RelEng workflow ===
At the moment, rel-eng is pushing updates manualy few times a week [citation needed, this is informal information which has not been audited].
The thing is, that we can provide the _at the moment 'pushable'_ subset of packages. When rel-eng will want to push from -pending to {stable, updates, ...}, they'll just 'list' the pushable subset (yes, resultsdb will help ;-)).
"The tool" will detect if there were any changes in the pushable subset, and "the tool" will inform rel-eng that they either need to wait for the results of the next depcheck run, or (if we're able to do it) request new depcheck run.
Why not just make a tool that re-reruns all "*-pending" updates through a battery of tests (depcheck and upgradepath ATM), ignoring any "PASSED" results and not posting any new comments to bodhi unless there was a change for an update? That way we don't have to worry about detecting if there were changes or not and we can be more confidant of the to-be-pushed set as a whole (which is what our end goal is, IIRC)
That's exactly what upgradepath does now. Depcheck is little different, it uses the concept of 'accepted updates'.
We can't remove the 'accepted updates' concept from depcheck, because that could cause a flood of emails going to maintainers' mailboxes through bodhi comments. If we don't use the concept of ever-growing accepted set of updates, it might happen that your update will be accepted at 1 PM, rejected at 2 PM (someone pushed conflicting update), accepted at 3 PM (someone removed the conflicting update), etc. Hence the Level 1 solution, which is not perfect, but doesn't spam maintainers.
My understanding to the original design goal with depcheck was that once an update was tested and accepted ... it must not be revoked. The set of accepted updates must always be dep-free. If new builds are added to an update, those builds must represent a new set of -pending updates for testing. If another update lands in -pending, and causes problems with some previously accepted update ... the problem is with the new update, not with something already accepted.
Does this design goal still hold?
After we have the concept of ResultDB and we abolish the concept of sending emails to maintainers for every single change in depcheck results, we can do the proper solution as described by Josef. That means emptying the 'accepted set' if some update from that set has been changed.
When new builds are added to an existing update, the previous builds may have already moved on to updates-[testing] and will be automatically included in future depcheck runs. If they aren't in updates[-testing] yet, will bodhi remove the -pending tags from the previous builds, or will there be two versions of the same package still in -pending?
One point I get confused no here is thinking about updates and builds. With depcheck, is it safe to say we test builds, but provide feedback against updates? When new builds are added to an existing update ... they are tagged with -pending right? As long as that's true, they'll be treated as new builds, right? It's our update feedback procedure that needs to accommodate for an already tested, but modified, update?
The issue in question now is if those new builds introduce a dependency conflict? In that case, we should note the failure in the bodhi update since the results are against newer builds (and remove any previously supplied karma </future>)?
It also means providing the latest never-outdated results directly to RelEng (or anyone else interested) on request, not continuously to package maintainers. If _in that moment of RelEng push_ some update is excluded because of failed dependencies, only after that we can send email to the maintainer. (Or some similar approach).
To sum it up, the really proper solution is not possible as long as we rely on sending emails to maintainers for every change. That's not viable, just temporary, because we could easily turn into spammers.
Agreed.
Thanks, James