My understanding to the original design goal with depcheck was that once an update was tested and accepted ... it must not be revoked. The set of accepted updates must always be dep-free. If new builds are added to an update, those builds must represent a new set of -pending updates for testing. If another update lands in -pending, and causes problems with some previously accepted update ... the problem is with the new update, not with something already accepted.
Does this design goal still hold?
Yes, it does. Unless some update in the accepted set changes.
Even though the updates should not influence each other (all the related builds should be grouped together in a single update), it's naive to suppose it's always the case. If we presume that sometimes two different updates can influence each other (which is the main presumption of our whole depcheck testing in the first case, btw), then we have to clear the accepted set and start building it again when some already accepted update has been changed. Because it could have broken not only itself, but also some other already accepted updates.
That's the Level 2 (long-term) solution.
After we have the concept of ResultDB and we abolish the concept of sending emails to maintainers for every single change in depcheck results, we can do the proper solution as described by Josef. That means emptying the 'accepted set' if some update from that set has been changed.
When new builds are added to an existing update, the previous builds may have already moved on to updates-[testing] and will be automatically included in future depcheck runs. If they aren't in updates[-testing] yet, will bodhi remove the -pending tags from the previous builds, or will there be two versions of the same package still in -pending?
I suppose that if I have update "foo-1.1,bar-1.1" and then edit the update and replace "foo-1.1" for "foo-1.2", then "foo-1.1" should be untagged from updates-testing and "foo-1.2" should be tagged into updates-testing. Confirmation with lmacken needed. I believe this is exactly the kind of issues we've been hitting lately with Bodhi (untagging wasn't done right).
This only concerns updates-testing. Once it is in stable updates, you can't edit it.
One point I get confused no here is thinking about updates and builds. With depcheck, is it safe to say we test builds, but provide feedback against updates?
Correct, we say that update has passed <=> all its builds have passed.
When new builds are added to an existing update ... they are tagged with -pending right? As long as that's true, they'll be treated as new builds, right?
They will, once we fix https://fedorahosted.org/autoqa/ticket/306
Currently, we would suppose the new build was already tested, because its parent update already has our comments.
It's our update feedback procedure that needs to accommodate for an already tested, but modified, update?
Yes, we need to accomodate bodhi_already_commented() to detected changed updates. Currently it doesn't. That's the Level 1 solution.
The issue in question now is if those new builds introduce a dependency conflict? In that case, we should note the failure in the bodhi update since the results are against newer builds
Yes, we want to post new results to bodhi because the update has changed.
(and remove any previously supplied karma </future>)?
We don't supply karma now and I don't think we will in the near future. And the far future is far ahead :)