#248: depcheck: simultaneous run may report incorrect results --------------------+------------------------------------------------------- Reporter: kparal | Owner: wwoods Type: task | Status: assigned Priority: major | Milestone: 0.4.4 Component: tests | Resolution: Keywords: | --------------------+------------------------------------------------------- Comment (by wwoods):
There should only be four reasons the pending/accepted sets will change during a test:
1. New updates arrived in pending.
This implies that the test has been scheduled again for the new update. So we can cancel our run and let the newer run override it.
2. Another depcheck test completed, and moved updates from pending to accepted.
Our results are invalid because our input data has changed. Restart the test if there's still something to test.
3. Updates were untagged (obsoleted by maintainer or removed by rel-eng)
Same as above - Inputs changed, restart the test.
4. Updates were pushed live.
Again, our inputs have changed, so we must restart. Technically, though, if the ''only'' change to either set is that accepted updates were pushed live, the test results will come out the same. So in that case restarting is unnecessary, but also harmless.
So the test wrapper can work sort of like this: {{{ result = None tries_remaining = 10 (pending, accepted) = get_pending_and_accepted_sets() while pending and tries_remaining and (result is None): tries_remaining -= 1 result = do_test(pending, accepted)
# did anything change? old_pending = pending old_accepted = accepted (pending, accepted) = get_pending_and_accepted_sets() if (old_pending == pending) and (old_accepted == accepted): post_results(result) else: result = None # invalidate result if pending.difference(old_pending): # new updates means another test scheduled test_exit() if result is None: test_error("too many retries") }}}
{{{tries_remaining}}} should be unnecessary but it might be a good idea to have it in there, just in case.
So, as far as I can tell the only place we might need locking is around {{{get_pending_and_accepted_sets()}}} and {{{post_results(result)}}} - we can't have the sets changing after the check, but before/during the posting of results.