#206: Update bodhi to enforce Package Update Acceptance policy
------------------------+--------------------------------------------------
Reporter: wwoods | Owner:
Type: task | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: production | Resolution:
Keywords: | Blocked By:
Blocking: |
------------------------+--------------------------------------------------
Comment (by kparal):
I occasionally see some questions or downright fixes related to AutoQA
results, which gladdens me. But these are the obstacles until we can
enable enforcing mode:
1. Depcheck needs to be re-written. Sorry to say it this roughly. Depcheck
doesn't work for packages that uses conflicts or obsoletes. Sometimes it
doesn't produce any result at all for selected packages. Fortunately these
issues concern just a fraction of packages (less than several percent, I'd
guess), but they are present. We are eagerly waiting for libsolve to
appear.[[BR]][[BR]]OTOH our second test, upgradepath, seems not to suffer
from any issues and probably could be enforcing.
2. There is no possibility to override results. That would probably be
implemented inside Bodhi, so it's not our problem. But we have to come up
with processes - who decides whether you can override, whether you need
some whitelist of packages that are often evaluated incorrectly, etc.
3. We need to implement the option to re-run any test from scratch.
Because something had failed or there had been a bug in our test, we could
have marked some package incorrectly. We (or the maintainer) should be
able to re-run it again. But to tell the truth, we probably don't need
this option for the current implementations of depcheck and upgradepath,
because they try to evaluate all available packages every time. But for
some future tests this will be needed for sure.
4. I think deeper integration of Bodhi and our ResultsDB is required. As
we were discussing a long time ago, in order to produce trustworthy
results we need to operate on package sets, not individual packages. We
can say "if you push this set of packages to stable, it won't break". If
one of those packages is unpushed, the whole set must be invalidated and
new results must be computed. We can't really achieve that just with Bodhi
comments. It needs development on our side and on Bodhi side. Releng team
must then push just the package set as indicated by Bodhi, nothing more,
nothing less.
5. If we start to experiment with enforcing mode, we'll probably discover
many more problems on the way, people will start to get louder.
As I currently see it, I would rather keep the permissive/informative mode
on, and a) develop the framework the way we need it b) encourage and
promote the informative results (making it more maintainer-friendly,
improve documentation, improve integration with Bodhi, etc). Our solution
would be currently half-baked at best. Our dependency checking is not good
enough, and our upgradepath checking is good enough, but misses the
infrastructure to make updates pushing reliable.
--
Ticket URL: <
https://fedorahosted.org/autoqa/ticket/206#comment:4>
AutoQA <
http://autoqa.fedorahosted.org>
Automated QA project