#206: Update bodhi to enforce Package Update Acceptance policy
------------------------+--------------------------------------------------
Reporter: wwoods | Owner:
Type: task | Status: new
Priority: major | Milestone: Package Update Acceptance Test Plan
Component: production | Resolution:
Keywords: | Blocked By:
Blocking: |
------------------------+--------------------------------------------------
Comment (by tflink):
Replying to [comment:4 kparal]:
I occasionally see some questions or downright fixes related to
AutoQA
results, which gladdens me. But these are the obstacles until we can
enable enforcing mode:
1. Depcheck needs to be re-written.
Another option would be to have a whitelist of failures that are ignored.
I'm not sure that's the best idea, but it would solve the biggest problem
that you brought up.
Another thought would be to re-think depcheck. I haven't had the chance to
try it out yet due to slight incompatibility with Fedora but EDOS
(currently used by debian) is an interesting and very different approach
to what we're trying to do with depcheck. [
http://arxiv.org/abs/0811.3620
A paper describing the process is available here]
2. There is no possibility to override results. That would probably
be
implemented inside Bodhi, so it's not our problem. But we have to come up
with processes - who decides whether you can override, whether you need
some whitelist of packages that are often evaluated incorrectly, etc.
Yeah, we would need this at the very least.
3. We need to implement the option to re-run any test from scratch.
I wonder if we can get away with contributors being able to request re-
runs instead of needing an interface for them to do it themselves.
Eventually, I agree that we need some self-service interface but I'm not
as sure it's blocking the enforcement right now.
4. I think deeper integration of Bodhi and our ResultsDB is required.
As
we were discussing a long time ago, in order to produce trustworthy
results we need to operate on package sets, not individual packages. We
can say "if you push this set of packages to stable, it won't break". If
one of those packages is unpushed, the whole set must be invalidated and
new results must be computed. We can't really achieve that just with Bodhi
comments. It needs development on our side and on Bodhi side. Releng team
must then push just the package set as indicated by Bodhi, nothing more,
nothing less.
I'm not quite sure I follow you on this one. Yeah, it would be great to
have better integration between ResultsDB and Bodhi but I'm not sure it's
a requirement to start enforcing the update plan. We probably need better
integration between AutoQA as a whole and Bodhi but I'm not sure how
ResultsDB fits in to making sure that an update is re-tested after
modification.
5. If we start to experiment with enforcing mode, we'll probably
discover many more problems on the way, people will start to get louder.
Well, I think people are going to complain no matter what we do. However,
I also think that taking reasonable steps to counter the loudness would be
wise on our part.
As I currently see it, I would rather keep the
permissive/informative
mode on, and a) develop the framework the way we need it b) encourage and
promote the informative results (making it more maintainer-friendly,
improve documentation, improve integration with Bodhi, etc).
Yeah, maintainer-friendlyness is something that occurred to me over the
weekend. I'm also not sure that we're friendly enough to maintainers in
order to start forcing them to use AutoQA results. We could suggest doing
so but I imagine that there would be a lot of pushback.
Our solution would be currently half-baked at best. Our dependency
checking is not good enough, and our upgradepath checking is good enough,
but misses the infrastructure to make updates pushing reliable.
Well, when do we think that AutoQA would be good enough to start
enforcing? I realize that there isn't a good answer to that question - I'm
more wondering if we should try to come up with a better answer instead of
just "real soon now".
--
Ticket URL: <
https://fedorahosted.org/autoqa/ticket/206#comment:5>
AutoQA <
http://autoqa.fedorahosted.org>
Automated QA project