On 04/29/2011 03:10 PM, Tim Flink wrote:
How would we deal with tests that change state between PASS and
FAIL?
- Either 'PASS -> FAIL' or 'FAIL -> PASS'
I think that both of these cases are important and I'm not sure how to
do them without at least some parsing and comprehension of update
comments as a whole.
This could get really complicated, really quick but I have some ideas
based on parsing comments, creating sequences of test result order and
making more holistic decisions.
I tried putting my thoughts into words, but it wasn't working. I'm not
convinced that my idea here is actually going to work with a reasonable
amount of work.
I'll see if I can spike something together for a rough concept. It might
not get done today, though.
What emails do we want to get rid of?
- Just certain 'PASS' messages?
- All results messages?
I hadn't put a whole lot of thought into the details of this, but my
initial thought on the most important messages we could send would be:
- Update PASS/FAIL status transition
- Package PASS/FAIL status transition
For now, I'm wondering if package state transition would be good enough.
How do we expect maintainers to learn of AutoQA results if we turn
off
email notifications?
I can't really think of anything here. I think that some pestering is a
good thing, just not too much and that turning off emails completely
would leave us with not enough pestering.
Can we do this in a way such that it is configurable by maintainer?
- If so, should we attempt this?
- What is the benefit? How big is it?
This is a hard one. I'm sure that we could do it but I'm more interested
in the should (at least for the near future).
It would be a decent amount of work and require some new concepts in
AutoQA - interfacing with FAS (at least indirectly), defining various
knobs to turn for update emails and looking that up before or during
commenting.
I'm thinking that this would be a cool feature but unless we start
getting requests for it, I'd vote for not now.
Can we just send a single email after all tests have passed?
- Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB? Sure,
we could code up some stuff to determine what packages are supposed to
pass what tests, but this seems to be quite a bit of overlap with PUATP
and ResultsDB to me.
I'm inclined to say wait on ResultsDB for this one. I think that it
might be a good idea to continue to use Bodhi comments to notify
maintainers of update test status after we implement ResultsDB but I
can't think of an easy way to do this until then.
Is there another approach that would be better?
I'm probably the wrong person to answer this question but I'm interested
in hearing what the alternatives would be.
I like this approach because it gives us the ability to determine when
emails are getting sent. Unless that functionality is already in bodhi
or elsewhere, I'm not sure it makes a whole lot of sense to do the
filtering outside AutoQA.