On 05/02/2011 09:54 AM, Kamil Paral wrote:
>> Can we just send a single email after all tests have passed?
-
>> Again, if so, should we?
>
> Is there a reasonable way that we could do this without ResultsDB?
> Sure, we could code up some stuff to determine what packages are
> supposed to pass what tests, but this seems to be quite a bit of
> overlap with PUATP and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we
don't have ResultDB (which will take a while) and we need some
reasonable workaround till that time. Of course some of that work
will be nicely usable even for ResultDB (like splitting one bug
test output into multiple per-item logs). But some stuff is just
temporary fix. Therefore I wouldn't really be afraid of a few
unclean hacks that will be removed once we have ResultDB.
I think that we're of the same mind on this. I'm not completely against
not-completely-clean hacks for the short term but I also think that it
would be wise to think about the level of effort we're putting in. In my
mind, we're shooting for somewhat short-term gains on some of this and
after a certain point, energy devoted to hacking stuff together could be
better directed towards finishing ResultsDB.
So I thought about this. Could we send an email only after all tests
have finished? That would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded
definitions.
We know that we currently execute only two tests, depcheck and
upgradepath. Upgradepath runs always on a noarch arch. Depcheck runs
on i386 and x86_64 in 99,9% [1].
If there are that few, we could hard code them into the bodhi commenting
code. Not sure that is the greatest idea ever, though. Another option
would be to document the issue and contact the affected maintainers.
So our code may default to email=False and change to email=True only
when all tests have been executed (counting also the current one that
is to be sent). That alone decreases the email load by 66%.
That works for me in the short term (the hardcoding part).
Let's go further. If all tests are completed and PASSED, I
wouldn't
send an email. If everything went fine, why bother the maintainer?
All tests completed and PASSED might be worth sending an email for as a
notification that everything is working but I see your point. We might
want to get some input from maintainers on that one.
Let's go further. Currently we re-send FAILED results after at
least
3 days from the first failure. I wouldn't send email notifications
for them. The purpose is to provide a fresh failure log, not the
notification (the maintainer already knows about it).
That makes sense to me unless we do really want to pester maintainers
with packages that are failing repeatedly. Maybe up it from 3 to 5 or 7
days?
Then there remains the question of PASS->FAIL and FAIL->PASS
transitions. I would probably send notifications for them. They seem
rather important.
At the very least, transitions are important. Agreed on this one.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages.
About 20 packages is 32bit only:
http://fpaste.org/BRwA/ and 4
packages is 64bit only:
http://fpaste.org/WVBO/ I'm willing to
sacrifice that, for now. We have much higher test crash ratio
anyway.