> Can we just send a single email after all tests have passed?
> - Again, if so, should we?
Is there a reasonable way that we could do this without ResultsDB?
Sure,
we could code up some stuff to determine what packages are supposed to
pass what tests, but this seems to be quite a bit of overlap with
PUATP
and ResultsDB to me.
From what I have understood we're doing 0.5.0 exactly because we
don't have ResultDB (which will take a while) and we need some reasonable workaround
till that time. Of course some of that work will be nicely usable even for ResultDB (like
splitting one bug test output into multiple per-item logs). But some stuff is just
temporary fix. Therefore I wouldn't really be afraid of a few unclean hacks that will
be removed once we have ResultDB.
So I thought about this. Could we send an email only after all tests have finished? That
would decrease email load substantially.
I believe we can, if we can live with some temporary hardcoded definitions.
We know that we currently execute only two tests, depcheck and upgradepath. Upgradepath
runs always on a noarch arch. Depcheck runs on i386 and x86_64 in 99,9% [1].
So our code may default to email=False and change to email=True only when all tests have
been executed (counting also the current one that is to be sent). That alone decreases the
email load by 66%.
Let's go further. If all tests are completed and PASSED, I wouldn't send an email.
If everything went fine, why bother the maintainer?
Let's go further. Currently we re-send FAILED results after at least 3 days from the
first failure. I wouldn't send email notifications for them. The purpose is to provide
a fresh failure log, not the notification (the maintainer already knows about it).
Then there remains the question of PASS->FAIL and FAIL->PASS transitions. I would
probably send notifications for them. They seem rather important.
What do you think?
[1] I've examined F14 repository, it contains about 16800 packages. About 20 packages
is 32bit only:
http://fpaste.org/BRwA/
and 4 packages is 64bit only:
http://fpaste.org/WVBO/
I'm willing to sacrifice that, for now. We have much higher test crash ratio anyway.