On Fri, 2017-03-17 at 23:00 -0400, Dusty Mabe wrote:
> I've just verified that this is actually working: when openQA tests the
> 'two-week Atomic' composes and a failure occurs, the mail does get
> sent. However, there hasn't been a failure of the test since - AFAICS -
> 2016-10-01. So that's why no such mails have been sent out recently. I
> got Mike to check, and he actually did get a mail on 20167-10-01, when
> the test last failed.
woot for passing tests! Adam, do you know if there is any way to send
a weekly report that basically states how many tests ran and how many
passed? That would at least let us know that they were there and still
running and passing.
I imagine now that the tests results are in resultsdb the answer is
going to be something related to that.
The thing that sends the per-compose reports is just a script that I
wrote, called check-compose:
https://pagure.io/fedora-qa/check-compose
there's a fedmsg consumer which notices when the last test for a
compose is run, and runs check-compose for that compose with a
configuration that results in the mails being sent.
check-compose currently has no ability to do a weekly summary report or
anything like it, it can only generate reports for exactly one compose
at a time. It wouldn't be too difficult to write such a thing (either
as an extension to check-compose or as a separate project), but someone
would have to do it.
check-compose queries openQA directly, rather than resultsdb; I wrote
it before we were forwarding results to resultsdb. It can't be ported
to only use resultsdb as it would still have to query openQA directly
for some stuff that isn't forwarded to resultsdb and really can't be
(like the logs it uses to report on system resource usage and stuff).
At some point I'd like to adjust it to use rdb as well as direct openQA
queries so it can include info on results from other test systems, but
it's not at the top of my priority list.
You could write a simple 'weekly summary' kinda script purely from the
info in resultsdb, and that way you could consider the results from
openQA and autocloud (currently) / taskotron (future?) together. I do
believe that's the general idea for this kind of status reporting
indeed - if you want that kind of info, build something that uses
resultsdb, whether it's some kinda web dashboard or script or whatever
you like.
Yes! we would love a UEFI test and I think it would actually be good
to run the atomic-host-tests against these images assuming openQA is
the right tool for that job. I thought openQA was more for interactive
install testing, so please let me know where I'm wrong. roshi knows
openQA and atomic-host-tests so he might be able to comment here.
You can do just about anything with openQA, really; we *could* do all
the testing we're currently doing in other systems in it. But it
wouldn't necessarily be the *best* way to do every kind of testing. :)
Interactive install testing is the classic example of a thing openQA
can do quite well that few other things can do at all.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net