openQA: run 'all' nightly instead of 'current'?

[Adam Williamson]

On Tue, 2015-08-18 at 08:26 +0200, Jan Sedlak wrote:
> I'm all for it. BOS is idle most of the time anyway and if we look at 
> it
> from time to time, perhaps we will notice that our tests are broken 
> sooner
> than later.
> 
> But it raises a question (and I've talked about it with Kamil 
> before):
> Where are we gonna put results of tests that aren't in test matrices
> (if we are even planning to put them anywhere else in the first 
> place)?
> This isn't specific merely to nightly testing, but for other tests 
> also.

Well, for now 'just leave 'em in openQA' seems fine to me. It's *meant*
to act as a results store - otherwise they wouldn't bother making it
look nice in the web UI. It works perfectly fine, at least in recent
updates the jobs group nicely by 'BUILD' value - all it takes is for us
to have a look at it each day and look into failures. It would be
pretty easy to have wikitcms create a validation page for each branch
each night and stuff the results there, but I'm not convinced it's of
much *value* to do that (except we could then use the testcase_stats
stuff to handily plot status over time, but...meh, that's a pretty
small win).

In the long term, I think the plan we're kinda looking towards is to
try and centralize result storage in resultsdb, but we can look into
the details of that as we move along.

As part of this 'compose status' reporting stuff I'm working on ATM, I
can include openQA test result status, though of course if the nightly
tests are running in BOS we'll have to run the script on a machine
behind the firewall. Shouldn't be a problem. The report could, say,
list out all test fails and highlight any that passed the previous day
('new fails').

Heck, if it seems to fit right we can have the openQA dispatcher run
the 'compose status' checks and send out the emails. But it doesn't
really matter exactly how the bits plug together, point is it's all
stuff we can do relatively easy right now. I'm about to push some new
bits to fedfind that implement checking any given release to see if any
'expected important images' are missing, and diffing the images present
for any one release against any other.

I'll wait to hear from Josef on the 'all' idea, and if he doesn't mind,
I'll poke the cron job (after making sure 'all' still works right of
course :>). Thanks for the feedback!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net