openQA crash reporting

[Adam Williamson]

hey folks! I mentioned this to jskladan on IRC, but just for the 
permanent record, I'm working on optional crash report submission for 
openQA.

at first I had the workers clicking through the graphical report 
submission process, but that has several problems:

a) needles and keypresses and blah
b) workers don't actually know the job ID or URL, so can't include it 
in the bug report
c) requires inventing some kind of way to get a BZ username and 
password into the workers without it being logged (doable, but just 
unnecessary work, when libreport-plugin-bugzilla already has this set 
up)

so instead I'm doing it in report_job_results.py in 
openqa_fedora_tools. It actually builds off D310, Jan's improvement to 
upload the contents of /var/tmp after a crash.

Given a job_id, we check if there's a var_tmp.tar.gz for that job, and 
if there is, we look for libreport 'problem directories' inside it. If 
we find any, we extract them from the tarball and run 'reporter-
bugzilla -d (directory)' on them.

That's really it in a nutshell, the rest is just error checks and glue 
and frills. There's an attempt to include the web UI job URL in the 
bug report for new crash reports (though so far I've been testing with 
a problem directory that shows up as a dupe of an existing report, so I
haven't tested this yet), and we capture the IDs of the bugs reported.

I also refactored the reporting functions a bit to avoid code 
duplication between calling report_job_results directly and using it 
from openqa_trigger, and made it possible to specify the openQA URL in 
a config file (so you can do result reporting from a system other than 
the openQA host itself - like, fr'instance, a Fedora system with 
libreport-plugin-bugzilla installed...)

To test it out you need a job in some openQA instance which has a 
var_tmp.tar.gz with a crash directory inside it: I've been testing 
with https://openqa.happyassassin.net/tests/2736 . You also need to 
put a valid BZ username and password in 
/etc/libreport/plugins/bugzilla.conf and, unless you're running on the 
openQA host itself (there *are* libreport packages for openSUSE in 
some OBS repository, but I haven't tried them), you'll want to create 
/etc/openqa_fedora.conf with this content:

[site]
url = https://openqa.happyassassin.net

(or whatever URL is appropriate).

Then you can do this:

python report_job_results.py --crashes 2736

(or whatever the job ID is).

This probably still needs a bit more testing and polish before I 
submit it as a differential, but I wanted to give people a heads-up 
that I was working on it and explain the general design. My current 
patch (against 'develop' branch, to which I've merged the 'live' work 
now) is attached.

In case you're wondering what happens with duplicate reports: I tested 
and it seems like 'not a lot'. When calling reporter-bugzilla in this 
way, if the crash has already been reported, it will only generate BZ 
activity if the BZ account in question isn't already on the CC list: 
it will add it. But if the BZ account is already on the CC list, it 
doesn't change the bug at all, it doesn't add the extra comment saying 
'another user encountered this issue'. I checked libreport and it 
actually only does that when some comment text has been provided, and 
we aren't providing one, so it gets skipped.

If we're still worried about noise on dupes it *is* possible to test 
if a bug is a dupe by checking the output of:

reporter-bugzilla -h $(cat duphash)

and completely skip the report submission step if it is, and I 
actually had that written, but took it out as it seemed unnecessary. 
Easy enough to put it back if we want to, though.

In the current version of the patch things are set up so that 
openqa_trigger current or openqa_trigger all or openqa_trigger compose 
--submit-results runs will try and report all crashes, but it's 
absolutely trivial to change that if we only want to report crashes 
via a separate invocation.

Comments welcome!

[Adam Williamson]

On Thu, 2015-03-19 at 01:15 -0700, Adam Williamson wrote:
> hey folks! I mentioned this to jskladan on IRC, but just for the  
> permanent record, I'm working on optional crash report submission 
> for  openQA.

To follow up on a question Jan had on IRC:

<garretraziel> 01:38:03> adamw: do you know whether reporter-bugzilla 
command is in OpenSUSE or not?

The answer is not officially, but jfilak maintains an OBS repo with 
builds:

https://build.opensuse.org/project/show/home:jfilak

Hasn't been updated for a year, though, don't know if it works on 13.2.

The other part of the answer is that this is why my patch allows you 
to specify the openQA URL: so you can run crash submission from a 
Fedora system. :) (in fact we should be able to make all of op
enqa_trigger usable from any system, if we make at least the 'client' 
command available in Fedora or just write a bit of Python to do API 
requests, the openQA API looks very simple).