ABRT, Faf and current state of bug reporting

[Richard Marko]

Hi all,

I'll try to explain how crash reporting currently works in Fedora.

Typical reporting process looks like this:
 - crash is reported to Faf server which responds with 'known' or
'unknown' reply;
 - in case it responds with 'known' and the bug was already reported to
both the server and bugzilla, the reporting is stopped and only report
counts on the server are updated;
 - if the crash is unknown, the reporting either continues or stops
depending on the configuration (for Gnome, only automated reporting to
faf is enabled);
 - if enabled, the rest of the process continues with local or remote
retracing, reporting to bugzilla and attaching bugzilla ticket to faf
report.


This allows us to get accurate statistics of crashing applications while
not forcing every user to report to bugzilla. This is a trade-off
between getting accurate statistics and quality of the reports as
automated reports are anonymous which is also the reason why they can't
contain full backtrace with data.

Then there are reports with no bugzilla attached as they were reported
automatically or no one finished the bugzilla reporting. These reports
get bugzilla ticket attached after there's person who finishes the
reporting or the ticket is created by the server.

The intermediate part of the stack, faf server, is still pretty new so
please bear with us as we are dealing with lots of data. The goal of the
server is to provide accurate statistics of crashing applications and
clustering of the incoming reports.

Hope this helps to clarify the situation a bit. Feedback is always
welcome, especially if you are receiving bug reports you are not happy with.

Please use [1] for reporting issues if our mailing list [2] is not an
option for you.

[1] https://github.com/abrt/faf/issues/new
[2] crash-catcher(a)lists.fedorahosted.org


Best regards,

-- 
Richard Marko

[Kevin Fenzi]

On Tue, 23 Apr 2013 15:27:50 +0200
Richard Marko <rmarko(a)redhat.com&gt; wrote:

>  Hi all,
>  
>  I'll try to explain how crash reporting currently works in Fedora.
>  
>  Typical reporting process looks like this:
>   - crash is reported to Faf server which responds with 'known' or
>  'unknown' reply;
>   - in case it responds with 'known' and the bug was already reported
>  to both the server and bugzilla, the reporting is stopped and only
>  report counts on the server are updated; 
Does the user get a link to any bugs associated with the crash? 
Or this happens without user interaction? 

>   - if the crash is unknown, the reporting either continues or stops
>  depending on the configuration (for Gnome, only automated reporting to
>  faf is enabled);
>   - if enabled, the rest of the process continues with local or remote
>  retracing, reporting to bugzilla and attaching bugzilla ticket to faf
>  report.
>  
>  This allows us to get accurate statistics of crashing applications
>  while not forcing every user to report to bugzilla. This is a
>  trade-off between getting accurate statistics and quality of the
>  reports as automated reports are anonymous which is also the reason
>  why they can't contain full backtrace with data. 
Well, it's nice to know when things crash, sure... but without more
information it's very difficult to figure out how to fix that crash. 

>  Then there are reports with no bugzilla attached as they were
> reported
>  automatically or no one finished the bugzilla reporting. These reports
>  get bugzilla ticket attached after there's person who finishes the
>  reporting or the ticket is created by the server. 
Perhaps we could make it always require a person to be willing to
file? Hopefully someone who can explain what happened and what they
have installed, etc? ie, "hey, look, 50 people saw this crash, but it
has no bug yet, well, I know exactly what I do to cause it, let me file
the bug and help all 50 of the other folks seeing it out" 

kevin

[Przemek Klosowski]

On 04/23/2013 01:46 PM, Kevin Fenzi wrote:
>  On Tue, 23 Apr 2013 15:27:50 +0200
>  Richard Marko <rmarko(a)redhat.com&gt; wrote: 
> > This allows us to get accurate statistics of crashing
> applications
> > while not forcing every user to report to bugzilla. This is a
> > trade-off between getting accurate statistics and quality of the
> > reports as automated reports are anonymous which is also the reason
> > why they can't contain full backtrace with data.
>
>  Well, it's nice to know when things crash, sure... but without more
>  information it's very difficult to figure out how to fix that crash. 
Well, here's an example why I think such information is highly useful: 
currently on my system every program using 3D (openGL/Mesa) crashes; 
that means everything: Blender, Avogadro, pymol, FreeCAD, openSCAD, and 
even everything that uses FLTK library because it renders both 2D and 3D 
through OpenGL:

Program received signal SIGSEGV, Segmentation fault.
intel_miptree_unmap (intel=0x18d1960, mt=0x0, level=0, slice=0) at 
intel_mipmap_tree.c:1752
1752	   if (mt->num_samples <= 1)

It's an Intel driver bug that seems pretty severe to me but I don't have 
the graphics chops to fix it:

https://bugzilla.redhat.com/show_bug.cgi?id=922224

I think it would help things if there were reliable statistics on its 
footprint. Is it just my Q45 or all Intel chipsets? is it just my weird 
configuration (two screens? update rather than fresh install? some 
configuration I did and forgot about?).

[Kevin Fenzi]

On Tue, 23 Apr 2013 15:45:34 -0400
Przemek Klosowski <przemek.klosowski(a)nist.gov&gt; wrote:

>  Well, here's an example why I think such information is highly
>  useful: currently on my system every program using 3D (openGL/Mesa)
>  crashes; that means everything: Blender, Avogadro, pymol, FreeCAD,
>  openSCAD, and even everything that uses FLTK library because it
>  renders both 2D and 3D through OpenGL:
>  
>  Program received signal SIGSEGV, Segmentation fault.
>  intel_miptree_unmap (intel=0x18d1960, mt=0x0, level=0, slice=0) at 
>  intel_mipmap_tree.c:1752
>  1752	   if (mt->num_samples <= 1)
>  
>  It's an Intel driver bug that seems pretty severe to me but I don't
>  have the graphics chops to fix it:
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=922224 
Which seems to be also:
https://bugzilla.redhat.com/show_bug.cgi?id=946960
and
https://retrace.fedoraproject.org/faf/problems/786379/
 
>  I think it would help things if there were reliable statistics on its
>
>  footprint. Is it just my Q45 or all Intel chipsets? is it just my
>  weird configuration (two screens? update rather than fresh install?
>  some configuration I did and forgot about?). 
That would be nice, but alas, we don't know and faf doesn't tell us.

We know that there have been 32 instances of this crash recorded by
faf. That doesn't tell us anything at all about the hardware or
installed packages or screens or other info about those instances. All
we know is that it happened 32 times (I suppose we don't even know it's
32 different machines even, could one person report a crash 32times?)

I suppose we can see 3 different programs that crashed in this place. I
suppose that might of help. 

kevin

[Richard Marko]

On 04/23/2013 09:53 PM, Kevin Fenzi wrote:
>  On Tue, 23 Apr 2013 15:45:34 -0400
>  Przemek Klosowski <przemek.klosowski(a)nist.gov&gt; wrote:
>
> > Well, here's an example why I think such information is highly
> > useful: currently on my system every program using 3D (openGL/Mesa)
> > crashes; that means everything: Blender, Avogadro, pymol, FreeCAD,
> > openSCAD, and even everything that uses FLTK library because it
> > renders both 2D and 3D through OpenGL:
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > intel_miptree_unmap (intel=0x18d1960, mt=0x0, level=0, slice=0) at 
> > intel_mipmap_tree.c:1752
> > 1752	   if (mt->num_samples <= 1)
> >
> > It's an Intel driver bug that seems pretty severe to me but I don't
> > have the graphics chops to fix it:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=922224
>  Which seems to be also:
>  https://bugzilla.redhat.com/show_bug.cgi?id=946960
>  and
>  https://retrace.fedoraproject.org/faf/problems/786379/
>   
> > I think it would help things if there were reliable statistics on its 
> > footprint. Is it just my Q45 or all Intel chipsets? is it just my
> > weird configuration (two screens? update rather than fresh install?
> > some configuration I did and forgot about?).
>  That would be nice, but alas, we don't know and faf doesn't tell us. 
It's not easy to implement this because of privacy issues. There was
already such request though so we'll consider implementing this.

>  We know that there have been 32 instances of this crash recorded by
>  faf. That doesn't tell us anything at all about the hardware or
>  installed packages or screens or other info about those instances. All
>  we know is that it happened 32 times (I suppose we don't even know it's
>  32 different machines even, could one person report a crash 32times?) 
Correct. List of installed packages related to the crash will be available.

-- 
Richard Marko

[Richard Marko]

On 04/23/2013 07:46 PM, Kevin Fenzi wrote:
>  On Tue, 23 Apr 2013 15:27:50 +0200
>  Richard Marko <rmarko(a)redhat.com&gt; wrote:
>
> > Hi all,
> >
> > I'll try to explain how crash reporting currently works in Fedora.
> >
> > Typical reporting process looks like this:
> >  - crash is reported to Faf server which responds with 'known' or
> > 'unknown' reply;
> >  - in case it responds with 'known' and the bug was already reported
> > to both the server and bugzilla, the reporting is stopped and only
> > report counts on the server are updated;
>  Does the user get a link to any bugs associated with the crash? 
>  Or this happens without user interaction?  
He does. Both the link to faf and link to bugzilla (if ticket exists).

> >  - if the crash is unknown, the reporting either continues or stops
> > depending on the configuration (for Gnome, only automated reporting to
> > faf is enabled);
> >  - if enabled, the rest of the process continues with local or remote
> > retracing, reporting to bugzilla and attaching bugzilla ticket to faf
> > report.
> >
> > This allows us to get accurate statistics of crashing applications
> > while not forcing every user to report to bugzilla. This is a
> > trade-off between getting accurate statistics and quality of the
> > reports as automated reports are anonymous which is also the reason
> > why they can't contain full backtrace with data.
>  Well, it's nice to know when things crash, sure... but without more
>  information it's very difficult to figure out how to fix that crash. 
>
> > Then there are reports with no bugzilla attached as they were reported
> > automatically or no one finished the bugzilla reporting. These reports
> > get bugzilla ticket attached after there's person who finishes the
> > reporting or the ticket is created by the server.
>  Perhaps we could make it always require a person to be willing to
>  file? Hopefully someone who can explain what happened and what they
>  have installed, etc? ie, "hey, look, 50 people saw this crash, but it
>  has no bug yet, well, I know exactly what I do to cause it, let me file
>  the bug and help all 50 of the other folks seeing it out"  
Yes, this sounds good to me. I'll create tickets for this.

-- 
Richard Marko

[Stephan Bergmann]

On 04/23/2013 03:27 PM, Richard Marko wrote:
>  This allows us to get accurate statistics of crashing applications
> while
>  not forcing every user to report to bugzilla. This is a trade-off
>  between getting accurate statistics and quality of the reports as
>  automated reports are anonymous which is also the reason why they can't
>  contain full backtrace with data. 
To clarify, is it so that from now on we'll always get (less detailed) 
"[faf]" bugzilla issues instead of the old (more detailed) "[abrt]" 
ones, or does that depend on how the user experiencing a crash interacts 
with ABRT?

Stephan

[Richard Marko]

On 04/24/2013 09:43 AM, Stephan Bergmann wrote:
>  On 04/23/2013 03:27 PM, Richard Marko wrote:
> > This allows us to get accurate statistics of crashing applications while
> > not forcing every user to report to bugzilla. This is a trade-off
> > between getting accurate statistics and quality of the reports as
> > automated reports are anonymous which is also the reason why they can't
> > contain full backtrace with data.
>
>  To clarify, is it so that from now on we'll always get (less detailed)
>  "[faf]" bugzilla issues instead of the old (more detailed) "[abrt]"
>  ones, or does that depend on how the user experiencing a crash
>  interacts with ABRT? 
It depends on both the configuration and how the user interacts with
ABRT. If there's a user who reports to bugzilla you'll get detailed
report. If not, you'll only get report from faf when number of reports
reaches certain threshold. We'll increase the threshold so there's
higher chance of someone creating more detailed report.

Even if you close these bugs with INSUFFICIENT DATA the server can point
people there and they might provide required data.

-- 
Richard Marko