Call for users of darkserver

[Pierre-Yves Chibon]

Good Morning Everyone,

The week before DevConf, a number of the members of the Fedora Infrastructure
met in Brno to discuss states and plans for the infrastructure.
One of the question that raised was about darkserver.

This application is available at: https://darkserver.fedoraproject.org/
and is meant to:

enable developer tools to identify exact package builds from which process
images (e.g. core dumps) come. This can enable their analysis, debugging
profiling, by finding out where the rpm / elf / dwarf files may be found, so
they can download them. (This is even better than
abrt-action-install-debuginfo-to-abrt-cache because that apparently cannot query
files no longer indexed by repodata.)

Source: https://fedoraproject.org/wiki/Darkserver


However, it seems this application has not been working for a long time now and
not many people asked about it.

So, is anyone using this service?

If there is little interest for this project, we will likely decommission it in
the coming weeks (say end of March).


Thanks for your attention and your feedback,

Pierre
For the Fedora Infrastructure team


_______________________________________________
devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
To unsubscribe send an email to devel-announce-leave(a)lists.fedoraproject.org

[Igor Gnatenko]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Wed, 2018-02-14 at 11:02 +0100, Pierre-Yves Chibon wrote:
>  Good Morning Everyone,
>  
>  The week before DevConf, a number of the members of the Fedora Infrastructure
>  met in Brno to discuss states and plans for the infrastructure.
>  One of the question that raised was about darkserver.
>  
>  This application is available at: https://darkserver.fedoraproject.org/
>  and is meant to:
>  
>  enable developer tools to identify exact package builds from which process
>  images (e.g. core dumps) come. This can enable their analysis, debugging
>  profiling, by finding out where the rpm / elf / dwarf files may be found, so
>  they can download them. (This is even better than
>  abrt-action-install-debuginfo-to-abrt-cache because that apparently cannot
>  query
>  files no longer indexed by repodata.)
>  
>  Source: https://fedoraproject.org/wiki/Darkserver
>  
>  
>  However, it seems this application has not been working for a long time now
>  and
>  not many people asked about it.
>  
>  So, is anyone using this service? 
I heard about this service, but never used it... But since debuginfo is now
parallel-installable, it would be nice to have a place where all debuginfo
(from all releases / builds) is available so people could install it even it
disappeared from repositories.

I don't know if existing service is doing exactly what I want..
- -- 
- -Igor Gnatenko
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCAAdFiEEhLFO09aHZVqO+CM6aVcUvRu8X0wFAlqEIHAACgkQaVcUvRu8
> X0wn1g/8C7z4bXpQCEAMaPRqw3IPlPd+pZqH3/Ilq05mOagrLfR3h6mM6HUqrgSu
> RshooEd17J1H9gtrlqIJuUrp9Hv8C7yf77AOTMvcc2Kgo9ZC5o2VL/AyMUyot7vw
> Hla9EUZtvGGHQAulyjg1XPRXB7K+p0C9LFs3x87bchpCnJeAwoU2QpyOpWP2tM8z
> Zipq4+4empZ0mVlPuXLZN1w4BNAfm/4YbvNQxQW6PYDKKfjZQemaDkKfiukTo8cH
> an65qQPOCxA1ohxF8m/zxnBOtubYxEvWEa9HGWVFqavihZvWBUuhvaed58RmHwY6
> 8V0/sUt9euN5xBQHiWdLm+Ovx8icMw0qHbr+YcrbS4PmqxB5i9RyHWX05lhVafy0
> c7dgT9dX76/xhzperIreMoBmLzkM5OWWxso6PRlIFWzeghf1nRTESVI2NjPiyRe4
> j0lNVGPiCsTj6LuoUQxyftYgNHnbwM8RDqF0tP7oo8QCxzu/21DgdUNchaXdGUSs
> CY6pSCtKjuLihkjnqC4OVr1b+ESFQytO1Af/HO81rAnxXexFMihIibzAB6evIbbX
> jCs4aXXO5XRaBAyIAyiWoE9/NQy24cs5gePsnLEQOZ/2TwifWbTLXvBqWM451jF2
> CqDG5S21dY9i93/0BshfSEX3CZLcuI+SBx/ZcAB6U6W2yUCUNmU=
> =VZxp
> -----END PGP SIGNATURE-----

[Pierre-Yves Chibon]

On Wed, Feb 14, 2018 at 12:41:36PM +0100, Igor Gnatenko wrote:
>  -----BEGIN PGP SIGNED MESSAGE-----
>  Hash: SHA256
>  
>  On Wed, 2018-02-14 at 11:02 +0100, Pierre-Yves Chibon wrote:
>  > Good Morning Everyone,
>  > 
>  > The week before DevConf, a number of the members of the Fedora Infrastructure
>  > met in Brno to discuss states and plans for the infrastructure.
>  > One of the question that raised was about darkserver.
>  > 
>  > This application is available at: https://darkserver.fedoraproject.org/
>  > and is meant to:
>  > 
>  > enable developer tools to identify exact package builds from which process
>  > images (e.g. core dumps) come. This can enable their analysis, debugging
>  > profiling, by finding out where the rpm / elf / dwarf files may be found, so
>  > they can download them. (This is even better than
>  > abrt-action-install-debuginfo-to-abrt-cache because that apparently cannot
>  > query
>  > files no longer indexed by repodata.)
>  > 
>  > Source: https://fedoraproject.org/wiki/Darkserver
>  > 
>  > 
>  > However, it seems this application has not been working for a long time now
>  > and
>  > not many people asked about it.
>  > 
>  > So, is anyone using this service?
>  
>  I heard about this service, but never used it... But since debuginfo is now
>  parallel-installable, it would be nice to have a place where all debuginfo
>  (from all releases / builds) is available so people could install it even it
>  disappeared from repositories. 
I just confirmed it with Kushal (the developer of darkserver), darkserver
provides you an URL but that's basically sending you to koji (which keeps a good
chunk of the RPMs shipped to our user, but not all).
darkserver itself doesn't store anything.


Pierre

[Jan Kratochvil]

On Wed, 14 Feb 2018 15:55:20 +0100, Pierre-Yves Chibon wrote:
>  darkserver itself doesn't store anything. 
Darkserver was at least in the past using debuginfo storage from ABRT server
which is much more rich than what Koji server keeps.


Jan

[Jan Kratochvil]

On Wed, 14 Feb 2018 11:02:52 +0100, Pierre-Yves Chibon wrote:
>  If there is little interest for this project, we will likely
> decommission it in
>  the coming weeks (say end of March). 
The darkserver project ws initiated AFAIK by me as there is always a problem
how to reproduce an ABRT bugreport.

The darkserver usability problems were related to each other,
a chicken-and-egg problem:
 * It never really started working.  When I tried to use it once upon few
   months when I found time to process some ABRT bugreports which were not
   obvious enough darkserver failed and after contacting Kushal Das
   (darkserver author) he found some new software or data bug why it did not
   work that time.
 * There was never a tool making it convenient enough to reconstruct the
   tree of files based on their build-ids.
 * There was never enough users (was there any besides me?) that started using
   darkserver, because of the two problems above.

So I believe darkserver would be great but not in its current state of
functionality.

Also I believe ABRT project already contains most of the infrastructure and
code required, I believe darkserver could be rather just few lines of code
added to the ABRT project - that is to interactively run the crashed program
with all matching versions of libraries - not just getting the non-interative
core file backtrace (which ABRT submits to Bugzilla).


Jan

[Pierre-Yves Chibon]

On Wed, Feb 14, 2018 at 06:13:53PM +0100, Jan Kratochvil wrote:
>  On Wed, 14 Feb 2018 11:02:52 +0100, Pierre-Yves Chibon wrote:
>  > If there is little interest for this project, we will likely decommission it in
>  > the coming weeks (say end of March).
>  
>  The darkserver project ws initiated AFAIK by me as there is always a problem
>  how to reproduce an ABRT bugreport.
>  
>  The darkserver usability problems were related to each other,
>  a chicken-and-egg problem:
>   * It never really started working.  When I tried to use it once upon few
>     months when I found time to process some ABRT bugreports which were not
>     obvious enough darkserver failed and after contacting Kushal Das
>     (darkserver author) he found some new software or data bug why it did not
>     work that time.
>   * There was never a tool making it convenient enough to reconstruct the
>     tree of files based on their build-ids.
>   * There was never enough users (was there any besides me?) that started using
>     darkserver, because of the two problems above.
>   
>  So I believe darkserver would be great but not in its current state of
>  functionality.
>  
>  Also I believe ABRT project already contains most of the infrastructure and
>  code required, I believe darkserver could be rather just few lines of code
>  added to the ABRT project - that is to interactively run the crashed program
>  with all matching versions of libraries - not just getting the non-interative
>  core file backtrace (which ABRT submits to Bugzilla). 
So what do you advice as course of action here, should we fix darkserver or move
its functionality to ABRT?
The later is tempting to me if it's just a few lines of code added on an app we
already maintain.


Pierre

PS: Kushal no longer work at RH so this email won't work (I've contacted him
before starting this conversation though)

[Jan Kratochvil]

On Fri, 16 Feb 2018 11:02:25 +0100, Pierre-Yves Chibon wrote:
>  So what do you advice as course of action here, should we fix
> darkserver or move
>  its functionality to ABRT? 
"move its functionality to ABRT"


>  The later is tempting to me if it's just a few lines of code
> added on an app we
>  already maintain. 
One problem of ABRT is that it cannot (or at least not easily) provide the
core file to Bug Assignee.  Users always fail to figure out how when I ask
them for that.  I do not know if it is just not user friendly enough or if the
core files gets deleted from disk too quickly.

And developers (that is me) could not use ABRT so far as it OOM-killed machine
on a first segfaulted program:
	-fsanitize=address locks up abrt-hook-ccpp
	https://bugzilla.redhat.com/show_bug.cgi?id=1164548
It is EOLed again but maybe it works now on F-27 after my quick local
re-check, I will check it later once I have access to an F-27 VM (travelling
now).

Then the ABRT bugreport does not contain build-ids of the shared libraries
involved, only build-ids of frames from a backtrace ("core_backtrace") which
is not sufficient to reproduce the same memory layout.

It seems to me interactive crash core file analysis is just not a goal of ABRT
(which is what was the goal of darkserver).

And currently after my reassignment in Red Hat I currently do not have any
ABRT bugreports to investigate so I am currently not involved in this
darkserver/ABRT build-id crash reproducibility (I may be again later).


Regards,
Jan

[Pierre-Yves Chibon]

On Fri, Feb 16, 2018 at 03:04:11PM +0100, Jan Kratochvil wrote:
>  On Fri, 16 Feb 2018 11:02:25 +0100, Pierre-Yves Chibon wrote:
>  > So what do you advice as course of action here, should we fix darkserver or move
>  > its functionality to ABRT?
>  
>  "move its functionality to ABRT"
>  
>  
>  > The later is tempting to me if it's just a few lines of code added on an app we
>  > already maintain.
>  
>  One problem of ABRT is that it cannot (or at least not easily) provide the
>  core file to Bug Assignee.  Users always fail to figure out how when I ask
>  them for that.  I do not know if it is just not user friendly enough or if the
>  core files gets deleted from disk too quickly.
>  
>  And developers (that is me) could not use ABRT so far as it OOM-killed machine
>  on a first segfaulted program:
>  	-fsanitize=address locks up abrt-hook-ccpp
>  	https://bugzilla.redhat.com/show_bug.cgi?id=1164548
>  It is EOLed again but maybe it works now on F-27 after my quick local
>  re-check, I will check it later once I have access to an F-27 VM (travelling
>  now).
>  
>  Then the ABRT bugreport does not contain build-ids of the shared libraries
>  involved, only build-ids of frames from a backtrace ("core_backtrace") which
>  is not sufficient to reproduce the same memory layout.
>  
>  It seems to me interactive crash core file analysis is just not a goal of ABRT
>  (which is what was the goal of darkserver). 
Did it ever meet this goal?
I never quite understood how it works, being far from this field, so I'm happy
to be enlighten on the topic.

>  And currently after my reassignment in Red Hat I currently do not
> have any
>  ABRT bugreports to investigate so I am currently not involved in this
>  darkserver/ABRT build-id crash reproducibility (I may be again later). 
Would you know if anyone in your old team would be interested by this?

I'm trying to assess if there is actually an interest for it.
You said you were interested by it but aren't working on this topic anymore, I
don't think anybody else than you noticed it not working over the course of last
year.
So I wonder if we shouldn't just call it a day, provide a dump of the DB
somewhere for people interested and decommission the app.
Fixing it on the other hand would require someone learning how the code works,
probably trying to poke at Kushal for inputs (but I don't want to assume he'll
fix it as I'm assuming he has other priorities) and finish deploying that new
version. It may not be quick.


Thoughts?

Pierre

[Jan Kratochvil]

On Thu, 01 Mar 2018 19:30:28 +0100, Pierre-Yves Chibon wrote:
>  On Fri, Feb 16, 2018 at 03:04:11PM +0100, Jan Kratochvil wrote:
>  > On Fri, 16 Feb 2018 11:02:25 +0100, Pierre-Yves Chibon wrote:
>  > It seems to me interactive crash core file analysis is just not a goal of ABRT
>  > (which is what was the goal of darkserver).
>  
>  Did it ever meet this goal? 
I do not know.


>  I never quite understood how it works, being far from this field, so
> I'm happy
>  to be enlighten on the topic. 
Darkserver itself will just give you a rpm build for given build-id.  So with
a core file (containing the build-ids) and a small shell script you can
reconstruct system (chroot) where you can load the core file for more thorough
crash analysis.


>  > And currently after my reassignment in Red Hat I currently do
> not have any
>  > ABRT bugreports to investigate so I am currently not involved in this
>  > darkserver/ABRT build-id crash reproducibility (I may be again later).
>  
>  Would you know if anyone in your old team would be interested by this? 
It is not specific to GDB team.  I am curious how all the Fedora package
developers deal with ABRT-filed Bugs where the backtrace contained therein is
not sufficient to understand the problem but one may be able to understand it
accessing more data with GDB using the core file.

The backtrace contained in the ABRT bugreport can rarely really debug the
problem.  Sure even the core file is not always sufficient but it gives more
chance to debug the crash (which requires an darkserver-like functionality).

But then I usually cannot get even just the core file from ABRT users so this
is why I ask how other package (*) maintainers deal with the ABRT bugreports.

(*) It may be confusing I am GDB package maintainer when GDB is used for the
crash analysis.  But here I talk about GDB as any other program (like bash,
firefox etc.) as even GDB crashes like any other program (bash, firefox, etc.).


>  I'm trying to assess if there is actually an interest for it.
>  You said you were interested by it but aren't working on this topic anymore, I
>  don't think anybody else than you noticed it not working over the course of last
>  year. 
Yes, because I think nobody knows one could troubleshoot ABRT-bugreported
problems better than with the information filed by ABRT into Bugzilla.


>  So I wonder if we shouldn't just call it a day, 
It won't change the current state of bugs troubleshooting.
But Fedora should not forget the Bugs troubleshooting could be improved.


Jan

[Jason Callaway]

I was unaware of it, but I think it looks pretty great!

OOC, is there an alternative service that one could use if this is
decommissioned? Or would we have to dig through Koji and Bodhi logs?

I'll bet the CHAOSS folks [0] would like it, if that helps. Dunno if
they're aware of it, though. I'll introduce them to it if there's no
objection.

[0] - https://chaoss.community/


On Wed, Feb 14, 2018 at 5:02 AM, Pierre-Yves Chibon <pingou(a)pingoured.fr&gt;
wrote:

>  Good Morning Everyone,
>
>  The week before DevConf, a number of the members of the Fedora
>  Infrastructure
>  met in Brno to discuss states and plans for the infrastructure.
>  One of the question that raised was about darkserver.
>
>  This application is available at: https://darkserver.fedoraproject.org/
>  and is meant to:
>
>  enable developer tools to identify exact package builds from which process
>  images (e.g. core dumps) come. This can enable their analysis, debugging
>  profiling, by finding out where the rpm / elf / dwarf files may be found,
>  so
>  they can download them. (This is even better than
>  abrt-action-install-debuginfo-to-abrt-cache because that apparently
>  cannot query
>  files no longer indexed by repodata.)
>
>  Source: https://fedoraproject.org/wiki/Darkserver
>
>
>  However, it seems this application has not been working for a long time
>  now and
>  not many people asked about it.
>
>  So, is anyone using this service?
>
>  If there is little interest for this project, we will likely decommission
>  it in
>  the coming weeks (say end of March).
>
>
>  Thanks for your attention and your feedback,
>
>  Pierre
>  For the Fedora Infrastructure team
>
>
>  _______________________________________________
>  devel-announce mailing list -- devel-announce(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-announce-leave@lists.
>  fedoraproject.org
>
>  

-- 
Jason Callaway | jcallaway(a)redhat.com | [phone] | GPG Key 0x81ED4A9A

[Pierre-Yves Chibon]

On Wed, Feb 14, 2018 at 03:17:57PM -0500, Jason Callaway wrote:
>     I was unaware of it, but I think it looks pretty great!
>     OOC, is there an alternative service that one could use if this is
>     decommissioned? Or would we have to dig through Koji and Bodhi logs?
>     I'll bet the CHAOSS folks [0] would like it, if that helps. Dunno if
>     they're aware of it, though. I'll introduce them to it if there's no
>     objection.
>     [0] -Â https://chaoss.community/ 
If that was me, I'd wait for the faith of darkserver to be a little clearer
before introducing that tool to other communities :)

Pierre