valgrind on Fedora

[Gordon Messmer]

I'm working on reducing memory use in packagekitd, and so far progress 
has been very good.  I've opened 4 memory-related PRs against PackageKit 
and libdnf this week, and locally I've reduced memory use at idle by 
almost 90% (I can reliably cause stock packagekitd to use ~ 700MB of RAM 
in a couple of minutes, but a patched version will free most of its 
memory and shrink to around 75MB.)

However, I've hit a possibly minor road block.  I haven't fixed all of 
the leaks originally reported; valgrind still reports some leaks in my 
patched builds, but most of the call stack is unresolved, so I can't 
locate the problems in order to address any remaining issues.

I'm running 'valgrind --leak-check=full --num-callers=25 
/usr/libexec/packagekitd', running for a while, and then exiting with 
Ctrl-C.  When I exit, I get a list of leaks, most of which contain 
unresolved symbols.  I see the same thing when I run Fedora's builds (in 
which case, debug info appears to be provided by debuginfod-client), or 
when I run my own builds (in which case I'm installing the debuginfo and 
debugsource packages produced by the local build.)

Does anyone have any hints for improving the information I get from 
valgrind?

[Florian Festi]

On 1/16/23 07:10, Gordon Messmer wrote:
>  Does anyone have any hints for improving the information I get from
>  valgrind? 
Have you installed the debuginfo packages for the packages involved?
See man debuginfo-install

Florian

[Tom Hughes]

On 16/01/2023 08:12, Florian Festi wrote:
>  On 1/16/23 07:10, Gordon Messmer wrote:
> > Does anyone have any hints for improving the information I get from
> > valgrind?
>  Have you installed the debuginfo packages for the packages involved?
>  See man debuginfo-install 
Making sure debuginfod fetching works should also do it as valgrind
has support for that.

If that doesn't work then some examples would help, at least if you're
getting a partial trace, so that we can get some idea of what component
it is not able to unwind.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Gordon Messmer]

On 2023-01-16 00:31, Tom Hughes via devel wrote:
>  On 16/01/2023 08:12, Florian Festi wrote:
> > Have you installed the debuginfo packages for the packages involved?
> > See man debuginfo-install
>
>  Making sure debuginfod fetching works should also do it as valgrind
>  has support for that. 

I've tried this both with debuginfo packages installed (using a long 
list of debuginfo packages helpfully provided by gdb when debuginfod 
support is declined), and with no debuginfo packages installed and 
relying entirely on debuginfod to fetch debug info. I'm sure that 
fetching is working at least in part, because I can remove 
~/.cache/debuginfod_client completely, and it will be repopulated by a 
subsequent run of valgrind.


>  If that doesn't work then some examples would help, at least if
> you're
>  getting a partial trace, so that we can get some idea of what component
>  it is not able to unwind.  

==29692== 30 bytes in 2 blocks are definitely lost in loss record 917 of 
2,602
==29692==    at 0x484386F: malloc (vg_replace_malloc.c:393)
==29692==    by 0x14806539: ???
==29692==    by 0x14BA7D87: ???
==29692==    by 0x14BA7DFC: ???
==29692==    by 0x14B86D88: ???
==29692==    by 0x146ED193: ???
==29692==    by 0x1475E205: ???
==29692==    by 0x1475E71A: ???
==29692==    by 0x1475ED49: ???
==29692==    by 0x146EF09F: ???
==29692==    by 0x4A5D0E7: g_type_create_instance (gtype.c:1931)
==29692==    by 0x4A42C1F: g_object_new_internal (gobject.c:2228)
==29692==    by 0x4A44247: g_object_new_with_properties (gobject.c:2391)
==29692==    by 0x4A44FF0: g_object_new (gobject.c:2037)
==29692==    by 0x146F5395: ???
==29692==    by 0x145D820C: ???
==29692==    by 0x145E06E3: ???
==29692==    by 0x1276D0: pk_backend_load (pk-backend.c:569)
==29692==    by 0x135D1A: pk_engine_load_backend (pk-engine.c:967)
==29692==    by 0x119468: main (pk-main.c:219)

==29692== 158,863 (8,192 direct, 150,671 indirect) bytes in 1 blocks are 
definitely lost in loss record 2,602 of 2,602
==29692==    at 0x48486AF: realloc (vg_replace_malloc.c:1451)
==29692==    by 0x14805D7B: ???
==29692==    by 0x148062D6: ???
==29692==    by 0x1480DA3F: ???
==29692==    by 0x1480FCAB: ???
==29692==    by 0x14B86DF5: ???
==29692==    by 0x146ED193: ???
==29692==    by 0x1475E205: ???
==29692==    by 0x1475E71A: ???
==29692==    by 0x1475ED49: ???
==29692==    by 0x146EF09F: ???
==29692==    by 0x4A5D0E7: g_type_create_instance (gtype.c:1931)
==29692==    by 0x4A42C1F: g_object_new_internal (gobject.c:2228)
==29692==    by 0x4A44247: g_object_new_with_properties (gobject.c:2391)
==29692==    by 0x4A44FF0: g_object_new (gobject.c:2037)
==29692==    by 0x146F5395: ???
==29692==    by 0x145D820C: ???
==29692==    by 0x145E06E3: ???
==29692==    by 0x1276D0: pk_backend_load (pk-backend.c:569)
==29692==    by 0x135D1A: pk_engine_load_backend (pk-engine.c:967)
==29692==    by 0x119468: main (pk-main.c:219)

[Tom Hughes]

On 16/01/2023 08:52, Gordon Messmer wrote:
>  On 2023-01-16 00:31, Tom Hughes via devel wrote:
>
> > If that doesn't work then some examples would help, at least if you're
> > getting a partial trace, so that we can get some idea of what component
> > it is not able to unwind. 
>  
>  
>  ==29692== 30 bytes in 2 blocks are definitely lost in loss record 917 of 
>  2,602
>  ==29692==    at 0x484386F: malloc (vg_replace_malloc.c:393)
>  ==29692==    by 0x14806539: ???
>  ==29692==    by 0x14BA7D87: ???
>  ==29692==    by 0x14BA7DFC: ???
>  ==29692==    by 0x14B86D88: ???
>  ==29692==    by 0x146ED193: ???
>  ==29692==    by 0x1475E205: ???
>  ==29692==    by 0x1475E71A: ???
>  ==29692==    by 0x1475ED49: ???
>  ==29692==    by 0x146EF09F: ???
>  ==29692==    by 0x4A5D0E7: g_type_create_instance (gtype.c:1931)
>  ==29692==    by 0x4A42C1F: g_object_new_internal (gobject.c:2228)
>  ==29692==    by 0x4A44247: g_object_new_with_properties (gobject.c:2391)
>  ==29692==    by 0x4A44FF0: g_object_new (gobject.c:2037)
>  ==29692==    by 0x146F5395: ???
>  ==29692==    by 0x145D820C: ???
>  ==29692==    by 0x145E06E3: ???
>  ==29692==    by 0x1276D0: pk_backend_load (pk-backend.c:569)
>  ==29692==    by 0x135D1A: pk_engine_load_backend (pk-engine.c:967)
>  ==29692==    by 0x119468: main (pk-main.c:219) 
I suspect this is a result of libraries being opened and closed
dynamically - for performance reasons valgrind doesn't resolve names
until it prints the trace and if the library in question has been
closed by then it will normally be unable to resolve names from it.

Try using --keep-debuginfo=yes to make valgrind cache debuginfo
for libraries that have been closed.

Tom

-- 
Tom Hughes (tom(a)compton.nu)
http://compton.nu/

[Gordon Messmer]

On 2023-01-16 01:38, Tom Hughes wrote:
>  I suspect this is a result of libraries being opened and closed
>  dynamically...
>  Try using --keep-debuginfo=yes to make valgrind cache debuginfo
>  for libraries that have been closed.  

Yes, that was it.  I did not know this about valgrind.  Thank you!

I'll resume work to shrink packagekitd.

[Richard W.M. Jones]

On Mon, Jan 16, 2023 at 12:52:38AM -0800, Gordon Messmer wrote:
>  ==29692== 30 bytes in 2 blocks are definitely lost in loss record
>  917 of 2,602
>  ==29692==    at 0x484386F: malloc (vg_replace_malloc.c:393)
>  ==29692==    by 0x14806539: ???
>  ==29692==    by 0x14BA7D87: ???
>  ==29692==    by 0x14BA7DFC: ???
>  ==29692==    by 0x14B86D88: ???
>  ==29692==    by 0x146ED193: ???
>  ==29692==    by 0x1475E205: ???
>  ==29692==    by 0x1475E71A: ???
>  ==29692==    by 0x1475ED49: ???
>  ==29692==    by 0x146EF09F: ??? 
This has the hallmark of missing glibc debuginfo.  What's the output
of this command?

$ rpm -qa | grep glibc | sort

Rich.

>  ==29692==    by 0x4A5D0E7: g_type_create_instance (gtype.c:1931)
>  ==29692==    by 0x4A42C1F: g_object_new_internal (gobject.c:2228)
>  ==29692==    by 0x4A44247: g_object_new_with_properties (gobject.c:2391)
>  ==29692==    by 0x4A44FF0: g_object_new (gobject.c:2037)
>  ==29692==    by 0x146F5395: ???
>  ==29692==    by 0x145D820C: ???
>  ==29692==    by 0x145E06E3: ???
>  ==29692==    by 0x1276D0: pk_backend_load (pk-backend.c:569)
>  ==29692==    by 0x135D1A: pk_engine_load_backend (pk-engine.c:967)
>  ==29692==    by 0x119468: main (pk-main.c:219)
>  
>  ==29692== 158,863 (8,192 direct, 150,671 indirect) bytes in 1 blocks
>  are definitely lost in loss record 2,602 of 2,602
>  ==29692==    at 0x48486AF: realloc (vg_replace_malloc.c:1451)
>  ==29692==    by 0x14805D7B: ???
>  ==29692==    by 0x148062D6: ???
>  ==29692==    by 0x1480DA3F: ???
>  ==29692==    by 0x1480FCAB: ???
>  ==29692==    by 0x14B86DF5: ???
>  ==29692==    by 0x146ED193: ???
>  ==29692==    by 0x1475E205: ???
>  ==29692==    by 0x1475E71A: ???
>  ==29692==    by 0x1475ED49: ???
>  ==29692==    by 0x146EF09F: ???
>  ==29692==    by 0x4A5D0E7: g_type_create_instance (gtype.c:1931)
>  ==29692==    by 0x4A42C1F: g_object_new_internal (gobject.c:2228)
>  ==29692==    by 0x4A44247: g_object_new_with_properties (gobject.c:2391)
>  ==29692==    by 0x4A44FF0: g_object_new (gobject.c:2037)
>  ==29692==    by 0x146F5395: ???
>  ==29692==    by 0x145D820C: ???
>  ==29692==    by 0x145E06E3: ???
>  ==29692==    by 0x1276D0: pk_backend_load (pk-backend.c:569)
>  ==29692==    by 0x135D1A: pk_engine_load_backend (pk-engine.c:967)
>  ==29692==    by 0x119468: main (pk-main.c:219)
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

[Richard W.M. Jones]

On Sun, Jan 15, 2023 at 10:10:24PM -0800, Gordon Messmer wrote:
>  I'm working on reducing memory use in packagekitd, and so far
>  progress has been very good.  I've opened 4 memory-related PRs
>  against PackageKit and libdnf this week, and locally I've reduced
>  memory use at idle by almost 90% (I can reliably cause stock
>  packagekitd to use ~ 700MB of RAM in a couple of minutes, but a
>  patched version will free most of its memory and shrink to around
>  75MB.)
>  
>  However, I've hit a possibly minor road block.  I haven't fixed all
>  of the leaks originally reported; valgrind still reports some leaks
>  in my patched builds, but most of the call stack is unresolved, so I
>  can't locate the problems in order to address any remaining issues.
>  
>  I'm running 'valgrind --leak-check=full --num-callers=25
>  /usr/libexec/packagekitd', running for a while, and then exiting
>  with Ctrl-C.  When I exit, I get a list of leaks, most of which
>  contain unresolved symbols.  I see the same thing when I run
>  Fedora's builds (in which case, debug info appears to be provided by
>  debuginfod-client), or when I run my own builds (in which case I'm
>  installing the debuginfo and debugsource packages produced by the
>  local build.)
>  
>  Does anyone have any hints for improving the information I get from
>  valgrind? 
Just wanted to report that valgrind works very well for me in Fedora
if you have correct debuginfo installed (or debuginfod).  I use it
very regularly.

Also want to mention that your valgrind command line is ... a little
naive.  Many other options are useful.  Here's what we use in nbdkit:

 
https://gitlab.com/nbdkit/nbdkit/-/blob/a5f804180240aea7031470cb8ed294f90...

(Best to check each of these options in the manual before using)

Also make use of suppressions:

  https://gitlab.com/nbdkit/nbdkit/-/tree/master/valgrind

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

[Kalev Lember]

On Mon, Jan 16, 2023 at 2:21 PM Richard W.M. Jones <rjones(a)redhat.com&gt;
wrote:

>  Also want to mention that your valgrind command line is ... a little
>  naive.  Many other options are useful.  Here's what we use in nbdkit:
>
>
>
> https://gitlab.com/nbdkit/nbdkit/-/blob/a5f804180240aea7031470cb8ed294f90...
>
>  (Best to check each of these options in the manual before using)
>
>  Also make use of suppressions:
>
>    https://gitlab.com/nbdkit/nbdkit/-/tree/master/valgrind 

Also, to add to this, glib2 has a suppressions file you can use to cut down
on the false positives: /usr/share/glib-2.0/valgrind/glib.supp

-- 
Kalev

[Gordon Messmer]

On 2023-01-16 05:24, Kalev Lember wrote:
>  On Mon, Jan 16, 2023 at 2:21 PM Richard W.M. Jones
> <rjones(a)redhat.com&gt; 
>  wrote:
>
>      Also make use of suppressions:
>
>      https://gitlab.com/nbdkit/nbdkit/-/tree/master/valgrind
>
>
>  Also, to add to this, glib2 has a suppressions file you can use to cut 
>  down on the false positives: /usr/share/glib-2.0/valgrind/glib.supp 

Thanks.  I've looked at the suppressions file for glib, and while I do 
get a couple of loss records from valgrind that look like they happen in 
glib, neither of them look like they match one of the existing 
suppressions.  :)

That leaves one loss record that's printed over a hundred times, which I 
find very confusing 
(https://sourceforge.net/p/valgrind/mailman/message/37763877/), and two 
more that might be leaks in sqlite3 that I still need to look into.