valgrinded executable seem to run far more slowly in Rawhide

[Richard W.M. Jones]

I haven't filed a bug yet because I'm not sure what component to
assign this too.  I upgraded to Rawhide this morning and noticed that
some tests that previously ran OK are now timing out.  It turned out
that the tests were failing because a valgrinded executable runs much
more slowly in Rawhide today.

Here's my test:

Upgrade to Rawhide and install these additional packages:

  # dnf install libnbd nbdkit valgrind

Unpack the certificates from
http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp

Run this command as NON-root (it's self-contained and just connects
the two commands together over localhost port 10809):

  $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run 'valgrind nbdinfo
nbds://localhost'

For me this takes over 40 seconds in Rawhide, versus a couple of
seconds in Fedora 33/34.  Also if you remove "valgrind" it runs
instantaneously even in Rawhide.  If you run it as root, it runs
quickly again (WTF!)

The perf flamegraph seems to indicate it's doing a lot of stuff with
DWARF 3 symbols.

  http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg

Uninstalling all relevant debuginfo packages didn't seem to help.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

[Mark Wielaard]

Hi Rich,

On Tue, Jun 08, 2021 at 04:09:33PM +0100, Richard W.M. Jones wrote:
>  I haven't filed a bug yet because I'm not sure what component
> to
>  assign this too.  I upgraded to Rawhide this morning and noticed that
>  some tests that previously ran OK are now timing out.  It turned out
>  that the tests were failing because a valgrinded executable runs much
>  more slowly in Rawhide today.
>  
>  Here's my test:
>  
>  Upgrade to Rawhide and install these additional packages:
>  
>    # dnf install libnbd nbdkit valgrind
>  
>  Unpack the certificates from
>  http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp
>  
>  Run this command as NON-root (it's self-contained and just connects
>  the two commands together over localhost port 10809):
>  
>    $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run 'valgrind
> nbdinfo nbds://localhost'
>  
>  For me this takes over 40 seconds in Rawhide, versus a couple of
>  seconds in Fedora 33/34.  Also if you remove "valgrind" it runs
>  instantaneously even in Rawhide.  If you run it as root, it runs
>  quickly again (WTF!)
>  
>  The perf flamegraph seems to indicate it's doing a lot of stuff with
>  DWARF 3 symbols.
>  
>    http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg
>  
>  Uninstalling all relevant debuginfo packages didn't seem to help. 
Thanks for using valgrind and sorry we made it even slower.  I am
working on it. As you already figured out, it has to do with
debuginfo. Basically valgrind should only really need to read the line
tables, but it does a lot more. This is the fedora bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1662656

This has become more urgent now that debuginfod has become the default
in rawhide (valgrind supports debuginfod since 3.17.0):
https://fedoraproject.org/wiki/Changes/DebuginfodByDefault

That also explains your observations, it depends on whether or not
DEBUGINFOD_URLS is set in the environment (and whether
elfutils-debuginfod-client is installed).

I'll work on making valgrind fast again, as far as valgrind is fast to
begin with :)

Cheers,

Mark

[Richard W.M. Jones]

On Wed, Jun 09, 2021 at 10:20:28AM +0200, Mark Wielaard wrote:
>  Hi Rich,
>  
>  On Tue, Jun 08, 2021 at 04:09:33PM +0100, Richard W.M. Jones wrote:
>  > I haven't filed a bug yet because I'm not sure what component to
>  > assign this too.  I upgraded to Rawhide this morning and noticed that
>  > some tests that previously ran OK are now timing out.  It turned out
>  > that the tests were failing because a valgrinded executable runs much
>  > more slowly in Rawhide today.
>  > 
>  > Here's my test:
>  > 
>  > Upgrade to Rawhide and install these additional packages:
>  > 
>  >   # dnf install libnbd nbdkit valgrind
>  > 
>  > Unpack the certificates from
>  > http://oirase.annexia.org/tmp/rawhide-valgrind-bug/pki.tar.gz into /tmp
>  > 
>  > Run this command as NON-root (it's self-contained and just connects
>  > the two commands together over localhost port 10809):
>  > 
>  >   $ time nbdkit --tls=require --tls-certificates=/tmp/pki null --run 'valgrind
> nbdinfo nbds://localhost'
>  > 
>  > For me this takes over 40 seconds in Rawhide, versus a couple of
>  > seconds in Fedora 33/34.  Also if you remove "valgrind" it runs
>  > instantaneously even in Rawhide.  If you run it as root, it runs
>  > quickly again (WTF!)
>  > 
>  > The perf flamegraph seems to indicate it's doing a lot of stuff with
>  > DWARF 3 symbols.
>  > 
>  >   http://oirase.annexia.org/tmp/rawhide-valgrind-bug/libnbd.svg
>  > 
>  > Uninstalling all relevant debuginfo packages didn't seem to help.
>  
>  Thanks for using valgrind and sorry we made it even slower.  I am
>  working on it. As you already figured out, it has to do with
>  debuginfo. Basically valgrind should only really need to read the line
>  tables, but it does a lot more. This is the fedora bug:
>  https://bugzilla.redhat.com/show_bug.cgi?id=1662656
>  
>  This has become more urgent now that debuginfod has become the default
>  in rawhide (valgrind supports debuginfod since 3.17.0):
>  https://fedoraproject.org/wiki/Changes/DebuginfodByDefault
>  
>  That also explains your observations, it depends on whether or not
>  DEBUGINFOD_URLS is set in the environment (and whether
>  elfutils-debuginfod-client is installed). 
FWIW yes it is (to "https://debuginfod.fedoraproject.org/")

Unsetting the environment variable does improve the performance back
to expected levels.  This also explains the mystery of why "sudo"
fixed the problem, since sudo won't pass this environment through to
the subprocess.

Rich.

>  I'll work on making valgrind fast again, as far as valgrind is
> fast to
>  begin with :)
>  
>  Cheers,
>  
>  Mark
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]
>  Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v