libguestfs tests running much slower

[Matthew Booth]

This is only subjective, and could yet be something entirely random on 
my machine. However, I just pulled changes 
3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe 
and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10 times 
slower. Is it just me?

Matt
-- 
Matthew Booth, RHCA, RHCSS
Red Hat Engineering, Virtualisation Team

M:       [phone]
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

[Jim Meyering]

Matthew Booth wrote:
>  This is only subjective, and could yet be something entirely random
> on
>  my machine. However, I just pulled changes
>  3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe
>  and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10
>  times slower. Is it just me? 
Hi Matt,

I've noticed that some programs (thinking of one of augeas' self tests,
in particular) can take far longer when there is limited memory than
when they don't hit the limit.

For example, one "make check" test takes just 5 seconds most of the
time, but occasionally it takes 30s on an idle system.  strace -c
showed that in that case, most of the time was spent in the many
sbrk syscalls.

[Jim Meyering]

Matthew Booth wrote:
>  This is only subjective, and could yet be something entirely random
> on
>  my machine. However, I just pulled changes
>  3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe
>  and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10
>  times slower. Is it just me? 
Hi Matt,

It's not just you.  I've confirmed it.

I reset back to 202e11543ead0d21a8485879654c927ec95ea7f0
[Don't dereference or free undefined "msg" upon OOM.],
ran make, then "make check" and the latter took 15 minutes.

Now, "make check" on the latest, f20854ec61eef1aea313920f0cf193a78c1a9219
[fish: handle some out-of-memory conditions], is at only 95/174, and it's
already taken 23 minutes (50% longer).  I noticed that this one

  35/174 test_zerofree_0

took a particularly long time, but wasn't watching the whole time,
so no hard numbers.

[Jim Meyering]

Jim Meyering wrote:

>  Matthew Booth wrote:
> > This is only subjective, and could yet be something entirely random on
> > my machine. However, I just pulled changes
> > 3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe
> > and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10
> > times slower. Is it just me?
>
>  Hi Matt,
>
>  It's not just you.  I've confirmed it.
>
>  I reset back to 202e11543ead0d21a8485879654c927ec95ea7f0
>  [Don't dereference or free undefined "msg" upon OOM.],
>  ran make, then "make check" and the latter took 15 minutes.
>
>  Now, "make check" on the latest, f20854ec61eef1aea313920f0cf193a78c1a9219
>  [fish: handle some out-of-memory conditions], is at only 95/174, and it's
>  already taken 23 minutes (50% longer).  I noticed that this one
>
>    35/174 test_zerofree_0
>
>  took a particularly long time, but wasn't watching the whole time,
>  so no hard numbers. 
It finished.  elapsed time: 33 minutes.
Over double the time from 6 commits ago.

Ironically, the commit that appears to introduce this
slow-down talks about improved performance:

    Change to use virtio_blk (virtio block device) by default

    virtio_blk is the fast, virt-native block device driver
    supported by qemu and KVM.  Note that virtio_blk device
    names are called /dev/vd*.

    Existing scripts should continue working because device name
    translation will silently change device names of the form
    /dev/sd* to /dev/vd* as required.

    See also:
    http://libguestfs.org/guestfs.3.html#block_device_naming

I did git rebase -i HEAD~7 and deleted the line for that commit,
rebuilt, and then "make check" took nearly 15 minutes again.
However, there were failures:

    /sbin/sfdisk /dev/sda: external command failed
    test_umount_all_1 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_lvcreate_0 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_vgcreate_0 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_pvcreate_0 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_lvs_1 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_vgs_1 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_pvs_1 FAILED
    /sbin/sfdisk /dev/sda: external command failed
    test_list_partitions_1 FAILED

    ***** 8 / 174 tests FAILED *****

[Richard W.M. Jones]

On Wed, Jul 01, 2009 at 06:23:14PM +0100, Matthew Booth wrote:
>  This is only subjective, and could yet be something entirely random
> on  
>  my machine. However, I just pulled changes  
>  3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe 
>  and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10 times  
>  slower. Is it just me? 
It turns out that two commands take a lot longer to run
with a virtio backend than with an IDE backend.  They are:

mkfs.ext2
zerofree

At the moment I have no idea why this is.

I'm about to push some very simple command-level profiling
into the daemon.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Daniel P. Berrange]

On Thu, Jul 02, 2009 at 02:38:33PM +0100, Richard W.M. Jones wrote:
>  On Wed, Jul 01, 2009 at 06:23:14PM +0100, Matthew Booth wrote:
>  > This is only subjective, and could yet be something entirely random on  
>  > my machine. However, I just pulled changes  
>  > 3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe 
>  > and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10 times 
>
>  > slower. Is it just me?
>  
>  It turns out that two commands take a lot longer to run
>  with a virtio backend than with an IDE backend.  They are:
>  
>  mkfs.ext2
>  zerofree
>  
>  At the moment I have no idea why this is. 
We've had that reported with F11 + virtio + sparse files, but unable to
reliably reproduce it thus far, so no resolution on the bug...

  https://bugzilla.redhat.com/show_bug.cgi?id=504376

"F-11 guest takes 45 minutes in mke2fs for 8gb fs on virtio disk with a sparse
backing file"

Regards,
Daniel
-- 
|: Red Hat, Engineering, London   -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org  -o-  http://virt-manager.org  -o-  http://ovirt.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-  F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Richard W.M. Jones]

On Thu, Jul 02, 2009 at 02:51:41PM +0100, Daniel P. Berrange wrote:
>  We've had that reported with F11 + virtio + sparse files, but
> unable to
>  reliably reproduce it thus far, so no resolution on the bug...
>  
>    https://bugzilla.redhat.com/show_bug.cgi?id=504376
>  
>  "F-11 guest takes 45 minutes in mke2fs for 8gb fs on virtio disk with a sparse
> backing file" 
I opened another bug because we see the same issue WITHOUT a
sparse backing disk.

We've also got it nailed with an easy reproducer script.

https://bugzilla.redhat.com/show_bug.cgi?id=509383

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

[Richard W.M. Jones]

On Thu, Jul 02, 2009 at 02:38:33PM +0100, Richard W.M. Jones wrote:
>  On Wed, Jul 01, 2009 at 06:23:14PM +0100, Matthew Booth wrote:
>  > This is only subjective, and could yet be something entirely random on  
>  > my machine. However, I just pulled changes  
>  > 3d15f7e652340777514ff30c3cfc560a90b612ec..99e28249d52ca5495b636e14ae3e4387ee62c8fe 
>  > and now 'make check' runs a *lot* slower. I'm guessing maybe 5-10 times 
>
>  > slower. Is it just me?
>  
>  It turns out that two commands take a lot longer to run
>  with a virtio backend than with an IDE backend.  They are:
>  
>  mkfs.ext2
>  zerofree
>  
>  At the moment I have no idea why this is.
>  
>  I'm about to push some very simple command-level profiling
>  into the daemon. 
Dan pointed me to this bug which looks relevant:

https://bugzilla.redhat.com/show_bug.cgi?id=504376

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://et.redhat.com/~rjones/libguestfs/
See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html

[Richard W.M. Jones]

On Thu, Jul 02, 2009 at 03:17:10PM +0100, Richard W.M. Jones wrote:
>  Dan pointed me to this bug which looks relevant:
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=504376 
However it's not sparseness which is the problem.  Even with
non-sparse backing files (see attached patch) I still see a slowdown.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

https://bugzilla.redhat.com/show_bug.cgi?id=509383

New bug created for this issue.

Rich.

-- 
Richard Jones, Emerging Technologies, Red Hat  http://et.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top