rawhide ext4/virtio FS performance hit (10x slower) ?

[Jim Meyering]

Today I noticed that some tests from coreutils' test suite
were taking far longer than they used to on rawhide.
For example, run this command in an empty directory:

  seq 200000|env time xargs touch

it takes < 5s on F17 3.3.4-1.fc17.x86_64 ext4/sdb8/SSD, yet
it takes 47s on rawhide 3.4.0-0.rc5.git5.1.fc18.x86_64 ext4/vda1/SSD

10x performance hit... ouch.

Does anyone know why?

[Josh Boyer]

On Mon, May 7, 2012 at 8:43 AM, Jim Meyering <jim(a)meyering.net&gt; wrote:
>  Today I noticed that some tests from coreutils' test suite
>  were taking far longer than they used to on rawhide.
>  For example, run this command in an empty directory:
>
>   seq 200000|env time xargs touch
>
>  it takes < 5s on F17 3.3.4-1.fc17.x86_64 ext4/sdb8/SSD, yet
>  it takes 47s on rawhide 3.4.0-0.rc5.git5.1.fc18.x86_64 ext4/vda1/SSD
>
>  10x performance hit... ouch.
>
>  Does anyone know why? 
Rawhide has all the kernel debug options turned on.  It isn't
worthwhile doing a comparison when those are enabled.  You could try
again with the first -rc6 build that shows up, as that will have the
options disabled.

josh

[Jim Meyering]

Josh Boyer wrote:
>  On Mon, May 7, 2012 at 8:43 AM, Jim Meyering <jim(a)meyering.net&gt;
> wrote:
> > Today I noticed that some tests from coreutils' test suite
> > were taking far longer than they used to on rawhide.
> > For example, run this command in an empty directory:
> >
> >  seq 200000|env time xargs touch
> >
> > it takes < 5s on F17 3.3.4-1.fc17.x86_64 ext4/sdb8/SSD, yet
> > it takes 47s on rawhide 3.4.0-0.rc5.git5.1.fc18.x86_64 ext4/vda1/SSD
> >
> > 10x performance hit... ouch.
> >
> > Does anyone know why?
>
>  Rawhide has all the kernel debug options turned on.  It isn't
>  worthwhile doing a comparison when those are enabled.  You could try
>  again with the first -rc6 build that shows up, as that will have the
>  options disabled. 
Thanks.  Will do.

I knew about the kernel debug options, but didn't realize they
could make such a large difference.

[Jim Meyering]

Josh Boyer wrote:

>  On Mon, May 7, 2012 at 8:43 AM, Jim Meyering <jim(a)meyering.net&gt;
> wrote:
> > Today I noticed that some tests from coreutils' test suite
> > were taking far longer than they used to on rawhide.
> > For example, run this command in an empty directory:
> >
> >  seq 200000|env time xargs touch
> >
> > it takes < 5s on F17 3.3.4-1.fc17.x86_64 ext4/sdb8/SSD, yet
> > it takes 47s on rawhide 3.4.0-0.rc5.git5.1.fc18.x86_64 ext4/vda1/SSD
> >
> > 10x performance hit... ouch.
> >
> > Does anyone know why?
>
>  Rawhide has all the kernel debug options turned on.  It isn't
>  worthwhile doing a comparison when those are enabled.  You could try
>  again with the first -rc6 build that shows up, as that will have the
>  options disabled. 
Hi Josh,

I've just rerun the test with 3.4.0-0.rc6.git1.1.fc18.x86_64,
and it takes just as long, so the hit is *not* due to debug options:

$ seq 200000|env time xargs touch
0.17user 41.98system 0:48.22elapsed 87%CPU (0avgtext+0avgdata 1424maxresident)k
88inputs+0outputs (0major+4502minor)pagefaults 0swaps

[drago01]

On Sat, May 12, 2012 at 11:04 PM, Jim Meyering <jim(a)meyering.net&gt; wrote:
>  Josh Boyer wrote:
>
> > On Mon, May 7, 2012 at 8:43 AM, Jim Meyering <jim(a)meyering.net&gt; wrote:
> >> Today I noticed that some tests from coreutils' test suite
> >> were taking far longer than they used to on rawhide.
> >> For example, run this command in an empty directory:
> >>
> >>  seq 200000|env time xargs touch
> >>
> >> it takes < 5s on F17 3.3.4-1.fc17.x86_64 ext4/sdb8/SSD, yet
> >> it takes 47s on rawhide 3.4.0-0.rc5.git5.1.fc18.x86_64 ext4/vda1/SSD
> >>
> >> 10x performance hit... ouch.
> >>
> >> Does anyone know why?
> >
> > Rawhide has all the kernel debug options turned on.  It isn't
> > worthwhile doing a comparison when those are enabled.  You could try
> > again with the first -rc6 build that shows up, as that will have the
> > options disabled.
>
>  Hi Josh,
>
>  I've just rerun the test with 3.4.0-0.rc6.git1.1.fc18.x86_64, 
This one has debug options enabled; 3.4.0-0.rc6.git0.1 is the one
without debug options.

[Reindl Harald]

Am 12.05.2012 23:38, schrieb drago01:
> > I've just rerun the test with
> 3.4.0-0.rc6.git1.1.fc18.x86_64,
>  
>  This one has debug options enabled; 3.4.0-0.rc6.git0.1 is the one
>  without debug options. 
how can someone find out if it is a debug-kernel?
even the satble ones seems to have a ton of debug options enabled
especially "CONFIG_DEBUG_KERNEL=y"

[root@srv-rhsoft:/boot]$ cat config-3.3.5-2.fc16.x86_64 | grep -i debug | grep
"=y"
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_SLUB_DEBUG=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_XEN_DEBUG_FS=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_CFG80211_DEBUGFS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_DEBUG_DEVRES=y
CONFIG_CB710_DEBUG_ASSUMPTIONS=y
CONFIG_IWMC3200TOP_DEBUGFS=y
CONFIG_DM_DEBUG=y
CONFIG_FIREWIRE_OHCI_DEBUG=y
CONFIG_MLX4_DEBUG=y
CONFIG_ATH5K_DEBUG=y
CONFIG_ATH9K_DEBUGFS=y
CONFIG_ATH6KL_DEBUG=y
CONFIG_IWLWIFI_DEBUG=y
CONFIG_IWLWIFI_DEBUGFS=y
CONFIG_IWLEGACY_DEBUG=y
CONFIG_IWLEGACY_DEBUGFS=y
CONFIG_RT2X00_LIB_DEBUGFS=y
CONFIG_SND_DEBUG=y
CONFIG_SND_PCM_XRUN_DEBUG=y
CONFIG_INFINIBAND_MTHCA_DEBUG=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
CONFIG_INFINIBAND_IPOIB_DEBUG_DATA=y
CONFIG_DRM_NOUVEAU_DEBUG=y
CONFIG_DLM_DEBUG=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_SCHED_DEBUG=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
CONFIG_DEBUG_SET_MODULE_RONX=y
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y

[Bruno Wolff III]

On Sun, May 13, 2012 at 00:00:24 +0200,
   Reindl Harald <h.reindl(a)thelounge.net&gt; wrote:
> Am 12.05.2012 23:38, schrieb drago01:
> >> I've just rerun the test with 3.4.0-0.rc6.git1.1.fc18.x86_64,
> >
> > This one has debug options enabled; 3.4.0-0.rc6.git0.1 is the one
> > without debug options.
>
> how can someone find out if it is a debug-kernel?
> even the satble ones seems to have a ton of debug options enabled
> especially "CONFIG_DEBUG_KERNEL=y" 
You can look at the changelog for the build. Current practice is that
the first build for a particular rc is a nodebug build.

The package set is also different. The debug builds don't include an
update of kernel-doc.

[Michal Schmidt]

On 05/13/2012 12:00 AM, Reindl Harald wrote:
>  how can someone find out if it is a debug-kernel?
>  even the satble ones seems to have a ton of debug options enabled
>  especially "CONFIG_DEBUG_KERNEL=y" 
See http://fedoraproject.org/wiki/KernelDebugStrategy to find what 
kernel config options to look for.

Michal