Memory Performance Issue with Fedora Core 2 Kernels

[James Foris]

James Wilkinson wrote:
> James Foris wrote:
> 
>>I have run into an issue with memory bandwidth using the Fedora Core
>>2 kernels and I need help.  I don't know what is wrong, but something
>>killed performance of my custom driver when I ported it from RedHat 7.3
>>to Fedora Core 2.  I believe I have narrowed it down to the kernel.
>>
>>My driver requires a large amount of contiguous physical memory for
>>DMA from a PCI device.  I use the 'mem=YYY' command line parameter to
>>reserve the top of physical RAM for my driver.  Then I allow mapping
>>via mmap() calls to user space.  The user space app then uses this
>>pointer to save the data to disk.
>>
>><snip> 
>>
>>I have tried this on several platforms and kernels and the results vary,
>>but the common denominator seems to be:
>>
>>   Fedora kernel + 32-bit Intel = poor performance (see below)
> 
> 
> I suggested:
> 
>>Wild guess: have you tried turning off the 4G+4G support in the kernel
>>(requires a recompile)?
> 
> 
> James replied
> 
>>I have rebuilt now w/o several patches; 4Kstacks, netdump, and 4g/4g.
>>
>>The problem has gone away.  I suspect that 4G/4G is broken.
> 
> 
> It sounds more "broken as designed", to be honest. It sounds as though
> your setup is doing a *lot* of context switches between user mode and
> kernel mode. The basic trade-off that 4G/4G flushes the TLB each
> context switch, so that kernel and user both get nearly 4 GB, and TLB
> flushes are expensive.
> 
> See
> http://lwn.net/Articles/39283/
> and
> http://kerneltrap.org/node/view/2891
> 
> May I recommend fedora-devel-list at redhat.com or the LKML? You might not
> have much done about it, but they might be able to recommend
> workarounds.

Actually, we have gotten some of the kernel group at Intel into the game, now.

It seems that the problem can be completely reproduced using the native
"/dev/mem" interface - no custom drivers needed.  The intel engineer that
has looked at this believes that either the 4g/4g patch is broken, or that there
are mem driver interactions with 4g/4g that no one has taken into account
(which to me says that potentially both the mem driver and 4g/4g patch need
fixing).

Certainly, removing the 4g/4g patch fixes this problem.

We have been told that Intel will be talking/working with the Red Hat people
directly to resolve this.  They seem to think that this is something significant
that should be resolved.

> 
> At the very least, it would be good if real-life problems with 4G/4G are
> reported.
> 
> James.
>