Memory Performance Issue with Fedora Core 2 Kernels

[James Foris]

John Dangler wrote:
> -----Original Message-----
> From: fedora-list-bounces at redhat.com [mailto:fedora-list-bounces at redhat.com]
> On Behalf Of James Foris
> Sent: Monday, July 26, 2004 3:22 PM
> To: fedora-list at redhat.com
> Subject: Memory Performance Issue with Fedora Core 2 Kernels
> ...[snip]...
> 
> 
>>My driver requires a large amount of contiguous physical memory for
>>DMA from a PCI device.  I use the 'mem=YYY' command line parameter to
>>reserve the top of physical RAM for my driver.  Then I allow mapping
>>via mmap() calls to user space.  The user space app then uses this
>>pointer to save the data to disk.
> 
> 
>>Normally the user space app writes to disk using the mmap()'d pointer as
>>the source.  With the new kernels these writes are taking way too long
>>(around 20 MB/s).  Even when the write goes to /dev/shm, the speed is
>>limited to around 20 MB/s.  A memcpy from the mmap()'d memory seems to
>>have no such slowdown.
> 
> 
> The bottleneck seems to be in i/o...

It certainly does.

> 
> check the hdparm settings on the drive.

Look again at the above example:  it was using the "/dev/shm" device - which
is the shared memory file system.  There is no disk hardware involved.

And this same test on a simple/cheap home PC had a transfer rate > 450 MBytes/sec.

It this beause 1) it should be as close to a memcpy as a file system transfer
can get, 2) it is portable, and 3) it eliminates the disk subsystem as an issue.


Also, this method has worked for several years using the 2.4 kernel series with
performance numbers in the 700+ MByte range on HP workstations.  It is only
in the 2.6 kernel series that we see a dramatic slowdown.

And this is the reason for this post - it really looks like something is broken
in the IA32 architecture in 2.6.

Jim Foris

> 
> (memcpy will definitely run faster than disc i/o)
> 
> ...[snip]...
>