File size limit exceeded

[Samuel Flory]

Samuel Flory wrote:
> Björn Persson wrote:
> 
>> Horacio Reyes wrote:
>>
>>> Fews weeks ago, we change our old AIX for a new intel/fedora machine. We
>>> make software in ansi C, we recompile all programs with gcc and works 
>>> fine
>>> with almost every input file, until we process a program with a 3Gb 
>>> input
>>> file (AFP file), then one of our programs say "File size limit exceeded"
>>>
>>> We re-run this on our old AIX and works fine with this file.
>>>
>>> Any idea?
>>
>>
>>
>> What file system do you use? Ext2 and Ext3 have a limit at about 2 GB. 
>> Some other file systems can handle larger files.
>>
> 
>   Most likely the error message in their program is wrong, and they are 
> hitting a memory addressing limit.
> 
   On second thought you may need to modify the program to use LFS, or 
recompile and add "-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE" to your 
compile options.

http://www.suse.de/~aj/linux_lfs.html

  Using LFS

For using LFS in user programs, the programs have to use the LFS API. 
This involves recompilation and changes of programs. The API is 
documented in the glibc manual (the libc info pages) which can be read 
with e.g. "info libc".

In a nutshell for using LFS you can choose either of the following:

     * Compile your programs with "gcc -D_FILE_OFFSET_BITS=64 
-D_LARGEFILE_SOURCE". This forces all file access calls to use the 64 
bit variants. Several types change also, e.g. off_t becomes off64_t. 
It's therefore important to always use the correct types and to not use 
e.g. int instead of off_t.
     * Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With these 
defines you can use the LFS functions like open64 directly.
     * Use the O_LARGEFILE flag with open to operate on large files.

A complete documentation of the feature test macros like 
_FILE_OFFSET_BITS and _LARGEFILE_SOURCE is in the glibc manual (run e.g. 
"info libc 'Feature Test Macros'").

The LFS API is also documented in the LFS standard which is available at 
http://ftp.sas.com/standards/large.file/x_open.20Mar96.html.
LFS and Libraries other than Glibc

Be careful when using _FILE_OFFSET_BITS=64 to compile a program that 
calls a library or a library if any of the interfaces uses off_t. With 
_FILE_OFFSET_BITS=64 glibc will change the type of off_t to off64_t. You 
can either change the interface to always use off64_t, use a different 
function if _FILE_OFFSET_BITS=64 is used (like glibc does). Otherwise 
take care that both library and program have the same _FILE_OFFSET_BITS 
setting. Note that glibc is aware of the _FILE_OFFSET_BITS setting, 
there's no problem with it but there might be problems with other libraries.

-- 
There is no such thing as obsolete hardware.
Merely hardware that other people don't want.
(The Second Rule of Hardware Acquisition)
Sam Flory  <sflory at rackable.com>