memory limits....

[gaf at blu.org (Jerry Feldman)]

On Thursday 19 May 2005 10:28 am, Stephen Adler wrote:
> Thanks for the reply. I'm using malloc, under gcc 3.2, red hat,
> enterprise linux 3, kernel 2.4. After allocating about 2.4 Gigs,
> subsequent malloc's return NULL. I check for NULL returns from
> my malloc's and exit my code if this condition is met. The OS
> seems to be happy, nothing grinds to a halt. My applications
> allocates all the needed memory at startup, so if not enough
> is allocated, it informs the user and exits.
Classicly, the way malloc works requires some explanation:
First your program is given a virtual memory map. This includes where the 
program text (eg code) is mapped. This is usually readonly, the program 
stack, the program's initialized data, and unititalized data. Then you get 
what is called a heap. In classic Unix and Linux, the heap grows upward 
using the brk(2) or sbrk(2) system calls. These grow the heap contiguously. 
At some point they may his a barrier. This could be the top of your virtual 
address space, or it could be that some other operation, such as mmap(2) or 
the loader has mapped something else up there. Malloc(3) periodically grabs 
a chunk of memory from the heap, normally in multiples of page sizes. More 
modern mallocs may take a different approach by using mmap(2) to allocate 
its chunks. The advantage of using mmap(2) is that it can allocate 
non-contiguous chunks. Malloc(3) is simply a set of library functions and 
has nothing to do with the compiler, BUT the compiler is responsible for 
setting up the initial virtual mapping in your program, and the loader is 
responsible for the dynamically loading of your program, and mapping in 
shared libraries, et. al. Also, remember that a 32 bit signed integer is 
limited to about 2GB. I think that you may be running into a 32-bit 
instrinsic limit. This could be imposed by malloc(3) or by the kernel where 
you have a 2GB limit on heap size. 
-- 
Jerry Feldman <gaf at blu.org>
Boston Linux and Unix user group
http://www.blu.org PGP key id:C5061EA9
PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9