problem initializing large double array

Question

Silly question from a new C programmer... I get a segmentation fault in the following code:

#include 
int main(void)
{
double YRaw[4000000]={0         


        

Answer 1

4000000 * sizeof(double) is likely to be on the order of 32MB. That is too large for the stack, which is why you are getting an exception. (In short, the stack overflowed.)

Either use malloc() to allocated it from the heap, make it a static, or make it a global.

Generally, automatic allocation should only be used for small to moderate sized objects. The threshold is difficult to characterize, but 32MB is well above it under most circumstances.

Update:

There are several regions of memory within a process. (Note that I'm going to simplify this for clarity, if you want the gory details, read the documentation for ld, and find the linker control file actually used to lay out memory of executables for your platform.)

First, there is the text (sometimes called code) segment. This contains the actual code that executes, and usually any constant data. Generally, the text segment is protected from accidental modifications, and in some systems the physical memory can actually be shared among processes that happen to be running the same program or using the same shared library.

Next, there are the data and bss segments. Together, these segments hold all of the statically allocated variables. The data segment contains initialized variables, and bss the uninitialized variables. They are distinct because the uninitialized variables are only known in the executable file by their individual addresses and their total size. That information is used to request blank pages from the operating system, which is one explanation for why uninitialized globals are known to have the value 0.

Then there are the stack and heap. Both of these segments are created at run time, from memory allocated to the process when it loads, and often extended during execution. The stack logically holds a record of call nesting, parameters, and local variables, although the details can be surprisingly different among platforms. The heap is the pool of memory managed by malloc() and its friends (and usually the operator new() in C++). In many platforms, the process memory map is arranged so that the stack and heap don't interact, but that can mean that the stack has an upper bound on its total size, while the heap is usually bounded by the virtual memory system.

With that as background, lets clarify where each declaration gets stored.

All global variables land in the data or bss segment, based solely on whether they are initialized or not. If they are in the data segment, then they will directly contribute to the file size of the executable. Either way, if the linker succeeded, then their storage is guaranteed to exist for the lifetime of the process.

Variables declared static are allocated the same as a global variable, but without a public symbol table entry. That means that an uninitialized static buffer is located in the bss segment, and an initialized static in the data segment, but its name is known only to the scope that can see its definition.

Allocations from the heap are not guaranteed to succeed at run time. There is an upper bound to the total size of a process, enforced by system policy and by the hardware architecture. On a typical 32-bit system, the architecture cannot allow more than 4GB of addresses to any single process, for instance.

Here are some concrete examples:

int foo(void)
{
    double YRaw[4000000]={0}; 
    // ... do something with huge buffer
}

Here, YRaw is a local variable and has automatic storage class. It is allocated on the stack when the function is entered, and automatically released when the function exits. However, this will only work if the stack has enough room for it. If it doesn't, then the stack overflows, and if you are very lucky you get some kind of run time error to indicate that fact. (If you are not as lucky, the stack still overflowed, but it wrote on top of memory allocated to some other segment, perhaps the text segment, and a clever hacker might be able to execute arbitrary code.)

static double YRaw2[4000000]={0}; 
int foo(void)
{
    static double YRaw3[4000000]={0}; 
    // ... do something with huge buffer
}

Here, YRaw2 and YRaw3 are initialized, both end up in the data segment, and on many platforms will make the actual executable file contain the 4 million 0.0 values you specified as their initial values. The only difference between the two buffers is a matter of scope. YRaw2 can be used by any function in the same module, while YRaw3 is only visible within the function.

static double YRaw4[4000000]; 
int foo(void)
{
    static double YRaw5[4000000]; 
    // ... do something with huge buffer
}

Here, YRaw4 and YRaw5 both end up in the bss segment, and usually will not enlarge the executable file itself. Again, the buffers only differ in scope of their names. They will be implicitly initialized to the same 0 value as was specified for YRaw2 and YRaw3 when the program starts.

double YRaw6[4000000]; 
int foo(void)
{
    // ... do something with huge buffer
}

Here, YRaw6 is similar to YRaw4 above, except that the name has global scope, and the buffer could be shared with other modules as well as with every function in this module. It is stored in the bss segment, so like YRaw4 it has no impact on the file size.

And finally, it can come from the heap. If it needs to exist for the entire run as if it were allocated at compile time, you can do something like this:

int foo(void)
{
    static double *YRaw7 = NULL;

    if (!YRaw7) {
        // allocate the buffer on the first use
        YRaw7 = calloc(4000000, sizeof(double));
    }
    // ... do something with huge buffer
}

Here, YRaw7 is stored in the heap, allocated at its first use, and never freed until the process exits. On most "reasonable" platforms, this usage pattern is both sensible and allowed.

int foo(void)
{
    double *YRaw8 = calloc(4000000, sizeof(double));
    assert(YRaw8 != NULL);
    // do something with huge buffer
    // ...
    // but be careful that all code paths that return also
    // free the buffer if it was allocated.
    free(YRaw8);
}

Here, YRaw8 has the same lifetime as an automatic variable as you intended with your original example, but is physically stored in the heap. It is probably wise to verify that the memory allocation succeeded as I've done with the call to assert(), but there may be no better response to the lack of memory than to allow the assertion to fail.

One other subtlety: I used calloc() to allocate the buffers. This has the nice property that the memory is guaranteed to be initialized to all zero bits if the allocation succeeds. However, this has the side effect that it (usually) has to write to every byte of the allocation to have that effect. That means that all of those pages of virtual memory not only get allocated to the process, but each had to be paged in and every byte written. Using malloc() instead will usually perform better, but at the cost that the memory is not guaranteed to be clear.

Finally, the obvious question is "Where should I allocate that buffer anyway?"

I can't give you a hard and fast rule, other than that large allocations never belong on the stack. If the buffer needs to exist for the lifetime of the process, then an uninitialized static (whether at module or function scope) is usually the right answer.

If the buffer needs a different lifetime, to have a size known only at run time, or to live and die in response to external events at run time, then it should be allocated on the heap with malloc() and friends and released eventually with free() or possibly the termination of the process.

Answer 2

You tried to declare the entire array on the stack. Even if you have a terabyte of RAM, only a small, fixed portion of it is going to be dedicated to stack space. Large amounts of data need to be allocated on the heap, using malloc:

#include <stdio.h>
int main(void)
{
  double* YRaw = malloc(4000000 * sizeof(double));
  memset(YRaw, 0, 4000000 * sizeof(double));

  /* ... use it ... */

  free(YRaw); /* Give the memory back to the system when you're done */

  return 0;
}

See also: "What and where are the stack and heap?"