Why does GCC subtract the wrong value to the stack pointer when allocating a big array with no subsequent function calls?

Question

Really bizarre gcc quirk. Check this out:

main() { int a[100]; a[0]=1; }

produces this assembly:

   0:   55                         


        

Answer 1

Your guess is correct. It is a "red zone". The red zone is the space from rsp-128 to rsp, which may be used by a function for local variables and for temporary storage. This space is untouched by interrupt and exception handlers. Obviously, the red zone is destroyed by function calls, so if any function is called, no local variable can be in the red zone.

The red zone can only be used in 64 bit Linux, BSD and Mac. It is not available in kernel code.

It may be used to optimize for space, since with the red zone you can reference up to 512 bytes of local variables with short instructions, based on only rsp and ebp. Without the red zone only 384 bytes are available. All local variables outside of this limit are accessed with longer code or with additional registers.

For your example, using the red zone is not necessary, but gcc prefers to use it for all "leaf" functions. It is just easier to implement compiler this way.

Answer 2

The x86-64 ABI mandates a 'red zone' of 128 bytes beyond the stack pointer that can be used without modifying %rsp. In the first example, main() is a leaf function, so the compiler is optimizing the use of stack space - i.e., there are no function calls, so this region will not be overwritten.