Set stack size with setrlimit() and provoke a stack overflow/segfault

Question

In the given example below I try to set the stacksize to 1kb.

Why is it now possible to allocate an array of ints on the stack with size 8kb in foo()

Answer 1

The limit is set immediately but only checked when trying to allocate a new stack or trying to grow the existing stack. A grep for RLIMIT_STACK (or a LXR identifier search) on the kernel sources should tell.

Apparently, the initial size of the stack is whatever is needed to the filename + env strings + arg strings plus some extra pages allocated on setup_arg_pages (20 pages in 2.6.33 1,2, 128 Kb on 2.6.34 3).

In summary:

initial stack size = MIN(size for filename + arg strings + env strings + extra pages, MAX(size for filename + arg strings + env strings, RLIMIT_STACK))

where

size for filename + arg strings + env strings <= MAX(ARG_MAX(32 pages), RLIMIT_STACK/4)

Additionally, kernels with Ingo Molnar's exec-shield patch (Fedora, Ubuntu, ...) have an additional EXEC_STACK_BIAS "(2MB more to cover randomization effects.)", see the call to the new function over_stack_limit() from acct_stack_growth() ([Ubuntu1], [Ubuntu2], [Ubuntu3]).

I've edited the original program to show this:

#include 
#include 

void foo(void);

int main(int argc, char *argv[]) {
        struct rlimit lim = {1, 1};


        if (argc > 1 && argv[1][0] == '-' && argv[1][8]=='l') {
                printf("limiting stack size\n");
                if (setrlimit(RLIMIT_STACK, &lim) == -1) {
                        printf("rlimit failed\n");
                        return 1;
                }
        }

        foo();

        return 0;
}

void foo() {
        unsigned ints[32768];

        printf("foo: %u\n", ints[2047]=42);
}

Which results in:

$./rl
foo: 42
$./rl -l
limiting stack size
Segmentation fault
$