Linux kernel behaviour on heap overrun or stack overflow

问题

I am trying to understand few important OS concepts (for simplicity, lets stick to Linux Kernel). Assume I run this in kernel mode , perhaps adding these lines (either caseA or caseB not both) into source code of some system call.

# Assume __malloc() here is a simple heap memory manager
void consume_heap_forever(void)
    {

      for (;;)
        (void) __malloc(PAGE_SIZE);         
    }

Case A: The above consumes heap in a loop. I will first start consuming memory and things will go normal. After a high enough consumption, what begins to happen (before a crash) ? I know that kernel space is within reserved chunk in process address space. Will I crash at point when I cross the stack portion the kernel uses? Or will this expand that reservation (and perhaps consume whole of virtual memory)?

# Vanilla Factorial logic
int factorial(int value)
    {
        if (value == 0)
           return 1;
         return value * factorial(value-1)
    }

Case B: I am aware that the kernel has a fixed (and small) amount of stack reserved for it. So perhaps when I give a value big enough -- I will run out of that predefined stack space. What kind of crash happens here? Will I cross into the heap section of kernel?

回答1:

In your example A, your application would loop forever. At some point malloc will be unable to map pages to the logical address space and will return 0.

In your example B: Each process has its own kernel mode stack (usually, there is one shared interrupt stack).

It's likely that you'd eventually hit a guard page at the end of the stack and get an access violation. You're not going to run over the kernel's memory pool.

来源：https://stackoverflow.com/questions/42217778/linux-kernel-behaviour-on-heap-overrun-or-stack-overflow