Why is ESP masked with 0xFFFFFFF0?

Question

I have disassembled a program. I see at the beginning an AND instruction with ESP and 0xFFFFFFF0.

What is the meaning of this

Answer 1

gcc for i386 Linux defaults to -mpreferred-stack-boundary=4 (meaning 24 = 16 byte alignment). (Other non-Linux systems also use ELF executables, but they also have the same stack-alignment defaults and SysV ABI.)

Unlike clang, gcc doesn't assume that the stack will be aligned on entry to main, so it uses an AND instruction to mask off the low bits of the stack pointer. This is the cheapest way to reserve enough padding on the stack to reach the next alignment boundary.

Stack alignment is the same reason you'll see a function that calls another function reserve some space on the stack that it doesn't use for anything:

extern int bar(void);
int foo(int x) { return x+bar(); }

  gcc 5.3 -O3
    sub     esp, 12                   # align the stack for another call
    call    bar
    add     eax, DWORD PTR [esp+16]   # add our arg (from the stack) to bar()'s return value (in eax)
    add     esp, 12
    ret

See this on the Godbolt compiler explorer, where you can try different compilers and options.

-mincoming-stack-boundary=3 (or less) causes the stack-alignment boilerplate to be added to every function (not just main). -mstackrealign and -mno-stackrealign has no effect on foo() or main(), with or without a small -mincoming-stack-boundary. Based on the documentation in the gcc manual, I thought it would enable or disable the alignment stuff for functions other than main, or for main.

---

The 32bit x86 SysV ABI used to only guarantee 4-byte stack alignment, but the calling convention now guarantees 16-byte alignment of %esp before a call instruction.

Section 2.2.2 The Stack Frame

... In other words, the value (%esp + 4) is always a multiple of 16 when control is transferred to the function entry point. (32 when a 32-byte ymm vector is passed by value)

So gcc's -mpreferred-stack-boundary=4 is not just a good idea, it's the law (on systems like Linux where the updated ABI version that includes this stronger guarantee is the official standard). This makes it safe for functions other than main to omit that alignment step before using aligned SSE stores/loads to the stack. Those instructions (like movaps) will fault on unaligned addresses. So alignment is required for correctness, not just performance.

---

The AND instruction isn't actually needed these days

The current version of the 32bit ABI does guarantee that a freshly-execveed 32bit process will start with %esp 16-byte aligned. (Section 2.3.1 Initial Stack and Register State, see the bullet point about the initial state of %esp itself, not the stack contents.)

This means that gcc's behaviour of aligning the stack at the start of main is now obsolete, assuming the CRT startup code that calls main doesn't misalign the stack.

clang does assume the stack is aligned at the start of main, like 64bit gcc does.

16B-alignment was part of the x86-64 SysV ABI from the start, not added later, so it was always a safe assumption and there are no old kernels that don't provide that at process startup.

The x86 tag wiki has links to other ABIs, and much more.