Syscall or sysenter on 32 bits Linux?

Question

Since MS‑DOS, I know system invocation using interrupts. In old papers, I saw reference to int 80h to invoke system functions on Linux. Since a rather long time now

Answer 1

After some web searching, I landed to this other topic on StackOverflow: Linux invoke a system call via sysenter tutorial. It says the recommended way to invoke the system, is neither using int 80h nor syscall nor sysenter, but linux-gate.so.

Still remains the question about the crash and core‑dump. My guess is finally that although either syscall or sysenter instructions are available as a CPU instruction, may be the Linux kernel just does not set‑up properly this “entry point” when it decide it's not really useful on a given hardware platform.

Seems on 32 bits platform, sysenter or syscall may be available, while it's always available, only on 64 bits platform.

Although I feel this answer my question, I still welcome more material, like an authoritative reference for my above guess.

-- update --

At least, I could find this which confirm the above. That's still not an authoritative reference but seems trustable enough I believe.

What is linux-gate.so.1?, says:

The preferred way of invoking a system call is determined by the kernel at boot time, and evidently this box uses sysenter.

Also, from another source, a sample FASM assembly source (needs some translations if you use NASM), to call a system function via linux-gate.so: Finding linux-gate.so.1 in Assembly .

Answer 2

The Intel manual says that syscall is invalid in compatibility (32-bit) mode, so it should not be used by the kernel.

This seems to be an Intel-only restriction however: https://stackoverflow.com/a/29784932/895245 that AMD does not have, but certainly Linux has to support Intel :-)

sysenter appears to be the best way to do it today as it is faster than int 0x80, but it should be used indirectly through VDSO as explained at How to invoke a system call via sysenter in inline assembly (x86/amd64 linux)?