The relation between privileged instructions, traps and system calls

Question

I am trying to understand how a virtual machine monitor (VMM) virtualizes the CPU.

My understanding right now is that the CPU issues a protection fault interrupt when a

Answer 1

I'm not an expert on computer architecture. But I have several opinions for your consideration:

1. The CPU has two kinds of instructions
   • normal instructions, e.g., add, sub, etc.
   • privileged instructions, e.g., initiate I/O, load/store from protected memory etc.
2. The machine (CPU) has two kinds of modes (set by status bit in a protected register):
   • user mode: processor executes normal instructions in the user’s program
   • kernel mode: processor executes both normal and privileged instructions (OS == kernel)
3. Operating systems hide privileged instructions as system calls. And if user program calls them, it will cause an exception (throws a software interrupt), which vectors to a kernel handler, trap to kernel modes and switch contexts.
4. Upon encountering a privileged instruction in user mode, processor trap to kernel mode. Depending on what happened it would be one of several traps, such as a memory access violation, an illegal instruction violation, or a register access violation. The trap switches the processor’s execution to kernel mode and switches control to the operating system, which then decides on a course of action. The address is defined by the trap vector, which is set up when the operating system starts up.

Answer 2

In no particular order:

Your confusion is mainly caused by the fact that the operating systems community does not have standardized vocabulary. Here are some terms that get slung around that sometimes mean the same thing, sometimes not: exception, fault, interrupt, system call, and trap. Any individual author will generally use the terms consistently, but different authors define them differently.

There are 3 different kinds of events that cause entry into privileged mode.

1. An asynchronous interrupt (caused, for example, by an i/o device needing service.)
2. A system call instruction (int on the x86). (More generally in the x86 manuals these are called traps and include a couple of other instructions (for debuggers mostly.))
3. An instruction that does something exceptional (illegal instruction, protection fault, divide-by-0, page fault, ...). (Different authors calls these exceptions, faults or traps. x86 manuals call these faults.)

Each interrupt, trap or fault has a different number associated with it.

In all cases:

1. The processor enters privileged mode.
2. The user-mode registers are saved somewhere.
3. The processor finds the base address of the interrupt vector table, and uses the interrupt/trap/fault number as an offset into the table. This gives a pointer to the service routine for that interrupt/trap/fault.
4. The processor jumps to the service routine. Now we are in protected mode, the user level state is all saved somewhere we can get at it, and we're in the correct code inside the operating system.
5. When the service routine is finished it calls an interrupt-return instruction (iret on x86.) (This is the subtle distinction between a fault and a trap on x86: faults return to the instruction that caused the fault, traps return to the instruction after the trap.)

Note the confusing name "interrupt vector table." Even though it is called an interrupt table, it is used for faults and traps as well. (Which leads some authors to call everything an interrupt.)

The popf issue is rather subtle. This is essentially a bug in the x86 architecture. When popf executes from user mode it does not cause a trap or fault (or exception or interrupt or whatever you want to call it.) It simply acts as a noop.

Does this matter? Well, for a normal OS it doesn't really matter. If, on the other hand, you are implementing a virtual machine monitor (like VMWare or Xen or Hyper-V), the VMM is running in protected mode, and you'd like to run the guest operating systems in user mode and efficiently emulate any protected mode code. When the guest operating system uses a popf instruction you want it to generate a general protection fault, but it doesn't. (The cli and sti instructions do generate a general protection fault if called from user mode, which is what you want.)