I would like to ask if it is GRUB that switch the CPU to protected mode during boot up or is it the Linux kernel that does it. And also I would like to ask - is the kernel itself (vmlinuz) an ELF or is it plain binary format? Thanks.
GRUB does drop you in protected mode.
The GRUB Multiboot Specification (version 0.6.96) Section 3.2 tells you this
‘CR0’
Bit 31 (PG) must be cleared. Bit 0 (PE) must be set. Other bits are all undefined.
And CR0 Register mapping tells you that the system should be in protected mode.
Linux is not a multiboot kernel and does not rely on some bootloader for switching in the PM, it follows the Linux Boot Protocol. But linux does the protected mode switch itself, and does not rely in the bootloader
check : http://lxr.linux.no/#linux+v2.6.39/arch/x86/boot/main.c
Here it calls go_to_protected_mode(); when then calls protected_mode_jump () which then does the CR0 stuff (sets the bit 0)
(The other bit says that paging is disabled)
EDIT
What i can figure out is that GRUB can detect linux boot protocol (GRUB2, and legacy should also) and loads linux on memory, but does not switch to protected mode. Have a look at this link: http://www.gnu.org/software/grub/manual/grub.html#GNU_002fLinux and section 16 of the page in this link.
According to http://www.moses.uklinux.net/patches/lki-1.html Linux kernel is an ELF binary. I'm pretty sure that it is Linux that switches to protected mode, not the boot loader. This page agrees: http://oss.sgi.com/LDP/HOWTO/Kernel-HOWTO/linux_boot_process.html
I would suggest that the answer here is that GRUB loader switches to "unreal mode" before loading the kernel (which explains why CR0 bit 0 isn't set at that point.) Full 32 bit addressing is enabled, a flat GDT is set up for [0,4Gigs>, the A20 line is enabled, but paging isn't enabled so the Linux kernel (or any other kernel) will still have to do that and switch back to protected mode.
来源：https://stackoverflow.com/questions/4821911/does-grub-switch-to-protected-mode