order for encoding x86 instruction prefix bytes

Question

I know that x86 instructions can have a maximum of 4 bytes of prefixes, e.g Lock, rep, segment overrides etc.

Is there any particular order in which they should appear,

Answer 1

the order can be found in volume 2A of the Intel Software Developer's Manual.

In a nutshell:

• the F2 and F3 prefixes cancel each other out. The one that comes later has precedence.
• the 66 prefix is ignored if either F2 or F3 are used (as mandatory prefixes in a long instruction). This of course doesn't apply to rep movsw where both those prefixes are simply prefixes, not part of the opcode.
• the REX escape may not be followed by any other prefixes.
• the VEX escape may not be preceded by REX, 66, F2 or F3

for the rest, the order shouldn't matter.

Answer 2

The architecture volume of the intel developer manuals details the layout at lenght, however, from what I remember last time I read it, the order for most didn't matter, except the REX/REX.W prefix which must occupy the slot closest to the start of the actual instruction bytes (aka it takes the slot most on the right)

Answer 3

Quote from Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M

Instruction prefixes are divided into four groups, each with a set of allowable prefix codes. For each instruction, it is only useful to include up to one prefix code from each of the four groups (Groups 1, 2, 3, 4). Groups 1 through 4 may be placed in any order relative to each other.