Can branch prediction cause illegal instruction?

Question

In the following pseudo-code:

if (rdtscp supported by hardware) {
    Invoke "rdtscp" instruction
} else {
    Invoke "rdtsc" instruction
}

Let's say the CPU does not support the rdtscp instruction and so we fallback to the else statement.

If CPU mispredicts the branch, is it possible for the instruction pipeline to try to execute rdtscp and throw an Illgal Instruction error?

Answer 1

It is explicitly documented for the #UD trap (Invalid Opcode Execution) in the Intel Processor Manuals, Volume 3A, chapter 6.15:

In Intel 64 and IA-32 processors that implement out-of-order execution microarchitectures, this exception is not generated until an attempt is made to retire the result of executing an invalid instruction; that is, decoding and speculatively attempting to execute an invalid opcode does not generate this exception. Likewise, in the Pentium processor and earlier IA-32 processors, this exception is not generated as the result of prefetching and preliminary decoding of an invalid instruction.

Answer 2

Instruction traps, such as the "Illegal instruction" trap, take effect when the instruction is executed, not before. Programs often depend on what state the program is at the moment the trap is thrown, so executing them early would be a serious bug in the architecture.

EDIT: okay, okay. From the Intel software developer's manual, volume 3A:

The ability of a P6 family processor to speculatively execute instructions does not affect the taking of interrupts by the processor. Interrupts are taken at instruction boundaries located during the retirement phase of instruction execution; so they are always taken in the “in-order” instruction stream.

Answer 3

It depends on how you define "cause". Can it have some observable effect? yes.

As Hans and Sneftel pointed out - exceptions, interrupts, traps, etc.. are all taking place at retirement, to which a wrong-path code would never reach. So a mispredicted branch can't cause a commit / retirement of wrong path instruction, including any illegal opcodes they may attempt to execute.

However, it is possible for a mispredicted code path to affect the micro architectural state in more subtle ways, you may have loads performed and cached, page walks done, and various other micro architectural events - in short, all the hard work your CPU does while you carelessly go and mispredict some branch (which is quite a lot in OOO CPUs). I highly doubt that any CPU out there is going to go and fish all this stuff out of your caches/TLBs, so take into account that these things can later have an effect on the performance of your program (for good or for worse), just not on its functional behavior.

It's probably not worthwhile if you're in the business of optimizing performance (and probably too complicated to make any use of), but it may be relevant to consider if you're in the business of securing your code - some hackers may have ways to sniff such data from your caches.