It's a compilation error in accordance with JLS §15.20.2:
If a cast of the RelationalExpression to the ReferenceType would be rejected as a compile-time error, then the instanceof relational expression likewise produces a compile-time error. In such a situation, the result of the instanceof expression could never be true.
RelationalExpression is the first operand of instanceof and ReferenceType is the second.
