Ok I must be overlooking something extremely simple but I am lost.
Given this
object val = -1;
var foo = (Int32)(val);
var bar = (Int64)(val);
This is because you can't unbox and perform a conversion in a single operation. You must unbox the Int32 value into an Int32, and then subsequently convert its type.
Because of that, this requires the object to be unboxed, then converted to Int64:
object val = -1;
int foo = (Int32)val;
Int64 bar = (Int64)(Int32)val;
Eric Lippert covered this in detail on his blog post titled Representation and Identity.