Following the spirit of @Jerry Coffin: create segments from rectangle corners to the point in question. Solve the linear equations. Slope is tan(a). Sum up all seq arctangents diff, if it is 2*PI and each diff < PI - point is inside the rectangle.
Edit Probably enough just check for each sequential difference < Pi...
