Simple solution that works in N dimensions for convex polyhedra, of which a 2-dimensional rectangle is a special case:
- Represent the polyhedron as the intersection of half-spaces, each defined by a unit normal vector and the distance of the surface hyperplane from the origin along the normal.
- For each of these half-spaces, take the dot product of point in question with the defining normal vector. The point is in the half-space if and only if the dot product is less than [or equal to] the defining distance.
- The point is inside the polyhedron if and only if it's in every one of the half-spaces.
For a rectangle defined as a counter-clockwise sequence of edges, step 1 amounts to rotating the edges each by 90 degrees clockwise to get the normals, then intersecting the normal line with the line containing the edge to find the distance to the origin.
Assuming step 1 is complete, testing a point takes at most 8 multiplications, 4 additions, and 4 comparisons.
If you want to, you can optimize the process a bit since you have rectangles (and thus opposite sides have opposite normals). Now you're only looking at 2 normals rather than 4, and a range of dot product values which indicate points that lie between the opposite sides. So now you're down to 4 multiplications, 2 additions, and 4 comparisons.
You can also get lucky if the first test you make shows that the point is outside the rectangle, in which case it's just 2 multiplications, 1 addition, and 1-2 comparisons.
