How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and

Answer 1

To follow up on Rons approach: You can find your z-values if you know how you've rotated your rectangle.

The trick is to find the projective matrix that did the projection. Fortunately this is possible and even cheap to do. The relevant math can be found in the paper "Projective Mappings for Image Warping" by Paul Heckbert.

http://pages.cs.wisc.edu/~dyer/cs766/readings/heckbert-proj.pdf

This way you can recover the homogenous part of each vertex back that was lost during projection.

Now you're still left with four lines instead of points (as Ron explained). Since you know the size of your original rectangle however nothing is lost. You can now plug the data from Ron's method and from the 2D approach into a linear equation solver and solve for z. You get the exact z-values of each vertex that way.

Note: This just works because:

1. The original shape was a rectangle
2. You know the exact size of the rectangle in 3D space.

It's a special case really.

Hope it helps, Nils