2D Coordinate to 3D world coordinate

Question

I want to convert 2D Image coordinates to 3D world coordinates. I am using the ZED camera which is a stereo camera and the sdk shipped with it, provides the disparity map. H

Answer 1

As you already know the depth D in meters for each pixel, you don't need the baseline B between cameras (which you would need to compute the depth from the disparity value). In fact, D is already your searched Z coordinate.

The general formula for the pinhole camera model (assuming there is no distortion) is:

u = fx * (X / Z) + cx
v = fy * (Y / Z) + cy

So it is then straightforward to compute the 3D-coordinates:

X = Z / fx * (u - cx)
Y = Z / fy * (v - cy)
[Z = D]

Note that this only is correct if you are working with a rectified image (or an image with low distortion).

Answer 2

Just stumbled upon this. I am also using the ZED camera.

Just FYI for anyone interesting, the ZED APIs (SDK v. 1.2) provide a nice function for this: you can simply retrieve the XYZ map of all points on the image using

sl::zed::Mat xyz = zed->retrieveMeasure(sl::zed::XYZABGR);

Note sure if this is what you were after or not, but definitely something useful :)