Kinect: Converting from RGB Coordinates to Depth Coordinates

Question

I am using the Windows Kinect SDK to obtain depth and RGB images from the sensor.

Since the depth image and the RGB images do not align, I would like to find a way of converting the coordinates of the RGB image to that of the depth image, since I want to use an image mask on the depth image I have obtained from some processing on the RGB image.

There is already a method for converting depth coordinates to the color space coordinates:

NuiImageGetColorPixelCoordinatesFromDepthPixel

unfortunately, the reverse does not exist. There is only an arcane call in INUICoordinateMapper:

HRESULT MapColorFrameToDepthFrame(
     NUI_IMAGE_RESOLUTION eColorResolution,
     NUI_IMAGE_RESOLUTION eDepthResolution,
     DWORD cDepthPixels,
     NUI_DEPTH_IMAGE_PIXEL *pDepthPixels,
     DWORD cDepthPoints,
     NUI_DEPTH_IMAGE_POINT *pDepthPoints
)

How this method works is not very well documented. Has anyone used it before?

I'm on the verge of performing a manual calibration myself to calculate a transformation matrix, so I would be very happy for a solution.

Answer 1

Thanks to commenter horristic, I got a link to msdn with some useful information (thanks also to T. Chen over at msdn for helping out with the API). Extracted from T. Chen's post, here's the code that will perform the mapping from RGB to depth coordinate space:

INuiCoordinateMapper* pMapper;

mNuiSensor->NuiGetCoordinateMapper(&pMapper);

pMapper->MapColorFrameToDepthFrame(
        NUI_IMAGE_TYPE_COLOR,
        NUI_IMAGE_RESOLUTION_640x480,
        NUI_IMAGE_RESOLUTION_640x480,
        640 * 480, 
        (NUI_DEPTH_IMAGE_PIXEL*)LockedRect.pBits,
        640 * 480, 
        depthPoints);

Note: the sensor needs to be initialized and a depth frame locked for this to work.

The transformed coordinates can, e.g., be queried as follows:

/// transform RGB coordinate point to a depth coordinate point 
cv::Point TransformRGBtoDepthCoords(cv::Point rgb_coords, NUI_DEPTH_IMAGE_POINT *    depthPoints)
{
    long index = rgb_coords.y * 640 + rgb_coords.x;
    NUI_DEPTH_IMAGE_POINT depthPointAtIndex = depthPoints[index];
    return cv::Point(depthPointAtIndex.x, depthPointAtIndex.y); 
}

Answer 2

As far as I can tell, MapColorFrameToDepthFrame effectively runs the co-ordinate system conversion on every pixel of your RGB image, storing the depth image coordinates resulting from the conversion and the resultant depth value in the output NUI_DEPTH_IMAGE_POINT array. The definition of that structure is here: http://msdn.microsoft.com/en-us/library/nuiimagecamera.nui_depth_image_point.aspx

Possibly this is overkill for your needs however, and I've no idea how fast that method is. XBOX Kinect developers have a very fast implementation of that function that runs on the GPU at frame rate, Windows developers might not be quite so lucky!