OpenCV warping image based on calcOpticalFlowFarneback

Question

I'm trying to perform a complex warp of an image using Dense Optical Flow (I am trying to wap the second image into roughly the same shape as the first image). I'm probably getting this all wrong but Ill post up what I've tried:

cv::Mat flow;
cv::calcOpticalFlowFarneback( mGrayFrame1, mGrayFrame2, flow, 0.5, 3, 15, 3, 5, 1.2, 0 );

cv::Mat newFrame = cv::Mat::zeros( frame.rows, frame.cols, frame.type() );
cv:remap( frame, newFrame, flow, cv::Mat(), CV_INTER_LINEAR );

The idea if that I am calculating the flow mat from 2 gray scale frames. I get back a flow mat which seems to make sense but now I'm trying to remap my original (ie non-grayscale) image using this flow information.

I assumed that the remap function is what I want but I get a very badly distorted image from it. None of my colour information survives at all. I simply end up with an orange and black image that bears a small resemblance to my original image.

I'm guessing that I'm misunderstanding the remap function and it either doesn't do what I want it to do or I'm getting something wrong with what I'm passing in.

If the former has anyone got any suggestions as to how I can solve this problem? If the latter, what am i getting wrong?

Any help hugely appreciated!

Answer 1

The remap function can't work with optical flow. The function remap transforms the source image using the specified map:

dst(x, y) = src(mapx(x, y), mapy(x, y))

Optical flow has another formula:

frame1(x, y) = frame2(x + flowx(x, y), y + flowy(x, y))

So to use remap function first you need to create a map from flow:

Mat flow; // backward flow
calcOpticalFlowFarneback(nextFrame, prevFrame, flow);

Mat map(flow.size(), CV_32FC2);
for (int y = 0; y < map.rows; ++y)
{
    for (int x = 0; x < map.cols; ++x)
    {
        Point2f f = flow.at<Point2f>(y, x);
        map.at<Point2f>(y, x) = Point2f(x + f.x, y + f.y);
    }
}

Mat newFrame;
remap(prevFrame, newFrame, map);