Having some difficulty in image stitching using OpenCV

Question

I'm currently working on Image stitching using OpenCV 2.3.1 on Visual Studio 2010, but I'm having some trouble.

Problem Description I'm trying to write a code for stitching multiple images derived from a few cameras(about 3~4), i,e, the code should keep executing image stitching until I ask it to stop.

The following is what I've done so far: (For simplification, I'll replace some part of the code with just a few words)

     1.Reading frames(images) from 2 cameras (Currently I'm just working on 2 cameras.)
     2.Feature detection, descriptor calculation (SURF)
     3.Feature matching using FlannBasedMatcher
     4.Removing outliers and calculate the Homography with inliers using RANSAC.
     5.Warp one of both images.

For step 5., I followed the answer in the following thread and just changed some parameters: Stitching 2 images in opencv

However, the result is terrible though. I just uploaded the result onto youtube and of course only those who have the link will be able to see it.

http://youtu.be/Oy5z_7LeaMk

My code is shown below: (Only crucial parts are shown)

VideoCapture cam1, cam2;
cam1.open(0);
cam2.open(1);

while(1)
{
    Mat frm1, frm2;

    cam1 >> frm1;
    cam2 >> frm2;

   //(SURF detection, descriptor calculation 
   //and matching using FlannBasedMatcher)


    double max_dist = 0; double min_dist = 100;

    //-- Quick calculation of max and min distances between keypoints
    for( int i = 0; i < descriptors_1.rows; i++ )
    { 
        double dist = matches[i].distance;
        if( dist < min_dist ) min_dist = dist;
        if( dist > max_dist ) max_dist = dist;
    }


    (Draw only "good" matches 
    (i.e. whose distance is less than 3*min_dist ))            

    vector<Point2f> frame1;
    vector<Point2f> frame2;

    for( int i = 0; i < good_matches.size(); i++ )
    {
      //-- Get the keypoints from the good matches
      frame1.push_back( keypoints_1[ good_matches[i].queryIdx ].pt );
      frame2.push_back( keypoints_2[ good_matches[i].trainIdx ].pt ); 
    }
    Mat H = findHomography( Mat(frame1), Mat(frame2), CV_RANSAC );
    cout << "Homography: " << H << endl;


    /* warp the image */
    Mat warpImage2;
    warpPerspective(frm2, warpImage2, 
    H, Size(frm2.cols, frm2.rows), INTER_CUBIC); 

    Mat final(Size(frm2.cols*3 + frm1.cols, frm2.rows),CV_8UC3); 

    Mat roi1(final, Rect(frm1.cols, 0, frm1.cols, frm1.rows)); 
    Mat roi2(final, Rect(2*frm1.cols, 0, frm2.cols, frm2.rows)); 

    warpImage2.copyTo(roi2); 
    frm1.copyTo(roi1); 
    imshow("final", final); 

What else should I do to make the stitching better?

Besides, is it reasonable to make the Homography matrix fixed instead of keeping computing it ? What I mean is to specify the angle and the displacement between the 2 cameras by myself so as to derive a Homography matrix that satisfies what I want.

Thanks. :)

Answer 1

It sounds like you are going about this sensibly, but if you have access to both of the cameras, and they will remain stationary with respect to each other, then calibrating offline, and simply applying the transformation online will make your application more efficient.

One point to note is, you say you are using the findHomography function from OpenCV. From the documentation, this function:

Finds a perspective transformation between two planes.

However, your points are not restricted to a specific plane as they are imaging a 3D scene. If you wanted to calibrate offline, you could image a chessboard with both cameras, and the detected corners could be used in this function.

Alternatively, you may like to investigate the Fundamental matrix, which can be calculated with a similar function. This matrix describes the relative position of the cameras, but some work (and a good textbook) will be required to extract them.

If you can find it, I would strongly recommend having a look at Part II: "Two-View Geometry" in the book "Multiple View Geometry in computer vision", by Richard Hartley and Andrew Zisserman, which goes through the process in detail.

Answer 2

I have been working lately on image registration. My algorithm takes two images, calculates the SURF features, find correspondences, find homography matrix and then stitch both images together, I did it with the next code:

void stich(Mat base, Mat target,Mat homography, Mat& panorama){


Mat corners1(1, 4,CV_32F);
Mat corners2(1,4,CV_32F);
Mat corners(1,4,CV_32F);
vector<Mat> planes;
/* compute corners 
of warped image
*/
corners1.at<float>(0,0)=0;
corners2.at<float>(0,0)=0;
corners1.at<float>(0,1)=0;
corners2.at<float>(0,1)=target.rows;
corners1.at<float>(0,2)=target.cols;
corners2.at<float>(0,2)=0;
corners1.at<float>(0,3)=target.cols;
corners2.at<float>(0,3)=target.rows;

planes.push_back(corners1);
planes.push_back(corners2);

merge(planes,corners);

perspectiveTransform(corners, corners, homography);

/* compute size of resulting 
image and allocate memory
*/
double x_start = min( min( (double)corners.at<Vec2f>(0,0)[0], (double)corners.at<Vec2f> (0,1)[0]),0.0);
double x_end   = max( max( (double)corners.at<Vec2f>(0,2)[0], (double)corners.at<Vec2f>(0,3)[0]), (double)base.cols);
double y_start = min( min( (double)corners.at<Vec2f>(0,0)[1], (double)corners.at<Vec2f>(0,2)[1]), 0.0);
double y_end   = max( max( (double)corners.at<Vec2f>(0,1)[1], (double)corners.at<Vec2f>(0,3)[1]), (double)base.rows);

/*Creating image
with same channels, depth
as target
and proper size
*/
panorama.create(Size(x_end - x_start + 1, y_end - y_start + 1), target.depth());

planes.clear();

/*Planes should
have same n.channels
as target
*/
for (int i=0;i<target.channels();i++){
planes.push_back(panorama);
}

merge(planes,panorama);
    // create translation matrix in order to copy both images to correct places
Mat T;
T=Mat::zeros(3,3,CV_64F);
T.at<double>(0,0)=1;
T.at<double>(1,1)=1;
T.at<double>(2,2)=1;
T.at<double>(0,2)=-x_start;
T.at<double>(1,2)=-y_start;

// copy base image to correct position within output image

 warpPerspective(base, panorama, T,panorama.size(),INTER_LINEAR| CV_WARP_FILL_OUTLIERS);
 // change homography to take necessary translation into account
gemm(T, homography,1,T,0,T);
    // warp second image and copy it to output image
warpPerspective(target,panorama, T, panorama.size(),INTER_LINEAR);
 //tidy
corners.release();
T.release();

}

Any question I will try