Compute coordinates from source images after stitching

Question

I use an algorithm of panorama stitching from opencv, in order to stitch 2 or 3 images into one new result image.

I have coordinates of points in each source image. I need to calculate what are the new coordinates for these points in the result image.

I describe below the algorithm. My code is similar to a sample "stitching_detailed" from opencv (branch 3.4). A result_mask of type Mat is produced, maybe it is the solution? But I don't know how to use it. I found a related question here but not on stitching.

Any idea?

Here is the algorithm (for detailed code: stitching_detailed.cpp):

Find features for each image:

Ptr<FeaturesFinder> finder = makePtr<SurfFeaturesFinder>()
vector<ImageFeatures> features(num_images);
for (int i = 0; i < num_images; ++i)
{
  (*finder)(images[i], features[i]);
}

Make pairwise_matches:

vector<MatchesInfo> pairwise_matches;
Ptr<FeaturesMatcher> matcher = makePtr<BestOf2NearestMatcher>(false, match_conf);
(*matcher)(features, pairwise_matches);

Reorder the images:

vector<int> indices = leaveBiggestComponent(features, pairwise_matches, conf_thresh);
# here some code to reorder 'images'

Estimate an homography in cameras:

vector<CameraParams> cameras;
Ptr<Estimator> estimator = makePtr<HomographyBasedEstimator>();
(*estimator)(features, pairwise_matches, cameras);

Convert to CV_32F:

for (size_t i = 0; i < cameras.size(); ++i)
{
  Mat R;
  cameras[i].R.convertTo(R, CV_32F);
  cameras[i].R = R;
}

Execute a BundleAdjuster:

Ptr<detail::BundleAdjusterBase> adjuster = makePtr<detail::BundleAdjusterRay>();
adjuster->setConfThresh(conf_thresh);
adjuster->setRefinementMask(refine_mask);
(*adjuster)(features, pairwise_matches, cameras);

Compute a value for warped_image_scale:

for (int i = 0; i < cameras.size(); ++i)
  focals.push_back(cameras[i].focal);
float warped_image_scale = static_cast<float>(focals[focals.size() / 2 - 1] + focals[focals.size() / 2]) * 0.5f;

Do wave correction:

vector<Mat> rmats;
for (size_t i = 0; i < cameras.size(); ++i)
  rmats.push_back(cameras[i].R.clone());
waveCorrect(rmats, wave_correct);
for (size_t i = 0; i < cameras.size(); ++i)
  cameras[i].R = rmats[i];

Create a warper:

Ptr<WarperCreator> warper_creator = makePtr<cv::SphericalWarper>();
Ptr<RotationWarper> warper = warper_creator->create(static_cast<float>(warped_image_scale * seam_work_aspect));

Create a blender and feed it:

Ptr<Blender> blender;

for (size_t i = 0; i < cameras.size(); ++i)
{
  full_img = input_imgs[img_idx];
  if (!is_compose_scale_set)
  {
    is_compose_scale_set = true;
    compose_scale = /* … */
  }
  if (abs(compose_scale - 1) > 1e-1)
    resize(full_img, img, Size(), compose_scale, compose_scale, INTER_LINEAR_EXACT);
  else
    img = full_img;

  // Warp the current image
  warper->warp(img, K, cameras[img_idx].R, INTER_LINEAR, BORDER_REFLECT, img_warped);

  // Warp the current image mask
  mask.create(img_size, CV_8U);
  mask.setTo(Scalar::all(255));
  warper->warp(mask, K, cameras[img_idx].R, INTER_NEAREST, BORDER_CONSTANT, mask_warped);

  // Compensate exposure
  compensator->apply(img_idx, corners[img_idx], img_warped, mask_warped);

  dilate(masks_warped[img_idx], dilated_mask, Mat());
  resize(dilated_mask, seam_mask, mask_warped.size(), 0, 0, INTER_LINEAR_EXACT);
  mask_warped = seam_mask & mask_warped;

  if (!blender)
  {
    blender = Blender::createDefault(blend_type, try_gpu);
    Size dst_sz = resultRoi(corners, sizes).size();
    float blend_width = sqrt(static_cast<float>(dst_sz.area())) * blend_strength / 100.f;
    MultiBandBlender *mb = dynamic_cast<MultiBandBlender *>(blender.get());
    mb->setNumBands(static_cast<int>(ceil(log(blend_width) / log(2.)) - 1.));
    blender->prepare(corners, sizes);
  }

  // Blend the current image
  blender->feed(img_warped_s, mask_warped, corners[i]);
}

Then, use the blender:

Mat result, result_mask;
blender->blend(result, result_mask);
// The result image is in 'result'

Answer 1

When I was a school boy, I foundopencv/samples/cpp/stitching_detailed.cpp in OpenCV samples folder. At that time, my programming skills were very poor. I can't understand it even though I racked my brains. This question attracts my attention, and arouses my memory. After a whole night of hard work and debugging, I finally get it.

---

Basic steps:

1. Given the three images: blue.png, green.png, and red.png

2. We can get the stitching result(result.png) using the stitching_detailed.cpp. .

blender->blend(result, result_mask);
imwrite("result.png", result);
imwrite("result_mask.png", result_mask);

3. I choose the centers from the three images, and calculate the corresponding coordinates (warped) on the stitching image, and draw in solid as follow:

---

Warping images (auxiliary)...
Compensating exposure...

Blending ...

Warp each center point, and draw solid circle.
[408, 204] => [532, 224]
[408, 204] => [359, 301]
[408, 204] => [727, 320]

Check `result.png`, `result_mask.png` and `result2.png`!

Done!

---

This is the function calcWarpedPoint I wrote to calculate the warped point on the stitching image:

cv::Point2f calcWarpedPoint(
    const cv::Point2f& pt,
    InputArray K,                // Camera K parameter             
    InputArray R,                // Camera R parameter                
    Ptr<RotationWarper> warper,  // The Rotation Warper    
    const std::vector<cv::Point> &corners,
    const std::vector<cv::Size> &sizes)
{
    // Calculate the wrapped point using camera parameter.
    cv::Point2f  dst = warper->warpPoint(pt, K, R);

    // Calculate the stitching image roi using corners and sizes.
    // the corners and sizes have already been calculated.
    cv::Point2f  tl = cv::detail::resultRoi(corners, sizes).tl();

    // Finally adjust the wrapped point to the stitching image.
    return cv::Point2f(dst.x - tl.x, dst.y - tl.y);
}

---

This is example code snippet：

std::cout << "\nWarp each center point, and draw solid circle.\n";
std::vector<cv::Scalar> colors = { {255,0,0}, {0, 255, 0}, {0, 0, 255} };
for (int idx = 0; idx < img_names.size(); ++idx) {
    img = cv::imread(img_names[idx]);
    Mat K;
    cameras[idx].K().convertTo(K, CV_32F);
    Mat R = cameras[idx].R;

    cv::Point2f cpt = cv::Point2f(img.cols / 2, img.rows / 2);
    cv::Point pt = calcWarpedPoint(cpt, K, R, warper, corners, sizes);
    cv::circle(result, pt, 5, colors[idx], -1, cv::LINE_AA);
    std::cout << cpt << " => " << pt << std::endl;
}

std::cout << "\nCheck `result.png`, `result_mask.png` and `result2.png`!\n";
imwrite("result2.png", result);

---

The full code:

/*
* Author   : Kinght-金(https://stackoverflow.com/users/3547485/)
* Created  : 2019/03/01 23:00 (CST)
* Finished : 2019/03/01 07:50 (CST)
*
* Modified on opencv401/samples/cpp/stitching_detailed.cpp
* From  https://github.com/opencv/opencv/blob/4.0.1/samples/cpp/stitching_detailed.cpp
*
* 
* Description: A simple opencv(4.0.1) image stitching code for Stack Overflow answers.
* For https://stackoverflow.com/questions/54904718/compute-coordinates-from-source-images-after-stitching/54953792#comment96681412_54953792
*
*/

#include <iostream>
#include <fstream>
#include <string>
#include "opencv2/opencv_modules.hpp"
#include <opencv2/core/utility.hpp>
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/stitching/detail/autocalib.hpp"
#include "opencv2/stitching/detail/blenders.hpp"
#include "opencv2/stitching/detail/camera.hpp"
#include "opencv2/stitching/detail/exposure_compensate.hpp"
#include "opencv2/stitching/detail/matchers.hpp"
#include "opencv2/stitching/detail/motion_estimators.hpp"
#include "opencv2/stitching/detail/seam_finders.hpp"
#include "opencv2/stitching/detail/warpers.hpp"
#include "opencv2/stitching/warpers.hpp"

using namespace std;
using namespace cv;
using namespace cv::detail;

//! img_names are the input image (full) paths
// You can download from using the links from the answer.
//! Blue: https://i.stack.imgur.com/Yz3U1.png
//! Green: https://i.stack.imgur.com/AbUTH.png
//! Red: https://i.stack.imgur.com/9wcGc.png
vector<String> img_names = {"D:/stitching/blue.png", "D:/stitching/green.png", "D:/stitching/red.png"};

//! The function to calculate the warped point on the stitching image.
cv::Point2f calcWarpedPoint(
    const cv::Point2f& pt,
    InputArray K,                // Camera K parameter
    InputArray R,                // Camera R parameter
    Ptr<RotationWarper> warper,  // The Rotation Warper
    const std::vector<cv::Point> &corners,
    const std::vector<cv::Size> &sizes)
{
    // Calculate the wrapped point
    cv::Point2f  dst = warper->warpPoint(pt, K, R);

    // Calculate the stitching image roi using corners and sizes,
    // the corners and sizes have already been calculated.
    cv::Point2f  tl = cv::detail::resultRoi(corners, sizes).tl();

    // Finally adjust the wrapped point
    return cv::Point2f(dst.x - tl.x, dst.y - tl.y);
}


int main(int argc, char* argv[])
{
    double work_megapix = 0.6;
    double seam_megapix = 0.1;
    double compose_megapix = -1;
    float conf_thresh = 1.f;
    float match_conf = 0.3f;
    float blend_strength = 5;


    // Check if have enough images
    int num_images = static_cast<int>(img_names.size());
    if (num_images < 2)
    {
        std::cout << "Need more images\n";
        return -1;
    }

    double work_scale = 1, seam_scale = 1, compose_scale = 1;
    bool is_work_scale_set = false, is_seam_scale_set = false, is_compose_scale_set = false;

    //(1) 创建特征查找器
    Ptr<Feature2D> finder = ORB::create();

    // (2) 读取图像，适当缩放，并计算图像的特征描述
    Mat full_img, img;
    vector<ImageFeatures> features(num_images);
    vector<Mat> images(num_images);
    vector<Size> full_img_sizes(num_images);
    double seam_work_aspect = 1;

    for (int i = 0; i < num_images; ++i)
    {
        full_img = imread(img_names[i]);
        full_img_sizes[i] = full_img.size();

        if (full_img.empty())
        {
            cout << "Can't open image " << img_names[i] << std::endl;
            return -1;
        }
        if (!is_work_scale_set)
        {
            work_scale = min(1.0, sqrt(work_megapix * 1e6 / full_img.size().area()));
            is_work_scale_set = true;
        }
        resize(full_img, img, Size(), work_scale, work_scale, INTER_LINEAR_EXACT);

        if (!is_seam_scale_set)
        {
            seam_scale = min(1.0, sqrt(seam_megapix * 1e6 / full_img.size().area()));
            seam_work_aspect = seam_scale / work_scale;
            is_seam_scale_set = true;
        }

        computeImageFeatures(finder, img, features[i]);
        features[i].img_idx = i;
        std::cout << "Features in image #" << i + 1 << ": " << features[i].keypoints.size() << std::endl;

        resize(full_img, img, Size(), seam_scale, seam_scale, INTER_LINEAR_EXACT);
        images[i] = img.clone();
    }

    full_img.release();
    img.release();


    // (3) 创建图像特征匹配器，计算匹配信息
    vector<MatchesInfo> pairwise_matches;
    Ptr<FeaturesMatcher>  matcher = makePtr<BestOf2NearestMatcher>(false, match_conf);
    (*matcher)(features, pairwise_matches);
    matcher->collectGarbage();

    //! (4) 剔除外点，保留最确信的大成分
    // Leave only images we are sure are from the same panorama
    vector<int> indices = leaveBiggestComponent(features, pairwise_matches, conf_thresh);
    vector<Mat> img_subset;
    vector<String> img_names_subset;
    vector<Size> full_img_sizes_subset;
    for (size_t i = 0; i < indices.size(); ++i)
    {
        img_names_subset.push_back(img_names[indices[i]]);
        img_subset.push_back(images[indices[i]]);
        full_img_sizes_subset.push_back(full_img_sizes[indices[i]]);
    }

    images = img_subset;
    img_names = img_names_subset;
    full_img_sizes = full_img_sizes_subset;

    // Check if we still have enough images
    num_images = static_cast<int>(img_names.size());
    if (num_images < 2)
    {
        std::cout << "Need more images\n";
        return -1;
    }

    //!(5) 估计 homography
    Ptr<Estimator> estimator = makePtr<HomographyBasedEstimator>();
    vector<CameraParams> cameras;
    if (!(*estimator)(features, pairwise_matches, cameras))
    {
        cout << "Homography estimation failed.\n";
        return -1;
    }

    for (size_t i = 0; i < cameras.size(); ++i)
    {
        Mat R;
        cameras[i].R.convertTo(R, CV_32F);
        cameras[i].R = R;
        std::cout << "\nInitial camera intrinsics #" << indices[i] + 1 << ":\nK:\n" << cameras[i].K() << "\nR:\n" << cameras[i].R << std::endl;
    }

    //(6) 创建约束调整器
    Ptr<detail::BundleAdjusterBase> adjuster = makePtr<detail::BundleAdjusterRay>();
    adjuster->setConfThresh(conf_thresh);
    Mat_<uchar> refine_mask = Mat::zeros(3, 3, CV_8U);
    refine_mask(0, 0) = 1;
    refine_mask(0, 1) = 1;
    refine_mask(0, 2) = 1;
    refine_mask(1, 1) = 1;
    refine_mask(1, 2) = 1;
    adjuster->setRefinementMask(refine_mask);
    if (!(*adjuster)(features, pairwise_matches, cameras))
    {
        cout << "Camera parameters adjusting failed.\n";
        return -1;
    }

    // Find median focal length
    vector<double> focals;
    for (size_t i = 0; i < cameras.size(); ++i)
    {
        focals.push_back(cameras[i].focal);
    }

    sort(focals.begin(), focals.end());
    float warped_image_scale;
    if (focals.size() % 2 == 1)
        warped_image_scale = static_cast<float>(focals[focals.size() / 2]);
    else
        warped_image_scale = static_cast<float>(focals[focals.size() / 2 - 1] + focals[focals.size() / 2]) * 0.5f;


    std::cout << "\nWarping images (auxiliary)... \n";

    vector<Point> corners(num_images);
    vector<UMat> masks_warped(num_images);
    vector<UMat> images_warped(num_images);
    vector<Size> sizes(num_images);
    vector<UMat> masks(num_images);

    // Preapre images masks
    for (int i = 0; i < num_images; ++i)
    {
        masks[i].create(images[i].size(), CV_8U);
        masks[i].setTo(Scalar::all(255));
    }

    // Warp images and their masks
    Ptr<WarperCreator> warper_creator = makePtr<cv::CylindricalWarper>();
    if (!warper_creator)
    {
        cout << "Can't create the warper \n";
        return 1;
    }

    //! Create RotationWarper
    Ptr<RotationWarper> warper = warper_creator->create(static_cast<float>(warped_image_scale * seam_work_aspect));

    //! Calculate warped corners/sizes/mask
    for (int i = 0; i < num_images; ++i)
    {
        Mat_<float> K;
        cameras[i].K().convertTo(K, CV_32F);
        float swa = (float)seam_work_aspect;
        K(0, 0) *= swa; K(0, 2) *= swa;
        K(1, 1) *= swa; K(1, 2) *= swa;
        corners[i] = warper->warp(images[i], K, cameras[i].R, INTER_LINEAR, BORDER_REFLECT, images_warped[i]);
        sizes[i] = images_warped[i].size();
        warper->warp(masks[i], K, cameras[i].R, INTER_NEAREST, BORDER_CONSTANT, masks_warped[i]);
    }

    vector<UMat> images_warped_f(num_images);
    for (int i = 0; i < num_images; ++i)
        images_warped[i].convertTo(images_warped_f[i], CV_32F);

    std::cout << "Compensating exposure... \n";

    //! 计算曝光度，调整图像曝光，减少亮度差异
    Ptr<ExposureCompensator> compensator = ExposureCompensator::createDefault(ExposureCompensator::GAIN_BLOCKS);
    if (dynamic_cast<BlocksCompensator*>(compensator.get()))
    {
        BlocksCompensator* bcompensator = dynamic_cast<BlocksCompensator*>(compensator.get());
        bcompensator->setNrFeeds(1);
        bcompensator->setNrGainsFilteringIterations(2);
        bcompensator->setBlockSize(32, 32);
    }

    compensator->feed(corners, images_warped, masks_warped);

    Ptr<SeamFinder> seam_finder = makePtr<detail::GraphCutSeamFinder>(GraphCutSeamFinderBase::COST_COLOR);
    seam_finder->find(images_warped_f, corners, masks_warped);

    // Release unused memory
    images.clear();
    images_warped.clear();
    images_warped_f.clear();
    masks.clear();

    Mat img_warped, img_warped_s;
    Mat dilated_mask, seam_mask, mask, mask_warped;
    Ptr<Blender> blender;
    double compose_work_aspect = 1;

    for (int img_idx = 0; img_idx < num_images; ++img_idx)
    {
        // Read image and resize it if necessary
        full_img = imread(img_names[img_idx]);
        if (!is_compose_scale_set)
        {
            is_compose_scale_set = true;
            compose_work_aspect = compose_scale / work_scale;

            // Update warped image scale
            warped_image_scale *= static_cast<float>(compose_work_aspect);
            warper = warper_creator->create(warped_image_scale);

            // Update corners and sizes
            for (int i = 0; i < num_images; ++i)
            {
                cameras[i].focal *= compose_work_aspect;
                cameras[i].ppx *= compose_work_aspect;
                cameras[i].ppy *= compose_work_aspect;

                Size sz = full_img_sizes[i];
                if (std::abs(compose_scale - 1) > 1e-1)
                {
                    sz.width = cvRound(full_img_sizes[i].width * compose_scale);
                    sz.height = cvRound(full_img_sizes[i].height * compose_scale);
                }

                Mat K;
                cameras[i].K().convertTo(K, CV_32F);
                Rect roi = warper->warpRoi(sz, K, cameras[i].R);

                corners[i] = roi.tl();
                sizes[i] = roi.size();
            }
        }

        if (abs(compose_scale - 1) > 1e-1)
            resize(full_img, img, Size(), compose_scale, compose_scale, INTER_LINEAR_EXACT);
        else
            img = full_img;
        full_img.release();
        Size img_size = img.size();

        Mat K, R;
        cameras[img_idx].K().convertTo(K, CV_32F);
        R = cameras[img_idx].R;

        // Warp the current image : img => img_warped
        warper->warp(img, K, cameras[img_idx].R, INTER_LINEAR, BORDER_REFLECT, img_warped);

        // Warp the current image mask
        mask.create(img_size, CV_8U);
        mask.setTo(Scalar::all(255));
        warper->warp(mask, K, cameras[img_idx].R, INTER_NEAREST, BORDER_CONSTANT, mask_warped);

        compensator->apply(img_idx, corners[img_idx], img_warped, mask_warped);
        img_warped.convertTo(img_warped_s, CV_16S);
        img_warped.release();
        img.release();
        mask.release();

        dilate(masks_warped[img_idx], dilated_mask, Mat());
        resize(dilated_mask, seam_mask, mask_warped.size(), 0, 0, INTER_LINEAR_EXACT);
        mask_warped = seam_mask & mask_warped;

        if (!blender)
        {
            blender = Blender::createDefault(Blender::MULTI_BAND, false);
            Size dst_sz = resultRoi(corners, sizes).size();
            float blend_width = sqrt(static_cast<float>(dst_sz.area())) * blend_strength / 100.f;
            if (blend_width < 1.f){
                blender = Blender::createDefault(Blender::NO, false);
            }
            else
            {
                MultiBandBlender* mb = dynamic_cast<MultiBandBlender*>(blender.get());
                mb->setNumBands(static_cast<int>(ceil(log(blend_width) / log(2.)) - 1.));
            }
            blender->prepare(corners, sizes);
        }

        blender->feed(img_warped_s, mask_warped, corners[img_idx]);
    }

    /* ===========================================================================*/
    // Blend image
    std::cout << "\nBlending ...\n";
    Mat result, result_mask;
    blender->blend(result, result_mask);
    imwrite("result.png", result);
    imwrite("result_mask.png", result_mask);

    std::cout << "\nWarp each center point, and draw solid circle.\n";
    std::vector<cv::Scalar> colors = { {255,0,0}, {0, 255, 0}, {0, 0, 255} };
    for (int idx = 0; idx < img_names.size(); ++idx) {
        img = cv::imread(img_names[idx]);
        Mat K;
        cameras[idx].K().convertTo(K, CV_32F);
        Mat R = cameras[idx].R;

        cv::Point2f cpt = cv::Point2f(img.cols / 2, img.rows / 2);
        cv::Point pt = calcWarpedPoint(cpt, K, R, warper, corners, sizes);
        cv::circle(result, pt, 5, colors[idx], -1, cv::LINE_AA);
        std::cout << cpt << " => " << pt << std::endl;
    }

    std::cout << "\nCheck `result.png`, `result_mask.png` and `result2.png`!\n";
    imwrite("result2.png", result);

    std::cout << "\nDone!\n";
    /* ===========================================================================*/

    return 0;
}

---

Some links maybe useful:

1. stitching_detailed.cpp : https://github.com/opencv/opencv/blob/4.0.1/samples/cpp/stitching_detailed.cpp

2. waper->warp(), warpPoint(), warpRoi() https://github.com/opencv/opencv/blob/master/modules/stitching/src/warpers.cpp#L153

3. resultRoi() https://github.com/opencv/opencv/blob/master/modules/stitching/src/util.cpp#L116

---

Other links maybe interesting:

1. Converting opencv remap code from c++ to python

2. Split text lines in scanned document

3. How do I use the relationships between Flann matches to determine a sensible homography?