iOS:Retrieve rectangle shaped image from the background image

Question

I am working on an implementation where I have a rectangle shaped image in an big background image. I am trying to programmatically retrieve the rectangle shaped image from

Answer 1

Here is a partial answer. It is not complete because I am attempting to do the exact same thing and experiencing huge difficulties every step of the way. My knowledge is quite strong on objective-c but really weak on C++

You should read this guide to wrapping c++

And everything on Ievgen Khvedchenia's Computer Vision Talks blog, especially the openCV tutorial. Ievgen has also posted an amazingly complete project on github to go with the tutorial.

Having said that, I am still having a lot of trouble getting openCV to compile and run smoothly.

For example, Ievgen's tutorial runs fine as a finished project, but if I try to recreate it from scratch I get the same openCV compile errors that have been plaguing me all along. It's probably my poor understanding of C++ and it's integration with obj-C.

Regarding squares.cpp

What you need to do

• remove int main(int /*argc*/, char** /*argv*/) from squares.cpp
• remove imshow(wndname, image); from drawSquares (obj-c will do the drawing)
• create a header file squares.h
• make one or two public functions in the header file which you can call from obj-c (or from an obj-c/c++ wrapper)

Here is what I have so far...

class squares
{
public:
         static cv::Mat& findSquares( const cv::Mat& image, cv::vector<cv::vector<cv::Point> >& squares );
         static cv::Mat& drawSquares( cv::Mat& image, const cv::vector<cv::vector<cv::Point> >& squares );

};

you should be able to reduce this to a single method, say processSquares with one input cv::Mat& image and one return cv::Mat& image. That method would declare squares and call findSquares and drawSquares within the .cpp file.

The wrapper will take an input UIImage, convert it to cv::Mat image, call processSquares with that input, and get a result cv::Mat image. That result it will convert back to NSImage and pass back to the objc calling function.

SO that's a neat sketch of what we need to do, I will try and expand this answer once I've actually managed to do any of it!

Answer 2

Here is a full answer using a small wrapper class to separate the c++ from objective-c code.

I had to raise another question on stackoverflow to deal with my poor c++ knowledge - but I have worked out everything we need to interface c++ cleanly with objective-c code, using the squares.cpp sample code as an example. The aim is to keep the original c++ code as pristine as possible, and to keep the bulk of the work with openCV in pure c++ files for (im)portability.

I have left my original answer in place as this seems to go beyond an edit. The complete demo project is on github

CVViewController.h / CVViewController.m

• pure Objective-C

• communicates with openCV c++ code via a WRAPPER... it neither knows nor cares that c++ is processing these method calls behind the wrapper.

CVWrapper.h / CVWrapper.mm

• objective-C++

does as little as possible, really only two things...

• calls to UIImage objC++ categories to convert to and from UIImage <> cv::Mat
• mediates between CVViewController's obj-C methods and CVSquares c++ (class) function calls

CVSquares.h / CVSquares.cpp

• pure C++
• CVSquares.cpp declares public functions inside a class definition (in this case, one static function).
  This replaces the work of main{} in the original file.
• We try to keep CVSquares.cpp as close as possible to the C++ original for portability.

CVViewController.m

//remove 'magic numbers' from original C++ source so we can manipulate them from obj-C
#define TOLERANCE 0.01
#define THRESHOLD 50
#define LEVELS 9

UIImage* image =
        [CVSquaresWrapper detectedSquaresInImage:self.image
                                       tolerance:TOLERANCE
                                       threshold:THRESHOLD
                                          levels:LEVELS];

CVSquaresWrapper.h

//  CVSquaresWrapper.h

#import <Foundation/Foundation.h>

@interface CVSquaresWrapper : NSObject

+ (UIImage*) detectedSquaresInImage:(UIImage*)image
                          tolerance:(CGFloat)tolerance
                          threshold:(NSInteger)threshold
                             levels:(NSInteger)levels;

@end

CVSquaresWrapper.mm

//  CVSquaresWrapper.mm
//  wrapper that talks to c++ and to obj-c classes

#import "CVSquaresWrapper.h"
#import "CVSquares.h"
#import "UIImage+OpenCV.h"

@implementation CVSquaresWrapper

+ (UIImage*) detectedSquaresInImage:(UIImage*) image
                          tolerance:(CGFloat)tolerance
                          threshold:(NSInteger)threshold
                             levels:(NSInteger)levels
{
    UIImage* result = nil;

        //convert from UIImage to cv::Mat openCV image format
        //this is a category on UIImage
    cv::Mat matImage = [image CVMat]; 


        //call the c++ class static member function
        //we want this function signature to exactly 
        //mirror the form of the calling method 
    matImage = CVSquares::detectedSquaresInImage (matImage, tolerance, threshold, levels);


        //convert back from cv::Mat openCV image format
        //to UIImage image format (category on UIImage)
    result = [UIImage imageFromCVMat:matImage]; 

    return result;
}

@end

CVSquares.h

//  CVSquares.h

#ifndef __OpenCVClient__CVSquares__
#define __OpenCVClient__CVSquares__

    //class definition
    //in this example we do not need a class 
    //as we have no instance variables and just one static function. 
    //We could instead just declare the function but this form seems clearer

class CVSquares
{
public:
    static cv::Mat detectedSquaresInImage (cv::Mat image, float tol, int threshold, int levels);
};

#endif /* defined(__OpenCVClient__CVSquares__) */

CVSquares.cpp

//  CVSquares.cpp

#include "CVSquares.h"

using namespace std;
using namespace cv;

static int thresh = 50, N = 11;
static float tolerance = 0.01;

    //declarations added so that we can move our 
    //public function to the top of the file
static void findSquares(  const Mat& image,   vector<vector<Point> >& squares );
static void drawSquares( Mat& image, vector<vector<Point> >& squares );

    //this public function performs the role of 
    //main{} in the original file (main{} is deleted)
cv::Mat CVSquares::detectedSquaresInImage (cv::Mat image, float tol, int threshold, int levels)
{
    vector<vector<Point> > squares;

    if( image.empty() )
        {
        cout << "Couldn't load " << endl;
        }

    tolerance = tol;
    thresh = threshold;
    N = levels;
    findSquares(image, squares);
    drawSquares(image, squares);

    return image;
}


// the rest of this file is identical to the original squares.cpp except:
// main{} is removed
// this line is removed from drawSquares: 
// imshow(wndname, image); 
// (obj-c will do the drawing)

UIImage+OpenCV.h

The UIImage category is an objC++ file containing the code to convert between UIImage and cv::Mat image formats. This is where you move your two methods -(UIImage *)UIImageFromCVMat:(cv::Mat)cvMat and - (cv::Mat)cvMatWithImage:(UIImage *)image

//UIImage+OpenCV.h

#import <UIKit/UIKit.h>

@interface UIImage (UIImage_OpenCV)

    //cv::Mat to UIImage
+ (UIImage *)imageFromCVMat:(cv::Mat&)cvMat;

    //UIImage to cv::Mat
- (cv::Mat)CVMat;


@end        

The method implementations here are unchanged from your code (although we don't pass a UIImage in to convert, instead we refer to self)