OpenCV template matching and transparency

Question

What's the way OpenCV handles transparency in image during template matching?

The problem is that the template image needs to have transparent parts, because in the original image there could be anything at those places.

I tried all of the methods, and none of them yielded positive results (e.g. position of template in original image wasn't detected correctly).

Answer 1

It doesn't seem like OpenCV handles alpha the way you want it to.

You have two options:

1. Write your own cross-correlation method that will use the alpha channel
2. Transform your images so your alpha channel becomes irrelevant

Since the first option is straightforward, I will explore the second option here. I'm going to re-use the sample code I provided to a similar question earlier. If you apply cross-correlation directly to your images, the background interferes with the template matching (in particular, light background parts). If you play around with color channels, you will find that matching in the blue channel gives the correct result. This depends on the image content and isn't a consistent way to solve the problem.

Another option is to perform edge detection (e.g. Sobel) on the image and template, and perform cross-correlation then. Here are the edge detected images (I used the Sobel edge detector on the Luma channel in GIMP, and then some intensity stretching).

As you can see, the alpha channel here has become irrelevant, as most of the terrain has become zero intensity and will not contribute to the cross-correlation calculation. So now cross-correlation can be directly applied, giving the desired result:

misha@misha-desktop:~/Desktop/stackoverflow$ python cross-correlation.py map-blue.png building-maskz-blue.png 
(163, 244)

Finally, here's another related question.

PS. What game is this?

Answer 2

I have a slightly more brain-dead solution to this issue which actually seems to work reasonably well: Replace the template image's alpha channel with noise, which more or less makes the transparent regions statistically insignificant during the matching process.

For example, my use case involved searching for emoji characters in screen captures from iOS. The iOS keyboard background changes colors depending on context, which makes the matching process problematic if you commit to a particular background color in your template image.

Here's the raw template image on alpha:

Here's the processed template with noise filling in for the alpha channel:

I sent the processed template image through the Template Matching sample code provided in the OpenCV documentation. On either dark or light background, the match is found with reasonable confidence.

Searching on dark backgrounds:

Searching on light backgrounds:

In comparison, leaving the template's alpha channel transparent — or committing to a dark or light background — did not return acceptable matches.

Answer 3

OpenCV 3.0 offers native support for template matching with masked templates. Refer to the new documentation:

Parameters:

image ...

templ ...

result ...

method ...

mask Mask of searched template. It must have the same datatype and size with templ. It is not set by default.

[Slight Digression]

Note that template matching with masked reference images (the larger image) is not possible though. And that makes sense, given OpenCV uses FFT based template matching.

Therefore, if you need to perform template matching only at specific regions of your reference images, you will need to implement your own method for that or mask the output of cv::matchTemplate.

Implementing it from scratch should compensate for cases where you only want to search for the template at very specific regions (i.e.: around harris corners).

Answer 4

The SQDIFF/SQDIFF_N option would be a solution if you tried to replace alpha channel with the black RGB color. At least this was my solution to same problem. From my result is obvious that this method is sensitive to brighter pixel values, and I took an opportunity of that.

Answer 5

I think you're trying to do what in OpenCV is called template matching with a mask. I think you could try setting a ROI (region of interest) on the template. This SO question shows how to do it. (note that in that question the ROI is set on the target image, not the template, but the procedure is the same).

Answer 6

I'm not sure, but the transparency channel is treated just like any other channel. If a pixel in a template is "transparent" it should be "transparent" on the main image as well. I'm just guessing here.

Answer 7

I came across the same problem and I thought of a solution. Asuming that referenceImageMask and templateMask have 1s in the good pixels and 0s in the bad ones. And that referenceImage and templateImage have already been masked and have 0s in the bad pixels as well.

Then, the first result of template matching will give the not normalized cross correlation between the images. However, a bunch of pixels were zero.

The second template matching will give for each possible offset the number of pixels that were at the same time different from zero (unmasked) in both images.

Then, normalizing the correlation by that number should give the value you (and I) wanted. The average product for the pixels that are not masked in both images.

Image<Gray, float> imCorr = referenceImage.MatchTemplate(templateImage,      Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCORR);
Image<Gray, float> imCorrMask = referenceImageMask.MatchTemplate(templateMask, Emgu.CV.CvEnum.TM_TYPE.CV_TM_CCORR);
_imCorr = _imCorr.Mul(_imCorrMask.Pow(-1));

UPDATE: actually, this solution does not work. Because the implementation of the cross correlation in opencv uses the DFT there will be numeric issues and you cannot use the second crosscorrelation to correct the first one.