Advanced square detection (with connected region)

Question

if the squares has connected region in image, how can I detect them.

I have tested the method mentioned in OpenCV C++/Obj-C: Advanced square detection

It did no

Answer 1

Solution 1:

Dilate your image to delete connected components. Find contours of detected components. Eliminate contours which are not rectangles by introducing some measure (ex. ratio perimeter / area).

This solution will not detect rectangles connected to borders.

Solution 2:

Dilate to delete connected components. Find contours. Approximate contours to reduce their points (for rectangle contour should be 4 points). Check if angle between contour lines is 90 degrees. Eliminate contours which have no 90 degrees.

This should solve problem with rectangles connected to borders.

Answer 2

Applying a Watershed Transform based on the Distance Transform will separate the objects:

Handling objects at the border is always problematic, and often discarded, so that pink rectangle at top left not separated is not a problem at all.

Given a binary image, we can apply the Distance Transform (DT) and from it obtain markers for the Watershed. Ideally there would be a ready function for finding regional minima/maxima, but since it isn't there, we can make a decent guess on how we can threshold DT. Based on the markers we can segment using Watershed, and the problem is solved. Now you can worry about distinguishing components that are rectangles from those that are not.

import sys
import cv2
import numpy
import random
from scipy.ndimage import label

def segment_on_dt(img):
    dt = cv2.distanceTransform(img, 2, 3) # L2 norm, 3x3 mask
    dt = ((dt - dt.min()) / (dt.max() - dt.min()) * 255).astype(numpy.uint8)
    dt = cv2.threshold(dt, 100, 255, cv2.THRESH_BINARY)[1]
    lbl, ncc = label(dt)

    lbl[img == 0] = lbl.max() + 1
    lbl = lbl.astype(numpy.int32)
    cv2.watershed(cv2.cvtColor(img, cv2.COLOR_GRAY2BGR), lbl)
    lbl[lbl == -1] = 0
    return lbl


img = cv2.cvtColor(cv2.imread(sys.argv[1]), cv2.COLOR_BGR2GRAY)
img = cv2.threshold(img, 0, 255, cv2.THRESH_OTSU)[1]
img = 255 - img # White: objects; Black: background

ws_result = segment_on_dt(img)
# Colorize
height, width = ws_result.shape
ws_color = numpy.zeros((height, width, 3), dtype=numpy.uint8)
lbl, ncc = label(ws_result)
for l in xrange(1, ncc + 1):
    a, b = numpy.nonzero(lbl == l)
    if img[a[0], b[0]] == 0: # Do not color background.
        continue
    rgb = [random.randint(0, 255) for _ in xrange(3)]
    ws_color[lbl == l] = tuple(rgb)

cv2.imwrite(sys.argv[2], ws_color)

From the above image you can consider fitting ellipses in each component to determine rectangles. Then you can use some measurement to define whether the component is a rectangle or not. This approach has a greater chance to work for rectangles that are fully visible, and will likely produce bad results for partially visible ones. The following image shows the result of such approach considering that a component is a rectangle if the rectangle from the fitted ellipse is within 10% of component's area.

# Fit ellipse to determine the rectangles.
wsbin = numpy.zeros((height, width), dtype=numpy.uint8)
wsbin[cv2.cvtColor(ws_color, cv2.COLOR_BGR2GRAY) != 0] = 255

ws_bincolor = cv2.cvtColor(255 - wsbin, cv2.COLOR_GRAY2BGR)
lbl, ncc = label(wsbin)
for l in xrange(1, ncc + 1):
    yx = numpy.dstack(numpy.nonzero(lbl == l)).astype(numpy.int64)
    xy = numpy.roll(numpy.swapaxes(yx, 0, 1), 1, 2)
    if len(xy) < 100: # Too small.
        continue

    ellipse = cv2.fitEllipse(xy)
    center, axes, angle = ellipse
    rect_area = axes[0] * axes[1]
    if 0.9 < rect_area / float(len(xy)) < 1.1:
        rect = numpy.round(numpy.float64(
                cv2.cv.BoxPoints(ellipse))).astype(numpy.int64)
        color = [random.randint(60, 255) for _ in xrange(3)]
        cv2.drawContours(ws_bincolor, [rect], 0, color, 2)

cv2.imwrite(sys.argv[3], ws_bincolor)

Answer 3

You have three problems:

1. The rectangles are not very strict rectangles (the edges are often somewhat curved)
2. There are a lot of them.
3. They are often connected.

It seems that all your rects are essentially the same size(?), and do not greatly overlap, but the pre-processing has connected them.

For this situation the approach I would try is:

1. dilate your image a few times (as also suggested by @krzych) - this will remove the connections, but result in slightly smaller rects.
2. Use scipy to label and find_objects - You now know the position and slice for every remaining blob in the image.
3. Use minAreaRect to find the center, orientation, width and height of each rectangle.

You can use step 3. to test whether the blob is a valid rectangle or not, by its area, dimension ratio or proximity to the edge..

This is quite a nice approach, as we assume each blob is a rectangle, so minAreaRect will find the parameters for our minimum enclosing rectangle. Further we could test each blob using something like humoments if absolutely neccessary.

Here is what I was suggesting in action, boundary collision matches shown in red.

Code:

import numpy as np
import cv2
from cv2 import cv
import scipy
from scipy import ndimage

im_col = cv2.imread('jdjAf.jpg')
im = cv2.imread('jdjAf.jpg',cv2.CV_LOAD_IMAGE_GRAYSCALE)

im = np.where(im>100,0,255).astype(np.uint8)
im = cv2.erode(im, None,iterations=8)
im_label, num = ndimage.label(im)
for label in xrange(1, num+1):
    points = np.array(np.where(im_label==label)[::-1]).T.reshape(-1,1,2).copy()
    rect = cv2.minAreaRect(points)
    lines = np.array(cv2.cv.BoxPoints(rect)).astype(np.int)
    if any([np.any(lines[:,0]<=0), np.any(lines[:,0]>=im.shape[1]-1), np.any(lines[:,1]<=0), np.any(lines[:,1]>=im.shape[0]-1)]):
        cv2.drawContours(im_col,[lines],0,(0,0,255),1)
    else:
        cv2.drawContours(im_col,[lines],0,(255,0,0),1)

cv2.imshow('im',im_col)
cv2.imwrite('rects.png',im_col)
cv2.waitKey()

I think the Watershed and distanceTransform approach demonstrated by @mmgp is clearly superior for segmenting the image, but this simple approach can be effective depending upon your needs.