How to find the coordinates of the outside corners of these 4 squares? (morphological closing/opening does not conserve squares if image is rotated)

Question

One of the first processing steps in a tool I'm coding is to find the coordinates of the outside corners of 4 big black squares. They will then be used to do a homographic transform, in order to deskew / unrotate the image (a.k.a perspective transform), to finally get a rectangular image. Here is an example of - rotated and noisy - input (download link here):

To keep the big squares only, I'm using morphological transformations like closing/opening:

import cv2, numpy as np
img = cv2.imread('rotatednoisy-cropped.png', cv2.IMREAD_GRAYSCALE)
kernel = np.ones((30, 30), np.uint8)
img = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)
cv2.imwrite('output.png', img)

Input file (download link):

Output, after morphological transform:

Problem: the output squares are not square anymore, and therefore the coordinates of the top left corner of the square will be not precise at all!

I could reduce the kernel size, but then it would keep more unwanted small elements.

Question: how to get a better detection of the corners of the squares?

---

Note:

• As a morphological closing is just a dilatation + an erosion, I found the culprit:

  import cv2, numpy as np
  img = cv2.imread('rotatednoisy-cropped.png', cv2.IMREAD_GRAYSCALE)
  kernel = np.ones((30, 30), np.uint8)
  img = cv2.dilate(img, kernel, iterations = 1)
  

  After this step, it's still ok:

  

  Then

  img = cv2.erode(img, kernel, iterations = 1)
  

  gives

  

  and it's not ok anymore!

Answer 1

See this link for detailed explanation on how to de-skew an image.

import cv2
import numpy as np

def corners(box):
    cx,cy,w,h,angle = box[0][0],box[0][1],box[1][0],box[1][1],box[2]
    CV_PI = 22./7.
    _angle = angle*CV_PI/180.;
    b = np.cos(_angle)*0.5;
    a = np.sin(_angle)*0.5;

    pt = []
    pt.append((int(cx - a*h - b*w),int(cy + b*h - a*w)));
    pt.append((int(cx + a*h - b*w),int(cy - b*h - a*w)));
    pt.append((int(2*cx - pt[0][0]),int(2*cy - pt[0][1])));
    pt.append((int(2*cx - pt[1][0]),int(2*cy - pt[1][1])));
    return pt

if __name__ == '__main__':

    image = cv2.imread('image.jpg',cv2.IMREAD_UNCHANGED)

    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    n = 3
    sigma = 0.3 * (n/2 - 1) + 0.8
    gray = cv2.GaussianBlur(gray, ksize=(n,n), sigmaX=sigma)

    ret,binary = cv2.threshold(gray, 0, 255, cv2.THRESH_OTSU+cv2.THRESH_BINARY)

    _,contours,_ = cv2.findContours(binary, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)

    contours.sort(key=lambda x: len(x), reverse=True)

    points = []
    for i in range(0,4):
        shape = cv2.approxPolyDP(contours[i], 0.05*cv2.arcLength(contours[i],True), True)
        if len(shape) == 4:
            points.append(shape)

    points = np.array(points,dtype=np.int32)
    points = np.reshape(points, (-1,2))
    box = cv2.minAreaRect(points)
    pt = corners(box)

    for i in range(0,4):
       image = cv2.line(image, (pt[i][0],pt[i][1]), (pt[(i+1)%4][0],pt[(i+1)%4][1]), (0,0,255))


    (h,w) = image.shape[:2]
    (center) = (w//2,h//2)
    angle = box[2]

    if angle < -45:
        angle = (angle+90)
    else:
        angle = -angle

    M = cv2.getRotationMatrix2D(center, angle, 1.0)
    rotated = cv2.warpAffine(image, M, (w,h), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT)

    cv2.imshow('image', image)
    cv2.imshow('rotated', rotated)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

Answer 2

You could try by searching and filtering out your specific contours (black rectangles) and sorting them with a key. Then select the extreme point for each contour (left, right, top, bottom) and you will get the points. Note that this approach is ok for this picture only and if the picture was roteted in other direction, you would have to change the code accordingly. I am not an expert but I hope this helps a bit.

import numpy as np
import cv2


img = cv2.imread("rotate.jpg")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
ret, threshold = cv2.threshold(gray,150,255,cv2.THRESH_BINARY)
im, contours, hierarchy = cv2.findContours(threshold,cv2.RETR_TREE,cv2.CHAIN_APPROX_NONE)
contours.sort(key=lambda c: np.min(c[:,:,1]))
j = 1

if len(contours) > 0:
    for i in range(0, len(contours)):
        size = cv2.contourArea(contours[i])
        if 90 < size < 140:
            if j == 1:
                c1 = contours[i]
                j += 1
            elif j == 2:
                c2 = contours[i]
                j += 1
            elif j == 3:
                c3 = contours[i]
                j += 1
            elif j == 4:
                c4 = contours[i]
                break

Top = tuple(c1[c1[:, :, 1].argmin()][0])
Right = tuple(c2[c2[:, :, 0].argmax()][0])
Left = tuple(c3[c3[:, :, 0].argmin()][0])
Bottom = tuple(c4[c4[:, :, 1].argmax()][0])

cv2.circle(img, Top, 2, (0, 255, 0), -1)
cv2.circle(img, Right, 2, (0, 255, 0), -1)
cv2.circle(img, Left, 2, (0, 255, 0), -1)
cv2.circle(img, Bottom, 2, (0, 255, 0), -1)

cv2.imshow("Image", img)
cv2.waitKey(0)

Result:

Answer 3

You can extract the squares as single blobs after binarization with a suitable threshold, and select the appropriate ones based on size. You can also first denoise with a median filter if you want.

Then a tight rotated bounding rectangle will give you the corners (you can obtain it by running Rotating Calipers on the Convex Hull).