Image Processing Edge Detection in Java

Question

This is my situation. It involves aligning a scanned image which will account for incorrect scanning. I must align the scanned image with my Java program.

These are more

Answer 1

I researched the libraries but in the end I found it more convenient to code up my own edge detection methods.

The class below will detect black/grayed out edges of a scanned sheet of paper that contains such edges, and will return the x and y coordinate of the edges of the sheet of paper, starting from the rightmost end (reverse = true) or from lower end (reverse = true) or from the top edge (reverse = false) or from left edge (reverse = false). Also...the program will take ranges along vertical edges (rangex) measured in pixels, and horizontal ranges (rangey) measured in pixels. The ranges determine outliers in the points received.

The program does 4 vertical cuts using the specified arrays, and 4 horizontal cuts. It retrieves the values of the dark dots. It uses the ranges to eliminate outliers. Sometimes, a little spot on the paper may cause an outlier point. The smaller the range, the fewer the outliers. However, sometimes the edge is slightly tilted, so you don't want to make the range too small.

Have fun. It works perfectly for me.

import java.awt.image.BufferedImage;
import java.awt.Color;
import java.util.ArrayList;
import java.lang.Math;
import java.awt.Point;
public class EdgeDetection {

    public App ap;
        public int[] horizontalCuts = {120, 220, 320, 420};
        public int[] verticalCuts = {300, 350, 375, 400};



    public void printEdgesTest(BufferedImage image, boolean reversex, boolean reversey, int rangex, int rangey){
        int[] mx = horizontalCuts;
        int[] my = verticalCuts;

            //you are getting edge points here
            //the "true" parameter indicates that it performs a cut starting at 0. (left edge)
        int[] xEdges = getEdges(image, mx, reversex, true);
        int edgex = getEdge(xEdges, rangex);
        for(int x = 0; x < xEdges.length; x++){
            System.out.println("EDGE = " + xEdges[x]);
        }
        System.out.println("THE EDGE = " + edgex);
            //the "false" parameter indicates you are doing your cut starting at the end (image.getHeight)
            //and ending at 0
            //if the parameter was true, it would mean it would start the cuts at y = 0
        int[] yEdges = getEdges(image, my, reversey, false);
        int edgey = getEdge(yEdges, rangey);
        for(int y = 0; y < yEdges.length; y++){
            System.out.println("EDGE = " + yEdges[y]);
        }
        System.out.println("THE EDGE = " + edgey);
    }

    //This function takes an array of coordinates...detects outliers, 
    //and computes the average of non-outlier points.

    public int getEdge(int[] edges, int range){
        ArrayList<Integer> result = new ArrayList<Integer>();
        boolean[] passes = new boolean[edges.length];
        int[][] differences = new int[edges.length][edges.length-1];
        //THIS CODE SEGMENT SAVES THE DIFFERENCES BETWEEN THE POINTS INTO AN ARRAY
        for(int n = 0; n<edges.length; n++){
            for(int m = 0; m<edges.length; m++){
                if(m < n){
                    differences[n][m] = edges[n] - edges[m];
                }else if(m > n){
                    differences[n][m-1] = edges[n] - edges[m];
                }
            }
        }
         //This array determines which points are outliers or nots (fall within range of other points)
        for(int n = 0; n<edges.length; n++){
            passes[n] = false;
            for(int m = 0; m<edges.length-1; m++){
                if(Math.abs(differences[n][m]) < range){
                    passes[n] = true;
                    System.out.println("EDGECHECK = TRUE" + n);
                    break;
                }
            }
        }
         //Create a new array only using valid points
        for(int i = 0; i<edges.length; i++){
            if(passes[i]){
                result.add(edges[i]);
            }
        }

        //Calculate the rounded mean... This will be the x/y coordinate of the edge
        //Whether they are x or y values depends on the "reverse" variable used to calculate the edges array
        int divisor = result.size();
        int addend = 0;
        double mean = 0;
        for(Integer i : result){
            addend += i;
        }
        mean = (double)addend/(double)divisor;

        //returns the mean of the valid points: this is the x or y coordinate of your calculated edge.
        if(mean - (int)mean >= .5){
            System.out.println("MEAN " + mean);
            return (int)mean+1;
        }else{
            System.out.println("MEAN " + mean);
            return (int)mean;
        }       
    }


     //this function computes "dark" points, which include light gray, to detect edges.
     //reverse - when true, starts counting from x = 0 or y = 0, and ends at image.getWidth or image.getHeight()
     //verticalEdge - determines whether you want to detect a vertical edge, or a horizontal edge
     //arr[] - determines the coordinates of the vertical or horizontal cuts you will do
     //set the arr[] array according to the graphical layout of your scanned image
     //image - this is the image you want to detect black/white edges of
    public int[] getEdges(BufferedImage image, int[] arr, boolean reverse, boolean verticalEdge){
        int red = 255;
        int green = 255;
        int blue = 255;
        int[] result = new int[arr.length];
        for(int n = 0; n<arr.length; n++){
            for(int m = reverse ? (verticalEdge ? image.getWidth():image.getHeight())-1:0; reverse ? m>=0:m<(verticalEdge ? image.getWidth():image.getHeight());){
                Color c = new Color(image.getRGB(verticalEdge ? m:arr[n], verticalEdge ? arr[n]:m));
                red = c.getRed();
                green = c.getGreen();
                blue = c.getBlue();
                        //determine if the point is considered "dark" or not.
                        //modify the range if you want to only include really dark spots.
                        //occasionally, though, the edge might be blurred out, and light gray helps
                if(red<239 && green<239 && blue<239){
                    result[n] = m;
                    break;
                }
                        //count forwards or backwards depending on reverse variable
                if(reverse){
                    m--;
                }else{
                    m++;
                }
            }
        }
    return result;
    }

}

Answer 2

Edge detection is something that is typically done by enhancing the contrast between neighboring pixels, such that you get a easily detectable line, which is suitable for further processing.

To do this, a "kernel" transforms a pixel according it the pixel's inital value, and the value of that pixel's neighbors. A good edge detection kernel will enhance the differences between neighboring pixels, and reduce the strength of a pixel with similar neigbors.

I would start by looking at the Sobel operator. This might not return results that are immediately useful to you; however, it will get you far closer than you would be if you were to approach the problem with little knowledge of the field.

After you have some crisp clean edges, you can use larger kernels to detect points where it seems that a 90% bend in two lines occurs, that might give you the pixel coordinates of the outer rectangle, which might be enough for your purposes.

With those outer coordinates, it still is a bit of math to make the new pixels be composted with the average values between the old pixels rotated and moved to "match". The results (especially if you do not know about anti-aliasing math) can be pretty bad, adding blur to the image.

Sharpening filters might be a solution, but they come with their own issues, mainly they make the picture sharper by adding graininess. Too much, and it is obvious that the original image is not a high-quality scan.

Answer 3

Try to start from a simple scenario and then improve the approach.

1. Detect corners.
2. Find the corners in the boundaries of the form.
3. Using the form corners coordinates, calculate the rotation angle.
4. Rotate/scale the image.
5. Map the position of each field in the form relative to form origin coordinates.
6. Match the textboxes.

The program presented at the end of this post does the steps 1 to 3. It was implemented using Marvin Framework. The image below shows the output image with the detected corners.

The program also outputs: Rotation angle:1.6365770416167182

Source code:

import java.awt.Color;
import java.awt.Point;
import marvin.image.MarvinImage;
import marvin.io.MarvinImageIO;
import marvin.plugin.MarvinImagePlugin;
import marvin.util.MarvinAttributes;
import marvin.util.MarvinPluginLoader;

public class FormCorners {

public FormCorners(){
    // Load plug-in
    MarvinImagePlugin moravec = MarvinPluginLoader.loadImagePlugin("org.marvinproject.image.corner.moravec");
    MarvinAttributes attr = new MarvinAttributes();

    // Load image
    MarvinImage image = MarvinImageIO.loadImage("./res/printedForm.jpg");

    // Process and save output image
    moravec.setAttribute("threshold", 2000);
    moravec.process(image, null, attr);
    Point[] boundaries = boundaries(attr);
    image = showCorners(image, boundaries, 12);
    MarvinImageIO.saveImage(image, "./res/printedForm_output.jpg");

    // Print rotation angle
    double angle =  (Math.atan2((boundaries[1].y*-1)-(boundaries[0].y*-1),boundaries[1].x-boundaries[0].x) * 180 / Math.PI);
    angle =  angle >= 0 ? angle : angle + 360;
    System.out.println("Rotation angle:"+angle);
}

private Point[] boundaries(MarvinAttributes attr){
    Point upLeft = new Point(-1,-1);
    Point upRight = new Point(-1,-1);
    Point bottomLeft = new Point(-1,-1);
    Point bottomRight = new Point(-1,-1);
    double ulDistance=9999,blDistance=9999,urDistance=9999,brDistance=9999;
    double tempDistance=-1;
    int[][] cornernessMap = (int[][]) attr.get("cornernessMap");

    for(int x=0; x<cornernessMap.length; x++){
        for(int y=0; y<cornernessMap[0].length; y++){
            if(cornernessMap[x][y] > 0){
                if((tempDistance = Point.distance(x, y, 0, 0)) < ulDistance){
                    upLeft.x = x; upLeft.y = y;
                    ulDistance = tempDistance;
                } 
                if((tempDistance = Point.distance(x, y, cornernessMap.length, 0)) < urDistance){
                    upRight.x = x; upRight.y = y;
                    urDistance = tempDistance;
                }
                if((tempDistance = Point.distance(x, y, 0, cornernessMap[0].length)) < blDistance){
                    bottomLeft.x = x; bottomLeft.y = y;
                    blDistance = tempDistance;
                }
                if((tempDistance = Point.distance(x, y, cornernessMap.length, cornernessMap[0].length)) < brDistance){
                    bottomRight.x = x; bottomRight.y = y;
                    brDistance = tempDistance;
                }
            }
        }
    }
    return new Point[]{upLeft, upRight, bottomRight, bottomLeft};
}

private MarvinImage showCorners(MarvinImage image, Point[] points, int rectSize){
    MarvinImage ret = image.clone();
    for(Point p:points){
        ret.fillRect(p.x-(rectSize/2), p.y-(rectSize/2), rectSize, rectSize, Color.red);
    }
    return ret;
}

public static void main(String[] args) {
    new FormCorners();
}
}

Answer 4

A similar such problem I've done in the past basically figured out the orientation of the form, re-aligned it, re-scaled it, and I was all set. You can use the Hough transform to to detect the angular offset of the image (ie: how much it is rotated), but you still need to detect the boundaries of the form. It also had to accommodate for the boundaries of the piece of paper itself.

This was a lucky break for me, because it basically showed a black and white image in the middle of a big black border.

1. Apply an aggressive, 5x5 median filter to remove some noise.
2. Convert from grayscale to black and white (rescale intensity values from [0,255] to [0,1]).
3. Calculate the Principal Component Analysis (ie: calculate the Eigenvectors of the covariance matrix for your image from the calculated Eigenvalues) (http://en.wikipedia.org/wiki/Principal_component_analysis#Derivation_of_PCA_using_the_covariance_method) 4) This gives you a basis vector. You simply use that to re-orient your image to a standard basis matrix (ie: [1,0],[0,1]).

Your image is now aligned beautifully. I did this for normalizing the orientation of MRI scans of entire human brains.

You also know that you have a massive black border around the actual image. You simply keep deleting rows from the top and bottom, and both sides of the image until they are all gone. You can temporarily apply a 7x7 median or mode filter to a copy of the image so far at this point. It helps rule out too much border remaining in the final image from thumbprints, dirt, etc.