image processing to improve tesseract OCR accuracy

Question

I\'ve been using tesseract to convert documents into text. The quality of the documents ranges wildly, and I\'m looking for tips on what sort of image processing might impr

Answer 1

Text Recognition depends on a variety of factors to produce a good quality output. OCR output highly depends on the quality of input image. This is why every OCR engine provides guidelines regarding the quality of input image and its size. These guidelines help OCR engine to produce accurate results.

I have written a detailed article on image processing in python. Kindly follow the link below for more explanation. Also added the python source code to implement those process.

Please write a comment if you have a suggestion or better idea on this topic to improve it.

https://medium.com/cashify-engineering/improve-accuracy-of-ocr-using-image-preprocessing-8df29ec3a033

Answer 2

Three points to improve the readability of the image:

1. Resize the image with variable height and width(multiply 0.5 and 1 and 2 with image height and width).

2. Convert the image to Gray scale format(Black and white).

3. Remove the noise pixels and make more clear(Filter the image).

Refer below code :

Resize

public Bitmap Resize(Bitmap bmp, int newWidth, int newHeight)
        {
         
                Bitmap temp = (Bitmap)bmp;
            
                Bitmap bmap = new Bitmap(newWidth, newHeight, temp.PixelFormat);
             
                double nWidthFactor = (double)temp.Width / (double)newWidth;
                double nHeightFactor = (double)temp.Height / (double)newHeight;

                double fx, fy, nx, ny;
                int cx, cy, fr_x, fr_y;
                Color color1 = new Color();
                Color color2 = new Color();
                Color color3 = new Color();
                Color color4 = new Color();
                byte nRed, nGreen, nBlue;

                byte bp1, bp2;

                for (int x = 0; x < bmap.Width; ++x)
                {
                    for (int y = 0; y < bmap.Height; ++y)
                    {

                        fr_x = (int)Math.Floor(x * nWidthFactor);
                        fr_y = (int)Math.Floor(y * nHeightFactor);
                        cx = fr_x + 1;
                        if (cx >= temp.Width) cx = fr_x;
                        cy = fr_y + 1;
                        if (cy >= temp.Height) cy = fr_y;
                        fx = x * nWidthFactor - fr_x;
                        fy = y * nHeightFactor - fr_y;
                        nx = 1.0 - fx;
                        ny = 1.0 - fy;

                        color1 = temp.GetPixel(fr_x, fr_y);
                        color2 = temp.GetPixel(cx, fr_y);
                        color3 = temp.GetPixel(fr_x, cy);
                        color4 = temp.GetPixel(cx, cy);

                        // Blue
                        bp1 = (byte)(nx * color1.B + fx * color2.B);

                        bp2 = (byte)(nx * color3.B + fx * color4.B);

                        nBlue = (byte)(ny * (double)(bp1) + fy * (double)(bp2));

                        // Green
                        bp1 = (byte)(nx * color1.G + fx * color2.G);

                        bp2 = (byte)(nx * color3.G + fx * color4.G);

                        nGreen = (byte)(ny * (double)(bp1) + fy * (double)(bp2));

                        // Red
                        bp1 = (byte)(nx * color1.R + fx * color2.R);

                        bp2 = (byte)(nx * color3.R + fx * color4.R);

                        nRed = (byte)(ny * (double)(bp1) + fy * (double)(bp2));

                        bmap.SetPixel(x, y, System.Drawing.Color.FromArgb
                (255, nRed, nGreen, nBlue));
                    }
                }

       

                bmap = SetGrayscale(bmap);
                bmap = RemoveNoise(bmap);

                return bmap;
            
        }

SetGrayscale

public Bitmap SetGrayscale(Bitmap img)
            {
    
                Bitmap temp = (Bitmap)img;
                Bitmap bmap = (Bitmap)temp.Clone();
                Color c;
                for (int i = 0; i < bmap.Width; i++)
                {
                    for (int j = 0; j < bmap.Height; j++)
                    {
                        c = bmap.GetPixel(i, j);
                        byte gray = (byte)(.299 * c.R + .587 * c.G + .114 * c.B);
    
                        bmap.SetPixel(i, j, Color.FromArgb(gray, gray, gray));
                    }
                }
                return (Bitmap)bmap.Clone();
    
            }

RemoveNoise

public Bitmap RemoveNoise(Bitmap bmap)
            {
    
                for (var x = 0; x < bmap.Width; x++)
                {
                    for (var y = 0; y < bmap.Height; y++)
                    {
                        var pixel = bmap.GetPixel(x, y);
                        if (pixel.R < 162 && pixel.G < 162 && pixel.B < 162)
                            bmap.SetPixel(x, y, Color.Black);
                        else if (pixel.R > 162 && pixel.G > 162 && pixel.B > 162)
                            bmap.SetPixel(x, y, Color.White);
                    }
                }
    
                return bmap;
            }

INPUT IMAGE

OUTPUT IMAGE

Answer 3

Reading text from image documents using any OCR engine have many issues in order get good accuracy. There is no fixed solution to all the cases but here are a few things which should be considered to improve OCR results.

1) Presence of noise due to poor image quality / unwanted elements/blobs in the background region. This requires some pre-processing operations like noise removal which can be easily done using gaussian filter or normal median filter methods. These are also available in OpenCV.

2) Wrong orientation of image: Because of wrong orientation OCR engine fails to segment the lines and words in image correctly which gives the worst accuracy.

3) Presence of lines: While doing word or line segmentation OCR engine sometimes also tries to merge the words and lines together and thus processing wrong content and hence giving wrong results. There are other issues also but these are the basic ones.

This post OCR application is an example case where some image pre-preocessing and post processing on OCR result can be applied to get better OCR accuracy.

Answer 4

Java version for Sathyaraj's code above:

// Resize
public Bitmap resize(Bitmap img, int newWidth, int newHeight) {
    Bitmap bmap = img.copy(img.getConfig(), true);

    double nWidthFactor = (double) img.getWidth() / (double) newWidth;
    double nHeightFactor = (double) img.getHeight() / (double) newHeight;

    double fx, fy, nx, ny;
    int cx, cy, fr_x, fr_y;
    int color1;
    int color2;
    int color3;
    int color4;
    byte nRed, nGreen, nBlue;

    byte bp1, bp2;

    for (int x = 0; x < bmap.getWidth(); ++x) {
        for (int y = 0; y < bmap.getHeight(); ++y) {

            fr_x = (int) Math.floor(x * nWidthFactor);
            fr_y = (int) Math.floor(y * nHeightFactor);
            cx = fr_x + 1;
            if (cx >= img.getWidth())
                cx = fr_x;
            cy = fr_y + 1;
            if (cy >= img.getHeight())
                cy = fr_y;
            fx = x * nWidthFactor - fr_x;
            fy = y * nHeightFactor - fr_y;
            nx = 1.0 - fx;
            ny = 1.0 - fy;

            color1 = img.getPixel(fr_x, fr_y);
            color2 = img.getPixel(cx, fr_y);
            color3 = img.getPixel(fr_x, cy);
            color4 = img.getPixel(cx, cy);

            // Blue
            bp1 = (byte) (nx * Color.blue(color1) + fx * Color.blue(color2));
            bp2 = (byte) (nx * Color.blue(color3) + fx * Color.blue(color4));
            nBlue = (byte) (ny * (double) (bp1) + fy * (double) (bp2));

            // Green
            bp1 = (byte) (nx * Color.green(color1) + fx * Color.green(color2));
            bp2 = (byte) (nx * Color.green(color3) + fx * Color.green(color4));
            nGreen = (byte) (ny * (double) (bp1) + fy * (double) (bp2));

            // Red
            bp1 = (byte) (nx * Color.red(color1) + fx * Color.red(color2));
            bp2 = (byte) (nx * Color.red(color3) + fx * Color.red(color4));
            nRed = (byte) (ny * (double) (bp1) + fy * (double) (bp2));

            bmap.setPixel(x, y, Color.argb(255, nRed, nGreen, nBlue));
        }
    }

    bmap = setGrayscale(bmap);
    bmap = removeNoise(bmap);

    return bmap;
}

// SetGrayscale
private Bitmap setGrayscale(Bitmap img) {
    Bitmap bmap = img.copy(img.getConfig(), true);
    int c;
    for (int i = 0; i < bmap.getWidth(); i++) {
        for (int j = 0; j < bmap.getHeight(); j++) {
            c = bmap.getPixel(i, j);
            byte gray = (byte) (.299 * Color.red(c) + .587 * Color.green(c)
                    + .114 * Color.blue(c));

            bmap.setPixel(i, j, Color.argb(255, gray, gray, gray));
        }
    }
    return bmap;
}

// RemoveNoise
private Bitmap removeNoise(Bitmap bmap) {
    for (int x = 0; x < bmap.getWidth(); x++) {
        for (int y = 0; y < bmap.getHeight(); y++) {
            int pixel = bmap.getPixel(x, y);
            if (Color.red(pixel) < 162 && Color.green(pixel) < 162 && Color.blue(pixel) < 162) {
                bmap.setPixel(x, y, Color.BLACK);
            }
        }
    }
    for (int x = 0; x < bmap.getWidth(); x++) {
        for (int y = 0; y < bmap.getHeight(); y++) {
            int pixel = bmap.getPixel(x, y);
            if (Color.red(pixel) > 162 && Color.green(pixel) > 162 && Color.blue(pixel) > 162) {
                bmap.setPixel(x, y, Color.WHITE);
            }
        }
    }
    return bmap;
}

Answer 5

I did these to get good results out of an image which has not very small text.

1. Apply blur to the original image.
2. Apply Adaptive Threshold.
3. Apply Sharpening effect.

And if the still not getting good results, scale the image to 150% or 200%.

Answer 6

This is somewhat ago but it still might be useful.

My experience shows that resizing the image in-memory before passing it to tesseract sometimes helps.

Try different modes of interpolation. The post https://stackoverflow.com/a/4756906/146003 helped me a lot.