Fuzzy screenshot comparison with Selenium

Question

I\'m using Selenium to automate webpage functional testing. It\'s important for us to do a pixel-by-pixel comparison when we roll out new code, so we\'re using Selenium to take

Answer 1

I've been using perceptualdiff which uses a model of the human visual system to try to avoid reporting unnoticeable changes (the authors used for renderer regression testing). Usage is quite simple:

perceptualdiff -output diff.ppm baseline.png test.png

(where diff.ppm is a PPM format image highlighting the areas of difference)

The needle regression testing framework has support for using pdiff to compare screenshots:

http://needle.readthedocs.org/en/latest/#engines

Answer 2

Use an image format that does not create artifacts (like BMP or PNG) then you can do a per-pixel comparison. I think you can check each pixel with a common Euclidean Distance. To improve performance a little, do not calculate the square root but check the squares of the distances

// Maximum color distance allowed to define pixel consistency.
const float maxDistanceAllowed = 5.0;

// Square of the distance, used in calculations.
float maxD = maxDistanceAllowed * maxDistanceAllowed;

public bool isPixelConsistent(Color pixel1, Color pixel2)
{
    // Euclidean distance in 3-dimensions.
    float distanceSquared = (pixel1.R - pixel2.R)*(pixel1.R - pixel2.R) + (pixel1.G - pixel2.G)*(pixel1.G - pixel2.G) + (pixel1.B - pixel2.B)*(pixel1.B - pixel2.B);

    // If the actual distance is less than the max allowed, the pixel is
    // consistent and the method returns TRUE
    return distanceSquared <= maxD;
}

Didn't test the C# code, but it should give you the idea. Give some tries and adjust the maxDistanceAllowed to your needs.

Answer 3

So I ended up going with the ImageMagick command-line tool (because why re-invent image comparison). The "Peak Absolute Error" metric of the "compare" tool tells you how much you have to fuzz pixels before two images are identical. This seems to work well... for an image with slight graphical distortions, there might be a lot of pixels that don't match, but slight fuzzing is enough to make them match; but for two images that are actually different, even though most pixels might match, the ones that don't tend to be very different. Right now I'm checking for a PAE of less than 15% to see if the images should be counted as identical. Command line I'm using is:

 compare -metric PAE  original.png new.png comparison.png

Documentation on ImageMagick's compare tool is here: http://www.imagemagick.org/script/compare.php

Answer 4

If anyone else is looking for something similar there is Depicted-dpxdt. It is designed to be used as part of a CI/CD process.

It combines perceptual diff with server, commandline tool, wrapper for phantom js.

It has functionality demonstrated like crawling entire site and comparing pages for differences.