Removing Duplicate Images [closed]

Answer 1

Try PerceptualDiff for comparing 2 images with the same dimensions. Allows threshholds such as considering images with only X number of pixels different to be visually indistinguishable.

If visual duplicates may have different dimensions due to scaling, or different filetypes, you may want to make a standard format for comparisons. For example, I might use ImageMagick to scale all images to 100x100 and save them as PNG files.

Answer 2

A quick hack at this is to write a program that will calculate the value of the average pixel in each image, in greyscale, sort by this value, and then compare them visually. Very similar images should occur near each other in the sorted order.

Answer 3

You will need a command line tool to deal with so much data.

Comparing every possible pair of images will not scale to such a large set of images. You need to sort the entire set of images according to some metric so that further comparisons are only needed on neighbouring images.

An example of a simple metric is the average value of all of the pixels in an image, expressed as a single greyscale value. This should work only if the duplicates have not had any visual alterations. Using a lossy file format can also result in visual alterations.

Answer 4

The most important part is to make the files comparable.

A generic solution might be to scale all images to a certain fixed size and greyscale. Then save the resulting images in a separate directory with same name for later reference. It would then be possible to sort by filesize and visually compare neighboring entries.

The resulting pictures might be quantified in certain ways to programatically detect similarities (averaging of blocks, lines etc.).

Answer 5

If you can come up with a way of comparing images that obeys the triangle inequality (eg, if d(a,b) is the difference between images a and b, then d(a,b) < d(a,c) + d(b,c) for all a,b,c), then a BK-Tree would be an effective way of indexing the images such that you can find matches in O(log n) time instead of O(n) time for each image.

If your matches are restricted to the same image after varying amounts of compression/resizing/etc, then converting to some canonical size/color balance/etc and simply summing the squares-of-differences of each pixel may be a good metric, and this obeys the triangle inequality, so you could use a BK-tree for efficient access.

Answer 6

This is still a research area, I believe. If you have some time in your hands, some relevant keywords are:

• Image copy detection
• Content based image retrieval
• Image indexing
• Image duplicate removal

Basically, each image is processed (indexed) to produce an "image signature". Similar images have similar signatures. If your images are just rescaled then probably their signature are nearly identical, so they cluster well. Some popular signatures are the MPEG-7 descriptors. To cluster, I think K-Means or any of its variants may be enough. However, you probably need to deal with millions of images, this may be a problem.

Here is a link to the main Wikipedia entry:
http://en.wikipedia.org/wiki/CBIR

Hope this helps.