how to do fuzzy search in big data

Question

I\'m new to that area and I wondering mostly what the state-of-the-art is and where I can read about it.

Let\'s assume that I just have a key/value store and I have some

Answer 1

I suggest you take a look at FLANN Fast Approximate Nearest Neighbors. Fuzzy search in big data is also known as approximate nearest neighbors.

This library offers you different metric, e.g Euclidian, Hamming and different methods of clustering: LSH or k-means for instance.

The search is always in 2 phases. First you feed the system with data to train the algorithm, this is potentially time consuming depending on your data. I successfully clustered 13 millions data in less than a minute though (using LSH).

Then comes the search phase, which is very fast. You can specify a maximum distance and/or the maximum numbers of neighbors.

As Lukas said, there is no good generic solution, each domain will have its tricks to make it faster or find a better way using the inner property of the data your using.

Shazam uses a special technique with geometrical projections to quickly find your song. In computer vision we often use the BOW: Bag of words, which originally appeared in text retrieval.

If you can see your data as a graph, there are other methods for approximate matching using spectral graph theory for instance.

Let us know.

Answer 2

Depends on what your key/values are like, the Levenshtein algorithm (also called Edit-Distance) can help. It calculates the least number of edit operations that are necessary to modify one string to obtain another string.

• http://en.wikipedia.org/wiki/Levenshtein_distance
• http://www.levenshtein.net/

Answer 3

There is no (fast) generic solution, each application will need different approach.

Neither of the two examples actually does traditional nearest neighbor search. AcoustID (I'm the author) is just looking for exact matches, but it searches in a very high number of hashes in hope that some of them will match. The phonetic search example uses metaphone to convert words to their phonetic representation and is also only looking for exact matches.

You will find that if you have a lot of data, exact search using huge hash tables is the only thing you can realistically do. The problem then becomes how to convert your fuzzy matching to exact search.

A common approach is to use locality-sensitive hashing (LSH) with a smart hashing method, but as you can see in your two examples, sometimes you can get away with even simpler approach.

Btw, you are looking specifically for text search, the simplest way you can do it split your input to N-grams and index those. Depending on how your distance function is defined, that might give you the right candidate matches without too much work.