fuzzy-comparison

I have a table containing 3 million people records on which I want to perform fuzzy matching using q-grams (on surname for instance). I have created a table of 2-grams linking to this, but search performance is not great on this data volume (around 5 minutes). I basically have two questions: (1) Can you suggest any ways to improve performance to avoid a table scan (i.e. having to count common q-grams between the search string and 3 million surnames) (2) With q-grams, if A is similar to B and C is similar to B, does it imply C is similar to A? Kind regards Peter I've been looking into fuzzy

I have an array of team names from NCAA, along with statistics associated with them. The school names are often shortened or left out entirely, but there is usually a common element in all variations of the name (like Alabama Crimson Tide vs Crimson Tide). These names are all contained in an array in no particular order. I would like to be able to take all variations of a team name by fuzzy matching them and rename all variants to one name. I'm working in python 2.7 and I have a numpy array with all of the data. Any help would be appreciated, as I have never used fuzzy matching before. I have

I have an implementation of the jaro-winkler algorithm in my database. I did not write this function. The function compares two values and gives the probability of match. So jaro(string1, string2, matchnoofchars) will return a result. Instead of comparing two strings, I want to send one string with a matchnoofchars and then get a result set with the probability higher than 95%. For example the current function is able to return 97.62% for jaro("Philadelphia","Philadelphlaa",9) I wish to tweak this function so that I am able to find "Philadelphia" for an input of "Philadelphlaa". What kind of

In an app that i'm creating, I want to add functionality that groups news stories together. I want to group news stories about the same topic from different sources into the same group. For example, an article on XYZ from CNN and MSNBC would be in the same group. I am guessing its some sort of fuzzy logic comparison. How would I go about doing this from a technical standpoint? What are my options? We haven't even started the app yet, so we aren't limited in the technologies we can use. Thanks, in advance for the help! This problem breaks down into a few subproblems from a machine learning

i have two data frames which are cleaned and merged as a single csv file , the data frames are like this **Source Master** chang chun petrochemical CHANG CHUN GROUP chang chun plastics CHURCH AND DWIGHT CO INC church dwight CITRIX SYSTEMS ASIA PACIFIC P L citrix systems pacific CNH INDUSTRIAL N.V now from these , i have to consider the first name and check with each name of master names and find a match that is relevant and print the output as another data frame. the above data frames are few , but i am working with 20k values as such. My output must look like this **Source Master Result**

Is there any way to speed up the fuzzy string match using fuzzywuzzy in pandas. I have a dataframe as extra_names which has names that I want to run fuzzy matches for with another dataframe as names_df . >> extra_names.head() not_matching 0 Vij Sales 1 Crom Electronics 2 REL Digital 3 Bajaj Elec 4 Reliance Digi >> len(extra_names) 6500 >> names_df.head() names types 0 Vijay Sales 1 1 Croma Electronics 1 2 Reliance Digital 2 3 Bajaj Electronics 2 4 Pai Electricals 2 >> len(names_df) 250 As of now, I'm running the logic using the following code, but its taking forever to complete. choices =