Fuzzy matching of product names

Question

I need to automatically match product names (cameras, laptops, tv-s etc) that come from different sources to a canonical name in the database.

For example \"

Answer 1

You might want to create logic that ignores the letter/number combination of model numbers (since they're nigh always extremely similar).

Answer 2

Not having any experience with this type of problem, but I think a very naive implementation would be to tokenize the search term, and search for matches that happen to contain any of the tokens.

"Canon PowerShot A20 IS", for example, tokenizes into:

• Canon
• Powershot
• A20
• IS

which would match each of the other items you want to show up in the results. Of course, this strategy will likely produce a whole lot of false matches as well.

Another strategy would be to store "keywords" with each item, such as "camera", "canon", "digital camera", and searching based on items that have matching keywords. In addition, if you stored other attributes such as Maker, Brand, etc., you could search on each of these.

Answer 3

The key understanding here is that you do have a proper distance metric. That is in fact not your problem at all. Your problem is in classification.

Let me give you an example. Say you have 20 entries for the Foo X1 and 20 for the Foo Y1. You can safely assume they are two groups. On the other hand, if you have 39 entries for the Bar X1 and 1 for the Bar Y1, you should treat them as a single group.

Now, the distance X1 <-> Y1 is the same in both examples, so why is there a difference in the classification? That is because Bar Y1 is an outlier, whereas Foo Y1 isn't.

The funny part is that you do not actually need to do a whole lot of work to determine these groups up front. You simply do an recursive classification. You start out with node per group, and then add the a supernode for the two closest nodes. In the supernode, store the best assumption, the size of its subtree and the variation in it. As many of your strings will be identical, you'll soon get large subtrees with identical entries. Recursion ends with the supernode containing at the root of the tree.

Now map the canonical names against this tree. You'll quickly see that each will match an entire subtree. Now, use the distances between these trees to pick the distance cutoff for that entry. If you have both Foo X1 and Foo Y1 products in the database, the cut-off distance will need to be lower to reflect that.

Answer 4

We can use the Datadecision service for matching products.

It will allow you to automatically match your product data using statistical algorithms. This operation is done after defining a threshold score of confidence.

All data that cannot be automatically matched will have to be manually reviewed through a dedicated user interface.

The online service uses lookup tables to store synonyms as well as your manual matching history. This allows you to improve the data matching automation next time you import new data.

Answer 5

I worked on the exact same thing in the past. What I have done is using an NLP method; TF-IDF Vectorizer to assign weights to each word. For example in your case:

Canon PowerShot a20IS

• Canon --> weight = 0.05 (not a very distinguishing word)
• PowerShot --> weight = 0.37 (can be distinguishing)
• a20IS --> weight = 0.96 (very distinguishing)

This will tell your model which words to care and which words to not. I had quite good matches thanks to TF-IDF. But note this: a20IS cannot be recognized as a20 IS, you may consider to use some kind of regex to filter such cases.

After that, you can use a numeric calculation like cosine similarity.