Algorithm to classify a list of products? [closed]

Answer 1

There's no easy solution for this kind of problem. Especially if your list is really large (millions of items). Maybe those two papers can point you in the right direction:

http://www.cs.utexas.edu/users/ml/papers/normalization-icdm-05.pdf http://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle_SchmidtThieme2006-Object_Identification_with_Constraints.pdf

Answer 2

MALLET has implementations of CRFs and MaxEnt that can probably do the job well. As someone said earlier you'll need to extract the features first and then feed them into your classifier.

Answer 3

You need at least two components:

First, you need something that does "feature" extraction, i.e. that takes your items and extracts the relevant information. For example, "new and shinny" is not as relevant as "500Go hard drive" and "seagate". A (very) simple approach would consist of a simple heuristic extracting manufacturers, technology names like "USB2.0" and patterns like "GB", "RPM" from each item.

You then end up with a set of features for each item. Some machine learning people like to put this into a "feature vector", i.e. it has one entry for each feature, being set to 0 or 1, depending on whether the feature exists or not. This is your data representation. On this vectors you can then do a distance comparison.

Note that you might end up with a vector of thousands of entries. Even then, you then have to cluster your results.

Possibly useful Wikipedia articles:

• Feature Extraction
• Nearest Neighbour Search

Answer 4

To actually classify a product, you could use somewhat of a "enhanced neural network" with a blackboard. (This is just a metaphore to get you thinking in the right direction, not a strict use of the terms.)

Imagine a set of objects that are connected through listeners or events (just like neurons and synapsis). Each object has a set of patterns and tests the input against these patterns.

An example:

• One object tests for ("seagate"|"connor"|"maxtor"|"quantum"| ...)
• Another object tests for [:digit:]*(" ")?("gb"|"mb")
• Another object tests for [:digit:]*(" ")?"rpm"

All these objects connect to another object that, if certain combinations of them fire, categorizes the input as a hard drive. The individual objects themselves would enter certain characterizations into the black board (common writing area to say things about the input) such as manufacturer, capacity, or speed.

So the neurons do not fire based on a threshhold, but on a recognition of a pattern. Many of these neurons can work highly parallel on the blackboard and even correct categorizations by other neurons (maybe introducing certainties?)

I used something like this in a prototype for a product used to classify products according to UNSPSC and was able to get 97% correct classification on car parts.

Answer 5

First I'd use a CountVectorizer to look at the vocabulary generated. There'd be words like 'from', 'laptop', 'fast', 'silver' etc. You can use stop words to discard such words that give us no information. I'd also go ahead and discard 'hard', 'drive', 'hard drive' etc. because I know this is a list of hard drives so they provide no information. Then we'd have list of words like

1. Seagate 500Go
2. Seagate 120Go
3. Seagate Barracuda 7200.12 ST3500418AS 500GB 7200 RPM SATA 3.0Gb/s
4. 500Go Seagate etc.

You can use list of features like things that end with RPM are likely to give RPM information, same goes with stuff ending with mb/s or Gb/s. Then I'd discard alphanumeric characters like '1234FBA5235' which is most likely model numbers etc. which won't give us much information. Now if you are already aware of brands of hard drives that are appearing in your list like 'Seagate' 'Kingston' you can use string similarity or simply check if they are present in the given sentence. Once that's done you can use Clustering to group similar objects together. Now objects with similary rpm, gb's, gb/s, brand name will be clustered together. Again, if you use something like KMeans you'd have to figure out the best value of K. You'll have to do some manual work. What you could do it use a scatter plot and eyeball for which value of K the data classifies the best.

But the problem in above approach is if you don't know before hand the list of brands then you'd be in trouble. Then I'd use Bayesian Classifier to look for every sentence and get the probability of it being a hard drive brand. I'd look for two things

1. Look at the data, most of the times the sentence would explicitly mention the word 'hard drive' then I'd know it's definitely talking about a hard drive. Chances for something like 'Mercedes Benz hard drive' are slim.
2. This is a bit laborious but I'd write a Python web scrapper over Amazon (or if you can't write one just Google for most used Hard Drive brands and create a list) It give me list like 'Seagate Barracuda 7200.12 ST3500418AS 500GB 7200 RPM SATA 3.0Gb/s' now for every sentence it'd use something like Naive Bayes to give me probability it's a brand. sklearn come pretty handy to do this stuff.

Answer 6

To be honest, this seems more like a Record Linkage problem than a classification problem. You don't know ahead of time what all of the classes are, right? But you do want to figure out which product names refer to the same products, and which refer to different ones?