NLP: any easy and good methods to find semantic similarity between words?

Question

I don\'t know whether StackOverflow covers NLP, so I am gonna give this a shot. I am interested to find the semantic relatedness of two words from a specific domain, i.e. \"imag

Answer 1

Take a look at Latent Semantic Indexing http://en.wikipedia.org/wiki/Latent_semantic_indexing it specifically addresses your problem. However you need to come up with some way to correlate these meta concepts with either positive or negative sentiments. Sentiment analysis http://en.wikipedia.org/wiki/Sentiment_analysis should help you.

Answer 2

Re your comment:

1. Classifiation through machine learning is being used for NLP all the time.
2. Regarding semantic similarity between concepts, see Dekang Lin's information theoretic definition of similarity.

Please also see these questions: finding related words,semantic similarity of two phrases.

Answer 3

Word-Space is definitely the way to go here.If LSA is to slow for your application and if the semantics in random-indexing is too shallow the you should consider api.cortical.io . This REST API can give you the semantic fingerprint representation of any word. This semantic fingerprint contains all the different contexts to which the words belong. You can disambiguate any word wit one call like "organ" returns (muscle, piano, church, membership...) And for each of the contexts you can get contextual terms: "piano" will give (organ, clarinet, violin, flute, cello, compositions, harpsichord, orchestral) Concerning your last aspect, these semantic fingerprints are fully language independent. Currently cortical.io API covers: English, Spanish, French, German, Danish, Arabic, Russian, Chinese. More languages are being published until the end of 2014.

Answer 4

In order to find semantic similarity between words, a word space model should do the trick. Such a model can be implemented very easily and fairly efficiently. Most likely, you will want to implement some sort of dimensionality reduction. The easiest one I can think of is Random Indexing, which has been used extensively in NLP.

Once you have your word space model, you can calculate distances (e.g. cosine distance) between words. In such a model, you should get the results you mentioned earlier (distance between "focus" and "Details" should be higher than "camera weight" vs "flash").

Hope this helps!

Answer 5

I saw word2vec on HackerNews a couple of weeks ago, looks pretty close to what you want.

Answer 6

You might want to take a look at the book Opinion mining and sentiment analysis. If you are only interested in similarity of words and phrases, this survey paper may help you: From Frequency to Meaning: Vector Space Models of Semantics