What are some good methods to find the “relatedness” of two bodies of text?

Question

Here\'s the problem -- I have a few thousand small text snippets, anywhere from a few words to a few sentences - the largest snippet is about 2k on disk. I want to be able to c

Answer 1

This is quite doable for reasonable large texts, however harder for smaller texts.

I did it once like this, and it worked pretty well:

• Filter all "general" words (like a, an, the, in, etc...) (filters about 10-30% of the words)
• Count the frequencies of the remaining words, store the top x of most frequent words, these are your topics.
• As an extra step you can create groups of 2/3/4 subsequent words and compare them with the groups in other texts. I used it as a measure for plagerism.