Efficient way to create term density matrix from pandas DataFrame

Question

I am trying to create a term density matrix from a pandas dataframe, so I can rate terms appearing in the dataframe. I also want to be able to keep the \'spatial\' aspect of my

Answer 1

herrfz provides a way to handle this but I just wanted to point out that creating a term density data structure using a Python set is counterproductive, seeing as a set is a collection of unique objects. You won't be able to capture the count for each word, only the presence of a word for a given row.

return set(nltk.wordpunct_tokenize(strin)).difference(sw)

In order to strip out the stopwords you could do something like

tokens_stripped = [token for token in tokens 
                   if token not in stopwords]

after tokenization.