How can I find the entropy of a term in a collection of documents?

Question

I have a dataframe with the column text and I\'d like to tokenize it and compute the entropy of each word so that I can remove the ones with higher entropy.

This is t