python nltk keyword extraction from sentence

Question

\"First thing we do, let\'s kill all the lawyers.\" - William Shakespeare

Given the quote above, I would like to pull out \"k

Answer 1

One simple approach would be to keep stop word lists for NN, VB etc. These would be high frequency words that usually don't add much semantic content to a sentence.

The snippet below shows distinct lists for each type of word token, but you could just as well employ a single stop word list for both verbs and nouns (such as this one).

stop_words = dict(
    NNP=['first', 'second'],
    NN=['thing'],
    VBP=['do','done'],
    VB=[],
    NNS=['lets', 'things'],
)


def filter_stop_words(pos_list):
    return [[token, token_type] 
            for token, token_type in pos_list 
            if token.lower() not in stop_words[token_type]]