removing stop words using spacy

二次信任 提交于 2020-07-05 11:41:05

问题


I am cleaning a column in my data frame, Sumcription, and am trying to do 3 things:

  1. Tokenize
  2. Lemmantize
  3. Remove stop words

    import spacy        
    nlp = spacy.load('en_core_web_sm', parser=False, entity=False)        
    df['Tokens'] = df.Sumcription.apply(lambda x: nlp(x))    
    spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS        
    spacy_stopwords.add('attach')
    df['Lema_Token']  = df.Tokens.apply(lambda x: " ".join([token.lemma_ for token in x if token not in spacy_stopwords]))
    

However, when I print for example:

df.Lema_Token.iloc[8]

The output still has the word attach in it: attach poster on the wall because it is cool

Why does it not remove the stop word?

I also tried this:

df['Lema_Token_Test']  = df.Tokens.apply(lambda x: [token.lemma_ for token in x if token not in spacy_stopwords])

But the str attach still appears.


回答1:


import spacy
import pandas as pd

# Load spacy model
nlp = spacy.load('en', parser=False, entity=False)        

# New stop words list 
customize_stop_words = [
    'attach'
]

# Mark them as stop words
for w in customize_stop_words:
    nlp.vocab[w].is_stop = True


# Test data
df = pd.DataFrame( {'Sumcription': ["attach poster on the wall because it is cool",
                                   "eating and sleeping"]})

# Convert each row into spacy document and return the lemma of the tokens in 
# the document if it is not a sotp word. Finally join the lemmas into as a string
df['Sumcription_lema'] = df.Sumcription.apply(lambda text: 
                                          " ".join(token.lemma_ for token in nlp(text) 
                                                   if not token.is_stop))

print (df)

Output:

   Sumcription                                   Sumcription_lema
0  attach poster on the wall because it is cool  poster wall cool
1                           eating and sleeping         eat sleep


来源:https://stackoverflow.com/questions/55817040/removing-stop-words-using-spacy

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!