How to preserve punctuation marks in Scikit-Learn text CountVectorizer or TfidfVectorizer?

Question

Is there any way for me to preserve punctuation marks of !, ?, \" and \' from my text documents using text CountVectorizer or TfidfVectorizer param

Answer 1

You should customize the token_pattern parameter when you instantiate the vectorizer. For example:

vent = CountVectorizer(token_pattern=r"(?u)\b\w\w+\b|!|\?|\"|\'")