How do I use sklearn CountVectorizer with both 'word' and 'char' analyzer? - python

Question

How do I use sklearn CountVectorizer with both \'word\' and \'char\' analyzer? http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectoriz

Answer 1

You can combine arbitrary feature extraction steps with the FeatureUnion estimator: http://scikit-learn.org/dev/modules/pipeline.html#featureunion-combining-feature-extractors

In this case this is probably less efficient than larsmans solution, but might be easier to use.

Answer 2

You can pass a callable as the analyzer argument to get full control over the tokenization, e.g.

>>> from pprint import pprint
>>> import re
>>> x = ['this is a foo bar', 'you are a foo bar black sheep']
>>> def words_and_char_bigrams(text):
...     words = re.findall(r'\w{3,}', text)
...     for w in words:
...         yield w
...         for i in range(len(w) - 2):
...             yield w[i:i+2]
...             
>>> v = CountVectorizer(analyzer=words_and_char_bigrams)
>>> pprint(v.fit(x).vocabulary_)
{'ac': 0,
 'ar': 1,
 'are': 2,
 'ba': 3,
 'bar': 4,
 'bl': 5,
 'black': 6,
 'ee': 7,
 'fo': 8,
 'foo': 9,
 'he': 10,
 'hi': 11,
 'la': 12,
 'sh': 13,
 'sheep': 14,
 'th': 15,
 'this': 16,
 'yo': 17,
 'you': 18}