I have a text file that contains tweets per line, that need to be altered for a machine learning format. Im using python and basic unix text manipulation (regex) to achieve
from string import punctuation as pnc
tokens = {':)', 'cool', 'happy', 'fun'}
tweets = ['this has been a fun day :)', 'i find python cool! it makes me happy']
for tweet in tweets:
s = [(word in tokens or word.strip(pnc) in tokens) for word in tweet.split()]
print(' '.join('1' if t else '0' for t in s))
Output:
0 0 0 0 1 0 1
0 0 0 1 0 0 0 1
The or
in the 4th line is there to handle :)
, as suggested by @EOL.
There are still cases that will not be handled correctly, such as with cool :), I like it
. The problem is inherent to the requirements.