Tokenize dataframe column and create new dataframe for result

Question

I have the following dataframe

pd.DataFrame({\'category\': [1,2,1], \'names\' : [\'ab c\', \'s\', \'dm ab aaa\']})

category   names
0   1      ab c
1   2      s