How to treat numbers inside text strings when vectorizing words?

问题

If I have a text string to be vectorized, how should I handle numbers inside it? Or if I feed a Neural Network with numbers and words, how can I keep the numbers as numbers?

I am planning on making a dictionary of all my words (as suggested here). In this case all strings will become arrays of numbers. How should I handle characters that are numbers? how to output a vector that does not mix the word index with the number character?

Does converting numbers to strings weakens the information i feed the network?

回答1:

Expanding your discussion with @user1735003 - Lets consider both ways of representing numbers:

Treating it as string and considering it as another word and assign an ID to it when forming a dictionary. Or
Converting the numbers to actual words : '1' becomes 'one', '2' as 'two' and so on.

Does the second one change the context in anyway?. To verify it we can find similarity of two representations using word2vec. The scores will be high if they have similar context.

For example, 1 and one have a similarity score of 0.17, 2 and two have a similarity score of 0.23. They seem to suggest that the context of how they are used is totally different.

By treating the numbers as another word, you are not changing the context but by doing any other transformation on those numbers, you can't guarantee its for better. So, its better to leave it untouched and treat it as another word.

Note: Both word-2-vec and glove were trained by treating the numbers as strings (case 1).

回答2:

The link you provide suggests that everything resulting from a .split(' ') is indexed -- words, but also numbers, possibly smileys, aso. (I would still take care of punctuation marks). Unless you have more prior knowledge about your data or your problem you could start with that.

EDIT

Example literally using your string and their code:

corpus = {'my car number 3'}
dictionary = {}
i = 1
for tweet in corpus:
  for word in tweet.split(" "):
    if word not in dictionary: dictionary[word] = i
    i += 1
print(dictionary)
# {'my': 1, '3': 4, 'car': 2, 'number': 3}