问题
i'm working on an Multi-Label Emotion Classification problem to be solved by word2vec. this is my code that i've learned from a couple of tutorials. now the accuracy is very low. about 0.02 which is telling me something is wrong in my code. but i cannot find it. i tried this code for TF-IDF and BOW (obviously except word2vec part) and i got much better accuracy scores such as 0.28, but it seems this one is somehow wrong:
np.set_printoptions(threshold=sys.maxsize)
wv = gensim.models.KeyedVectors.load_word2vec_format("E:\\GoogleNews-vectors-negative300.bin", binary=True)
wv.init_sims(replace=True)
#Pre-Processor Function
pre_processor = TextPreProcessor(
omit=['url', 'email', 'percent', 'money', 'phone', 'user',
'time', 'url', 'date', 'number'],
normalize=['url', 'email', 'percent', 'money', 'phone', 'user',
'time', 'url', 'date', 'number'],
segmenter="twitter",
corrector="twitter",
unpack_hashtags=True,
unpack_contractions=True,
tokenizer=SocialTokenizer(lowercase=True).tokenize,
dicts=[emoticons]
)
#Averaging Words Vectors to Create Sentence Embedding
def word_averaging(wv, words):
all_words, mean = set(), []
for word in words:
if isinstance(word, np.ndarray):
mean.append(word)
elif word in wv.vocab:
mean.append(wv.syn0norm[wv.vocab[word].index])
all_words.add(wv.vocab[word].index)
if not mean:
logging.warning("cannot compute similarity with no input %s", words)
# FIXME: remove these examples in pre-processing
return np.zeros(wv.vector_size,)
mean = gensim.matutils.unitvec(np.array(mean).mean(axis=0)).astype(np.float32)
return mean
def word_averaging_list(wv, text_list):
return np.vstack([word_averaging(wv, post) for post in text_list ])
#Secondary Word-Averaging Method
def get_mean_vector(word2vec_model, words):
# remove out-of-vocabulary words
words = [word for word in words if word in word2vec_model.vocab]
if len(words) >= 1:
return np.mean(word2vec_model[words], axis=0)
else:
return []
#Loading data
raw_train_tweets = pandas.read_excel('E:\\train.xlsx').iloc[:,1] #Loading all train tweets
train_labels = np.array(pandas.read_excel('E:\\train.xlsx').iloc[:,2:13]) #Loading corresponding train labels (11 emotions)
raw_test_tweets = pandas.read_excel('E:\\test.xlsx').iloc[:,1] #Loading 300 test tweets
test_gold_labels = np.array(pandas.read_excel('E:\\test.xlsx').iloc[:,2:13]) #Loading corresponding test labels (11 emotions)
print("please wait")
#Pre-Processing
train_tweets=[]
test_tweets=[]
for tweets in raw_train_tweets:
train_tweets.append(pre_processor.pre_process_doc(tweets))
for tweets in raw_test_tweets:
test_tweets.append(pre_processor.pre_process_doc(tweets))
#Vectorizing
train_array = word_averaging_list(wv,train_tweets)
test_array = word_averaging_list(wv,test_tweets)
#Predicting and Evaluating
clf = LabelPowerset(LogisticRegression(solver='lbfgs', C=1, class_weight=None))
clf.fit(train_array,train_labels)
predicted= clf.predict(test_array)
intersect=0
union=0
accuracy=[]
for i in range(0,3250): #i have 3250 test tweets.
for j in range(0,11): #11 emotions
if predicted[i,j]&test_gold_labels[i,j]==1:
intersect+=1
if predicted[i,j]|test_gold_labels[i,j]==1:
union+=1
accuracy.append(intersect/union) if union !=0 else accuracy.append(0.0)
intersect=0
union=0
print(np.mean(accuracy))
The Result:
0.4674498168498169
And i printed predicted variable (for tweet 0 to 10) to see how it looks like:
(0, 0) 1
(0, 2) 1
(2, 0) 1
(2, 2) 1
(3, 4) 1
(3, 6) 1
(4, 0) 1
(4, 2) 1
(5, 0) 1
(5, 2) 1
(6, 0) 1
(6, 2) 1
(7, 0) 1
(7, 2) 1
(8, 4) 1
(8, 6) 1
(9, 3) 1
(9, 8) 1
as you can see, it only show 1's. for example (6,2) means in tweet number 6, emotion number 2 is 1. (9,8) means in tweet number 9, emotion number 8 is 1. the other emotions considered as 0. but you can imagine it like this to better understand what i've done in Accuracy method:
gold emotion for tweet 0: [1 1 0 0 0 0 1 0 0 0 1]
predicted emotion for tweet 0: [1 0 1 0 0 0 0 0 0 0 0]
i've used union and intersect for the indexes one by one. 1 to 1. 1 to 1. 0 to 1, until gold emotion 11 to predicted emotion 11. i did this for all tweets in two for loops.
Creating Word2Vec vectors on my tweets:
now i want to use gensim to create Word2Vec vectors on my tweet dataset. i changed some parts of the code above as below:
#Averaging Words Vectors to Create Sentence Embedding
def word_averaging(wv, words):
all_words, mean = set(), []
for word in words:
if isinstance(word, np.ndarray):
mean.append(word)
elif word in wv.vocab:
mean.append(wv.syn0norm[wv.vocab[word].index])
all_words.add(wv.vocab[word].index)
if not mean:
logging.warning("cannot compute similarity with no input %s", words)
# FIXME: remove these examples in pre-processing
return np.zeros(wv.vector_size,)
mean = gensim.matutils.unitvec(np.array(mean).mean(axis=0)).astype(np.float32)
return mean
def word_averaging_list(wv, text_list):
return np.vstack([word_averaging(wv, post) for post in text_list ])
#Loading data
raw_aggregate_tweets = pandas.read_excel('E:\\aggregate.xlsx').iloc[:,0] #Loading all train tweets
raw_train_tweets = pandas.read_excel('E:\\train.xlsx').iloc[:,1] #Loading all train tweets
train_labels = np.array(pandas.read_excel('E:\\train.xlsx').iloc[:,2:13]) #Loading corresponding train labels (11 emotions)
raw_test_tweets = pandas.read_excel('E:\\test.xlsx').iloc[:,1] #Loading 300 test tweets
test_gold_labels = np.array(pandas.read_excel('E:\\test.xlsx').iloc[:,2:13]) #Loading corresponding test labels (11 emotions)
print("please wait")
#Pre-Processing
aggregate_tweets=[]
train_tweets=[]
test_tweets=[]
for tweets in raw_aggregate_tweets:
aggregate_tweets.append(pre_processor.pre_process_doc(tweets))
for tweets in raw_train_tweets:
train_tweets.append(pre_processor.pre_process_doc(tweets))
for tweets in raw_test_tweets:
test_tweets.append(pre_processor.pre_process_doc(tweets))
print(len(aggregate_tweets))
#Vectorizing
w2v_model = gensim.models.Word2Vec(aggregate_tweets, min_count = 10, size = 300, window = 8)
print(w2v_model.wv.vectors.shape)
train_array = word_averaging_list(w2v_model.wv,train_tweets)
test_array = word_averaging_list(w2v_model.wv,test_tweets)
but i get this error:
TypeError Traceback (most recent call last)
<ipython-input-1-8a5fe4dbf144> in <module>
110 print(w2v_model.wv.vectors.shape)
111
--> 112 train_array = word_averaging_list(w2v_model.wv,train_tweets)
113 test_array = word_averaging_list(w2v_model.wv,test_tweets)
114
<ipython-input-1-8a5fe4dbf144> in word_averaging_list(wv, text_list)
70
71 def word_averaging_list(wv, text_list):
---> 72 return np.vstack([word_averaging(wv, post) for post in text_list ])
73
74 #Averaging Words Vectors to Create Sentence Embedding
<ipython-input-1-8a5fe4dbf144> in <listcomp>(.0)
70
71 def word_averaging_list(wv, text_list):
---> 72 return np.vstack([word_averaging(wv, post) for post in text_list ])
73
74 #Averaging Words Vectors to Create Sentence Embedding
<ipython-input-1-8a5fe4dbf144> in word_averaging(wv, words)
58 mean.append(word)
59 elif word in wv.vocab:
---> 60 mean.append(wv.syn0norm[wv.vocab[word].index])
61 all_words.add(wv.vocab[word].index)
62
TypeError: 'NoneType' object is not subscriptable
回答1:
It's not clear what your TextPreProcessor or SocialTokenizer classes might do. You should edit your question to either show their code, or show a few examples of the resulting texts to make sure it's doing what you expect. (For example: show the first few and last few entries of all_tweets.)
It's not likely that your line all_tweets = train_tweets.append(test_tweets) is doing what you expect. (It'll put the entire list test_tweets as the final element of all_tweets – but then return None which you assign to all_tweets. Your Word2Vec model might then be empty - you should enable INFO logging to watch its progress & review the output for anomalies, and add code post-training to print some details about the model that confirm useful training occurred.)
Are you sure train_tweets is the right format for your pipeline to .fit() against? (The texts sent to Word2Vec training seem to have been tokenized via a .split(), but the texts in the pandas.Series train_tweets may never have been tokenized.)
Generally, a good idea is to enable logging, and add more code after each step confirming, via checking property values or printing excerpts of the longer collections, that each step has had the intended effect.
来源：https://stackoverflow.com/questions/60738355/classification-accuracy-is-too-low-word2vec