Label not x is present in all training examples

问题

Hello, I have come across an issue when trying to predict tag/label on my project. I am currently using similar tutorial (with my own data) to predict complain in complaint register based on given tag such as 1 Complaint --> many Genre (Warranty, Refund, Air Conditioning)

DF -> Tag No of Columns -> 4 (original), 2 (clean-up) > genre_new and clean_plot Column Names ->ID, Plot, Title, Genre, genre_new, clean_plot

I used this tutorial: https://www.analyticsvidhya.com/blog/2019/04/predicting-movie-genres-nlp-multi-label-classification/. This is to predict movies with multiple Genre such as 1 movies has multiple Genre

I also found solution on UserWarning: Label not :NUMBER: is present in all training examples

Problem: The issue is likely to be that some tags occur just in a few documents . When you split the dataset into train and test to validate your model, it may happen that some tags are missing from the training data.

Error: label warning and 0 prediction

But I am not sure how to do write this workaround to cater my code as I am not a coder. Please help.

Please refer to my google drive link https://drive.google.com/drive/folders/10yLOVWZPgl1shVwwM5qDy7iyMCm7cS9A?usp=sharing

回答1:

from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
import pandas as pd
from sklearn.model_selection import train_test_split

mlb = MultiLabelBinarizer()
vect = CountVectorizer()
tfidf = TfidfTransformer()

lr = LogisticRegression()
clf = OneVsRestClassifier(lr)

df = pd.read_excel("Building Compliants in 2018 for training(1).xls")
df['Genre'] = df['Genre'].apply(lambda x: x.split(','))

y = mlb.fit_transform(df['Genre'])

train_data_vect = vect.fit_transform(df['Plot'])
train_data_tfidf = tfidf.fit_transform(train_data_vect)

x_train, x_test, y_train, y_test=train_test_split(train_data_tfidf,y, test_size=0.25)

clf.fit(x_train,y_train) #train your model on train data
print(clf.score(x_test,y_test)) #check score on test data
#op


Out[29]:
0.3333333333333333

#now for predicting , taking first element of Plot column

text =  df['Plot'][0]
vect_transform = vect.transform([text])
tfidf_transform = tfidf.transform(vect_transform)

clf.predict(tfidf_transform)
#array([[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0]])

mlb.inverse_transform(clf.predict(tfidf_transform))
#op
[(' Warranty', 'Airconditioning')]

def infer_tags(q):
    q = clean_text(q)
    q = remove_stopwords(q)
    q_vec = tfidf.transform([q])
    q_pred = clf.predict(q_vec)
    #print(q)
    return MultiLabelBinarizer.inverse_transform(q_pred)


 for i in range(100): 
     k = x_test.sample(i).index[2] 
     #print("Trader: ", Tag['Title'][k])
     print("Trader: ", Tag['Title'][k], "\nPredicted genre: ",infer_tags(x_test[k]))
     print("Actual genre: ",Tag['Genre'][k], "\n")

#op
   Traceback (most recent call last):
       File "<ipython-input-70-28cc8e8a7204>", line 11, in <module>
       k = x_test.sample(i).index[2]
       File "C:\Users\LAUJ3\Documents\Python Project\env\lib\site- 
       packages\scipy\sparse\base.py", line 688, in __getattr__
       raise AttributeError(attr + " not found")

    AttributeError: sample not found

来源：https://stackoverflow.com/questions/59349516/label-not-x-is-present-in-all-training-examples

标签

python

scikit-learn

multilabel-classification