Tweet with Disaster（Kaggle NLP项目实战）
项目介绍（Real or Not? NLP with Disaster Tweets）
项目kaggle链接：https://www.kaggle.com/c/nlp-getting-started/overview
在紧急情况下，Twitter已经成为一个重要的沟通渠道。智能手机的普及使人们能够实时宣布正在观察的紧急情况。正因为如此，越来越多的机构对程序化监控Twitter(即救灾组织和新闻机构)感兴趣。但是，人们并不总是清楚一个人的话是否真的在宣告一场灾难。比如下面的例子：
作者明确地使用了“燃烧”这个词，但它的意思是隐喻性的。这一点对人类来说是显而易见的，特别是在视觉辅助下。但对机器来说就不那么清楚了。
在这场竞争中，你面临着建立一个机器学习模型的挑战，该模型可以预测哪些Tweets是关于真正的灾难的，哪些Tweets不是。
EDA
数据预处理部分
1 导入数据
train = pd.read_csv('../input/nlp-getting-started/train.csv')
test = pd.read_csv('../input/nlp-getting-started/test.csv')
sample_submission = pd.read_csv('../input/nlp-getting-started/sample_submission.csv')
# Print the shape of the training data
print('{} rows and {} cols in training dataset.'.format(train.shape[0], train.shape[1]))
print('{} rows and {} cols in training dataset.'.format(test.shape[0], test.shape[1]))
# Inspecting the training data
train.head(10)
2 描述性分析
查看标签0和1的分布情况
# Frequency for taget variable
count_table = train.target.value_counts()
display(count_table)
# Plot class distribution
plt.figure(figsize=(6,5))
plt.bar('False',count_table[0],label='False',width=0.6)
plt.bar('True', count_table[1],label='True',width=0.6)
plt.legend()
plt.ylabel('Count of examples')
plt.xlabel('Category')
plt.title('Class Distribution')
plt.ylim([0,4700])
plt.show()
每条推特长度的分布
# Plot the frequency of tweets length
bins = 150
plt.figure(figsize=(18,5))
plt.hist(train[train['target']==0]['length'], label= 'False',bins=bins,alpha=0.8)
plt.hist(train[train['target']==1]['length'], label= 'True', bins=bins,alpha=0.8)
plt.xlabel('Length of text (characters)')
plt.ylabel('Count')
plt.title('Frequency of tweets length')
plt.legend(loc='best')
plt.show()
两种推特的长度分布情况对比
# Frequency of tweets length in 2 classes
fg, (ax1, ax2)=plt.subplots(1,2,figsize=(14,5))
ax1.hist(train[train['target']==0]['length'],color='red')
ax1.set_title('Distribution of fake tweets')
ax1.set_xlabel('Tweets length (characters)')
ax1.set_ylabel('Count')
ax2.hist(train[train['target']==1]['length'],color='blue')
ax2.set_title('Distribution of true tweets')
ax2.set_xlabel('Tweets length (characters)')
ax2.set_ylabel('Count')
fg.suptitle('Characater in classes')
plt.show()
两种推特出现的词的数量分布
# Plot the distribution of count of words
words_true = train[train['target']==1]['text'].str.split().apply(len)
words_false = train[train['target']==0]['text'].str.split().apply(len)
plt.figure(figsize=(10,5))
plt.hist(words_false, label='False',alpha=0.8,bins=15)
plt.hist(words_true, label='True',alpha=0.6,bins=15)
plt.legend(loc='best')
plt.title('Count of words in tweets')
plt.xlabel('Count of words')
plt.ylabel('Count')
plt.show()
3 数据清洗
定义去除所有停用词，语气符号，html符号，表情符号的函数
# Define a function to remove URL
def remove_url(text):
url = re.compile(r'https?://\S+|www\.\S+')
return url.sub(r'',text)
# Test function
test = 'Address of this kernel: https://www.kaggle.com/lilstarboy/kernel4d04fe5667/edit'
print(remove_url(test))
# Define a function to remove html tag
def remove_html(text):
html = re.compile(r'<.*?>')
return html.sub(r'',text)
# Test function
test = """<div>
<h1>Real or Fake</h1>
<p>Kaggle </p>
<a href="https://www.kaggle.com/c/nlp-getting-started">getting started</a>
</div>"""
print(remove_html(test))
# Define a function to remove emojis
def remove_emoji(text):
emoji_pattern = re.compile("["
u"\U0001F600-\U0001F64F" # emoticons
u"\U0001F300-\U0001F5FF" # symbols & pictographs
u"\U0001F680-\U0001F6FF" # transport & map symbols
u"\U0001F1E0-\U0001F1FF" # flags (iOS)
u"\U00002702-\U000027B0"
u"\U000024C2-\U0001F251"
"]+", flags=re.UNICODE)
return emoji_pattern.sub(r'', text)
remove_emoji("To test 🚀")
# Define a function to remove punctuations
def remove_punct(text):
table=str.maketrans('','',string.punctuation)
return text.translate(table)
# Define a function to convert abbreviations to text
abbreviations = {
"$" : " dollar ",
"€" : " euro ",
"4ao" : "for adults only",
"a.m" : "before midday",
"a3" : "anytime anywhere anyplace",
"aamof" : "as a matter of fact",
"acct" : "account",
"adih" : "another day in hell",
"afaic" : "as far as i am concerned",
"afaict" : "as far as i can tell",
"afaik" : "as far as i know",
"afair" : "as far as i remember",
"afk" : "away from keyboard",
"app" : "application",
"approx" : "approximately",
"apps" : "applications",
"asap" : "as soon as possible",
"asl" : "age, sex, location",
"atk" : "at the keyboard",
"ave." : "avenue",
"aymm" : "are you my mother",
"ayor" : "at your own risk",
"b&b" : "bed and breakfast",
"b+b" : "bed and breakfast",
"b.c" : "before christ",
"b2b" : "business to business",
"b2c" : "business to customer",
"b4" : "before",
"b4n" : "bye for now",
"b@u" : "back at you",
"bae" : "before anyone else",
"bak" : "back at keyboard",
"bbbg" : "bye bye be good",
"bbc" : "british broadcasting corporation",
"bbias" : "be back in a second",
"bbl" : "be back later",
"bbs" : "be back soon",
"be4" : "before",
"bfn" : "bye for now",
"blvd" : "boulevard",
"bout" : "about",
"brb" : "be right back",
"bros" : "brothers",
"brt" : "be right there",
"bsaaw" : "big smile and a wink",
"btw" : "by the way",
"bwl" : "bursting with laughter",
"c/o" : "care of",
"cet" : "central european time",
"cf" : "compare",
"cia" : "central intelligence agency",
"csl" : "can not stop laughing",
"cu" : "see you",
"cul8r" : "see you later",
"cv" : "curriculum vitae",
"cwot" : "complete waste of time",
"cya" : "see you",
"cyt" : "see you tomorrow",
"dae" : "does anyone else",
"dbmib" : "do not bother me i am busy",
"diy" : "do it yourself",
"dm" : "direct message",
"dwh" : "during work hours",
"e123" : "easy as one two three",
"eet" : "eastern european time",
"eg" : "example",
"embm" : "early morning business meeting",
"encl" : "enclosed",
"encl." : "enclosed",
"etc" : "and so on",
"faq" : "frequently asked questions",
"fawc" : "for anyone who cares",
"fb" : "facebook",
"fc" : "fingers crossed",
"fig" : "figure",
"fimh" : "forever in my heart",
"ft." : "feet",
"ft" : "featuring",
"ftl" : "for the loss",
"ftw" : "for the win",
"fwiw" : "for what it is worth",
"fyi" : "for your information",
"g9" : "genius",
"gahoy" : "get a hold of yourself",
"gal" : "get a life",
"gcse" : "general certificate of secondary education",
"gfn" : "gone for now",
"gg" : "good game",
"gl" : "good luck",
"glhf" : "good luck have fun",
"gmt" : "greenwich mean time",
"gmta" : "great minds think alike",
"gn" : "good night",
"g.o.a.t" : "greatest of all time",
"goat" : "greatest of all time",
"goi" : "get over it",
"gps" : "global positioning system",
"gr8" : "great",
"gratz" : "congratulations",
"gyal" : "girl",
"h&c" : "hot and cold",
"hp" : "horsepower",
"hr" : "hour",
"hrh" : "his royal highness",
"ht" : "height",
"ibrb" : "i will be right back",
"ic" : "i see",
"icq" : "i seek you",
"icymi" : "in case you missed it",
"idc" : "i do not care",
"idgadf" : "i do not give a damn fuck",
"idgaf" : "i do not give a fuck",
"idk" : "i do not know",
"ie" : "that is",
"i.e" : "that is",
"ifyp" : "i feel your pain",
"IG" : "instagram",
"iirc" : "if i remember correctly",
"ilu" : "i love you",
"ily" : "i love you",
"imho" : "in my humble opinion",
"imo" : "in my opinion",
"imu" : "i miss you",
"iow" : "in other words",
"irl" : "in real life",
"j4f" : "just for fun",
"jic" : "just in case",
"jk" : "just kidding",
"jsyk" : "just so you know",
"l8r" : "later",
"lb" : "pound",
"lbs" : "pounds",
"ldr" : "long distance relationship",
"lmao" : "laugh my ass off",
"lmfao" : "laugh my fucking ass off",
"lol" : "laughing out loud",
"ltd" : "limited",
"ltns" : "long time no see",
"m8" : "mate",
"mf" : "motherfucker",
"mfs" : "motherfuckers",
"mfw" : "my face when",
"mofo" : "motherfucker",
"mph" : "miles per hour",
"mr" : "mister",
"mrw" : "my reaction when",
"ms" : "miss",
"mte" : "my thoughts exactly",
"nagi" : "not a good idea",
"nbc" : "national broadcasting company",
"nbd" : "not big deal",
"nfs" : "not for sale",
"ngl" : "not going to lie",
"nhs" : "national health service",
"nrn" : "no reply necessary",
"nsfl" : "not safe for life",
"nsfw" : "not safe for work",
"nth" : "nice to have",
"nvr" : "never",
"nyc" : "new york city",
"oc" : "original content",
"og" : "original",
"ohp" : "overhead projector",
"oic" : "oh i see",
"omdb" : "over my dead body",
"omg" : "oh my god",
"omw" : "on my way",
"p.a" : "per annum",
"p.m" : "after midday",
"pm" : "prime minister",
"poc" : "people of color",
"pov" : "point of view",
"pp" : "pages",
"ppl" : "people",
"prw" : "parents are watching",
"ps" : "postscript",
"pt" : "point",
"ptb" : "please text back",
"pto" : "please turn over",
"qpsa" : "what happens", #"que pasa",
"ratchet" : "rude",
"rbtl" : "read between the lines",
"rlrt" : "real life retweet",
"rofl" : "rolling on the floor laughing",
"roflol" : "rolling on the floor laughing out loud",
"rotflmao" : "rolling on the floor laughing my ass off",
"rt" : "retweet",
"ruok" : "are you ok",
"sfw" : "safe for work",
"sk8" : "skate",
"smh" : "shake my head",
"sq" : "square",
"srsly" : "seriously",
"ssdd" : "same stuff different day",
"tbh" : "to be honest",
"tbs" : "tablespooful",
"tbsp" : "tablespooful",
"tfw" : "that feeling when",
"thks" : "thank you",
"tho" : "though",
"thx" : "thank you",
"tia" : "thanks in advance",
"til" : "today i learned",
"tl;dr" : "too long i did not read",
"tldr" : "too long i did not read",
"tmb" : "tweet me back",
"tntl" : "trying not to laugh",
"ttyl" : "talk to you later",
"u" : "you",
"u2" : "you too",
"u4e" : "yours for ever",
"utc" : "coordinated universal time",
"w/" : "with",
"w/o" : "without",
"w8" : "wait",
"wassup" : "what is up",
"wb" : "welcome back",
"wtf" : "what the fuck",
"wtg" : "way to go",
"wtpa" : "where the party at",
"wuf" : "where are you from",
"wuzup" : "what is up",
"wywh" : "wish you were here",
"yd" : "yard",
"ygtr" : "you got that right",
"ynk" : "you never know",
"zzz" : "sleeping bored and tired"
}
def convert_abbrev(word):
return abbreviations[word.lower()] if word.lower() in abbreviations.keys() else word
def convert_abbrev_in_text(text):
tokens = word_tokenize(text)
tokens = [convert_abbrev(word) for word in tokens]
text = ' '.join(tokens)
return text
# Test function
test = 'This is very complex!!!!!??'
print(remove_punct(test))
4 用词云进行可视化展示
# Wordcloud for not disaster tweets
corpus_all_0 = create_corpus(df, 0)
# Plot the wordcloud
plt.figure(figsize=(15,8))
word_cloud = WordCloud(
background_color='white',
max_font_size = 80
).generate(" ".join(corpus_all_0))
plt.imshow(word_cloud)
plt.axis('off')
plt.show()
# Wordcloud for disaster tweets
corpus_all_1 = create_corpus(df, 1)
# Plot the wordcloud
plt.figure(figsize=(15,8))
word_cloud = WordCloud(
background_color='white',
max_font_size = 80
).generate(" ".join(corpus_all_1))
plt.imshow(word_cloud)
plt.axis('off')
plt.show()
没有提及真实的灾难的推特的词云：
提及真实灾难的推特的词云
导入Bert预训练模型
介绍下Bert预训练模型：
用Bert进行迁移学习和fine-tuning的原理大家可以参考这篇论文https://arxiv.org/abs/1810.04805
这里用的是Bert-based Uncased模型，是一个12层神经网络，768个hidden layer，110M个参数的小模型（在Bert模型里面确实算小了狗头）
# Define hyperparameters
MAXLEN = 128
BATCH_SIZE = 32
NUM_EPOCHS = 5
LEARNING_RATE = 3e-6
# Import bert tokenizer, config and model
tokenizer = BertTokenizer.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt")
config = BertConfig.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json")
bert_model = TFBertModel.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-tf_model.h5",config=config)
接下来我们使用Bert自带的分词器生成词向量看看效果
# Convert the first sentence in 'text' column into word vector
text = train['text'][0]
print(text)
input_ids = tokenizer.encode(text,max_length=MAXLEN)
print(input_ids)
print(tokenizer.convert_ids_to_tokens(input_ids))
构造Bert模型输入
接下来我们就要构造Bert模型的输入层
这里的Bert预训练模型有三个输入：
- 一个二维数组（batch_size,input_length）
- 每个单词的index
- 相应的attention_mask和对应的token_type_id
输出层有两个输出
- 每个时刻的hidden state(batch_size,input_length,hidden_size),是一个三维数组
- 每个句子的向量表示（batch_size,input_length），由上一个hidden_state得到
具体设置和预设参数请参考Bert的官方GitHub：https://github.com/google-research/bert
这里我们进行了一个Bert模型输入的简单构造，每一句句子的词向量不够的长度用0补充，由于都是单个句子，所以token type都是0
# Build input values on the training data
train_input_ids = []
train_attension_mask = []
train_token_type_ids = []
for text in train['text']:
input_ids = tokenizer.encode(text,max_length=MAXLEN)
padding_length = MAXLEN-len(input_ids)
train_input_ids.append(input_ids+[0]*padding_length)
train_attension_mask.append([1]*len(input_ids)+[0]*padding_length)
train_token_type_ids.append([0]*MAXLEN)
train_input_ids = np.array(train_input_ids)
train_attension_mask = np.array(train_attension_mask)
train_token_type_ids = np.array(train_token_type_ids)
# Build input values on the testing data
test_input_ids = []
test_attension_mask = []
test_token_type_ids = []
for text in test['text']:
input_ids = tokenizer.encode(text,max_length=MAXLEN)
padding_length = MAXLEN-len(input_ids)
test_input_ids.append(input_ids+[0]*padding_length)
test_attension_mask.append([1]*len(input_ids)+[0]*padding_length)
test_token_type_ids.append([0]*MAXLEN)
test_input_ids = np.array(test_input_ids)
test_attension_mask = np.array(test_attension_mask)
test_token_type_ids = np.array(test_token_type_ids)
y_train = np.array(train['target'])
建立模型并训练
接下来我们就构造Bert模型，由于二分类任务激活函数是sigmoid，Adam优化器其他没啥好说的
# Build the Bert-base-Uncased model
input_ids = keras.layers.Input(shape=(MAXLEN,),dtype='int32')
attension_mask = keras.layers.Input(shape=(MAXLEN,),dtype='int32')
token_type_ids = keras.layers.Input(shape=(MAXLEN,),dtype='int32')
_, x = bert_model([input_ids,attension_mask,token_type_ids])
outputs = keras.layers.Dense(1,activation='sigmoid')(x)
model = keras.models.Model(inputs=[input_ids,attension_mask,token_type_ids],outputs=outputs)
model.compile(loss='binary_crossentropy',optimizer=keras.optimizers.Adam(lr=LEARNING_RATE),metrics=['accuracy'])
接下来训练
# Fit the Bert-base-Uncased model
(train_input_ids,valid_input_ids,
train_attension_mask,valid_attension_mask,
train_token_type_ids,valid_token_type_ids,y_train,y_valid) = train_test_split(train_input_ids,train_attension_mask,
train_token_type_ids,y_train,test_size=0.1,
stratify=y_train, random_state=0)
early_stopping = keras.callbacks.EarlyStopping(patience=3,restore_best_weights=True)
model.fit([train_input_ids,train_attension_mask,train_token_type_ids],y_train,
validation_data=([valid_input_ids,valid_attension_mask,valid_token_type_ids],y_valid),
batch_size = BATCH_SIZE,epochs=NUM_EPOCHS,callbacks=[early_stopping])
看看summary
model.summary()
提交结果
# Use the model to do prediction
y_pred = model.predict([test_input_ids,test_attension_mask,test_token_type_ids],batch_size=BATCH_SIZE,verbose=1).ravel()
y_pred = (y_pred>=0.5).astype(int)
# Export to submission
submission = pd.read_csv("../input/nlp-getting-started/sample_submission.csv")
submission['target'] = y_pred
submission.to_csv('nlp_prediction.csv',index=False)
调参过程这里就不详细说了，经过几次提交，得到最好的成绩是accuracy：0.83742
具体流程可以参阅我们的kaggle网页https://www.kaggle.com/lilstarboy/pig-budt758b-project-notebook?scriptVersionId=33280711
来源：oschina
链接：https://my.oschina.net/u/4302850/blog/4294785