Bert Embedding Layer raises `Type Error: unsupported operand type(s) for +: 'None Type' and 'int'` with BiLSTM

Question

I\'ve problems integrating Bert Embedding Layer in a BiLSTM model for word sense disambiguation task,

Windows 10
Python 3.6.4
TenorFlow 1.12
Keras 2.2.4
No          


        

Answer 1

First of all, the results by "mean" or "first" pooling is not for all the tokens, so you got to change in call() function:

elif self.pooling == "mean": 
    result = self.bert(inputs=bert_inputs, signature="tokens", as_dict=True)["sequence_output" ] 
    pooled = result

In build_model, change to:

embedding_size = 768
in_id = Input(shape=(max_seq_length,), name="input_ids") 
in_mask = Input(shape=(max_seq_length,), name="input_masks")
in_segment = Input(shape=(max_seq_length,), name="segment_ids")

bert_inputs = [in_id, in_mask, in_segment] 
bert_output = BertLayer(n_fine_tune_layers=12, pooling="mean")(bert_inputs) 
bert_output = Reshape((max_seq_length, embedding_size))(bert_output) 

bilstm = Bidirectional(LSTM(128, dropout=0.2,recurrent_dropout=0.2,return_sequences=True))(bert_output)
output = Dense(output_size, activation="softmax")(bilstm)