问题
I am trying to measure token importance for BERT
via comparing token embedding grad value. So, to get the grad, I've copied the 2.8.0
forward of BertModel and changed it a bit:
huggingface transformers 2.8.0 BERT
https://github.com/huggingface/transformers/blob/11c3257a18c4b5e1a3c1746eefd96f180358397b/src/transformers/modeling_bert.py
Code:
embedding_output = self.embeddings(
input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
)
embedding_output = embedding_output.requires_grad_(True) # my code
encoder_outputs = self.encoder(
embedding_output,
attention_mask=extended_attention_mask,
head_mask=head_mask,
encoder_hidden_states=encoder_hidden_states,
encoder_attention_mask=encoder_extended_attention_mask,
)
sequence_output = encoder_outputs[0]
sequence_output.mean().backward() # my code
assert(embedding_output.grad is not None) # my code
Colab
link: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD
But it gives assertion error. I do not understand why and it seems to be a bug for me.
Please, help!
回答1:
I needed to add this line:
embedding_output = torch.tensor(embedding_output, requires_grad=True)
It seems, that I used .requires_grad_ method incorrectly.
来源:https://stackoverflow.com/questions/61286574/bert-token-importance-measuring-issue-grad-is-none