BERT token importance measuring issue. Grad is none

我只是一个虾纸丫 提交于 2020-04-30 06:36:26

问题


I am trying to measure token importance for BERT via comparing token embedding grad value. So, to get the grad, I've copied the 2.8.0 forward of BertModel and changed it a bit:

huggingface transformers 2.8.0 BERT https://github.com/huggingface/transformers/blob/11c3257a18c4b5e1a3c1746eefd96f180358397b/src/transformers/modeling_bert.py

Code:

        embedding_output = self.embeddings(
            input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
        )
        embedding_output = embedding_output.requires_grad_(True) # my code
        encoder_outputs = self.encoder(
            embedding_output,
            attention_mask=extended_attention_mask,
            head_mask=head_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_extended_attention_mask,
        )
        sequence_output = encoder_outputs[0]
        sequence_output.mean().backward() # my code
        assert(embedding_output.grad is not None) # my code

Colab link: https://colab.research.google.com/drive/1MggBUaDWAAZNuXbTDM11E8jvdMGEkuRD But it gives assertion error. I do not understand why and it seems to be a bug for me. Please, help!


回答1:


I needed to add this line:

embedding_output = torch.tensor(embedding_output, requires_grad=True)

It seems, that I used .requires_grad_ method incorrectly.



来源:https://stackoverflow.com/questions/61286574/bert-token-importance-measuring-issue-grad-is-none

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!