attention-model | 易学教程

Getting alignment/attention during translation in OpenNMT-py

阅读更多关于 Getting alignment/attention during translation in OpenNMT-py

问题 Does anyone know how to get the alignments weights when translating in Opennmt-py? Usually the only output are the resulting sentences and I have tried to find a debugging flag or similar for the attention weights. So far, I have been unsuccessful. 回答1: You can get the attention matrices. Note that it is not the same as alignment which is a term from statistical (not neural) machine translation. There is a thread on github discussing it. Here is a snippet from the discussion. When you get the

PyTorch: How to implement attention for graph attention layer

阅读更多关于 PyTorch: How to implement attention for graph attention layer

问题 I have implemented the attention (Eq. 1) of https://arxiv.org/pdf/1710.10903.pdf but it's clearly not memory efficient and can run only a single model on my GPU (it takes 7-10GB). Currently, I have class MyModule(nn.Module): def __init__(self, in_features, out_features): super(MyModule, self).__init__() self.in_features = in_features self.out_features = out_features self.W = nn.Parameter(nn.init.xavier_uniform(torch.Tensor(in_features, out_features).type(torch.cuda.FloatTensor if torch.cuda

LSTM with Attention

阅读更多关于 LSTM with Attention

问题 I am trying to add attention mechanism to stacked LSTMs implementation https://github.com/salesforce/awd-lstm-lm All examples online use encoder-decoder architecture, which I do not want to use (do I have to for the attention mechanism?). Basically, I have used https://webcache.googleusercontent.com/search?q=cache:81Q7u36DRPIJ:https://github.com/zhedongzheng/finch/blob/master/nlp-models/pytorch/rnn_attn_text_clf.py+&cd=2&hl=en&ct=clnk&gl=uk def __init__(self, rnn_type, ntoken, ninp, nhid,

LSTM with Attention

阅读更多关于 LSTM with Attention

I am trying to add attention mechanism to stacked LSTMs implementation https://github.com/salesforce/awd-lstm-lm All examples online use encoder-decoder architecture, which I do not want to use (do I have to for the attention mechanism?). Basically, I have used https://webcache.googleusercontent.com/search?q=cache:81Q7u36DRPIJ:https://github.com/zhedongzheng/finch/blob/master/nlp-models/pytorch/rnn_attn_text_clf.py+&cd=2&hl=en&ct=clnk&gl=uk def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5, dropouth=0.5, dropouti=0.5, dropoute=0.1, wdrop=0, tie_weights=False): super

Getting alignment/attention during translation in OpenNMT-py

阅读更多关于 Getting alignment/attention during translation in OpenNMT-py

Does anyone know how to get the alignments weights when translating in Opennmt-py? Usually the only output are the resulting sentences and I have tried to find a debugging flag or similar for the attention weights. So far, I have been unsuccessful. You can get the attention matrices. Note that it is not the same as alignment which is a term from statistical (not neural) machine translation. There is a thread on github discussing it. Here is a snippet from the discussion. When you get the translations from the mode, the attentions are in the attn field. import onmt import onmt.io import onmt

Visualizing attention activation in Tensorflow

阅读更多关于 Visualizing attention activation in Tensorflow

问题 Is there a way to visualize the attention weights on some input like the figure in the link above(from Bahdanau et al., 2014), in TensorFlow's seq2seq models? I have found TensorFlow's github issue regarding this, but I couldn't find out how to fetch the attention mask during the session. 回答1: I also want to visualize the attention weights of Tensorflow seq2seq ops for my text summarization task. And I think the temporary solution is to use session.run() to evaluate the attention mask tensor