上两篇已经稍微解析了注意力ATTN和编码器EncoderRNN
现在我们来看seq2seq模型最后的一个部件。decoder解码器
class LuongAttnDecoderRNN(nn.Module):
def __init__(self, attn_model, embedding, hidden_size, output_size, n_layers=1, dropout=0.1):
super(LuongAttnDecoderRNN, self).__init__()
# Keep for reference
self.attn_model = attn_model
self.hidden_size = hidden_size
self.output_size = output_size
self.n_layers = n_layers
self.dropout = dropout
# Define layers
self.embedding = embedding
self.embedding_dropout = nn.Dropout(dropout)
self.gru = nn.GRU(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout))
self.concat = nn.Linear(hidden_size * 2, hidden_size)
self.out = nn.Linear(hidden_size, output_size)
# Choose attention model
if attn_model != 'none':
self.attn = Attn(attn_model, hidden_size)
def forward(self, input_seq, last_hidden, encoder_outputs):
# Note: we run this one step at a time
# Get the embedding of the current input word (last output word)
embedded = self.embedding(input_seq)
embedded = self.embedding_dropout(embedded) #[1, 64, 512]
if(embedded.size(0) != 1):
raise ValueError('Decoder input sequence length should be 1')
# Get current hidden state from input word and last hidden state
rnn_output, hidden = self.gru(embedded, last_hidden)
# Calculate attention from current RNN state and all encoder outputs;
# apply to encoder outputs to get weighted average
attn_weights = self.attn(rnn_output, encoder_outputs) #[64, 1, 14]
# encoder_outputs [14, 64, 512]
context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) #[64, 1, 512]
# Attentional vector using the RNN hidden state and context vector
# concatenated together (Luong eq. 5)
rnn_output = rnn_output.squeeze(0) #[64, 512]
context = context.squeeze(1) #[64, 512]
concat_input = torch.cat((rnn_output, context), 1) #[64, 1024]
concat_output = torch.tanh(self.concat(concat_input)) #[64, 512]
# Finally predict next token (Luong eq. 6, without softmax)
output = self.out(concat_output) #[64, output_size]
# Return final output, hidden state, and attention weights (for visualization)
return output, hidden, attn_weights
首先同样的初始化
init参数attn_model, embedding, hidden_size, output_size, n_layers=1, dropout=0.1
这段没事好讲的
self.attn_model = attn_model ~ attn的模型 str类型参考上篇ATTN初始化参数
self.hidden_size = hidden_size ~rnn的隐藏层大小
self.output_size = output_size ~rnn的输出层大小
self.n_layers = n_layers ~rnn的层数
self.dropout = dropout ~遗忘率
self.embedding = embedding ~嵌入层
self.embedding_dropout = nn.Dropout(dropout) ~嵌入层的遗忘
self.gru = nn.GRU(hidden_size, hidden_size, n_layers, dropout=(0 if n_layers == 1 else dropout)) ~创建GRU
self.concat = nn.Linear(hidden_size * 2, hidden_size) ~创建全连接层
self.out = nn.Linear(hidden_size, output_size) ~创建全连接层
~选择注意模型 attn_model是str类型参考上篇ATTN初始化参数
if attn_model != 'none':
self.attn = Attn(attn_model, hidden_size)
好的这样我们初始化就大概了解了
接着我们到具体的执行方法forward
传入的参数input_seq,~ 输入句子
last_hidden, ~最后层的大小
encoder_outputs~编码输出
Get the embedding of the current input word (last output word)
获取当前输入字(最后输出字)的嵌入
根据翻译知道 input_seq是最后一个字
第一层嵌入层
embedded = self.embedding(input_seq)
embedded = self.embedding_dropout(embedded) #[1, 64, 512]
if(embedded.size(0) != 1): 假如第一个不是1则直接报错
raise ValueError('Decoder input sequence length should be 1')
报错内容：解码器输入序列长度应为1
第二层GRURNN
rnn_output, hidden = self.gru(embedded, last_hidden)
第三层注意力层
attn_weights = self.attn(rnn_output, encoder_outputs)#[64, 1, 14]
获取到注意力权重
第四层 关系关联层
context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) #[64, 1, 512]
bmm计算两个tensor的矩阵乘法
转置的encoder_outputs乘以attn_weights ，两个数据关联
第五层 全连接层
维度压缩，在0起的指定位置N，去掉维数为1的的维度
rnn_output = rnn_output.squeeze(0)
context = context.squeeze(1)
concat_input = torch.cat((rnn_output, context), 1) #[64, 1024]按维数1拼接(横着拼)
concat_output = torch.tanh(self.concat(concat_input) #[64, 512]全连接并用激活函数激活
全连接
output = self.out(concat_output)
最后输出 output, hidden, attn_weights三个东西
好了，大概的解析完了，往下的篇章以后会有更深刻的理解。
