### 编码器-解码器架构在Transformer模型中的概述 编码器-解码器架构是Transformer模型的核心组成部分之一。此架构由两部分组成：编码器和解码器，二者均基于自注意力机制构建[^1]。 #### 编码器工作原理 编码器负责处理输入序列并将其转换成上下文表示形式。具体来说，在T5 Transformer中，编码器接收tokenized inputs作为输入，并通过多层堆叠的self-attention layers来捕捉不同位置之间的关系。每一层都会更新隐藏状态向量，最终形成一个富含语义信息的矩阵输出给解码器。 ```python class Encoder(nn.Module): def __init__(self, num_layers, d_model, num_heads,dff, input_vocab_size,maximum_position_encoding, rate=0.1): super(Encoder, self).__init__() self.d_model = d_model self.num_layers = num_layers self.embedding = tf.keras.layers.Embedding(input_vocab_size,d_model) self.pos_encoding = positional_encoding(maximum_position_encoding,self.d_model) self.enc_layers = [EncoderLayer(d_model,num_heads,dff,rate)for _ in range(num_layers)] self.dropout = tf.keras.layers.Dropout(rate) def call(self,x,training,mask): seq_len = tf.shape(x)[1] # adding embedding and position encoding. x = self.embedding(x) # (batch_size, input_seq_len, d_model) x *= tf.math.sqrt(tf.cast(self.d_model,tf.float32)) x += self.pos_encoding[:,:seq_len,:] x = self.dropout(x, training=training) for i in range(self.num_layers): x = self.enc_layers[i](x, training, mask) return x # (batch_size, input_seq_len, d_model) ``` #### 解码器功能描述 解码器的任务是从编码器传递过来的信息以及先前预测的结果逐步生成目标序列。它同样采用了类似的分层设计，不过除了常规的self-attention外还引入了cross-attention layer用于关注源端特征。这使得解码过程能够有效地利用来自两个方向的知识——既有关于当前正在产生的词项也有整个输入句子的整体理解。 ```python class Decoder(nn.Module): def __init__(self,num_layers,d_model,num_heads,dff,target_vocab_size, maximum_position_encoding,rate=0.1): super(Decoder, self).__init__() self.d_model=d_model self.num_layers=num_layers self.embedding=tf.keras.layers.Embedding(target_vocab_size,d_model) self.pos_encoding=positional_encoding(maximum_position_encoding,d_model) self.dec_layers=[DecoderLayer(d_model,num_heads,dff,rate)for _inrange(num_layers)] self.dropout=tf.keras.layers.Dropout(rate) defcall(self,x,enc_output,training,look_ahead_mask,padding_mask): seq_len=tf.shape(x)[1] attention_weights={} x=self.embedding(x)#(batch_size,tar_seq_len,d_model) x*=tf.math.sqrt(tf.cast(self.d_model,tf.float32)) x+=self.pos_encoding[:, :seq_len , :] x=self.dropout(x,training=training) foriinrange(self.num_layers): x, block1, block2 = self.dec_layers[i](x, enc_output, training, look_ahead_mask, padding_mask) attention_weights[f'decoder_layer{i+1}_block1'] = block1 attention_weights[f'decoder_layer{i+1}_block2'] = block2 returnx,attention_weights #(batch_size,tar_seq_len,d_model),{} ```