### 学习 Transformer 模型源码实现原理 #### 一、Transformer架构概述 Transformer是一种基于自注意力机制的序列到序列模型，在自然语言处理领域取得了显著的成功。该模型摒弃了传统的RNN结构，完全依赖于自注意机制来捕捉输入和输出之间的全局依赖关系[^1]。 #### 二、编码器(Encoder) 编码器由多个相同的层堆叠而成，每一层主要包含两个子层：一个多头自注意力机制(Multi-head Self-Attention)，用于计算输入序列中各个位置的重要性；另一个是全连接前馈网络(Fully Connected Feed-Forward Network)[^2]。具体来说： - 多头自注意力机制允许模型在同一时间关注不同位置的信息，从而更好地理解上下文语义； - 前馈神经网络则负责对每个位置上的特征进行非线性变换。 为了提高训练稳定性并加速收敛速度，在这两个子层之后还加入了残差连接(Residual Connection)以及Layer Normalization操作[^3]。 ```python class EncoderBlock(nn.Module): def __init__(self, d_model, num_heads, dropout=0.1): super().__init__() self.self_attn = MultiHeadedAttention(num_heads=num_heads, d_model=d_model) self.feed_forward = PositionwiseFeedForward(d_model=d_model, hidden_size=hidden_size) self.layer_norm_1 = nn.LayerNorm(d_model) self.layer_norm_2 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(dropout) def forward(self, x, mask=None): attn_output = self.self_attn(x, x, x, mask) output = self.layer_norm_1(x + self.dropout(attn_output)) ff_output = self.feed_forward(output) output = self.layer_norm_2(output + self.dropout(ff_output)) return output ``` #### 三、解码器(Decoder) 类似于编码器的设计思路，解码器也包含了类似的两部分——遮蔽后的多头自注意力机制（Masked Multi-Head Attention）与编解码交互式的多头注意力机制(Cross-Multihead Attention)。前者用来防止当前位置看到未来时刻的内容；后者则是让当前时刻能够获取到来自整个输入序列的有效信息。 ```python class DecoderBlock(nn.Module): def __init__(self, d_model, num_heads, dropout=0.1): super().__init__() self.masked_self_attn = MaskedMultiHeadedAttention(num_heads=num_heads, d_model=d_model) self.cross_attn = CrossMultiHeadedAttention(num_heads=num_heads, d_model=d_model) self.feed_forward = PositionwiseFeedForward(d_model=d_model, hidden_size=hidden_size) self.layer_norm_1 = nn.LayerNorm(d_model) self.layer_norm_2 = nn.LayerNorm(d_model) self.layer_norm_3 = nn.LayerNorm(d_model) self.dropout = nn.Dropout(dropout) def forward(self, tgt, memory, src_mask=None, tgt_mask=None): masked_self_attn_output = self.masked_self_attn(tgt, tgt, tgt, tgt_mask) normed_tgt = self.layer_norm_1(tgt + self.dropout(masked_self_attn_output)) cross_attn_output = self.cross_attn(normed_tgt, memory, memory, src_mask) normed_cross_attn_output = self.layer_norm_2(normed_tgt + self.dropout(cross_attn_output)) ff_output = self.feed_forward(normed_cross_attn_output) output = self.layer_norm_3(normed_cross_attn_output + self.dropout(ff_output)) return output ``` #### 四、整体框架搭建 按照论文中的描述，完整的Transformer模型通过重复上述所提及的基础组件而建立起来。通常情况下会设置N个相同配置下的编码单元(encoder block)串联在一起形成最终版本的编码模块；同样地也会有相应数量级大小的解码单元(decoder blocks)依次排列组成了解码端的整体逻辑流程图。 ```python def make_transformer(N=6, d_model=512, num_heads=8, dropout=0.1): c = copy.deepcopy encoder_block = EncoderBlock(d_model=d_model, num_heads=num_heads, dropout=dropout) decoder_block = DecoderBlock(d_model=d_model, num_heads=num_heads, dropout=dropout) model = EncoderDecoder( Encoder(layer=c(encoder_block), N=N), Decoder(layer=c(decoder_block), N=N), nn.Sequential(Embeddings(d_model, vocab_src), PositionalEncoding(d_model, dropout)), nn.Sequential(Embeddings(d_model, vocab_trg), PositionalEncoding(d_model, dropout)), Generator(d_model, vocab_trg) ) for p in model.parameters(): if p.dim() > 1: nn.init.xavier_uniform_(p) return model ```