基于提供的参考资料，《Attention Is All You Need》论文提出了一种基于注意力机制的Transformer架构，该架构摒弃了传统的循环神经网络和卷积神经网络，实现了序列到序列任务的并行化处理[ref_2]。 Transformer模型的核心组件包括： **编码器-解码器结构** - 编码器由N个相同层堆叠而成，每层包含多头自注意力机制和前馈神经网络 - 解码器在编码器基础上增加编码器-解码器注意力层 - 各子层均采用残差连接和层归一化[ref_3] **多头注意力机制** ```python class MultiHeadAttention(nn.Module): def __init__(self, d_model, num_heads): super().__init__() self.num_heads = num_heads self.d_k = d_model // num_heads self.w_q = nn.Linear(d_model, d_model) self.w_k = nn.Linear(d_model, d_model) self.w_v = nn.Linear(d_model, d_model) self.w_o = nn.Linear(d_model, d_model) def scaled_dot_product_attention(self, q, k, v, mask=None): scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k) if mask is not None: scores = scores.masked_fill(mask == 0, -1e9) attention = F.softmax(scores, dim=-1) return torch.matmul(attention, v) ``` [ref_3] **位置编码** ```python class PositionalEncoding(nn.Module): def __init__(self, d_model, max_len=5000): super().__init__() pe = torch.zeros(max_len, d_model) position = torch.arange(0, max_len).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model)) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) self.register_buffer('pe', pe.unsqueeze(0)) def forward(self, x): return x + self.pe[:, :x.size(1)] ``` [ref_3] 该模型在机器翻译任务中表现出色，通过自注意力机制有效捕捉长距离依赖关系，实现了训练效率和性能的双重提升[ref_1][ref_4]。