### 使用 Transformer 模型实现图片分类与识别的最佳实践 Transformer 模型最初设计用于自然语言处理任务，但由于其强大的建模能力以及自注意力机制的优势，在计算机视觉领域也得到了广泛应用。以下是关于如何使用 Transformer 实现图像分类与识别的一些最佳实践： #### 1. 数据预处理的重要性 在训练基于 Transformer 的图像分类模型之前，数据预处理是一个至关重要的环节。通常需要对输入图像进行标准化、裁剪和调整大小操作，以便它们能够适配到固定尺寸的补丁序列中[^3]。 ```python import tensorflow as tf def preprocess_image(image, label): image = tf.image.resize(image, (224, 224)) # 调整至统一尺寸 image /= 255.0 # 归一化像素值范围 [0, 1] return image, label ``` #### 2. 图像分块（Patching） 为了使 Transformer 更好地适应二维图像结构，通常会将图像划分为多个不重叠的小块（patches），并将其展平成向量形式作为输入特征。这种策略借鉴了 ViT（Vision Transformer）的设计理念[^2]。 ```python patch_size = 16 # 假设每一块为 16x16 像素 num_patches = (image_height // patch_size) * (image_width // patch_size) def extract_patches(images): batch_size = tf.shape(images)[0] patches = tf.image.extract_patches( images=images, sizes=[1, patch_size, patch_size, 1], strides=[1, patch_size, patch_size, 1], rates=[1, 1, 1, 1], padding="VALID", ) patches = tf.reshape(patches, [batch_size, num_patches, -1]) return patches ``` #### 3. 添加位置编码 由于 Transformer 并未显式考虑输入顺序的信息，因此需引入位置编码来增强模型对于空间关系的理解能力。这一步骤同样是模仿 NLP 领域中的做法[^1]。 ```python class PositionEmbedding(tf.keras.layers.Layer): def __init__(self, num_patches, projection_dim): super(PositionEmbedding, self).__init__() self.position_embedding_layer = tf.keras.layers.Embedding(input_dim=num_patches, output_dim=projection_dim) def call(self, inputs): positions = tf.range(start=0, limit=self.num_patches, delta=1) encoded_positions = self.position_embedding_layer(positions) return inputs + encoded_positions ``` #### 4. 构建多头自注意模块 核心部分在于构建 Multi-head Self-Attention 层次，这是让 Transformer 可以捕捉全局依赖性的关键所在。 ```python from tensorflow.keras import layers class MultiHeadSelfAttention(layers.Layer): def __init__(self, embed_dim, num_heads=8): super(MultiHeadSelfAttention, self).__init__() self.embed_dim = embed_dim self.num_heads = num_heads if embed_dim % num_heads != 0: raise ValueError(f"embedding dimension={embed_dim} should be divisible by number of heads={num_heads}") self.projection_dim = embed_dim // num_heads self.query_dense = layers.Dense(embed_dim) self.key_dense = layers.Dense(embed_dim) self.value_dense = layers.Dense(embed_dim) self.combine_heads = layers.Dense(embed_dim) def attention(self, query, key, value): score = tf.matmul(query, key, transpose_b=True) dim_key = tf.cast(tf.shape(key)[-1], dtype=tf.float32) scaled_score = score / tf.math.sqrt(dim_key) weights = tf.nn.softmax(scaled_score, axis=-1) output = tf.matmul(weights, value) return output def separate_heads(self, x, batch_size): x = tf.reshape(x, (batch_size, -1, self.num_heads, self.projection_dim)) return tf.transpose(x, perm=[0, 2, 1, 3]) def call(self, inputs): batch_size = tf.shape(inputs)[0] query = self.separate_heads(self.query_dense(inputs), batch_size) key = self.separate_heads(self.key_dense(inputs), batch_size) value = self.separate_heads(self.value_dense(inputs), batch_size) attention_output = self.attention(query, key, value) attention_output = tf.transpose(attention_output, perm=[0, 2, 1, 3]) concat_attention = tf.reshape(attention_output, (batch_size, -1, self.embed_dim)) outputs = self.combine_heads(concat_attention) return outputs ``` #### 5. 定义完整的 Vision Transformer 结构 最后通过堆叠若干层上述组件即可完成整个网络架构定义工作，并利用标准优化算法对其进行端到端训练过程。 ```python def create_vit_classifier(): input_shape = (224, 224, 3) patch_size = 16 num_classes = 1000 hidden_size = 768 transformer_layers = 12 inputs = layers.Input(shape=input_shape) augmented = data_augmentation(inputs) patches = Patches(patch_size)(augmented) encoded_patches = PatchEncoder(num_patches, hidden_size)(patches) for _ in range(transformer_layers): x1 = layers.LayerNormalization(epsilon=1e-6)(encoded_patches) attention_output = MultiHeadSelfAttention(hidden_size, num_heads=12)(x1) x2 = layers.Add()([attention_output, encoded_patches]) x3 = layers.LayerNormalization(epsilon=1e-6)(x2) x3 = mlp(x3, hidden_units=[hidden_size*4], dropout_rate=0.1) encoded_patches = layers.Add()([x3, x2]) representation = layers.LayerNormalization(epsilon=1e-6)(encoded_patches) representation = layers.Flatten()(representation) representation = layers.Dropout(0.5)(representation) logits = layers.Dense(num_classes)(representation) model = tf.keras.Model(inputs=inputs, outputs=logits) return model ``` ---