### 批量梯度下降、随机梯度下降和小批量梯度下降的比较 #### 1. 方法介绍 批量梯度下降（Batch Gradient Descent）、随机梯度下降（Stochastic Gradient Descent, SGD）以及小批量梯度下降（Mini-batch Gradient Descent）都是用于优化损失函数的技术。它们的核心思想是利用梯度信息逐步调整模型参数，使损失函数达到最小值。 - **批量梯度下降**：在每一步更新中使用整个训练集计算梯度[^1]。这种方法的优点在于它能稳定地收敛到全局最优解；然而，当数据规模较大时，其计算成本较高。 - **随机梯度下降**：每次仅选取单个样本进行梯度计算并更新参数[^2]。这种方式显著降低了内存需求和计算复杂度，但在接近最优解的过程中可能会表现出较大的波动。 - **小批量梯度下降**：综合上述两种方法的特点，在每一次迭代过程中采用一小部分样本来估算梯度[^3]。此策略既保留了SGD的速度优势又减少了噪音干扰，因此成为实践中最为常用的一种变体。 #### 2. 实现过程中的关键点 为了完成从均方误差出发构建损失函数直至实现回归模型预测的任务，可以按照如下方式编写代码： ```python import numpy as np from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error import matplotlib.pyplot as plt # 数据生成 np.random.seed(42) X = 2 * np.random.rand(100, 1) y = 4 + 3 * X + np.random.randn(100, 1) # 划分训练集与测试集 X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2) m = len(X_train) # 样本数量 def compute_cost(theta, X_b, y): """计算MSE""" predictions = X_b.dot(theta) errors = predictions - y mse = (errors.T).dot(errors)[0][0]/len(y) return mse def batch_gradient_descent(X_b, y, theta_init, learning_rate=0.1, n_iterations=1000): m = len(y) cost_history = [] thetas = [theta_init] for iteration in range(n_iterations): gradients = 2/m * X_b.T.dot(X_b.dot(thetas[-1]) - y) next_theta = thetas[-1] - learning_rate * gradients thetas.append(next_theta) current_cost = compute_cost(next_theta, X_b, y) cost_history.append(current_cost) return thetas, cost_history def stochastic_gradient_descent(X_b, y, theta_init, learning_rate=0.1, n_epochs=50): m = len(y) cost_history = [] thetas = [theta_init] for epoch in range(n_epochs): indices = np.arange(m) np.random.shuffle(indices) X_b_shuffled = X_b[indices] y_shuffled = y[indices] for i in range(m): random_index = np.random.randint(m) xi = X_b[random_index:random_index+1] yi = y[random_index:random_index+1] gradients = 2 * xi.T.dot(xi.dot(thetas[-1]) - yi) next_theta = thetas[-1] - learning_rate * gradients thetas.append(next_theta) current_cost = compute_cost(next_theta, X_b, y) cost_history.append(current_cost) return thetas, cost_history def mini_batch_gradient_descent(X_b, y, theta_init, learning_rate=0.1, n_epochs=50, batch_size=16): t0, t1 = 5, 50 def learning_schedule(t): return t0 / (t + t1) m = len(y) cost_history = [] thetas = [theta_init] batches_per_epoch = int(np.ceil(m/batch_size)) for epoch in range(n_epochs): shuffled_indices = np.random.permutation(m) X_b_shuffled = X_b[shuffled_indices] y_shuffled = y[shuffled_indices] for batch_idx in range(batches_per_epoch): start_ix = batch_idx * batch_size end_ix = min(start_ix + batch_size, m) Xi = X_b_shuffled[start_ix:end_ix] yi = y_shuffled[start_ix:end_ix] gradients = 2/len(Xi) * Xi.T.dot(Xi.dot(thetas[-1]) - yi) lr = learning_schedule(epoch*batches_per_epoch + batch_idx) next_theta = thetas[-1] - lr * gradients thetas.append(next_theta) current_cost = compute_cost(next_theta, X_b, y) cost_history.append(current_cost) return thetas, cost_history # 添加偏置项 X_b = np.c_[np.ones((len(X_train), 1)), X_train] # 初始化Theta theta_initial = np.random.randn(2,1) thetas_bgd, costs_bgd = batch_gradient_descent(X_b, y_train, theta_initial) thetas_sgd, costs_sgd = stochastic_gradient_descent(X_b, y_train, theta_initial) thetas_mbgd, costs_mbgd = mini_batch_gradient_descent(X_b, y_train, theta_initial) plt.figure(figsize=(12,8)) for method_name, costs in zip(['BGD', 'SGD', 'MBGD'], [costs_bgd, costs_sgd, costs_mbgd]): plt.plot(costs[:100], label=f'{method_name} Cost') plt.xlabel('Iteration') plt.ylabel('Cost Function Value') plt.legend() plt.title('Comparison of Convergence Rates Across Methods') plt.show() final_thetas = {name: eval(f'thetas_{name.lower()}')[-1].flatten().tolist() for name in ['BGD','SGD','MBGD']} print(final_thetas) ``` #### 3. 结果分析 通过绘制不同方法下前几次迭代的成本变化曲线图可以看出各自特点及其适用场景。通常来说，如果数据集较小或者硬件资源充足，则可以选择批处理形式；而对于大规模数据集而言，考虑到效率因素更倾向于选用在线版本即随机或小批次模式[^4]。 ---