当然可以。为了更好地演示如何通过去除异常点并评估模型性能，我们将使用Python编写一段完整的代码示例。在这个例子中，我们首先生成一组包含噪声的数据，然后尝试识别和移除其中的异常值，最后分别建立线性回归模型并比较其性能。 我们将重点展示以下内容： 1. 数据准备：创建带有异常点的人工数据集。 2. 异常检测与删除：利用Z-score 或 IQR 方法找出并排除异常观测。 3. 构建基础线性回归模型。 4. 性能评估：计算MSE、RMSE 和 R-squared 来衡量不同模型的效果差异。 以下是具体的实现步骤及对应代码片段： ```python # 导入必要的库 import numpy as np import pandas as pd from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_squared_error, r2_score import matplotlib.pyplot as plt def generate_data(n_samples=50): """生成带有一些随机分布的异常点的模拟数据""" # 正规数据部分 X_normal = 6 * np.random.rand(n_samples - int(0.1*n_samples)) + 1 y_normal = 3*X_normal + 2*np.random.randn(len(X_normal)) # 添加一定比例的异常点 X_outliers = 8 + 4 * (np.random.rand(int(0.1*n_samples)) > 0.5) y_outliers = (-9) * X_outliers + 7 return np.concatenate([X_normal, X_outliers]), np.concatenate([y_normal, y_outliers]) def detect_and_remove_outliers(x, method='zscore', threshold_z=3, factor_iqr=1.5): """ 根据指定的方法来探测并清除异常值，默认采用 Z 分数法； 可选地也可以基于四分位距(IQR). 参数: x : 输入数组形式的一维数据序列 method : 'zscore'|'iqr' ... 其他参数... 返回: 清洗后的非异常数据 """ if method == 'zscore': z_scores = abs((x - x.mean()) / x.std()) filtered_x = x[z_scores < threshold_z] elif method=='iqr': Q1,Q3=np.percentile(x,[25,75]) iqr=Q3-Q1 lower_bound = Q1 - factor_iqr*iqr upper_bound = Q3 + factor_iqr*iqr mask=(lower_bound<x)&(x<upper_bound) filtered_x=x[mask] else: raise ValueError("Unsupported outlier detection method.") print(f"Original length {len(x)}; After filtering out outliers: Length={len(filtered_x)}.") return filtered_x.reshape(-1, 1) if __name__=="__main__": # Step 1: Generate Data with Outliers X_all, y_all = generate_data() data_with_outlier = {'Feature': X_all, 'Target': y_all} df_initial = pd.DataFrame(data=data_with_outlier) # Visualization Before Cleaning fig, ax = plt.subplots(figsize=(8, 6)) df_initial.plot(kind="scatter", x="Feature", y="Target", alpha=.7, title="Data Distribution With Possible Outliers", color='blue') plt.show() # Step 2: Detect and Remove Outliers using Z-Score Method cleaned_X = detect_and_remove_outliers(df_initial['Feature'].values,method='zscore') # Matching the target variable after removing outliers from features idx_to_keep=df_initial.index[df_initial['Feature'].isin(cleaned_X.flatten())] cleaned_y=y_all[idx_to_keep].reshape(-1, 1) # Visualization Post Cleaning cleaned_df={'Cleaned Feature':cleaned_X[:,0],'Cleaned Target':cleaned_y[:,0]} df_cleaned=pd.DataFrame(data=cleaned_df) fig, bx=plt.subplots(figsize=(8,6)) df_cleaned.plot(kind="scatter", x="Cleaned Feature", y="Cleaned Target",alpha=.7, title="Distribution Without Detected Outliers", color='green') plt.show() # Training Baseline Model on All Data Including Potential Outliers model_full_dataset = LinearRegression().fit(df_initial[['Feature']],df_initial['Target']) predictions_baseline=model_full_dataset.predict(df_initial[['Feature']]) mse_original = mean_squared_error(y_true=df_initial['Target'],y_pred=predictions_baseline) rmse_original=np.sqrt(mse_original) r2_original=r2_score(y_true=df_initial['Target'],y_pred=predictions_baseline) # Training Improved Model Using Clean Dataset Only improved_model=LinearRegression().fit(cleaned_X,cleaned_y) predictions_improved=improved_model.predict(cleaned_X) mse_cleaned =mean_squared_error(y_true=cleaned_y,y_pred=predictions_improved ) rmse_cleaned=np.sqrt(mse_cleaned) r2_cleaned=r2_score(y_true=cleaned_y,y_pred=predictions_improved ) comparison={"Metric":["Mean Squared Error","Root Mean Squared Error","R-Squared"], "With Original Data":[mse_original ,rmse_original,r2_original ], "After Removing Outliers":[mse_cleaned ,rmse_cleaned,r2_cleaned]} comp_table=pd.DataFrame(comparison).set_index('Metric') print("\nPerformance Comparison Between Models:\n",comp_table) ``` ### 输出预期 此脚本会依次完成以下几个任务: - 创建一个人造二维空间内的散点图可视化原始含噪音的数据; - 应用`detect_and_remove_outliers()` 函数筛选掉潜在的离群点并将结果再次绘制成图表显示变化情况; - 利用 Scikit-Learn 的 `LinearRegression` 类别分别针对全部样本来训练基准模型以及只保留正常样本后改进型模型; - 计算每种情形下的 MSE、RMSE 以及 \( R^2 \); - 打印出一张表格对比两者的各项评价指标得分高低; 通过这种方式你能直观感受到剔除异常点对于提高预测精度所带来的正面影响。同时，在实践中还可以进一步探索更多复杂的预处理技术和机器学习算法组合以寻求最佳解决方案。