### 使用 Python 编写爬虫程序，通过正则表达式和 XPath 方法爬取影评数据 以下是一个完整的 Python 程序，结合正则表达式和 XPath 方法爬取《星际穿越》与《上海堡垒》的影评及评分各 1000 条，并将结果保存为 CSV 文件。 #### 爬虫代码实现 ```python import requests import re from lxml import etree import csv # 定义函数用于通过 XPath 提取影评数据 def fetch_reviews_xpath(movie_name, url): headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" } response = requests.get(url, headers=headers) if response.status_code == 200: html = etree.HTML(response.text) # 使用 XPath 提取评论内容和评分 reviews = html.xpath('//div[@class="review"]/text()')[:1000] # 根据实际网页结构调整XPath表达式[^1] ratings = html.xpath('//span[@class="rating"]/text()')[:1000] # 根据实际网页结构调整XPath表达式[^1] # 清洗数据并存储到列表中 cleaned_reviews = [] for i in range(min(len(reviews), len(ratings))): review_text = reviews[i].strip() rating_value = ratings[i].strip() cleaned_reviews.append({"movie": movie_name, "rating": rating_value, "content": review_text}) return cleaned_reviews else: print(f"Failed to fetch data for {movie_name}. Status code: {response.status_code}") return [] # 定义函数用于通过正则表达式提取影评数据 def fetch_reviews_regex(movie_name, url): headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" } response = requests.get(url, headers=headers) if response.status_code == 200: content = response.text # 使用正则表达式提取评论内容和评分 reviews = re.findall(r'<div class="review">(.*?)</div>', content, re.S)[:1000] # 根据实际网页结构调整正则表达式[^2] ratings = re.findall(r'<span class="rating">(.*?)</span>', content, re.S)[:1000] # 根据实际网页结构调整正则表达式[^2] # 清洗数据并存储到列表中 cleaned_reviews = [] for i in range(min(len(reviews), len(ratings))): review_text = reviews[i].strip() rating_value = ratings[i].strip() cleaned_reviews.append({"movie": movie_name, "rating": rating_value, "content": review_text}) return cleaned_reviews else: print(f"Failed to fetch data for {movie_name}. Status code: {response.status_code}") return [] # 定义函数用于保存数据到 CSV 文件 def save_to_csv(data, filename): with open(filename, 'w', newline='', encoding='utf-8') as file: writer = csv.DictWriter(file, fieldnames=["movie", "rating", "content"]) writer.writeheader() writer.writerows(data) # 爬取《星际穿越》的影评数据（XPath方法） interstellar_url_xpath = "https://example.com/interstellar-reviews" # 替换为目标网站的实际 URL interstellar_reviews_xpath = fetch_reviews_xpath("Interstellar (XPath)", interstellar_url_xpath) save_to_csv(interstellar_reviews_xpath, "interstellar_reviews_xpath.csv") # 爬取《星际穿越》的影评数据（正则表达式方法） interstellar_url_regex = "https://example.com/interstellar-reviews" # 替换为目标网站的实际 URL interstellar_reviews_regex = fetch_reviews_regex("Interstellar (Regex)", interstellar_url_regex) save_to_csv(interstellar_reviews_regex, "interstellar_reviews_regex.csv") # 爬取《上海堡垒》的影评数据（XPath方法） shanghai_fortress_url_xpath = "https://example.com/shanghai-fortress-reviews" # 替换为目标网站的实际 URL shanghai_fortress_reviews_xpath = fetch_reviews_xpath("Shanghai Fortress (XPath)", shanghai_fortress_url_xpath) save_to_csv(shanghai_fortress_reviews_xpath, "shanghai_fortress_reviews_xpath.csv") # 爬取《上海堡垒》的影评数据（正则表达式方法） shanghai_fortress_url_regex = "https://example.com/shanghai-fortress-reviews" # 替换为目标网站的实际 URL shanghai_fortress_reviews_regex = fetch_reviews_regex("Shanghai Fortress (Regex)", shanghai_fortress_url_regex) save_to_csv(shanghai_fortress_reviews_regex, "shanghai_fortress_reviews_regex.csv") ``` #### 代码说明 - **`fetch_reviews_xpath`** 函数：使用 XPath 提取影评数据。需要根据目标网站的实际 HTML 结构调整 XPath 表达式。 - **`fetch_reviews_regex`** 函数：使用正则表达式提取影评数据。需要根据目标网站的实际 HTML 结构调整正则表达式。 - **`save_to_csv`** 函数：将爬取到的数据保存为 CSV 文件，便于后续分析和使用。 - **正则表达式与 XPath 的对比**：正则表达式适用于简单的字符串匹配，而 XPath 更适合处理结构化的 HTML 数据，能够更精确地定位目标元素[^3]。 #### 注意事项 在运行上述代码前，请确保替换所有 `url` 变量为目标网站的实际 URL。此外，应遵守目标网站的 `robots.txt` 文件规定以及相关法律法规，确保爬取行为合法合规。