### Python爬虫案例与代码实现 以下是几个典型的Python爬虫案例，涵盖从基础到高级的多个场景。这些案例展示了如何使用Python编写网络爬虫来抓取网页数据。 #### 案例1：基本网页数据抓取 以下示例展示如何使用`requests`和`BeautifulSoup`库抓取网页中的特定信息[^1]： ```python import requests from bs4 import BeautifulSoup url = "https://example.com" response = requests.get(url) # 解析HTML内容 soup = BeautifulSoup(response.text, 'html.parser') # 提取所有段落文本 paragraphs = soup.find_all('p') for paragraph in paragraphs: print(paragraph.text) ``` #### 案例2：分页爬取 如果目标网站包含分页功能，可以使用循环结构抓取多页数据[^2]： ```python import requests from bs4 import BeautifulSoup base_url = "https://example.com/page={}" all_data = [] for page in range(1, 6): # 假设共有5页 url = base_url.format(page) response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') items = soup.find_all('div', class_='item') # 假设每个项目都在<div class="item">中 for item in items: all_data.append(item.text) print(all_data) ``` #### 案例3：登录验证爬取 某些网站需要用户登录后才能访问特定内容。可以通过`requests.Session()`保持会话状态以完成登录操作： ```python import requests session = requests.Session() login_url = "https://example.com/login" credentials = { 'username': 'your_username', 'password': 'your_password' } # 执行登录请求 session.post(login_url, data=credentials) # 抓取登录后的页面 protected_page = "https://example.com/protected" response = session.get(protected_page) print(response.text) ``` #### 案例4：异步爬取 对于大规模数据抓取任务，可以使用`aiohttp`和`asyncio`实现异步爬取以提高效率[^2]： ```python import aiohttp import asyncio async def fetch(session, url): async with session.get(url) as response: return await response.text() async def main(): urls = ["https://example.com/page1", "https://example.com/page2"] tasks = [] async with aiohttp.ClientSession() as session: for url in urls: tasks.append(fetch(session, url)) results = await asyncio.gather(*tasks) for result in results: print(result) asyncio.run(main()) ``` #### 案例5：动态网页爬取（Selenium） 当目标网站通过JavaScript动态加载内容时，可以使用`Selenium`模拟浏览器行为[^1]： ```python from selenium import webdriver from selenium.webdriver.common.by import By driver = webdriver.Chrome() driver.get("https://example.com") # 等待元素加载 elements = driver.find_elements(By.CLASS_NAME, "dynamic-content") for element in elements: print(element.text) driver.quit() ``` ### 注意事项 在开发爬虫程序时，请务必遵守目标网站的`robots.txt`文件规则，并合理设置请求间隔以避免对服务器造成过大压力。