在 Python 3.8 中实现绕过人机验证的爬虫，通常需要结合多种技术手段来模拟真实用户行为，并降低被目标网站识别为爬虫的概率。以下是一个完整的代码示例，涵盖请求延迟、User-Agent 随机化、代理 IP 切换、Selenium 自动化处理 CAPTCHA 页面等关键策略。 ### 使用 `requests` 和反爬策略类进行基本请求 ```python import requests import random import time class AntiCrawler: """反爬虫策略类，包含各种避免被网站识别为爬虫的技术""" def __init__(self, user_agents=None, proxy_list=None, min_delay=0.5, max_delay=2.0, randomize=True): self.user_agents = user_agents or [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.2 Safari/605.1.15' ] self.proxy_list = proxy_list or [] self.min_delay = min_delay self.max_delay = max_delay self.randomize = randomize self.last_request_time = 0 self.current_proxy = None self.error_count = 0 self.success_count = 0 def apply_strategy(self, url): if self.randomize: delay = random.uniform(self.min_delay, self.max_delay) time.sleep(delay) headers = { 'User-Agent': random.choice(self.user_agents), 'Referer': 'https://www.google.com/' } proxies = {} if self.proxy_list: self.current_proxy = random.choice(self.proxy_list) proxies = { 'http': self.current_proxy, 'https': self.current_proxy } try: response = requests.get(url, headers=headers, proxies=proxies) self.success_count += 1 return response except Exception as e: self.error_count += 1 print(f"请求失败: {e}") return None ``` ### 使用 Selenium 自动化处理 CAPTCHA 页面 当检测到页面出现验证码时，可以切换至 Selenium 模拟浏览器操作，并允许人工干预完成验证： ```python from selenium import webdriver from selenium.webdriver.chrome.options import Options import time def handle_captcha(url): chrome_options = Options() chrome_options.add_argument("--headless") # 无头模式 chrome_options.add_argument("--disable-gpu") chrome_options.add_argument("--no-sandbox") driver = webdriver.Chrome(options=chrome_options) driver.get(url) if "captcha" in driver.current_url: print("需要手动处理验证码，请在页面上完成验证") time.sleep(10) # 留出时间进行人工干预 content = driver.page_source driver.quit() return content ``` ### 完整调用流程示例 ```python if __name__ == "__main__": anti_crawler = AntiCrawler( proxy_list=[ 'http://user:pass@10.10.1.10:3128', 'http://10.10.1.10:1080' ] ) target_url = "https://example.com" response = anti_crawler.apply_strategy(target_url) if response and "captcha" in response.url: html_content = handle_captcha(target_url) print(html_content) elif response: print(response.text) else: print("无法获取有效响应") ``` 上述代码展示了如何通过组合使用请求策略和浏览器自动化来绕过基本的人机验证机制[^1]。 --- ###