### 完整的Python程序用于爬取番茄小说的内容 以下是一个完整的 Python 程序，旨在从类似于番茄小说这样的网站中抓取数据。此脚本涵盖了 URL 构建、HTML 解析以及数据存储等功能模块。 #### 1. 导入库 首先导入必要的库以支持 HTTP 请求和 HTML 解析功能[^2]。 ```python import requests from bs4 import BeautifulSoup import time import random ``` #### 2. 设置基础变量 定义目标站点的基础URL以及其他配置参数，例如 User-Agent 头部信息等[^3]。 ```python base_url = "https://www.tomatonovel.com" type_links = [f"{base_url}/list/{i}-1.html" for i in range(1, 9)] # 假设共有8种类别 headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' } output_file = "./novels_data.txt" ``` #### 3. 获取单页上的所有书籍链接 编写函数用来访问指定类型的第一页，并返回该类别下的所有书目链接列表[^4]。 ```python def get_books_from_page(page_link): books = [] try: res = requests.get(page_link, headers=headers) soup = BeautifulSoup(res.content.decode(), 'lxml') book_list_tag = soup.select('.book-list .item') # 替换为实际匹配的选择器 for item in book_list_tag: link = base_url + item.a['href'] name = item.img['alt'] or '' author = item.span.string.strip() books.append({ 'name': name, 'author': author, 'link': link }) except Exception as e: print(f"Error occurred while fetching {page_link}: ", str(e)) finally: return books ``` #### 4. 抓取具体一本书籍详情 对于每一本书，进入其主页获取更详细的资料，包括简介、状态和其他相关信息[^5]。 ```python def fetch_book_details(book_info): details = {} try: detail_res = requests.get(book_info["link"], headers=headers) detail_soup = BeautifulSoup(detail_res.content.decode(), 'lxml') intro = detail_soup.find('meta', property="og:description")['content'].strip() status = detail_soup.select_one('#info p:nth-of-type(2)').text.split(":")[1].strip() details.update({ 'introduction': intro, 'status': status }) except Exception as ex: print("Failed to load detailed info:", str(ex)) return dict(**book_info, **details) ``` #### 5. 主循环逻辑 遍历所有的分类链接，逐个调用前面定义好的辅助方法完成最终的数据采集工作[^6]。 ```python all_novels = [] for category_index, cat_link in enumerate(type_links): novels_in_cat = get_books_from_page(cat_link) enriched_novels = list(map(fetch_book_details, novels_in_cat)) all_novels.extend(enriched_novels) sleep_time = round(random.uniform(1, 3), 2) print(f"Crawled Category #{category_index+1}, sleeping for {sleep_time} seconds...") time.sleep(sleep_time) ``` #### 6. 输出结果至文件 最后将收集到的所有小说记录保存成易于阅读的形式输出到外部文本文件当中去[^7]。 ```python with open(output_file, 'a+', encoding='utf-8') as out_f: for novel_entry in all_novels: line = "\t".join([str(v).replace("\n", "") for v in novel_entry.values()]) out_f.write(line + '\n') print("Data successfully written into file.") ``` --- ###