可以按照以下步骤使用Python爬虫实现数字化转型相关词频统计： ### 数据获取 使用Python的`requests`库和`BeautifulSoup`库来抓取网页内容。以下是一个简单示例，用于抓取网页的HTML内容： ```python import requests from bs4 import BeautifulSoup url = 'https://example.com' # 替换为实际要抓取的网页URL response = requests.get(url) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') text = soup.get_text() ``` ### 数据处理 使用`re`库去除文本中的特殊字符和标点符号，再使用`nltk`库进行分词处理： ```python import re import nltk from nltk.corpus import stopwords from nltk.tokenize import word_tokenize nltk.download('punkt') nltk.download('stopwords') # 去除特殊字符和标点符号 text = re.sub(r'[^\w\s]', '', text) # 分词 tokens = word_tokenize(text.lower()) # 去除停用词 stop_words = set(stopwords.words('english')) filtered_tokens = [token for token in tokens if token not in stop_words] ``` ### 词频统计 使用`collections.Counter`来统计词频： ```python from collections import Counter word_freq = Counter(filtered_tokens) # 打印出现频率最高的前10个词 print(word_freq.most_common(10)) ``` ### 筛选数字化转型相关词汇 可以根据事先确定的数字化转型相关词汇列表进行筛选： ```python # 假设这是数字化转型相关词汇列表 digital_transformation_words = ['digital', 'transformation', 'data', 'technology'] filtered_freq = {word: freq for word, freq in word_freq.items() if word in digital_transformation_words} print(filtered_freq) ``` ### 扩展到多个网页 如果要抓取多个网页，可以将上述步骤封装成函数，并使用循环来处理多个URL： ```python def crawl_and_count(url): response = requests.get(url) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') text = soup.get_text() text = re.sub(r'[^\w\s]', '', text) tokens = word_tokenize(text.lower()) stop_words = set(stopwords.words('english')) filtered_tokens = [token for token in tokens if token not in stop_words] word_freq = Counter(filtered_tokens) digital_transformation_words = ['digital', 'transformation', 'data', 'technology'] filtered_freq = {word: freq for word, freq in word_freq.items() if word in digital_transformation_words} return filtered_freq return {} urls = ['https://example1.com', 'https://example2.com'] # 替换为实际的URL列表 total_freq = Counter() for url in urls: freq = crawl_and_count(url) total_freq.update(freq) print(total_freq) ```