MapReduce是一种用于大规模数据处理的编程模型，适合对包含省份、景点类型等信息的数据进行计算。以下是一个使用Python和Hadoop Streaming实现MapReduce计算的示例，以统计每个省份不同景点类型的数量为例。 #### 1. 编写Mapper代码（`mapper.py`） ```python import sys for line in sys.stdin: # 假设输入数据是JSON格式，这里简单分割处理 try: fields = line.strip().split('"') province = None attraction_type = None for i in range(len(fields)): if fields[i] == "province": province = fields[i + 2] if fields[i] == "attraction_type": attraction_type = fields[i + 2] if province and attraction_type: print(f"{province}\t{attraction_type}\t1") except: continue ``` #### 2. 编写Reducer代码（`reducer.py`） ```python import sys current_province = None current_attraction_type = None current_count = 0 for line in sys.stdin: line = line.strip() province, attraction_type, count = line.split('\t') try: count = int(count) except ValueError: continue if current_province == province and current_attraction_type == attraction_type: current_count += count else: if current_province and current_attraction_type: print(f"{current_province}\t{current_attraction_type}\t{current_count}") current_province = province current_attraction_type = attraction_type current_count = count if current_province and current_attraction_type: print(f"{current_province}\t{current_attraction_type}\t{current_count}") ``` #### 3. 运行MapReduce作业 在Hadoop集群上，可以使用Hadoop Streaming来运行上述MapReduce作业： ```bash hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar \ -input input_data.txt \ -output output \ -mapper "python mapper.py" \ -reducer "python reducer.py" ``` 其中，`input_data.txt` 是包含给定数据的文件，`output` 是输出结果的目录。