### 使用 Docker 部署 Hive 集群 #### 准备工作 为了成功部署 Hive 集群，需先准备好环境并下载必要的镜像文件。确保已安装 Docker 和 Docker Compose 工具。 #### 创建项目结构 创建一个新的目录用于存放所有的配置文件和服务定义： ```bash mkdir -p ~/hive-cluster/{conf,data,hdfs,namenode,datanode} cd ~/hive-cluster/ ``` #### 编写 `docker-compose.yml` 文件 编写一个完整的 `docker-compose.yml` 文件来描述整个集群的服务架构[^1]： ```yaml version: '3' services: namenode: image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 container_name: namenode ports: - "9870:9870" - "9000:9000" volumes: - ./hdfs/namenode:/data environment: - CLUSTER_NAME=testcluster datanode: image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 container_name: datanode depends_on: - namenode ports: - "9864:9864" volumes: - ./hdfs/datanode:/hadoop/dfs/data environment: SERVICE_PRECONDITION: "namenode:9870" resourcemanager: image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 container_name: resourcemanager depends_on: - namenode ports: - "8088:8088" - "8032:8032" environment: SERVICE_PRECONDITION: "namenode:9870" nodemanager: image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 container_name: nodemanager depends_on: - resourcemanager ports: - "8042:8042" environment: SERVICE_PRECONDITION: "resourcemanager:8088" historyserver: image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 container_name: historyserver depends_on: - datanode ports: - "19888:19888" environment: SERVICE_PRECONDITION: "datanode:9864" volumes: - ./hdfs/historyserver:/tmp/logs hive-server: image: bde2020/hive:2.3.6-postgresql-metastore container_name: hiveserver depends_on: - datanode ports: - "10000:10000" - "10002:10002" environment: HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: jdbc:postgresql://metastore-db/metastore HIVE_CORE_CONF_javax_jdo_option_ConnectionDriverName: org.postgresql.Driver HIVE_CORE_CONF_javax_jdo_option_ConnectionUserName: postgres HIVE_CORE_CONF_javax_jdo_option_ConnectionPassword: mysecretpassword METASTORE_HOST: metastore-db MYSQL_SERVICE_HOST: mysql-service MYSQL_SERVICE_PORT: 3306 MYSQL_ROOT_PASSWORD: rootpassword MYSQL_DATABASE: metadb MYSQL_USER: hiveuser MYSQL_PASSWORD: hivepassword metastore-db: image: postgres:alpine container_name: metastore-db restart: always environment: POSTGRES_DB: metastore POSTGRES_USER: postgres POSTGRES_PASSWORD: mysecretpassword volumes: - ./data/postgres:/var/lib/postgresql/data mysql-service: image: mysql/mysql-server:latest container_name: mysql-service environment: MYSQL_ROOT_PASSWORD: rootpassword MYSQL_DATABASE: metadb MYSQL_USER: hiveuser MYSQL_PASSWORD: hivepassword ports: - "3306:3306" volumes: - ./data/mysql:/var/lib/mysql ``` 此配置文件定义了一个包含 NameNode、DataNode、ResourceManager、NodeManager、HistoryServer 的 Hadoop 集群以及 Hive Server 和 Metastore 数据库服务。其中 MySQL 被用来存储元数据信息[^2]。 #### 启动集群 完成上述设置后，在命令行工具中进入该目录并通过以下指令启动所有容器: ```bash docker-compose up -d ``` 这将会以后台模式运行所有指定的服务实例，并自动处理依赖关系。 #### 测试连接 等待几分钟让各个组件初始化完成后，可以通过浏览器访问 Web UI 或者使用客户端工具测试与各节点之间的通信状况。对于 Hive 来说，默认情况下可以在本地机器上通过 Beeline 连接器进行交互查询操作: ```bash beeline -u jdbc:hive2://localhost:10000/default -n username -p password ```