Hadoop environment deployment document

Keywords: Docker HBase Hadoop MySQL

I. Deployment environment

Local IP: 192.168.0.222
System: CentOS Linux release 7.6.1810 (Core)
Kernel: 3.10.0-957.el7.x86_64

II. Installation of docker-ce

yum install -y yum-utils device-mapper-persistent-data lvm2 && yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && yum install -y docker-ce docker-ce-cli containerd.io

Configuration of docker warehouse

mkdir /etc/docker/ && vim /etc/docker/daemon.json 

Add the following

{
  "registry-mirrors": ["https://1e7waog4.mirror.aliyuncs.com"],
  "insecure-registries":["192.168.0.20:5000"]
}

Start and Set Up Self-Start

systemctl start docker && systemctl enable docker

IV. Configuring docker network

docker network create --subnet=172.10.0.0/16 hadoop_net && docker network ls

5. Download Mirror

docker pull 192.168.0.20:5000/hadoop-hbase:master && docker pull 192.168.0.20:5000/hadoop-hbase:slave1 && docker pull 192.168.0.20:5000/hadoop-hbase:slave2 && docker pull 192.168.0.20:5000/hadoop-hbase:mysql

6. Start Container

Start master

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.2 \
-h master \
-p 10022:22 \
-p 2181:2181 \
-p 2887:2888 \
-p 3887:3888 \
-p 9870:9870 \
-p 8088:8088 \
-p 16010:16010 \
-p 10000:10000 \
--restart always \
--name master \
192.168.0.20:5000/hadoop-hbase:master

Start slave1

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.3 \
-h slave1 \
-p 20022:22 \
-p 2182:2181 \
-p 2888:2888 \
-p 3888:3888 \
-p 9864:9864 \
-p 8042:8042 \
-p 16011:16010 \
-p 16030:16030 \
--restart always \
--name slave1 \
192.168.0.20:5000/hadoop-hbase:slave1

Start slave2

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host mysql:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.4 \
-h slave2 \
-p 30022:22 \
-p 2183:2181 \
-p 2889:2888 \
-p 3889:3888 \
-p 9865:9864 \
-p 8043:8042 \
-p 16031:16030 \
--restart always \
--name slave2 \
192.168.0.20:5000/hadoop-hbase:slave2

Start MySQL

docker run -d \
--add-host master:172.10.0.2 \
--add-host slave1:172.10.0.3 \
--add-host slave2:172.10.0.4 \
--add-host slave2:172.10.0.5 \
--net hadoop_net \
--ip 172.10.0.5 \
-h mysql \
-p 40022:22 \
-p 23306:3306 \
--privileged \
--restart always \
--name mysql \
192.168.0.20:5000/hadoop-hbase:mysql \
/usr/sbin/init

Open three terminals into master, slave1, slave2 containers, default account root password

VII. Start-up Services

Start zookeeper

master: /opt/zookeeper-3.4.10/bin/zkServer.sh start
slave1: /opt/zookeeper-3.4.10/bin/zkServer.sh start
slave2: /opt/zookeeper-3.4.10/bin/zkServer.sh start

Start hdfs

master: /opt/hadoop-3.0.3/sbin/start-dfs.sh

Start yarn

master: /opt/hadoop-3.0.3/sbin/start-yarn.sh

Start hbase

master: /opt/hbase-2.1.0/bin/start-hbase.sh

Start hiveserver2

master: hiveserver2

VIII. Access Services

Visit hdfs
ip:9870
ip:9864
ip:9865

Visit yarn
ip:8088
ip:8042
ip:8043

Visit hbase
ip:16010
ip:16011
ip:16030
ip:16031
Visit hive

Posted by wannasub on Fri, 11 Oct 2019 12:32:49 -0700