centos7 hadoop+hive installation

Keywords: hive Hadoop vim MySQL

Prepare four virtual machines

Virtual Machine Installation

1.Create a new virtual machine
2.Click on Typical Installation(Recommend)
3.Select Chinese and click on your own partition
# Partition Configuration (JD Usage)
/boot 200M
swap 512M  # Not enough native memory, swap
/ # root directory
4.Configure others, as shown below

Update yum

yum install update -y

ip of four hosts

One Master and Three Subordinates Password: virtual machine hadoop01 corresponding to hadoop01 Password: virtual machine hadoop02 corresponding to hadoop02 Password: virtual machine hadoop03 corresponding to hadoop03 Password: virtual machine hadoop04 corresponding to hadoop04

#Reset the password for root
passwd root

hadoop installation


Configure DNS

Each node is configured

vim /etc/hosts   hadoop01  hadoop02  hadoop03  hadoop04

Close Firewall

# Close Firewall
systemctl stop firewalld

# Turn off self-start
systemctl disable firewalld

Configure Secret-Free Login


Configure DNS

Generate ssh key
# Generate ssh key
ssh-keygen -t rsa

cd /root/.ssh

# Copy the public key to a specific file authorized_keys on the primary node (hadoop01)
cp id_rsa.pub authorized_keys

# Copy authorized_keys to hadoop02
scp authorized_keys root@hadoop02:/root/.ssh/

# Log on to hadoop02 host
cd .ssh/
cat id_rsa.pub >> authorized_keys
# On copying authorized_keys to hadoop03
scp authorized_keys root@hadoop03:/root/.ssh/

# Log on to the hadoop03 host
cd .ssh/
cat id_rsa.pub >> authorized_keys
# Copying authorized_keys to hadoop04
scp authorized_keys root@hadoop04:/root/.ssh/

# Log on to hadoop04 host
cd .ssh/
cat id_rsa.pub >> authorized_keys
# Copy the generated authorized_keys to hadoop01, hadoop02, hadoop03
scp authorized_keys root@hadoop01:/root/.ssh/
scp authorized_keys root@hadoop02:/root/.ssh/
scp authorized_keys root@hadoop03:/root/.ssh/

# Verify Secret Login
//Verify password-free login using ssh username@node name or ssh ip address command
ssh root@hadoop02

wget download jdk8

# Copy the downloaded jdk to hadoop02, hadoop03, hadoop04
scp -r -P 22 jdk.tar.gz root@hadoop02:~/
scp -r -P 22 jdk.tar.gz root@hadoop03:~/
scp -r -P 22 jdk.tar.gz root@hadoop04:~/

Configuring the JDK environment

tar -zxvf jdk.tat.gz
mv jdk1.8.0_241 /opt/
# Create Soft Connections
ln -s /opt/jdk1.8.0_241 /opt/jdk

# Configure the java environment
vim /etc/profile
# Java
export JAVA_HOME=/opt/jdk
export PATH=$PATH:$JAVA_HOME/bin

# Make environment variables valid
source /etc/profile

# Verify java installation
java -version

Setting up a fully distributed Hadoop cluster

hadoop version download

Download hadoop
wget http://us.mirrors.quenda.co/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
1. Configure hadoop environment variables (per node)
# Unzip under opt
tar -zxvf hadoop-3.2.0.tar.gz -C /opt/

vim /etc/profile
# hadoop
export HADOOP_HOME=/opt/hadoop-3.2.0
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop

# Save to make profile valid
source /etc/profile
2. Configure JAVA_HOME parameters in the Hadoop environment script file
cd /opt/hadoop-3.2.0/etc/hadoop
#Add or modify the following parameters in the hadoop-env.sh, mapred-env.sh, yarn-env.sh files, respectively
vim hadoop-env.sh
vim mapred-env.sh
vim yarn-env.sh
export JAVA_HOME="/opt/jdk"
3. Modify Hadoop Profile

cd /opt/hadoop-3.2.0/etc/hadoop

In the etc/hadoop directory under the Hadoop installation directory, modify the core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml, workers files to modify the configuration information as appropriate

create folder

mkdir -p /opt/hadoop/tmp

core-site.xml (Configure Common Component Properties)
      <!-- To configure hdfs address -->
      <!-- To save a temporary file directory, you first need to/opt/hadoop Create under tmp Catalog -->
hdfs-site.xml (configure HDFS component properties)
         <!-- Primary Node Address -->
        <!-- Number of backups is default 3 -->
            <description>Configuration false After that, you can allow generation without checking permissions dfs The files on the are convenient, but you need to prevent them from being deleted by mistake.</description>
mapred-site.xml (configure Map-Reduce component properties)
        <!--#Set MapReduce to run on yarn-->
yarn-site.xml (Configure Resource Scheduling Properties)
        <!--#Specify the address of yarn's esourceManager management interface, and if not, Active Node will always be 0-->
        <!--#Reducr How to Get Data -->
        <description>Configuring an external network only requires replacing the external network ip Is True ip,Otherwise defaults to localhost:8088</description>
        <description>Available memory per node,Company MB,Default 8182 MB</description>
        <description>Ignore virtual memory checks. This configuration is useful if you are installing on a virtual machine, and subsequent operations are not likely to cause problems.</description>
vim workers
# Add something
4. Copy the configured folder to another slave node
scp -r /opt/hadoop-3.2.0 root@hadoop02:/opt/
scp -r /opt/hadoop-3.2.0 root@hadoop03:/opt/

scp -r /opt/hadoop root@hadoop02:/opt/
scp -r /opt/hadoop root@hadoop03:/opt/
5. Configure startup scripts, add HDFS and Yarn permissions
# Add HDFS permissions: Edit the following script and add HDFS permissions in the empty space on the second line
cd /opt/hadoop-3.2.0/sbin
vim start-dfs.sh 
vim stop-dfs.sh

# Add Yarn permissions: Edit the following script and add Yarn permissions in the empty space on the second line
cd /opt/hadoop-3.2.0/sbin
vim start-yarn.sh 
vim stop-yarn.sh 

6. Initialization & Startup
cd /opt/hadoop-3.2.0
# init 
# Format
bin/hdfs namenode -format wmqhadoop


# Back Open

# Stop it
7. Verify Hadoop started successfully
Open ResourceManager page at browser input: http://hadoop01:8088

Open the Hadoop Namenode page in the browser by typing: http://hadoop01:50070

mysql-5.7 installation


wget http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql57-community-release-el7-10.noarch.rpm

rpm -ivh mysql57-community-release-el7-10.noarch.rpm

Use the yum command to complete the installation

1,Installation command:
yum -y install mysql-community-server

2,start-up msyql: 
systemctl start mysqld #Start MySQL

3,Get the temporary password for the installation (which is used the first time you log on):
grep 'temporary password' /var/log/mysqld.log

3.Set up boot-up
systemctl enable mysqld

Sign in

mysql -uroot -p
# Enter the password you just entered

Change Password

ALTER USER 'root'@'localhost' IDENTIFIED BY 'Mysql123!';

Set to allow remote login

1. Execute authorization commands


2. Exit mysql Operations Console


3. Open port 3306

Open Firewall

sudo systemctl start firewalld.service

Open port 3306 permanently

sudo firewall-cmd --add-port=3306/tcp --permanent


sudo firewall-cmd --reload

Close Firewall

sudo systemctl stop firewalld.service

Set default encoding to utf8

View mysql encoding before modification

show variables like '%chara%';

Modify the / etc/my.cnf file to add the following two lines

vim /etc/my.cnf
init_connect='SET NAMES utf8'

After modification, restart mysql

sudo systemctl restart mysqld

hive installation

https://blog.csdn.net/qq_39315740/article/details/98626518 #Recommended


hive Download


hive installation

tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /opt/

Configuring environment variables

vim /etc/profile
# hive
export HIVE_HOME=/opt/apache-hive-3.1.2-bin
export PATH=$PATH:$HIVE_HOME/bin

source /etc/profile

Create hive-site.xml file

cd /opt/apache-hive-3.1.2-bin/conf
cp hive-default.xml.template hive-site.xml

For the following HDFS-related settings in hive-site.xml, we need to create the corresponding directory in HDFS now

    <description>location of default database for the warehouse</description>

    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>

Create HDFS Folder

hadoop fs -mkdir -p /user/hive/warehouse   # create folder
hadoop fs -mkdir -p /tmp/hive    # create folder
hadoop fs -chmod -R 777 /user/hive/warehouse   # Grant privileges
hadoop fs -chmod -R 777 /tmp/hive   # Grant privileges

# Check to see if the creation was successful
hadoop fs -ls /

Hive-related configuration

Change {system:java.io.tmpdir} in hive-site.xml to hive's local temporary directory and {system:user.name} to user name.
Create temp directory

cd /opt/apache-hive-3.1.2-bin
mkdir temp
chmod -R 777 temp
# Replace ${system:java.io.tmpdir} with / opt/apache-hive-3.1.2-bin/temp
# Replace ${system:user.name} with root
vim hive-site.xml

Database Related Configuration

# Database jdbc address, modified as host ip address in value tag
# Driver class name for database
# The new version 8.0 driver is com.mysql.cj.jdbc.Driver
# Older version 5.x is driven by com.mysql.jdbc.Driver
# Database User Name
# Database Password
    <value>Mysql123!</value> #Modify your own mysql password


Configure hive-log4j2.properties

cd /opt/apache-hive-3.1.2-bin/conf
cp hive-log4j2.properties.template hive-log4j2.properties

vim hive-log4j2.properties
# Modify Content
property.hive.log.dir = /opt/apache-hive-3.1.2-bin/temp/root

Configure hive-env.sh file

cd /opt/apache-hive-3.1.2-bin/conf
cp hive-env.sh.template hive-env.sh
vim hive-env.sh

Add the following:

export JAVA_HOME=/opt/jdk
export HADOOP_HOME=/opt/hadoop-3.2.0
export HIVE_CONF_DIR=/opt/apache-hive-3.1.2-bin/conf
export HIVE_AUX_JARS_PATH=/opt/apache-hive-3.1.2-bin/lib

Hive Start

Download Database 5.7 Driver
//Database Driven Download

wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz

# Move the database driver to lib in hive
cp -r mysql-connector-java-5.1.48-bin.jar /opt/apache-hive-3.1.2-bin/lib
schematool -dbType mysql -initSchema
# hive initialization error
# Compare the guava-27.0-jre.jar versions in hadoop and hive
cd /opt/hadoop-3.2.0/share/hadoop/common/lib
ll | grep guava*

cd /opt/apache-hive-3.1.2-bin/lib
ll | grep guava*

//Replace the higher version of guava-27.0-jre.jar with the lower version of guava-19.0-jre.jar

# There are also questions to refer to

Hadoop 3.1.2 + Hive 3.1.1 Installation



Modify vim comment color

Create a new.vimrc profile under User~Home folder
vim ~/.vimrc
 #Add the content and save it
hi Comment ctermfg =blue

vim replacement

Twenty-nine original articles were published, 11 were praised, and 10,000 visits were received.
Private letter follow

Posted by Derek on Fri, 31 Jan 2020 20:33:17 -0800