Install Sqoop on Linux (and connect mysql test)

Keywords: MySQL Hadoop HBase Zookeeper

Article directory

Environment description

Software Edition
operating system CentOS 7
sqoop 1.4.6
hadoop 2.7.7
mysql 5.7.26

Download and unzip

sqoop download address
Use the command: tar - zxvf spoon-1.4.7.bin_uouhadoop-2.6.0.tar.gz to decompress.

Change configuration sqoop-env.sh

After decompression

[root@centos2020 hadoop-2.7.7]# cd /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/
[root@centos2020 sqoop-1.4.7.bin__hadoop-2.6.0]# ls
bin            COMPILING.txt  ivy      LICENSE.txt  README.txt             sqoop-test-1.4.7.jar
build.xml      conf           ivy.xml  NOTICE.txt   sqoop-1.4.7.jar        src
CHANGELOG.txt  docs           lib      pom-old.xml  sqoop-patch-review.py  testdata

Modify sqoop-env.sh

Because there is a template profile, after the name change:

[root@centos2020 sqoop-1.4.7.bin__hadoop-2.6.0]# cd conf
[root@centos2020 conf]# vim sqoop-env.sh 

sqoop-env.sh add the following configuration:

[root@centos2020 conf]# vim sqoop-env.sh 

# limitations under the License.

# included in all the hadoop scripts with source command
# should not be executable directly
# also should not be passed any arguments, since we need original $*

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/usr/hadoop/hadoop-2.7.7

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/usr/hadoop/hadoop-2.7.7

#set the path to where bin/hbase is available
#export HBASE_HOME=

#Set the path to where bin/hive is available
export HIVE_HOME=/usr/hive/apache-hive-2.3.6-bin

#Set the path for where zookeper config dir is
#export ZOOCFGDIR=

Configure environment variables

[root@centos2020 conf]# vim /etc/profile
[root@centos2020 conf]# source /etc/profile

The configuration content is:

#sqoop environment
export SQOOP_HOME=/usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin
export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib

Copy mysql driver

Copy the mysql driver to the directory / usr / sqoop / sqoop-1.4.7.bin_uouhadoop-2.6.0/lib.
The driver package can be from the maven repository, or Official website Download.

What I use here is:
mysql-connector-java-5.1.48.jar

mysql start

service mysql start

View sqoop version

[root@centos2020 sqoop-1.4.7.bin__hadoop-2.6.0]# sqoop version
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/02/25 05:41:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

Test with mysql

[root@centos2020 sqoop-1.4.7.bin__hadoop-2.6.0]# sqoop list-databases --connect jdbc:mysql://centos2020:3306/ --username root -P
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
20/02/25 05:42:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password: 
20/02/25 05:42:19 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Tue Feb 25 05:42:20 EST 2020 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
mysql
performance_schema
sys

See that the table name in mysql is printed out, which proves that the configuration is successful.

141 original articles published, 122 praised, 30000 visitors+
Private letter follow

Posted by possiblyB9 on Tue, 25 Feb 2020 07:36:16 -0800