Linux builds Sqoop development environment

Linux builds Sqoop development environment

1.Sqoop

Sqoop is a tool for transferring data between Hadoop and relational database servers. It is used to export data from relational databases such as MySQL , Oracle to Hadoop HDFS, and from Hadoop file system to relational databases.

Traditional application management systems, that is, interactions with relational databases using RDBMS applications, are one of the sources of big data. Such large data, generated by relational databases, are stored in relational database structures in relational database servers.

When big data stores and analyzers such as MapReduce, Hive, HBase, Cassandra, Pig, etc. came into being in the Hadoop ecosystem, they needed a tool to import and export the relational data in which the big data resided. interact with the database server. Here, Sqoop dominates the Hadoop ecosystem to provide a viable interaction between relational database servers and Hadoop HDFS.

Sqoop: "SQL to Hadoop and Hadoop to SQL"

Sqoop is a tool for transferring data between Hadoop and relational database servers. It is used to export data from relational databases like MySQL, Oracle to Hadoop HDFS from Hadoop file system to relational databases. It is provided by the Apache Software Foundation.

 

2. The workflow of Sqoop:

3. Install Sqoop

[1] Download the Sqoop installation package: sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz ,

[2] Use the Xftp tool to upload the installation package to the server: /usr/local/sqoop, where sqoop-1.4.7 supports hadoop-2.6.0 and above, I chose sqoop-1.4.6 here, where sqoop-1.4. 6 Support haoop- 2.0.4 and above :

[3] Xshell uses the cd command to enter the sqoop installation directory: cd /usr/local/sqoop:

[4] Use tar -xvf to decompress the xx.tar.gz archive: tar -xvf sqoop-1.4.6.tar.gz

[5] Copy the mysql jdbc driver mysql-connector-java-5.1.46.jar to the lib directory of the sqoop installation package:

[6] Xhell uses the cd command to enter: cd /usr/local/sqoop/sqoop-1.4.6/conf, and execute the command: mv sqoop-env-template.sh sqoop-env.sh

[7] Configure environment variables: vim /etc/profile, enter: source /etc/profile to make environment variables take effect

#Setting SQOOP_HOME Path

export  SQOOP_HOME=/usr/local/sqoop/sqoop-1.4.6

export  PATH=${PATH}:${SQOOP_HOME}/bin

 

 

[8] Modify the configuration file sqoop-env.sh:

#Set path to where bin/hadoop is available

#export HADOOP_COMMON_HOME=

export HADOOP_COMMON_HOME=/usr/local/hadoop/hadoop-2.7.5

 

#Set path to where hadoop-*-core.jar is available

#export HADOOP_MAPRED_HOME=

export HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-2.7.5

 

#set the path to where bin/hbase is available

#export HBASE_HOME=

export HBASE_HOME=/usr/local/hadoop/hadoop-2.7.5

 

#Set the path to where bin/hive is available

#export HIVE_HOME=

export HIVE_HOME=/usr/local/hive/hive-2.3.2

 

#Set the path for where zookeper config dir is

#export ZOOCFGDIR=

export ZOOCFGDIR=/usr/local/sqoop/repository/zookeper

 

4. Test Sqoop:

[1] List all databases in the mysql database: sqoop list-databases --connect  database link string  - username  database username- password  database password

  例如：sqoop list-databases --connect jdbc: mysql://localhost:3306/ -username root -password 123456Abc

 

[2] Connect to mysql and list the tables in the database: sqoop list-tables --connect database link string  --username database username  --password  database password

例如： sqoop list-tables --connect jdbc: mysql://localhost:3306/hive -username root -password 123456Abc