sqoop 1.x 整合mysql jdbc

1、安装成功后,需要整合mysql 进行数据的读取。

[root@hadoop01 lib]# cp mysql-connector-java-8.0.17.jar  /data/bigData/sqoop/sqoop-1.4.7/lib/

2、进行测试:sqoop

Warning: /data/bigData/sqoop/sqoop-1.4.7/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /data/bigData/sqoop/sqoop-1.4.7/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/bigData/hadoop-2.9.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hBase/pseduo_hbase-0.98.17-hadoop2/hbase-0.98.17-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/bigData/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Try 'sqoop help' for usage.

这个错误前有个hcatalog 未识别,配置hcatalog。,在sqoop的 confi 文件中

注意这个hcatalog 在hive中;

[root@hadoop01 conf]# pwd
/data/bigData/sqoop/sqoop-1.4.7/conf
[root@hadoop01 conf]# vi sqoop-env.sh 
#set the path for where hcatalog
export HCAT_HOME=/data/bigData/hive/hcatalog

---测试从mysql 导出数据到hadoop 中

[root@hadoop01 sqoop-1.4.7]# sqoop import --connect jdbc:mysql://hadoop03.fandong.com:3306/hive2 --table DBS -username nodejs1 -password nodejs1 --hive-import

20/01/04 16:40:16 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/beeb03a7d6de2ee19ce2e78f14f05a25/DBS.jar
20/01/04 16:40:16 WARN manager.MySQLManager: It looks like you are importing from mysql.
20/01/04 16:40:16 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
20/01/04 16:40:16 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
20/01/04 16:40:16 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
20/01/04 16:40:16 INFO mapreduce.ImportJobBase: Beginning import of DBS
20/01/04 16:40:16 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
20/01/04 16:40:18 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
20/01/04 16:40:18 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

可以看出执行的是一个mapreduce任务。

---导入到hive 也很简单,查看官网上提示的

Sqoop’s import tool’s main function is to upload your data into files in HDFS. If you have a Hive metastore associated with your HDFS cluster, Sqoop can also import the data into Hive by generating and executing a CREATE TABLE statement to define the data’s layout in Hive. Importing data into Hive is as simple as adding the --hive-import option to your Sqoop command line.

-- 在上面的导入语句中添加hive-import 语句即可。

2 rows selected (0.236 seconds)
0: jdbc:hive2://hadoop01.fandong.com:10000> show tables;
+-----------+
| tab_name  |
+-----------+
| dbs       |
+-----------+
1 row selected (0.213 seconds)
0: jdbc:hive2://hadoop01.fandong.com:10000> select * from dbs;
+------------+------------------------+----------------------------------------------------+-----------+-----------------+-----------------+
| dbs.db_id  |        dbs.desc        |                dbs.db_location_uri                 | dbs.name  | dbs.owner_name  | dbs.owner_type  |
+------------+------------------------+----------------------------------------------------+-----------+-----------------+-----------------+
| 1          | Default Hive database  | hdfs://hadoop01.fandong.com:9001/user/hive/warehouse | default   | public          | ROLE            |
| 2          | null                   | hdfs://hadoop01.fandong.com:9001/user/hive/warehouse/portrait.db | portrait  | anonymous       | USER            |
+------------+------------------------+----------------------------------------------------+-----------+-----------------+-----------------+
2 rows selected (0.527 seconds)
0: jdbc:hive2://hadoop01.fandong.com:10000> 

我们发现这个表导入成功了,但是这种方式,我们需要改造,一般都是在hive上创建表的,在执行导入操作的。

发布了61 篇原创文章 · 获赞 1 · 访问量 644

猜你喜欢

转载自blog.csdn.net/u012842247/article/details/103833386
今日推荐