版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/jjshouji/article/details/83857861
1.问题描述
sqoop list-tables 工具正常，但是import 工具有问题。奇怪！！！！
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://172.1.1.96:3306/test -m 1 --delete-target-dir --table student_exam_score_20170802 --fields-terminated-by '\t' --hive-import --hive-database bond_edu --hive-table student_exam_score_20170802
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:31:13 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:31:13 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:31:13 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:31:13 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:31:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:31:14 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:31:14 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/98d200728b6a446d170eba746e2c9a4c/student_exam_score_20170802.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:31:18 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/98d200728b6a446d170eba746e2c9a4c/student_exam_score_20170802.jar
18/11/08 11:31:19 INFO tool.ImportTool: Destination directory student_exam_score_20170802 is not present, hence not deleting.
18/11/08 11:31:19 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:31:19 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:31:19 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:31:19 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:31:19 INFO mapreduce.ImportJobBase: Beginning import of student_exam_score_20170802
18/11/08 11:31:20 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:31:20 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:31:20 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:31:24 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:31:24 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:31:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0018
18/11/08 11:31:25 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0018
18/11/08 11:31:25 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0018/
18/11/08 11:31:25 INFO mapreduce.Job: Running job: job_1541640011445_0018
18/11/08 11:31:37 INFO mapreduce.Job: Job job_1541640011445_0018 running in uber mode : false
18/11/08 11:31:37 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:31:45 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:31:51 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:31:57 INFO mapreduce.Job: Task Id : attempt_1541640011445_0018_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2222)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2017)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:32:05 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:32:06 INFO mapreduce.Job: Job job_1541640011445_0018 failed with state FAILED due to: Task failed task_1541640011445_0018_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
18/11/08 11:32:06 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=20616
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20616
Total vcore-seconds taken by all map tasks=20616
Total megabyte-seconds taken by all map tasks=21110784
18/11/08 11:32:06 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/11/08 11:32:06 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 46.2311 seconds (0 bytes/sec)
18/11/08 11:32:06 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/11/08 11:32:06 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/11/08 11:32:06 ERROR tool.ImportTool: Error during import: Import job failed!
[hadoop@master ~]$ sqoop list-tables --username test --password test --connect jdbc:mysql://172.1.1.96:3306/test
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:35:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:35:27 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:35:27 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
a_jw_coursejoin_detail_f
a_test
bk_fee_common
bk_student_rect
bk_subjectdet
bonus
dept
emp
exam_ticket
gg
gg_tets
id
incre_table
p_bk_real_lession_cost_detail
registration
salgrade
student_exam_score_20170802
student_exam_score_20170803
student_exam_score_20170906
student_exam_term
t
t_card_package
t_dictionary
t_sms_send
t_time
t_tracelog_cgi
t_tracelog_cgi_20180920
t_tracelog_cgi_20180921
t_tracelog_cgi_tmp
tbl_name
test_1
test_2
test_3
test_4
test_5
test_x
treenodes
tt1
tt2
tt3
tt4
user
z
[hadoop@master ~]$ 2.问题解决
2.1 mysql 在本地（sqoop本地）
这个查询资料：
1. 把url localhost 改成ip(配置/etc/hosts 文件即可)
sqoop import --username test --password test --connect jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
2.给用户赋权（如果有权限，忽略）
grant all privileges on test to test@'%' IDENTIFIED BY 'test' WITH GRANT OPTION;
flush PRIVILEGES;locahost导入失败：
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://127.0.0.1:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:53:58 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:53:58 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:53:58 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:53:58 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:53:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:53:59 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:53:59 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/910a58c86a0bd49d993fb3c8e9860e5f/t.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:54:01 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/910a58c86a0bd49d993fb3c8e9860e5f/t.jar
18/11/08 11:54:03 INFO tool.ImportTool: Destination directory t is not present, hence not deleting.
18/11/08 11:54:03 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:54:03 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:54:03 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:54:03 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:54:03 INFO mapreduce.ImportJobBase: Beginning import of t
18/11/08 11:54:03 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:54:03 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:54:03 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:54:07 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:54:07 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:54:07 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0023
18/11/08 11:54:07 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0023
18/11/08 11:54:07 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0023/
18/11/08 11:54:07 INFO mapreduce.Job: Running job: job_1541640011445_0023
18/11/08 11:54:19 INFO mapreduce.Job: Job job_1541640011445_0023 running in uber mode : false
18/11/08 11:54:19 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:54:31 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:341)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:211)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:300)
... 26 more
18/11/08 11:54:41 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.sql.SQLException: Access denied for user 'test'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2191)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
18/11/08 11:54:51 INFO mapreduce.Job: Task Id : attempt_1541640011445_0023_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:897)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:886)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2097)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2022)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:779)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:389)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
... 10 more
Caused by: java.sql.SQLException: Access denied for user 'test'@'localhost' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2191)
at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2038)
... 24 more
18/11/08 11:55:03 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:55:03 INFO mapreduce.Job: Job job_1541640011445_0023 failed with state FAILED due to: Task failed task_1541640011445_0023_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
18/11/08 11:55:03 INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=36276
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=36276
Total vcore-seconds taken by all map tasks=36276
Total megabyte-seconds taken by all map tasks=37146624
18/11/08 11:55:03 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/11/08 11:55:03 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 60.1829 seconds (0 bytes/sec)
18/11/08 11:55:03 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/11/08 11:55:03 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/11/08 11:55:03 ERROR tool.ImportTool: Error during import: Import job failed!
[hadoop@master ~]$ IP导入成功：
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://10.0.1.118:3306/test?autoReconnect=true --delete-target-dir --table t --fields-terminated-by '\t' -m 1 --hive-import --hive-database test --hive-table t
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:51:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:51:29 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:51:30 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:51:30 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:51:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:30 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:30 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/8dc5c288374d370426b87eca124b15e0/t.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:51:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/8dc5c288374d370426b87eca124b15e0/t.jar
18/11/08 11:51:34 INFO tool.ImportTool: Destination directory t deleted.
18/11/08 11:51:34 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:51:34 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:51:34 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:51:34 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:51:34 INFO mapreduce.ImportJobBase: Beginning import of t
18/11/08 11:51:34 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:51:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:51:34 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:51:37 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:51:37 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:51:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0022
18/11/08 11:51:38 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0022
18/11/08 11:51:38 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0022/
18/11/08 11:51:38 INFO mapreduce.Job: Running job: job_1541640011445_0022
18/11/08 11:51:49 INFO mapreduce.Job: Job job_1541640011445_0022 running in uber mode : false
18/11/08 11:51:49 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:51:57 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:51:57 INFO mapreduce.Job: Job job_1541640011445_0022 completed successfully
18/11/08 11:51:57 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=160855
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=76
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=5208
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=5208
Total vcore-seconds taken by all map tasks=5208
Total megabyte-seconds taken by all map tasks=5332992
Map-Reduce Framework
Map input records=10
Map output records=10
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=98
CPU time spent (ms)=820
Physical memory (bytes) snapshot=107737088
Virtual memory (bytes) snapshot=2091540480
Total committed heap usage (bytes)=18919424
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=76
18/11/08 11:51:57 INFO mapreduce.ImportJobBase: Transferred 76 bytes in 22.9794 seconds (3.3073 bytes/sec)
18/11/08 11:51:57 INFO mapreduce.ImportJobBase: Retrieved 10 records.
18/11/08 11:51:57 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t` AS t LIMIT 1
18/11/08 11:51:57 INFO hive.HiveImport: Loading uploaded data into Hive
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:52:08 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/11/08 11:52:09 INFO hive.HiveImport:
18/11/08 11:52:09 INFO hive.HiveImport: Logging initialized using configuration in file:/home/hadoop/apache-hive-2.1.1/conf/hive-log4j2.properties Async: true
18/11/08 11:52:18 INFO hive.HiveImport: OK
18/11/08 11:52:18 INFO hive.HiveImport: Time taken: 1.476 seconds
18/11/08 11:52:19 INFO hive.HiveImport: Loading data to table test.t
18/11/08 11:52:19 INFO hive.HiveImport: OK
18/11/08 11:52:19 INFO hive.HiveImport: Time taken: 1.235 seconds
18/11/08 11:52:20 INFO hive.HiveImport: Hive import complete.
18/11/08 11:52:20 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.2.2 mysql 不在本地（远程）
hadoop 是集群（多个节点），多数据节点，务必保证N个数据节点都能访问mysql 数据库（N 取决于sqoop 指定的 -m参数后的 N，大于等于-m 后面的参数），sqoop 底层运行的MR，MR运行在hadoop 集群上，所以hadoop 每个节点都必须能连接mysql 数据库，可用mysql -h***** -u***** -p***** 测试。
我的环境：
三台虚拟机，每个两块网卡，一块私有（集群间通讯），一块公用桥接（与外面机器通讯）
| 节点 | IP |
| master | 10.0.1.118 |
| slave-1 | 10.0.1.227 |
| slave-2 | 10.0.1.226 |
slave-1 连接远程数据库:
slave-2 连接远程数据库：
导入远程172.1.1.96 数据表test.student_exam_score_20170802 到hive bond_edu.student_exam_score_20170802
[hadoop@master ~]$ sqoop import --username test --password test --connect jdbc:mysql://172.1.1.96:3306/test -m 1 --delete-target-dir --table student_exam_score_20170802 --fields-terminated-by '\t' --hive-import --hive-database bond_edu --hive-table student_exam_score_20170802
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/11/08 11:22:28 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
18/11/08 11:22:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/11/08 11:22:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/08 11:22:28 INFO tool.CodeGenTool: Beginning code generation
18/11/08 11:22:29 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:22:29 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:22:29 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.8.1/share/hadoop/mapreduce
Note: /tmp/sqoop-hadoop/compile/47ceab507f5a9890950cf26e7da06fdd/student_exam_score_20170802.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/11/08 11:22:32 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/47ceab507f5a9890950cf26e7da06fdd/student_exam_score_20170802.jar
18/11/08 11:22:34 INFO tool.ImportTool: Destination directory student_exam_score_20170802 is not present, hence not deleting.
18/11/08 11:22:34 WARN manager.MySQLManager: It looks like you are importing from mysql.
18/11/08 11:22:34 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
18/11/08 11:22:34 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
18/11/08 11:22:34 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
18/11/08 11:22:34 INFO mapreduce.ImportJobBase: Beginning import of student_exam_score_20170802
18/11/08 11:22:34 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/11/08 11:22:34 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/11/08 11:22:35 INFO client.RMProxy: Connecting to ResourceManager at master/10.0.1.118:18040
18/11/08 11:22:39 INFO db.DBInputFormat: Using read commited transaction isolation
18/11/08 11:22:39 INFO mapreduce.JobSubmitter: number of splits:1
18/11/08 11:22:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541640011445_0015
18/11/08 11:22:40 INFO impl.YarnClientImpl: Submitted application application_1541640011445_0015
18/11/08 11:22:40 INFO mapreduce.Job: The url to track the job: http://master:18088/proxy/application_1541640011445_0015/
18/11/08 11:22:40 INFO mapreduce.Job: Running job: job_1541640011445_0015
18/11/08 11:22:53 INFO mapreduce.Job: Job job_1541640011445_0015 running in uber mode : false
18/11/08 11:22:53 INFO mapreduce.Job: map 0% reduce 0%
18/11/08 11:23:03 INFO mapreduce.Job: map 100% reduce 0%
18/11/08 11:23:03 INFO mapreduce.Job: Job job_1541640011445_0015 completed successfully
18/11/08 11:23:03 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=161078
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=420640
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=6613
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=6613
Total vcore-seconds taken by all map tasks=6613
Total megabyte-seconds taken by all map tasks=6771712
Map-Reduce Framework
Map input records=3956
Map output records=3956
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=92
CPU time spent (ms)=1370
Physical memory (bytes) snapshot=106192896
Virtual memory (bytes) snapshot=2092593152
Total committed heap usage (bytes)=17985536
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=420640
18/11/08 11:23:03 INFO mapreduce.ImportJobBase: Transferred 410.7812 KB in 28.5838 seconds (14.3711 KB/sec)
18/11/08 11:23:03 INFO mapreduce.ImportJobBase: Retrieved 3956 records.
18/11/08 11:23:03 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `student_exam_score_20170802` AS t LIMIT 1
18/11/08 11:23:03 INFO hive.HiveImport: Loading uploaded data into Hive
18/11/08 11:23:19 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings.
18/11/08 11:23:19 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/apache-hive-2.1.1/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:23:19 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:23:19 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/home/hadoop/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
18/11/08 11:23:19 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
18/11/08 11:23:22 INFO hive.HiveImport:
18/11/08 11:23:22 INFO hive.HiveImport: Logging initialized using configuration in file:/home/hadoop/apache-hive-2.1.1/conf/hive-log4j2.properties Async: true
18/11/08 11:23:38 INFO hive.HiveImport: OK
18/11/08 11:23:38 INFO hive.HiveImport: Time taken: 4.372 seconds
18/11/08 11:23:39 INFO hive.HiveImport: Loading data to table bond_edu.student_exam_score_20170802
18/11/08 11:23:40 INFO hive.HiveImport: OK
18/11/08 11:23:40 INFO hive.HiveImport: Time taken: 1.812 seconds
18/11/08 11:23:41 INFO hive.HiveImport: Hive import complete.
18/11/08 11:23:41 INFO hive.HiveImport: Export directory is contains the _SUCCESS file only, removing the directory.
[hadoop@master ~]$ 总结：
1.sqoop 远程导入mysql数据到hive 必须保证N个数据节点能正常连接改远程数据库（N 大于等于sqoop 并行数）。
2.sqoop 本地导入 用户可能需要用IP或者修改host文件。
3.sqoop list-tables 不需要数据节点？（存疑？？）。