备注:
Hive 版本 2.1.1
一.问题描述
之前开始归档，然后job卡住，我直接其它CTRL + C退出，后面运行mr job都卡在提交job的环节
hive> select * from emp where empno = 7369;
Query ID = root_20201224123252_93210ca3-928f-49e7-9cf4-d442d2a6ca3e
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
20/12/24 12:32:54 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Starting Job = job_1608780340033_0003, Tracking URL = http://hp3:8088/proxy/application_1608780340033_0003/
Kill Command = /opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hadoop/bin/hadoop job -kill job_1608780340033_0003
二.问题解决
2.1 首先想到的是重启大法
因为维护hadoop的经验暂缺，所以想到的是重启的方法。
首先尝试重启hive，发现问题依旧。
后面重启整个CDH集群，发现问题依旧。
最后尝试了重启CDH集群所在的操作系统，发现问题依旧。
2.1 查看正在执行的job
使用 hadoop job -list 和 yarn application -list 查看，发现没有job在执行。
[root@hp1 ~]# hadoop job -list
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:38:49 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:38:49 INFO conf.Configuration: resource-types.xml not found
20/12/24 13:38:49 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
Total jobs:4
JobId JobName State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
job_1606698967173_0358 Archiving ods_fact_s PREP 1608631165868 root root.users.root DEFAULT 0 0 0M 0M 0M http://hp3:8088/proxy/application_1606698967173_0358/
job_1606698967173_0359 Archiving ods_fact_s PREP 1608686680739 root root.users.root DEFAULT 0 0 0M 0M 0M http://hp3:8088/proxy/application_1606698967173_0359/
job_1606698967173_0360 obtain_user_short_me PREP 1608711418647 root root.users.root DEFAULT 0 0 0M 0M 0M http://hp3:8088/proxy/application_1606698967173_0360/
job_1608716823585_0001 obtain_user_short_me PREP 1608717111853 root root.users.root DEFAULT 0 0 0M 0M 0M http://hp3:8088/proxy/application_1608716823585_0001/
[root@hp1 ~]#
[root@hp1 ~]#
[root@hp1 ~]# yarn application -list
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:45:34 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):8
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
application_1606698967173_0358 Archiving ods_fact_sa...sale_date=2010-04-12 MAPREDUCE root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1606698967173_0359 Archiving ods_fact_sa...sale_date=2010-04-12 MAPREDUCE root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1606698967173_0357 Hive on Spark (hiveSessionId = 6644365a-1e85-450f-9408-0dc223239cde) SPARK anonymous root.users.anonymous ACCEPTED UNDEFINED 0% N/A
application_1606698967173_0360 obtain_user_short_message.jar MAPREDUCE root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1608780340033_0001 Hive on Spark (hiveSessionId = ed520b3f-4779-4614-86cb-ea94d43bef85) SPARK root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1608716823585_0003 Hive on Spark (hiveSessionId = 4562d25b-0456-4bc0-86bd-78a95c6a489c) SPARK root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1608716823585_0001 obtain_user_short_message.jar MAPREDUCE root root.users.root ACCEPTED UNDEFINED 0% N/A
application_1608779426561_0001 Hive on Spark (hiveSessionId = e2f00acc-cc4f-4ab3-ba16-6e56bae1a3db) SPARK root root.users.root ACCEPTED UNDEFINED 0% N/A
[root@hp1 ~]#
通过kill命令杀掉卡住的job
语法:
-- kill mr job
hadoop job -kill job_1606698967173_0358
-- kill yarn application
yarn application -kill application_1606698967173_0357
测试记录:
[root@hp1 ~]# hadoop job -kill job_1606698967173_0358
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:47:41 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:47:42 INFO impl.YarnClientImpl: Killed application application_1606698967173_0358
Killed job job_1606698967173_0358
[root@hp1 ~]#
[root@hp1 ~]# hadoop job -kill job_1606698967173_0359
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:47:56 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:47:56 INFO impl.YarnClientImpl: Killed application application_1606698967173_0359
Killed job job_1606698967173_0359
[root@hp1 ~]#
[root@hp1 ~]# hadoop job -kill job_1606698967173_0360
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:48:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:48:22 INFO impl.YarnClientImpl: Killed application application_1606698967173_0360
Killed job job_1606698967173_0360
[root@hp1 ~]#
[root@hp1 ~]# hadoop job -kill job_1606698967173_0361
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:48:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:48:26 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
20/12/24 13:48:26 INFO tools.CLI: Could not obtain job info after 1 attempt(s). Sleeping for 2 seconds and retrying.
20/12/24 13:48:28 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
20/12/24 13:48:28 INFO tools.CLI: Could not obtain job info after 2 attempt(s). Sleeping for 2 seconds and retrying.
20/12/24 13:48:30 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
20/12/24 13:48:30 INFO tools.CLI: Could not obtain job info after 3 attempt(s). Sleeping for 2 seconds and retrying.
20/12/24 13:48:32 INFO mapred.ClientServiceDelegate: Could not get Job info from RM for job job_1606698967173_0361. Redirecting to job history server.
Could not find job job_1606698967173_0361
[root@hp1 ~]#
[root@hp1 ~]# hadoop job -kill job_1608716823585_0001
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:49:27 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:49:27 INFO impl.YarnClientImpl: Killed application application_1608716823585_0001
Killed job job_1608716823585_0001
[root@hp1 ~]#
[root@hp1 ~]# hadoop job -list
WARNING: Use of this script to execute job is deprecated.
WARNING: Attempting to execute replacement "mapred job" instead.
20/12/24 13:49:37 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
20/12/24 13:49:38 INFO conf.Configuration: resource-types.xml not found
20/12/24 13:49:38 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
Total jobs:0
JobId JobName State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
[root@hp1 ~]#
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1606698967173_0357
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Killing application application_1606698967173_0357
20/12/24 13:51:26 INFO impl.YarnClientImpl: Killed application application_1606698967173_0357
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608780340033_0001
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608780340033_0001 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608716823585_0003
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608716823585_0003 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608779426561_0001
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:52:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608779426561_0001 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -list
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:52:06 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
[root@hp1 ~]#
三.验证
可以看到现在运行mr job已经没有问题了
[root@hp1 ~]# yarn application -kill application_1606698967173_0357
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:26 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Killing application application_1606698967173_0357
20/12/24 13:51:26 INFO impl.YarnClientImpl: Killed application application_1606698967173_0357
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608780340033_0001
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608780340033_0001 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608716823585_0003
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:51:53 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608716823585_0003 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -kill application_1608779426561_0001
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:52:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Application application_1608779426561_0001 has already finished
[root@hp1 ~]#
[root@hp1 ~]# yarn application -list
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/12/24 13:52:06 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm69
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):0
Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URL
[root@hp1 ~]#
参考
1.https://blog.csdn.net/weixin_43930715/article/details/109765454