关于hadoop群起脚本走过的那些坑

其他 2021-01-24 23:42:20 阅读次数: 0

1、dos与unix不匹配

当脚本在远程客户端写得，或者从其它地方粘贴过来时可能会出现的问题，原因是粘贴过来的文本内容不是unix格式（一般是dos格式），需要在vi编辑器上将文件类型改为unix

# 按esc，输入：
:set fileformat=unix
:wq

2、zookeeper或journalnode在群起脚本上启动也没有保错，很正常，但jps就是没启动

zookeeper和journalnode有点特殊，我也忘记是为啥了，反正就是群起脚本执行前要加source /etc/profile

如：

#!/bin/bash

case $1 in
"start"){
	echo "------------------------$1 zookeeper------------------------"
	for i in hadoop104 hadoop105 hadoop106;
	do
		echo "------------------------$1 $i zookeeper------------------------"
		ssh $i "source /etc/profile;/opt/modules/zookeeper-3.4.5-cdh5.3.6/bin/zkServer.sh start"
	done
};;
"stop"){
    echo "------------------------$1 zookeeper------------------------"
    for i in hadoop104 hadoop105 hadoop106;
    do
        echo "------------------------$1 $i zookeeper------------------------"
        ssh $i "source /etc/profile;/opt/modules/zookeeper-3.4.5-cdh5.3.6/bin/zkServer.sh stop"
    done
};;
esac

3、hdfs无法启动

可能是没有格式化namenode

格式化可能会出现的问题：java.io.IOException: Cannot create directory /opt/module/hadoop-2.7.2/data/tmp/dfs/name/current

即无法创建文件夹，原因是权限不够切换到root用户执行bin/hdfs namenode -format

STARTUP_MSG:   build = http://github.com/cloudera/hadoop -r 6743ef286bfdd317b600adbdb154f982cf2fac7a; compiled by 'jenkins' on 2015-07-28T22:14Z
STARTUP_MSG:   java = 1.8.0_121
************************************************************/
20/10/28 18:25:22 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
20/10/28 18:25:22 INFO namenode.NameNode: createNameNode [-format]
20/10/28 18:25:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-50438531-b2f8-42c2-92dd-f7abb9ee1f4f
20/10/28 18:25:22 INFO namenode.FSNamesystem: No KeyProvider found.
20/10/28 18:25:22 INFO namenode.FSNamesystem: fsLock is fair:true
20/10/28 18:25:22 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
20/10/28 18:25:22 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
20/10/28 18:25:22 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
20/10/28 18:25:22 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Oct 28 18:25:22
20/10/28 18:25:22 INFO util.GSet: Computing capacity for map BlocksMap
20/10/28 18:25:22 INFO util.GSet: VM type       = 64-bit
20/10/28 18:25:22 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
20/10/28 18:25:22 INFO util.GSet: capacity      = 2^21 = 2097152 entries
20/10/28 18:25:22 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
20/10/28 18:25:22 INFO blockmanagement.BlockManager: defaultReplication         = 3
20/10/28 18:25:22 INFO blockmanagement.BlockManager: maxReplication             = 512
20/10/28 18:25:22 INFO blockmanagement.BlockManager: minReplication             = 1
20/10/28 18:25:22 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
20/10/28 18:25:22 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
20/10/28 18:25:22 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
20/10/28 18:25:22 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
20/10/28 18:25:22 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
20/10/28 18:25:22 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
20/10/28 18:25:22 INFO namenode.FSNamesystem: supergroup          = supergroup
20/10/28 18:25:22 INFO namenode.FSNamesystem: isPermissionEnabled = true
20/10/28 18:25:22 INFO namenode.FSNamesystem: HA Enabled: false
20/10/28 18:25:22 INFO namenode.FSNamesystem: Append Enabled: true
20/10/28 18:25:22 INFO util.GSet: Computing capacity for map INodeMap
20/10/28 18:25:22 INFO util.GSet: VM type       = 64-bit
20/10/28 18:25:22 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
20/10/28 18:25:22 INFO util.GSet: capacity      = 2^20 = 1048576 entries
20/10/28 18:25:22 INFO namenode.NameNode: Caching file names occuring more than 10 times
20/10/28 18:25:22 INFO util.GSet: Computing capacity for map cachedBlocks
20/10/28 18:25:22 INFO util.GSet: VM type       = 64-bit
20/10/28 18:25:22 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
20/10/28 18:25:22 INFO util.GSet: capacity      = 2^18 = 262144 entries
20/10/28 18:25:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
20/10/28 18:25:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
20/10/28 18:25:22 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
20/10/28 18:25:22 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
20/10/28 18:25:22 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
20/10/28 18:25:22 INFO util.GSet: Computing capacity for map NameNodeRetryCache
20/10/28 18:25:22 INFO util.GSet: VM type       = 64-bit
20/10/28 18:25:22 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
20/10/28 18:25:22 INFO util.GSet: capacity      = 2^15 = 32768 entries
20/10/28 18:25:22 INFO namenode.NNConf: ACLs enabled? false
20/10/28 18:25:22 INFO namenode.NNConf: XAttrs enabled? true
20/10/28 18:25:22 INFO namenode.NNConf: Maximum size of an xattr: 16384
20/10/28 18:25:22 INFO namenode.FSImage: Allocated new BlockPoolId: BP-924336988-192.168.1.104-1603880722785
20/10/28 18:25:22 INFO common.Storage: Storage directory /opt/module/hadoop-2.7.2/data/tmp/dfs/name has been successfully formatted.
20/10/28 18:25:22 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/10/28 18:25:22 INFO util.ExitUtil: Exiting with status 0
20/10/28 18:25:22 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop104/192.168.1.104
************************************************************/

初始化成功

4、hdfs无法启动

两种情况，一种是没有格式化，一种是格式化过多

hadoop namenode -format后，发现再次启动hadoop后，datanode节点无法正常启动。

原因是格式化次数过多，把几个集群里的data目录下的东西都清掉再格式化就行了

5、最后再附上我的群起脚本吧，希望对你们有帮助

#!/bin/bash

case $1 in
"start"){
	echo "**********$1 all zkServer**********"
	ssh hadoop104 "/opt/modules/zookeeper-3.4.5-cdh5.3.6/bin/allzkServer.sh start";
	
	echo "**********$1 HDFS**********"
	echo "----------$1 hadoop104 namenode----------"
	ssh hadoop104 "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh start namenode"
	
	echo "----------$1 hadoop106 Secondarynamenode----------"
	ssh hadoop106 "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh start secondarynamenode"
	
	echo "**********$1 datanode**********"
	for i in hadoop104 hadoop105 hadoop106;
	do
		echo "----------$1 $i datanode----------"
		ssh $i "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh start datanode"
	done
	
	echo "**********$1 YARN**********"
	echo "----------$1 hadoop105 resourcemanager----------"
	ssh hadoop105 "/opt/modules/hadoop-2.7.2/sbin/yarn-daemon.sh start resourcemanager"
	
	echo "**********$1 nodemanager**********"
	for i in hadoop104 hadoop105 hadoop106;
	do
		echo "----------$1 $i nodemanager----------"
		ssh $i "/opt/modules/hadoop-2.7.2/sbin/yarn-daemon.sh start nodemanager"
	done
	
	echo "**********$1 historyserver**********"
	ssh hadoop106 "/opt/modules/hadoop-2.7.2/sbin/mr-jobhistory-daemon.sh start historyserver"
};;
# -------------------------------------------------------------------------------------------------------------------
# -------------------------------------------------------------------------------------------------------------------
"stop"){
	echo "**********$1 historyserver**********"
	ssh hadoop106 "/opt/modules/hadoop-2.7.2/sbin/mr-jobhistory-daemon.sh stop historyserver"
	
	echo "**********$1 YARN**********"
	echo "----------$1 hadoop105 resourcemanager----------"
	ssh hadoop105 "/opt/modules/hadoop-2.7.2/sbin/yarn-daemon.sh stop resourcemanager"
	
	echo "**********$1 nodemanager**********"
	for i in hadoop104 hadoop105 hadoop106;
	do
		echo "----------$1 $i nodemanager----------"
		ssh $i "/opt/modules/hadoop-2.7.2/sbin/yarn-daemon.sh stop nodemanager"
	done
	echo "**********$1 HDFS**********"
	echo "----------$1 hadoop104 namenode----------"
	ssh hadoop104 "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh stop namenode"
	
	echo "----------$1 hadoop105 Secondarynamenode----------"
	ssh hadoop106 "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh stop secondarynamenode"
	
	echo "**********$1 datanode**********"
	for i in hadoop104 hadoop105 hadoop106;
	do
		echo "----------$1 $i datanode----------"
		ssh $i "/opt/modules/hadoop-2.7.2/sbin/hadoop-daemon.sh stop datanode"
	done
	
	echo "**********$1 all zkServer**********"
	ssh hadoop104 "/opt/modules/zookeeper-3.4.5-cdh5.3.6/bin/allzkServer.sh stop";
};;
esac
扫描二维码关注公众号，回复： 12258005 查看本文章

猜你喜欢

转载自blog.csdn.net/tyh1579152915/article/details/109334218
今日推荐
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
周排行
Java基础复习_day13_Collection集合
2018.11.16 c语言学习经验
且看Java内置四大核心函数式接口
小程序云开发中数据库的数据分段和显示图片
python的函数
Web-JS进阶
【干货】C++常用代码积累笔记大全
Spring的ioc操作 与 IOC底层原理
构建之法20191121-11 Scrum立会报告+燃尽图 07
Spring boot之Hello World访问404
每日归档
更多
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022