本文是我学习Hadoop搭建过程中的各种笔记,内容来自于各种公开的教程,起点非常低,从Linux基础开始,直至在PC上搭建Hadoop成功,是真正的从零开始。
感谢过程中帮助我的各位认识的和不认识的老师。
52、Hdfs的ha测试:
解压后:
配置环境变量:
[root@hadoop01 zookeeper-3.4.12]# cd ../hadoop-2.7.1
[root@hadoop01 hadoop-2.7.1]# rm -rf ./share/doc/
[root@hadoop01 hadoop-2.7.1]# vi /etc/profile
export JAVA_HOME=/usr/local/jdk1.8.0_144/ (配这个)
[root@hadoop01 hadoop-2.7.1]# vi ./etc/hadoop/core-site.xml
[root@hadoop01 hadoop-2.7.1]# vi ./etc/hadoop/hdfs-site.xml
做免登陆:
在hadoop01上分发到hadoop02上:
[root@hadoop01 hadoop-2.7.1]# scp -r ../hadoop-2.7.1 hadoop02:/usr/local/
此时在hadoop03上which hadoop是没有的:
[root@hadoop03 ~]# which hadoop
/usr/bin/which: no hadoop in (/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/jdk1.8.0_144//bin:/usr/local/hadoop-2.7.1//bin:/usr/local/hadoop-2.7.1//sbin:/usr/local/zookeeper-3.4.12//bin::/root/bin)
在hadoop01上分发到hadoop03上:
[root@hadoop01 hadoop-2.7.1]# scp -r ../hadoop-2.7.1 hadoop03:/usr/local/
做免登陆的配置,做2到1,2,3的,因为2也有老大:
[root@hadoop02 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
fe:bc:6a:98:93:a0:76:26:9c:ec:5d:45:9f:38:22:c7 root@hadoop02
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . |
| . . o . |
| . E S o |
| .o + . |
| o o ..+. |
| B.o.= .o |
| o.+. o..+. |
+-----------------+
此时登录自己,也需要输密码:
[root@hadoop02 ~]# ssh hadoop02
The authenticity of host 'hadoop02 (192.168.216.112)' can't be established.
RSA key fingerprint is 58:0e:71:78:09:8c:54:ed:43:16:e3:71:eb:5c:20:57.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop02,192.168.216.112' (RSA) to the list of known hosts.
root@hadoop02's password:
Last login: Thu May 3 13:58:12 2018 from 192.168.216.1
退出去:
[root@hadoop02 ~]# exit
logout
Connection to hadoop02 closed.
[root@hadoop02 ~]# ssh-copy-id hadoop02
root@hadoop02's password:
Now try logging into the machine, with "ssh 'hadoop02'", and check in:
.ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
现在再登录就可以了,不需要再输密码:
[root@hadoop02 ~]# ssh hadoop02
Last login: Thu May 3 19:05:27 2018 from hadoop02
退出去,同样的方法配置hadoop01免登陆配置
[root@hadoop02 ~]# exit
logout
Connection to hadoop02 closed.
[root@hadoop02 ~]# ssh-copy-id hadoop01
输入: 一次yes 一次root
[root@hadoop02 ~]# ssh hadoop01
Last login: Thu May 3 16:49:43 2018 from 192.168.216.1
退出去,同样的方法配置hadoop03免登陆配置
[root@hadoop01 ~]# exit
[root@hadoop02 ~]# ssh-copy-id hadoop03
[root@hadoop02 ~]# ssh hadoop03
Last login: Thu May 3 13:58:21 2018 from 192.168.216.1
现在1和2老大间的通信也配好:(在hadoop01,可到达hadoop02)
[root@hadoop01 hadoop-2.7.1]# ssh hadoop02
Last login: Thu May 3 19:08:43 2018 from hadoop02
[root@hadoop02 ~]#
[root@hadoop02 ~]# exit
logout
Connection to hadoop02 closed.
远程发送配置好的hadoop到另外的服务器上 (也做好)
按照步骤依次执行并启动:
1、 启动zk
2、 启动journalnode服务(单个启动,多个进程启动)
./sbin/hadoop-deamon.sh start journalnode
./sbin/hadoop-deamons.sh start journalnode
3、 挑选两个namenode之中的一台格式化,并且启动
4、 在另外一台namenode的机子上拉去元数据(也可以使用复制)
hdfs namenode –bootstrapStandby
5、 格式化zkfc
hdfs zkfc –formatZK
6、 启动
7、 测试
先看对应的进程是否启动
再看web ui是否正常
在hdfs中读写文件
关闭一个namenode失败,查看是否能自动切换(制造一个namenode失败,看能不能转换过去)
第一次配高可用要执行的东西:↓
1、启动zk
启用前,保证三台机子的zk都在,分别查看jps:
[root@hadoop01 hadoop-2.7.1]# jps
14145 QuorumPeerMain
[root@hadoop02 hadoop-2.7.1]# jps
14145 QuorumPeerMain
[root@hadoop03 hadoop-2.7.1]# jps
14145 QuorumPeerMain
2、启动journalnode服务(多个进程启动)
./sbin/hadoop-deamons.sh start journalnode
[root@hadoop01 hadoop-2.7.1]# hadoop-daemons.sh start journalnode
hadoop01: starting journalnode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop01.out
hadoop02: starting journalnode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop02.out
hadoop03: starting journalnode, logging to /usr/local/hadoop-2.7.1/logs/hadoop-root-journalnode-hadoop03.out
查看hadoop01 02 03 jps: 都起起来了JournalNode
[root@hadoop01 hadoop-2.7.1]# jps
14145 JournalNode
3、挑选两个namenode之中的一台格式化,并且启动
[root@hadoop01 hadoop-2.7.1]# hdfs namenode -format
[root@hadoop01 hadoop-2.7.1]# ll /home/hahadoopdata/dfs/name (下面有东西)
此时hadoop02下面没有,(02下面也有hahadoopdata,但是没有,原因:没有同步元数据)
[root@hadoop01 hadoop-2.7.1]# ll /home/hahadoopdata/dfs/name
单个启动,不起动起来,去同步是不行的:
[root@hadoop01 hadoop-2.7.1]# hadoop-daemon.sh start namenode
[root@hadoop01 hadoop-2.7.1]# jps
18618 NameNode (namenode启动起来了)
4、在另外一台namenode的机子上拉去元数据(也可以使用复制)
hdfs namenode –bootstrapStandby
[root@hadoop02 ~]# hdfs namenod -bootstrapStandby
[root@hadoop01 hadoop-2.7.1]# ll /home/hahadoopdata/dfs/name
此时hadoop02上就有了和hadoop01一样的元数据
5、格式化zkfc
hdfs zkfc –formatZK
在克隆的hadoop01上:
此处有问题,前面就有,没能解决
2018-05-03 20:31:42,583 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181, initiating session
正确的应该出现下面这个目录:
[zk: hadoop01:2181 (CONNECTED) 0]
[zk: hadoop01:2181 (CONNECTED) 0] ls /
[name0000000002,name0000000003, zookeeper]
删除:name0000000002
[zk: hadoop01:2181 (CONNECTED) 1] rmr /name0000000002
删除:name0000000003
[zk: hadoop01:2181 (CONNECTED) 2] rmr /name0000000003
查看:
[zk: hadoop01:2181 (CONNECTED) 3] ls /
[zookeeper]
在hadoop01上格式化:
[root@hadoop01 hadoop-2.7.1]# hdfs zkfc -formatZK
在克隆的hadoop01上查看:
[zk: hadoop01:2181 (CONNECTED) 4] ls /
[hadoop-ha,zookeeper]
[zk: hadoop01:2181 (CONNECTED) 5] ls /hadoop-ha
[qf]
[zk: hadoop01:2181 (CONNECTED) 6] ls /hadoop-ha/qf
[ ]
6、启动
[root@hadoop01 hadoop-2.7.1]# start-dfs.sh
hadoop01配置了5个,但上面有6个,有一个是连接客户端
[root@hadoop01 hadoop-2.7.1]# jps
hadoop02上5个:
[root@hadoop02 hadoop-2.7.1]# jps
hadoop03上3个:
[root@hadoop02 hadoop-2.7.1]# jps
7、测试
先看对应的进程是否启动; 再看web ui是否正常
在hdfs中读写文件
在hadoop03上查看(现在没有任何东西,因为是重新搭了一个高可用)
[root@hadoop03 ~]# hdfs dfs –ls
上传:
[root@hadoop03 ~]# hdfs dfs –put /home/shell/if.sh /
此时在web ui可以看到
也可以查看,能查看的,就没问题:
[root@hadoop03 ~]# hdfs dfs –cat /if.sh
关闭一个namenode失败,查看是否能自动切换(制造一个namenode失败,看能不能转换过去)
在hadoop01上:
[root@hadoop01 hadoop-2.7.1]# jps
18618 NameNode
[root@hadoop01 hadoop-2.7.1]# kill -9 18618
再查看jps,就没有NameNode了
但是,在hadoop03上仍然能读取,这就是高可用
老大坏一个,立马把另一个起起来
[root@hadoop03 ~]# hdfs dfs –cat /if.sh
在hadoop01上重新启动:
[root@hadoop01 hadoop-2.7.1]# hadoop-daemon.sh start namenode
此时jps,有namenode:
[root@hadoop01 hadoop-2.7.1]# jps
19451 NameNode
在网址上查看,此时hadoop01是standby
在hadoop02上,杀掉namenode:
[root@hadoop01 hadoop-2.7.1]# jps
14456 NameNode
[root@hadoop01 hadoop-2.7.1]# kill -9 14456
在网址上查看,此时hadoop01就变成active,hadoop02就访问不进来了。
在hadoop03上读取文件,仍然正常
[root@hadoop03 ~]# hdfs dfs –cat /if.sh