本文是我学习Hadoop搭建过程中的各种笔记，内容来自于各种公开的教程，起点非常低，从Linux基础开始，直至在PC上搭建Hadoop成功，是真正的从零开始。

感谢过程中帮助我的各位认识的和不认识的老师。

53、Yarn的ha配置和测试：

官网：http://hadoop.apache.org/docs/current/

YARN模块，ResourceManager HA （HA一般都是老大的HA）网址：http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

架构：

yarn的HA：

规划：

IP地址	主机名称	规划
192.168.216.111	hadoop01	resourcemanager、nodemanager、quroumPeerMain
192.168.216.112	hadoop02	resourcemanager、nodemanager、quroumPeerMain
192.168.216.113	hadoop03	nodemanager、quroumPeerMain

[没有自动失败转移的东西，内部有切换的机制]

简单的配置：

yarn-site.xml

<name>yarn.resourcemanager.ha.enabled</name>

</property>

<name>yarn.resourcemanager.cluster-id</name>

<value>cluster1</value>

</property>

<name>yarn.resourcemanager.ha.rm-ids</name>

</property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>hadoop01</value>

</property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>hadoop02</value>

</property>

<name>yarn.resourcemanager.webapp.address.rm1</name>

<value>hadoop01:8088</value>

</property>

<name>yarn.resourcemanager.webapp.address.rm2</name>

<value>hadoop02:8088</value>

</property>

<name>yarn.resourcemanager.zk-address</name>

<value>hadoop01:2181,hadoop02:2181,hadoop03:2181</value>

</property>

到上面的yarn的完成，跑作业还需要加点东西：

mapred-site.xml

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>mapreduce.framework.name</name>

</property >

</configuration>

远程发送：

[root@hadoop01 hadoop-2.7.1]# scp -r ./etc/hadoop/mapred-site.xml. ./etc/hadoop/yarn-site.xml hadoop02:/usr/local/hadoop-2.7.1/etc/hadoop/

mapred-site.xml 100% 893 0.9KB/s 00:00

yarn-site.xml 100% 1647 1.6KB/s 00:00

[root@hadoop01 hadoop-2.7.1]# scp -r ./etc/hadoop/mapred-site.xml. ./etc/hadoop/yarn-site.xml hadoop03:/usr/local/hadoop-2.7.1/etc/hadoop/

mapred-site.xml 100% 893 0.9KB/s 00:00

yarn-site.xml 100% 1647 1.6KB/s 00:00

[还需要做ssh免登陆，前面做了，这就不用做了]

直接启动：start-yarn.sh

[root@hadoop01 hadoop-2.7.1]# start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-resourcemanager-hadoop01.out

hadoop03: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop03.out

hadoop02: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop02.out

hadoop01: starting nodemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root-nodemanager-hadoop0.out

手动的HA：yarn-daemon.sh start resourcemanager

hadoop02上，手动启动resourcemanager：

[root@hadoop02 ~]# yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /usr/local/hadoop-2.7.1/logs/yarn-root- resourcemanager-hadoop03.out

在hadoop01,02上查看jps，都多了ResourceManager和NodeManager；在hadoop03上查看jps，多了NodeManager

查看web ui监控，能看到：

http://192.168.216.111:8088/ 处于active状态

http://192.168.216.112:8088/ 处于standby状态

两台上面都有resourcemanager，要不然web ui进不来

测试：

yara的高可用测试：yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/00

[关闭active的rm，然后查看是否正常切换]

跑一个作业：

到hadoop01，在home下建一个words vi /home/words

[root@hadoop01 hadoop-2.7.1]# vi /home/words

hadoop is nice

hadoop good

hadoop is better

建好，下面统计单词

把/home/words放到 / 根目录下来：hdfs dfs -put /home/words /

[root@hadoop01 hadoop-2.7.1]# hdfs dfs -put /home/words /

用yarn的集群测试能不能跑，能跑，yarn也是OK的

[root@hadoop01 hadoop-2.7.1]# yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/00

跑完，查看00的结果：

[root@hadoop01 hadoop-2.7.1]# hdfs dfs -cat /out/00/part-r-00000

better 1

good 1

hadoop 3

is 2

nice 1

关闭hadoop01的ResourceManager

[root@hadoop01 hadoop-2.7.1]# kill -9 20444

在网址web ui监控上查看，原来是standby状态的切换成了active

再次跑结果，改成01：

[root@hadoop01 hadoop-2.7.1]# yarn jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount/words /out/01

因为高可用切过去了，所以依然能跑出结果

查看jps，hadoop01,02,03的NodeManager都在，没有问题

查看01的结果，跑出来了：

[root@hadoop01 hadoop-2.7.1]# hdfs dfs -cat /out/01/part-r-00000

better 1

good 1

hadoop 3

is 2

nice 1

到此yarn的高可用搭建好↑

实体机

1、将centos写入U盘，按F8，使用U盘启动，安装虚拟机；

2、添加网卡驱动：

网址：https://blog.csdn.net/tidaomazi/article/details/79345746 按照此步骤执行即可；

注意：

① 下载：e1000e-3.4.0.2.tar.gz制到centos中的Computer Filesystem中的srv中；

② 将U盘中的Packages，复制到centos中的Computer Filesystem中

③ 进入network-scripts目录：

[root@hadoop001 Desktop]# cd /etc/sysconfig/network-scripts/

④ 新建一个eth0：

[root@localhost network-scripts]# vim ifcfg-eth0

⑤ 查看HWaddr：（每台机子都有固定的一个，硬件地址，唯一标识）

[root@hadoop001 network-scripts]#cat /etc/udev/rules.d/70-persistent-net.rules 或

[root@hadoop001 network-scripts]#ifconfig

⑥ 配置eth0（编辑eth0）:

[root@localhost network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

TYPE=Ethernet

UUID=b06ea46c-aced-4e34-aaf6-5e57e9b3015d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=static （static静态IP ；dhcp动态IP；none无（不指定））

HWADDR=00:0C:29:B0:A4:A5 （要改）

IPADDR=192.168.216.111 （要改）

PREFIX=24

GATEWAY=192.168.216.2

DNS1=192.168.216.2

DEFROUTE=yes

IPV4_FAILURE_FATAL=yes

IPV6INIT=no

NAME="System eth0"

⑦ 新建：（读取：cat network）

[root@hadoop001 network-scripts]#vim network

NETWORKING=yes

HOSTNAME=hadoop001

3、连接外网：

[root@hadoop001 Desktop]#route –n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

192.168.216.0 0.0.0.0 255.255.255.0 U 1 0 0 eth0

0.0.0.0 192.168.216.2 0.0.0.0 UG 0 0 0 eth0

[root@hadoop001 Desktop]# route add default gw 192.168.216.2

[root@hadoop001 Desktop]# vim /etc/resolv.conf

nameserver 114.114.114.114

nameserver 8.8.8.8 （添加）

[root@localhost network-scripts]# vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

TYPE=Ethernet

UUID=b06ea46c-aced-4e34-aaf6-5e57e9b3015d

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=dhcp

HWADDR=00:0C:29:B0:A4:A5

#IPADDR=192.168.216.111

#PREFIX=24

#GATEWAY=192.168.216.2

#DNS1=192.168.216.2

DEFROUTE=yes

IPV4_FAILURE_FATAL=yes

IPV6INIT=no

NAME="System eth0"

4、连接XShell

Hadoop搭建笔记（30）

实体机

猜你喜欢