运维企业实战——hadoop+zookeeper高可用

一、实验环境
清除实验环境：

[hadoop@server1 hadoop]$ sbin/stop-yarn.sh 
[hadoop@server1 hadoop]$ sbin/stop-dfs.sh 

确保每台机子上都安装了jdk hadoop并配置了环境

删除server1，server2，server3, server4 tmp中的内容

[hadoop@server1 hadoop]$ cd /tmp/
[hadoop@server1 tmp]$ rf -rm *
[hadoop@server2 ~]$ rm -rf /tmp/*
[hadoop@server3 ~]$ rm -rf /tmp/*
[hadoop@server4 ~]$ rm -rf /tmp/*

二、在server2上搭建zookepper

[hadoop@server2 ~]$ ls
hadoop               java                        zookeeper-3.4.9.tar.gz
hadoop-3.0.3         jdk1.8.0_181
hadoop-3.0.3.tar.gz  jdk-8u181-linux-x64.tar.gz
[hadoop@server2 ~]$ tar zxf zookeeper-3.4.9.tar.gz
[hadoop@server2 ~]$ cd zookeeper-3.4.9
[hadoop@server2 zookeeper-3.4.9]$ cd conf/
[hadoop@server2 conf]$ ls
configuration.xsl  log4j.properties  zoo_sample.cfg\

添加从节点信息

[hadoop@server2 conf]$  cp zoo_sample.cfg zoo.cfg
[hadoop@server2 conf]$ vim zoo.cfg #添加以下代码到文件末尾

配置id 并启动zookeeper
各节点配置文件相同,并且需要在/tmp/zookeeper 目录中创建 myid 文件,写入一个唯一的数字,取值范围在 1-255

[hadoop@server2 ~]$ mkdir /tmp/zookeeper
[hadoop@server3 ~]$ mkdir /tmp/zookeeper
[hadoop@server4 ~]$ mkdir /tmp/zookeeper
   
[hadoop@server2 conf]$ echo 1 > /tmp/zookeeper/myid
[hadoop@server2 zookeeper-3.4.9]$ bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[hadoop@server3 ~]$ echo 2 > /tmp/zookeeper/myid
[hadoop@server3 zookeeper-3.4.9]$ bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED


[hadoop@server4 ~]$ echo 3 > /tmp/zookeeper/myid
[hadoop@server4 ~]$ cd zookeeper-3.4.9
[hadoop@server4 zookeeper-3.4.9]$ bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

查看各节点的状态

[hadoop@server2 zookeeper-3.4.9]$ bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: follower

[hadoop@server3 zookeeper-3.4.9]$ bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: leader

[hadoop@server4 zookeeper-3.4.9]$ bin/zkServer.sh status 
#确保java环境，用java -version查看，如果环境有问题可以重新加载一下，使用[hadoop@server4 ~]$ source .bash_profile 

ZooKeeper JMX enabled by default
Using config: /home/hadoop/zookeeper-3.4.9/bin/../conf/zoo.cfg
Mode: follower

在server2进入命令行

[hadoop@server2 bin]$ ls
README.txt    zkCli.cmd  zkEnv.cmd  zkServer.cmd
zkCleanup.sh  zkCli.sh   zkEnv.sh   zkServer.sh
[hadoop@server2 bin]$ pwd
/home/hadoop/zookeeper-3.4.9/bin
[hadoop@server2 bin]$ ./zkCli.sh #连接zookeeper

在server1上配置hadoop

cd /home/hadoop/hadoop/etc/hadoop
[hadoop@server1 hadoop]$ vim core-site.xml 
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://masters</value>
        </property>
        <property>
                <name>ha.zookeeper.quorum</name>
                <value>172.25.60.2:2181,172.25.60.3:2181,172.25.60.4:2181</value>
        </property>
</configuration>


 [hadoop@server1 hadoop]$ vim hdfs-site.xml
<configuration>
 <property>
      <name>dfs.replication</name>
      <value>3</value>
 </property>
 
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.nameservices</name>
<value>masters</value>
</property>

<property>
<name>dfs.ha.namenodes.masters</name>
<value>h1,h2</value>
</property>

<property>
<name>dfs.namenode.rpc-address.masters.h1</name>
<value>172.25.76.1:9000</value>
</property>

<property>
<name>dfs.namenode.http-address.masters.h1</name>
<value>172.25.76.1:9870</value>
</property>

<property>
<name>dfs.namenode.rpc-address.masters.h2</name>
<value>172.25.76.5:9000</value>
</property>

<property> 
<name>dfs.namenode.shared.edits.dir</name> 
<value>qjournal://172.25.76.2:8485;172.25.76.3:8485;172.25.76.4:8485/masters</value>
</property>

<property>
<name>dfs.journalnode.edits.dir</name>
<value>/tmp/journaldata</value>
</property>

<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>

<property> 
<name>dfs.client.failover.proxy.provider.masters</name> 
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> 
</property>

<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>

<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>

</configuration>

在server 2 3 4 上启动 zookeeper 集群节点

[hadoop@server2 hadoop]$ bin/hdfs --daemon start journalnode
WARNING: /home/hadoop/hadoop-3.0.3/logs does not exist. Creating.
[hadoop@server2 hadoop]$ jps
1920 Jps
1768 QuorumPeerMain
1902 JournalNode

[hadoop@server3 hadoop]$ bin/hdfs --daemon start journalnode
WARNING: /home/hadoop/hadoop-3.0.3/logs does not exist. Creating.
[hadoop@server3 hadoop]$ jps
1808 Jps
1677 QuorumPeerMain
1790 JournalNode

[hadoop@server4 hadoop]$ bin/hdfs --daemon start journalnode
WARNING: /home/hadoop/hadoop-3.0.3/logs does not exist. Creating.
[hadoop@server4 hadoop]$ jps
1462 Jps
1432 JournalNode
1322 QuorumPeerMain

传递配置文件搭建高可用

[hadoop@server1 hadoop]$ cd /home/hadoop/hadoop
[hadoop@server1 hadoop]$ bin/hdfs namenode -format
[hadoop@server1 hadoop]$ scp -r /tmp/hadoop-hadoop 172.25.70.5:/tmp

格式化 zookeeper (只需在 h1 上执行即可)

[hadoop@server1 hadoop]$ pwd
/home/hadoop/hadoop
[hadoop@server1 hadoop]$ bin/hdfs zkfc -formatZK

启动 hdfs 集群(只需在 h1 上执行即可)

[hadoop@server1 hadoop]$ sbin/start-dfs.sh 
Starting namenodes on [server1 server5]
server5: Warning: Permanently added 'server5' (ECDSA) to the list of known hosts.
Starting datanodes
Starting journal nodes [172.25.60.2 172.25.60.3 172.25.60.4]
172.25.60.2: journalnode is running as process 11612.  Stop it first.
172.25.60.3: journalnode is running as process 11399.  Stop it first.
172.25.60.4: journalnode is running as process 11977.  Stop it first.
Starting ZK Failover Controllers on NN hosts [server1 server5]
[hadoop@server1 hadoop]$ jps
17074 DFSZKFailoverController

16725 NameNode
17125 Jps

关闭server1，server5的状态就变成了active

 [hadoop@server1 hadoop]$ kill 2365
[hadoop@server1 hadoop]$ jps
4261 Jps
2711 DFSZKFailoverController

在server1上传文件成功 通过server5

[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir -p /user/hadoop
[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir input
[hadoop@server1 hadoop]$ bin/hdfs dfs -put etc/hadoop/* input

再次打开server1

[hadoop@server1 hadoop]$ bin/hdfs --daemon start namenode
[hadoop@server1 hadoop]$ jps
4500 NameNode
2711 DFSZKFailoverController
4572 Jps