Hadoop NameNode HA搭建

其他 2019-01-09 21:07:16 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/u010719917/article/details/79789210

hadoop namenode HA的搭建

需要:

192.168.44.128   centos128
192.168.44.129   centos129
192.168.44.130   centos130
192.168.44.131   centos131
centos7.3
hadoop2.7
zookeeper3.4

各集群节点分布图

          |   NN | DN |  ZK | ZKFC | JN | RM | DM
---------------------------------- --------------
centos128 |   1  |    |  1  |  1   |    | 1  |
---------------------------------- --------------
centos129 |   1  | 1  |  1  |  1   |  1 |    |  1
---------------------------------- --------------
centos130 |      | 1  |  1  |      |  1 |    |  1
---------------------------------- --------------
centos131 |      | 1  |     |      |  1 |    |  1

namenode集群
datanode集群
zeookeepre集群
zkfc集群
jouralnode集群
resouce manage进程
data manage集群

一、jdk1.8安装
jdk安装在/usr/src/jkd
在/etc/profile文件添加以下内容：
JAVA_HOME=/usr/src/jdk/
PATH= J A V A H O M E / b i n : / u s r / l o c a l / x t r a b a c k u p / b i n : PATH
CLASSPATH=.: J A V A H O M E / l i b / d t . j a r : JAVA_HOME/lib/tools.jar
执行source /etc/profile

二、zookeeper集群安装
1、在/etc/profile文件添加以下内容：
export ZOOKEEPER_HOME=/root/hadoop-0.20.2/zookeeper-3.3.1
export PATH= P A T H : ZOOKEEPER_HOME/bin:$ZOOKEEPER_HOME/conf

在/etc/hosts中添如下内容：

     192.168.44.128  centos128
     192.168.44.129  centos129
     192.168.44.130  centos130
     192.168.44.131  centos131
zookeeper解压在/usr/src/zookeeper.

2、设置集群配置文件/usr/src/zookeeper/conf/zoo.cfg：

    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/usr/src/zookeeper/data
    clientPort=2181
    #maxClientCnxns=60
    #autopurge.snapRetainCount=3
    #autopurge.purgeInterval=1
    server.1=centos128:2888:3888
    server.2=centos129:2888:3888
    server.3=centos130:2888:3888

注释：

 tickTime=2000：每个tick设置为2000毫秒
 initLimit=10 ：初始同步阶段10个tick时间
 syncLimit=5:发送请求和得到确认之间5个tick间隔时间，follower之间的时间限制，超过了就会放弃
 dataDir=/usr/src/zookeeper/data：snapshot存储路径在/usr/src/zookeeper/data下
clientPor=2181： 客户端通2181端口连接
maxClientCnxns=60：最大的clients连接数为60，可以根据实际情况调整
autopurge.snapRetainCount=3：在dataDir路径保存3份snapshot(快照)
autopurge.purgeInterval=1 ：每一个小时消除任务，设置为0为关闭自动清除特征
 server.1=centos128:2888:3888  ：zookeeper集群的server.id ,其中id值必须与dataDir目录下的myid文件里的内容一致，都 为1，
host:port:port其中host就是服务器，两个port端口，第一个follower用一个端口2888，第二个follower用一个端口3888
server.2=centos129:2888:3888
server.3=centos130:2888:3888

3、在每个节点上开启zookeeper并查看进程

[root@centos128 bin]# /usr/src/zookeeper/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /usr/src/zookeeper/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [root@centos128 bin]# jps
    14962 QuorumPeerMain
    [root@centos129 bin]# /usr/src/zookeeper/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /usr/src/zookeeper/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [root@centos129 bin]# jps
    14962 QuorumPeerMain
    [root@centos130 bin]# /usr/src/zookeeper/bin/zkServer.sh start
    ZooKeeper JMX enabled by default
    Using config: /usr/src/zookeeper/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED
    [root@centos130 bin]# jps
    14962 QuorumPeerMain

三、hadoop安装
1,ssh trust
ssh trusts设置

    ssh-keygen -t rsa
    ssh-keygen -t dsa
    cd ~/.ssh
    ssh-copy-id -i  id_rsa.pub  centos128
    ssh-copy-id -i  id_dsa.pub  centos128
    ssh-copy-id -i  id_rsa.pub  centos129
    ssh-copy-id -i  id_dsa.pub  centos129
    ssh-copy-id -i  id_rsa.pub  centos130
    ssh-copy-id -i  id_dsa.pub  centos130
    ssh-copy-id -i  id_rsa.pub  centos131
    ssh-copy-id -i  id_dsa.pub  centos131
在其它服务器做同样的设置

**2、存放路径的创建**

    mkdir  /data/hadoop/name -p
    mkdir  /data/hadoop/tmp -p
    mkdir  /Data1 -p
    mkdir  /Data2 -p
**3、设置配置文件**
主要几个配置文件
**a,在hadoop-env.sh中将**

    export JAVA_HOME=${JAVA_HOME}
改成

    export JAVA_HOME=/usr/src/jdk
b,

etc/hadoop/core-site.xml  配置NameNode URI
etc/hadoop/hdfs-site.xml  配置NameNode ，配置DataNode，
etc/hadoop/yarn-site.xml  配置ResourceManager  ，配置NodeManager ，配置History Server
etc/hadoop/mapred-site.xml  配置MapReduce Applications，配置 MapReduce JobHistory Server
 etc/hadoop/slaves         添加slave的IP

b.1,etc/hadoop/core-site.xml 配置如下：

扫描二维码关注公众号，回复： 4846532 查看本文章 
  <!-- Put site-specific property overrides in this file. -->
     <configuration>
        <property>
            <name>fs.defaultFS</name>
            <value>hdfs://centos128:9000</value>
        </property>
      <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/hadoop/tmp</value>
      </property>
        <property>
            <name>io.file.buffer.size</name>
            <value>131072</value>
        </property>
      <property>
        <name>fs.trash.interval</name>
        <value>10080</value>
      </property>
    </configuration>
其中hdfs://centos128:9000是 DataName uri地址

b.2, etc/hadoop/hdfs-site.xml 配置如下：

    <configuration>
        <property>
            <name>dfs.namenode.name.dir</name>
            <value>/data/hadoop/name</value>
        </property>
        <property>
            <name>dfs.blocksize</name>
            <value>268435456</value>
        </property>
        <property>
            <name>dfs.namenode.handler.count</name>
            <value>100</value>
        </property>
       <!--
        <property>
            <name>dfs.replication</name>
            <value>3</value>
        </property>
       -->
        <property>
            <name>dfs.datanode.data.dir</name>
            <value>/Data1,/Data2</value>
        </property>
    </configuration>
dfs.namenode.name.dir  namenode物理路径
dfs.replication        默认为3个副本
dfs.datanode.data.dir  datanode放存物理路径

b.3, etc/hadoop/yarn-site.xml 配置如下：
含义参考：http://blog.csdn.net/u010719917/article/details/73917217

<!-- Site specific YARN configuration properties -->
     <!--
     ResourceManager
      -->
     <property>
       <name>yarn.log-aggregation-enable</name>
       <value>true</value>
     </property>
    <!--Configurations for ResourceManager -->
     <property>
       <name>yarn.resourcemanager.hostname</name>
       <value>centos128</value>
     </property>
     <property>
       <name>yarn.resourcemanager.address</name>
 <value>${yarn.resourcemanager.hostname}:8032</value>
     </property>
     <property>
     <name>yarn.resourcemanager.scheduler.address</name>
       <value>${yarn.resourcemanager.hostname}:8030</value>
     </property>
     <property>
       <name>yarn.resourcemanager.resource-tracker.address</name>
       <value>${yarn.resourcemanager.hostname}:8031</value>
     </property>
     <property>
       <name>yarn.resourcemanager.admin.address</name>
       <value>${yarn.resourcemanager.hostname}:8033</value>
     </property>
     <property>
       <name>yarn.resourcemanager.webapp.address</name>
       <value>${yarn.resourcemanager.hostname}:8088</value>
     </property>
     <property>
       <name>yarn.resourcemanager.scheduler.class</name>
       <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
     </property>
     <!--
     <property>
       <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
       <value>50</value>
     </property>
     <property>
       <name>yarn.resourcemanager.scheduler.client.thread-count</name>
       <value>50</value>
     </property>
     -->
     <property>
       <name>yarn.scheduler.minimum-allocation-mb</name>
       <value>0</value>
     </property>
     <property>
       <name>yarn.scheduler.maximum-allocation-mb</name>
       <value>512</value>
     </property>
     <!--
     <property>
       <name>yarn.scheduler.minimum-allocation-vcores</name>
       <value>1</value>
     </property>
     <property>
       <name>yarn.scheduler.maximum-allocation-vcores</name>
       <value>2</value>
     </property>
     <property>
       <name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>
       <value>1000</value>
     </property>
     -->
     <!--
     nodemanager
     -->
     <property>
       <name>yarn.nodemanager.resource.memory-mb</name>
       <value>1024</value>
     </property>
     <property>
       <name>yarn.nodemanager.vmem-pmem-ratio</name>
       <value>2.1</value>
     </property>
     <property>
       <name>yarn.nodemanager.local-dirs</name>
       <value>${hadoop.tmp.dir}/nm-local-dir</value>
     </property>
     <property>
       <name>yarn.nodemanager.log-dirs</name>
       <value>${yarn.log.dir}/userlogs</value>
     </property>
      <property>
         <name>yarn.nodemanager.log.retain-seconds</name>
       <value>10800</value>
     </property>
      <property>
       <name>yarn.nodemanager.remote-app-log-dir</name>
       <value>/logs</value>
     </property>
     <property>
       <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
       <value>logs</value>
     </property>
      <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
     </property>
     <!--
      History Serve
      -->
       <property>
       <name>yarn.log-aggregation.retain-seconds</name>
       <value>-1</value>
     </property>
     <property>
       <name>yarn.log-aggregation.retain-check-interval-seconds</name>
       <value>-1</value>
     </property>

     <!--
     <property>
       <name>yarn.nodemanager.resource.cpu-vcores</name>
       <value>4</value>
     </property>

     -->
   </configuration>

b.4. etc/hadoop/mapred-site.xml 配置如下：
含义参考：http://blog.csdn.net/u010719917/article/details/73917217

<configuration>
       <!--
        MapReduce Applications
        -->
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
       <property>
           <name>mapreduce.map.memory.mb</name>
           <value>1536</value>
       </property>
       <property>
           <name>mapreduce.map.java.opts</name>
           <value>-Xmx1024M</value>
       </property>
       <property>
           <name>mapreduce.reduce.memory.mb</name>
           <value>3072</value>
       </property>
       <property>
           <name>mapreduce.reduce.java.opts</name>
           <value>Xmx2560M</value>
       </property>
       <property>
           <name>mapreduce.task.io.sort.mb</name>
           <value>512</value>
       </property>
       <property>
           <name>mapreduce.task.io.sort.factor</name>
           <value>100</value>
       </property>
       <property>
           <name>mapreduce.reduce.shuffle.parallelcopies</name>
           <value>50</value>
       </property>
       <!--
       MapReduce JobHistory Server
       -->
       <property>
           <name>mapreduce.jobhistory.address</name>
           <value>centos128:10020</value>
       </property>
       <property>
           <name>mapreduce.jobhistory.webapp.address</name>
           <value>centos128:19888</value>
       </property>
       <property>
           <name>mapreduce.jobhistory.intermediate-done-dir</name>
           <value>/mr-history/tmp</value>
       </property>
       <property>
           <name>mapreduce.jobhistory.done-dir</name>
           <value>/mr-history/done</value>
       </property>
   </configuration>

b.5. etc/hadoop/slaves 如下

  [root@centos128 hadoop]# cat slaves
   centos129
   centos130
   centos131

如果设置SecondaryNameNode,需要在同路径下生成master文件，添加SecondaryNameNode所在hostname
HA中是不必要设置SecondaryNameNode

b.6，日志路径：

   [root@centos128 logs]# pwd
   /usr/src/hadoop/logs
   [root@centos128 logs]# ll

7.其它服务安装hadoop
将配置好的hadoop包，jdk,profile,host 复制到centos129,centos130

 cd /
   tar cvf hd.tar.gz /usr/src/hadoop/  /usr/src/jdk/ /etc/profile /etc/hosts        
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 **8、其它服务安装hadoop**

将配置好的hadoop包，jdk,profile,host 复制到centos129,centos130

 cd /
   tar cvf hd.tar.gz /usr/src/hadoop/  /usr/src/jdk/ /etc/profile /etc/hosts

c. hadoop脚本及页面
Hadoop Startup
要启动hdfs的HDFS，首先HDFS 格式化：

 [hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

开启HDFS的 NameNode：

[hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode

开启HDFS的 DataNode：

  [hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the HDFS processes can be started with a utility script. As hdfs:
如果etc/hadoop/slaves和ssh trusted都被配置，以上所有进程，可以用如下脚本开启

[hdfs]$ $HADOOP_PREFIX/sbin/start-dfs.sh

开启YARN的ResourceManager：

 [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

开启YARN的NodeManager：

  [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR start nodemanager

   Start a standalone WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:
   [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start proxyserver
   If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the YARN processes can be started with a utility script. As yarn:

直接开启yarn全部功能：

 [yarn]$ $HADOOP_PREFIX/sbin/start-yarn.sh

开启mapred的MapReduce JobHistory Server服务：

 [mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver

Hadoop Shutdown
关闭hdfs的namenode：

  [hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode

关闭hdfs的DataNode:

 [hdfs]$ $HADOOP_PREFIX/sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the HDFS processes may be stopped with a utility script. As hdfs:

[hdfs]$ $HADOOP_PREFIX/sbin/stop-dfs.sh

关闭yarn的ResourceManager：

 [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager
   关闭yarn的NodeManager：
   [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemons.sh --config $HADOOP_CONF_DIR stop nodemanager

If etc/hadoop/slaves and ssh trusted access is configured (see Single Node Setup), all of the YARN processes can be stopped with a utility script. As ya [yarn] HADOOP_PREFIX/sbin/stop-yarn.shStop the WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be run on each of them:

  [yarn]$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop proxyserver

关闭mapred的MapReduce JobHistory Serve：

  [mapred]$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR stop historyserver

Web Interfaces

 Once the Hadoop cluster is up and running check the web-ui of the components as described below:
   Daemon                         Web Interface                Notes
   NameNode                     http://nn_host:port/         Default HTTP port is 50070.
   ResourceManager              http://rm_host:port/       Default HTTP port is 8088.
   MapReduce JobHistory Server  http://jhs_host:port/      Default HTTP port is 19888.

   http://192.168.44.128:50070
   http://192.168.44.128:8088
   http://192.168.44.128:19888

四、NAMENODE HA设置
为了配置HA NameNodes需要添加些参数到hdfs-site.xml中，设置这些参数的顺序并不重要。但是dfs.nameservices和 dfs.ha.namenodes.[nameservice ID]的值在以后决定很重要。
你应该在接下来设置之前决定这些值

**1. dfs.nameservices**  ：作为新的nameservice 的逻辑名，可任意取。用在集群重要组件的HDFS路径。如果HDFS 是分布式，要用逗号分隔nameservice的列表。

   <property>                     
      <name>dfs.nameservices</name>
      <value>mycluster</value>     
    </property>
**2. dfs.ha.namenodes.[nameservice ID]** ： nameservice中每个namenode的唯一标识符。多个NameNode IDs，用逗号分隔,在集群中namenode用于识别所有的namenode.
例如：用mycluster作为nameservice id ,用nn1和nn2作为个别的namenode的id . 注意，目录每个nameservie最多只支持两个namenode

    <property>                               
      <name>dfs.ha.namenodes.mycluster</name>
      <value>nn1,nn2</value>                 
    </property>     
**3.dfs.namenode.rpc-address.[nameservice ID].[name node ID]** :每个namenode监听的rpc通信地址。
用前面的namenode id ,为每个namenode 进程设置rpc地址和端口。每个namenode设置一个配置。

 <property>                                                                 
      <name>dfs.namenode.rpc-address.mycluster.nn1</name>
      <value>centos128:8020</value>           
    </property>                                          
    <property>                                           
      <name>dfs.namenode.rpc-address.mycluster.nn2</name>
      <value>centos129:8020</value>           
    </property>   
**4.dfs.namenode.http-address.[nameservice ID].[name node ID]** ：每个namenoder 监听http的 地址。类似如上面。

 <property>                                            
      <name>dfs.namenode.http-address.mycluster.nn1</name>
      <value>centos128:50070</value>           
    </property>                                           
    <property>                                            
      <name>dfs.namenode.http-address.mycluster.nn2</name>
      <value>centos12:50070</value>           
    </property>  
**5.dfs.namenode.shared.edits.dir**  : JournalNodes 集群的uri,用于edits.log的读与写。 
这是JournalNodes的提供共享存储edits的地址，被Active nameNode 写入，Standby NameNode为了与Active nameNode对文件系统更改保持一致不断地读到最新更改。 
虽然你必须指定几个JournalNode地址，你应该仅仅只配置uri中的一个。URI的格式：`qjournal://*host1:port1*;*host2:port2*;*host3:port3*/*journalId*.` 
Journal ID是nameservice的唯一ID，允许单Journal Node的集为多分布式namesystems提供服务。
例如：这个集群的JournalNodes 运行机器"centos129","centos130","centos131"上并且nameservice ID是“mycluster”，JournalNode的默认端口8485。

   <property>                                                                                                
      <name>dfs.namenode.shared.edits.dir</name>                                                              
      <value>qjournal://centos129:8485;centos130:8485;centos131:8485/mycluster</value>
    </property> 
**6.dfs.client.failover.proxy.provider.[nameservice ID]** : HDFS clients用这java class连系   Active NameNode 
DFS Client连接的Java class 用于决定哪个NameNode 是当前的active 和 哪个NameNode就当前服务client requests。这个功能由ConfiguredFailoverProxyProvider 
和RequestHedgingProxyProvider 实现。在毫无头绪的情况，并行介入决定哪个是active NameNode 并处理随后的请求。

  <property>                                                                                
      <name>dfs.client.failover.proxy.provider.mycluster</name>                               
      <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>       
**7.dfs.ha.fencing.methods** ：scripts或java class 用于fence the Active NameNode 在failover期间。
在任何时间 NameNode 只有一个Active 状态。当用Quorum Journal Manager时， Importantly, when using the Quorum Journal Manager, only one NameNode will ever be allowed to write to the JournalNodes, 
so there is no potential for corrupting the file system metadata from a split-brain scenario。然而，当failover时，早先的Active NameNode处理client读请求，
当正在努力往JournalNodes写时可能会导致读请求超时直到NameNode关闭，可能会导致split-brain。正因如此，当使用Quorum Journal Manager时配置些fencing methods。
然而在the fencing mechanisms失败时，为了提升系统的可用性被建议配置成在最近的fencing method列表担保能返回成功的fencing方法。

在failover期间fence方法被用，fencing方法被配置成carriage-return-separated 列表，会按照顺序尝试直到fencing返回成功。hadoop两个fencing方法：ssh和sshfence

sshfence :ssh 登录上Active NameNode 并kill 进程。
sshfence 选SSHes 登录目标端并用fuser kill进程监听service'tcp端口。为了让fencing option工作，能够不需要密码ssh到目标端。因此、必须配置 dfs.ha.fencing.ssh.private-key-files 
用逗号分隔SSH private key file列表。

 <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>

    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/root/.ssh/id_rsa</value>
    </property>
用户的不同private-key-files文件的路径不同。

**8.fs.defaultFS** ：  the Hadoop FS client使用的默认路径的前缀
为了使用新的HA-enabled logical URI，可能要配置默认的路径为Hadoop clients。“mycluster” 作为 nameservice ID，也作为 HDFS path个部份。在core-site.xml 文件
如下配置：

 <property>                       
      <name>fs.defaultFS</name>      
      <value>hdfs://mycluster</value>
    </property>       
**9.dfs.journalnode.edits.dir**   ：  JournalNode daemon 存储本状态的文件的路径
在JournalNode机器上路径是绝对的，用于存放JNs会的edits和other local state。可以用单路径，也可冗余多路径。

 <property>                                        
      <name>dfs.journalnode.edits.dir</name>         
      <value>/journal/node/local/data</value>
    </property>   
**10. Automatic Failover** 
当在配置 Automatic Failover 之前必段先关闭cluster，不在cluster运行状态下把手动切换成自动。

**10.1Configuring automatic failover**   需要配置两个参数dfs.ha.automatic-failover.enabled 和 ha.zookeeper.quorum。
**在hdfs-site.xml里添加：**

   <property>                                      
       <name>dfs.ha.automatic-failover.enabled</name>                    
       <value>true</value>                           
     </property> 
 **在core-site.xml添加：**

    <property>                                                                                                     
        <name>ha.zookeeper.quorum</name>                                             
        <value>centos128:2181,centos129:2181,centos130:2181</value>
     </property>
 **11. Initializing HA state in ZooKeeper**  
 [hdfs]$ $HADOOP_PREFIX/bin/hdfs zkfc -formatZK

 Deployment details                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
 After all of the necessary configuration options have been set, you must start the JournalNode daemons on the set of machines where they will run. 
 This can be done by running the command “hadoop-daemon.sh start journalnode” and waiting for the daemon to start on each of the relevant machines.                                                                                                                                                                                                   
 Once the JournalNodes have been started, one must initially synchronize the two HA NameNodes’ on-disk metadata.                                                                                                                                                                                                                                                                                                                                                                                          
     If you are setting up a fresh HDFS cluster, you should first run the format command (hdfs namenode -format) on one of NameNodes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
     If you have already formatted the NameNode, or are converting a non-HA-enabled cluster to be HA-enabled, you should now copy over the contents 
     of your NameNode metadata directories to the other, unformatted NameNode by running the command “hdfs namenode -bootstrapStandby” on the 
     unformatted NameNode. Running this command will also ensure that the JournalNodes (as configured by dfs.namenode.shared.edits.dir) contain 
     sufficient edits transactions to be able to start both NameNodes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
     If you are converting a non-HA NameNode to be HA, you should run the command “hdfs namenode -initializeSharedEdits”, which will initialize the 
     JournalNodes with the edits data from the local NameNode edits directories.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
 At this point you may start both of your HA NameNodes as you normally would start a NameNode.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
 You can visit each of the NameNodes’ web pages separately by browsing to their configured HTTP addresses. You should notice that next to the configured 
 address will be the HA state of the NameNode (either “standby” or “active”.) Whenever an HA NameNode starts, it is initially in the Standby state.                                                                                                                                                                                           

 **11.1 先在jouralnode节点运行hadoop-daemon.sh start journalnod。11.2 未format节点运行hdfs namenode -bootstrapStandby**

各配置文件core-site.xml：

<configuration>
<!--
  <property>
        <name>fs.defaultFS</name>
        <value>hdfs://centos128:9000</value>
        <description>NameNode URI </description>
  </property>
-->
  <property>
        <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
  </property>

  <property>
         <name>hadoop.tmp.dir</name>
         <value>/data/hadoop/tmp</value>
         <description></description>
  </property>
  <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
        <description>Size of read/write buffer used in SequenceFiles.</description>
  </property>
  <property> 
        <name>fs.trash.interval</name> 
        <value>10080</value>
  </property>
  <property>
         <name>fs.checkpoint.period</name>
         <value>3600</value>
         <description>The number of seconds between two periodic checkpoints.
         </description>
  </property>
  <property>
         <name>ha.zookeeper.quorum</name>
         <value>centos128:2181,centos129:2181,centos130:2181</value>
   </property>
</configuration>

配置文件hdfs-site.xml：

  <configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/data/hadoop/name</value>
    </property>
    <property>
        <name>dfs.blocksize</name>
        <value>268435456</value>
    </property>
    <property>
        <name>dfs.namenode.handler.count</name>
        <value>100</value>
        <description>More NameNode server threads to handle RPCs from large number of DataNodes. </description>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/Data1,/Data2</value>
    </property>
<!--
    <property>
       <name>dfs.http.address</name>
       <value>centos128:50070</value>
       <description>
       The address and the base port where the dfs namenode web ui will listen on.
       If the port is 0 then the server will start on a free port.
       </description>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>centos129:50090</value>
    </property>
-->
    <property>
       <name>dfs.nameservices</name>
       <value>mycluster</value>
    </property> 
    <property>
       <name>dfs.ha.namenodes.mycluster</name>
       <value>nn1,nn2</value>
    </property> 
    <property>
       <name>dfs.namenode.rpc-address.mycluster.nn1</name>
       <value>centos128:8020</value>
   </property>
   <property>
       <name>dfs.namenode.rpc-address.mycluster.nn2</name>
       <value>centos129:8020</value>
   </property>    
   <property>
     <name>dfs.namenode.http-address.mycluster.nn1</name>
     <value>centos128:50070</value>
   </property>
   <property>
     <name>dfs.namenode.http-address.mycluster.nn2</name>
     <value>centos129:50070</value>
   </property> 
   <property>
     <name>dfs.namenode.shared.edits.dir</name>
     <value>qjournal://centos129:8485;centos130:8485;centos131:8485/mycluster</value>
   </property>
   <property>
     <name>dfs.client.failover.proxy.provider.mycluster</name>
     <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
   </property>
   <property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
    </property>
    <property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/root/.ssh/id_rsa</value>
    </property>
    <property>
       <name>dfs.journalnode.edits.dir</name>
     <value>/journalnode/local/data</value>
    </property>
    <property>
      <name>dfs.ha.automatic-failover.enabled</name>
      <value>true</value>
    </property>
</configuration>

配置文件mapred-site.xml：

<configuration>
    <!-- 
     MapReduce Applications 
     -->
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.map.memory.mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>mapreduce.job.ubertask.enable</name>
        <value>true</value>
    </property>
    <property>
        <name>mapreduce.map.java.opts</name>
        <value>-Xmx800m</value>
    </property>
    <property>
        <name>mapreduce.reduce.memory.mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>mapreduce.reduce.java.opts</name>
        <value>Xmx800m</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.mb</name>
        <value>100</value>
    </property>
    <property>
        <name>mapreduce.task.io.sort.factor</name>
        <value>10</value>
    </property>
    <property>
        <name>mapreduce.reduce.shuffle.parallelcopies</name>
        <value>5</value>
    </property>
    <!--
    MapReduce JobHistory Server
    -->
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>centos128:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>centos128:19888</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/mr-history/tmp</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.done-dir</name>
        <value>/mr-history/done</value>
    </property>
 <property>
        <name>mapreduce.job.reduce.slowstart.completedmaps</name>
        <value>0.6</value>
  </property> 

  <property>
        <name>mapreduce.map.output.compress</name>
        <value>true</value>
  </property> 
  <property>
     <name>mapreduce.job.jvm.numtasks</name>
     <value>-1</value>
  </property>
  <property>  
           <name>mapreduce.application.classpath</name>  
           <value>  
                /usr/src/hadoop/etc/hadoop,  
                /usr/src/hadoop/share/hadoop/common/*,  
                /usr/src/hadoop/share/hadoop/common/lib/*,  
                /usr/src/hadoop/share/hadoop/hdfs/*,  
                /usr/src/hadoop/share/hadoop/hdfs/lib/*,  
                /usr/src/hadoop/share/hadoop/mapreduce/*,  
                /usr/src/hadoop/share/hadoop/mapreduce/lib/*,  
                /usr/src/hadoop/share/hadoop/yarn/*,  
                /usr/src/hadoop/share/hadoop/yarn/lib/*  
           </value>  
   </property>  
</configuration>

配置文件yarn-site.xml:

<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>centos128</value>
</property>
<!--
<property>
  <name>yarn.web-proxy.address</name>
  <value>centos128:8888</value>
</property>
-->
<property>
  <name>yarn.log-aggregation-enable</name>
  <value>true</value>
</property>
<property>
  <name>yarn.log-aggregation.retain-seconds</name>
  <value>604800</value>
</property>
<property>
  <name>yarn.log.dir</name>
  <value>/usr/src/hadoop/logs</value>
</property>
<!-- 根据自己硬件配置可调整，建议最低就我这配置 -->
<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>1600</value>
</property>
<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.maximum-allocation-mb</name>
  <value>1600</value>
</property>
<property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>1</value>
</property>
<property>
  <name>yarn.scheduler.minimum-allocation-mb</name>
  <value>256</value>
</property>
<property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
</property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
    <value>50</value>
</property>
<property>
    <name>yarn.resourcemanager.scheduler.client.thread-count</name>
    <value>50</value>
</property>
<property>
    <name>yarn.resourcemanager.nodemanagers.heartbeat-interval-ms</name>
    <value>1000</value>
</property>
<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>${yarn.log.dir}/userlogs</value>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
</property>
<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>${hadoop.tmp.dir}/nm-local-dir</value>
</property>
<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>${yarn.log.dir}/userlogs</value>
</property>
<property>
      <name>yarn.nodemanager.log.retain-seconds</name>
    <value>10800</value>
</property>
<property>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/tmp/logs</value>
</property>
<property>
    <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
    <value>logs</value>
</property>
<property>
    <name>yarn.nodemanager.log-container-debug-info.enabled</name>
    <value>true</value>
</property>
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>-1</value>
</property>
<property>
    <name>yarn.log-aggregation-status.time-out.ms</name>
    <value>600000</value>
</property>
<property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
    <value>-1</value>
</property>
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<property>
    <name>yarn.log.server.url</name>
    <value>http://centos128:19888/jobhistory/logs</value>
</property>
</configuration>

六、HA操作
启动
1). zkServer.sh start #开zookeeper集群节点

首先分别启动centos129,centos130,centos131上的JournalNode

  [root@centos129 ~]# cd /usr/src/hadoop/sbin/
   [root@centos129 sbin]# ./hadoop-daemon.sh start journalnode
   [root@centos130 sbin]# ./hadoop-daemon.sh start journalnode
   [root@centos131 sbin]# ./hadoop-daemon.sh start journalnode

2). 将第一个NameNode上初始化的元数据文件拷贝到第二个NameNode上,要先启动第一个NameNode

 [root@centos128 ]# sbin/hadoop-daemon.sh start namenode
   [root@centos129 ]# bin/hdfs namenode -bootstrapStandby

4)在journal Node上

   hdfs namenode -initializeSharedEdits

5).在第一个NameNode上关闭所有hdfs服务

   [root@centos128]# sbin/stop-dfs.sh

6). 初始化zookeeper

   [root@centos128]# bin/hdfs zkfc -formatZK

7).启动HDFS

   [root@centos128]# sbin/start-dfs.sh

**HA部分操作：
zkServer.sh start #开zookeeper集群节点**
#开启journal Node节点:

   centos129: hadoop-daemon.sh start journalnode
   centos130: hadoop-daemon.sh start journalnode        
   centos131: hadoop-daemon.sh start journalnode    
  initially synchronize the two HA NameNodes’ on-disk metadata:
   copy copy over the contents of your NameNode metadata directories to the other
   hadoop-daemon.sh start namenode        #开启active namenode
   hdfs namenode -bootstrapStandby        #sync NN 

   Initialize the JournalNodes with the edits data from the local NameNode edits directories:
   hdfs namenode -initializeSharedEdits (同步gap from edit log of journal node )

stop-dfs.sh
hdfs zkfc -formatZK
start-dfs.sh

H A D O O P P R E F I X / s b i n / h a d o o p − d a e m o n . s h − − s c r i p t HADOOP_PREFIX/bin/hdfs start zkfc

hadoop-daemon.sh start namenode

Administrative commands:
Usage: haadmin
    [-transitionToActive <serviceId>]
    [-transitionToStandby <serviceId>]
    [-failover [--forcefence] [--forceactive] <serviceId> <serviceId>]
    [-getServiceState <serviceId>]
    [-checkHealth <serviceId>]
    [-help <command>]

journalnode上的问题：
出理WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: There appears to be a gap in the edit log.如下操作：
如果是首次启动，在namenode1上运行format命令
bin/hadoop namenode -format
如果是非首次启动，则在namenode1上运行以下命令
hdfs namenode -initializeSharedEdits 

原因是namenode元数据被破坏了，需要修复
输入命令：hadoop namenode -recover

手动切换ACTIVE与STANDBY的NN:

  11.active
      hdfs haadmin -transitionToActive nn2  --forcemanual
      hdfs haadmin -transitionToStandby nn1  --forcemanual
      hdfs haadmin -getServiceState nn1
      hdfs haadmin -getServiceState nn2

      failover:
      没操作成功

hdfs haadmin -transitionToStandby nn1  --forcemanual
hdfs haadmin -transitionToStandby nn2  --forcemanual
hdfs haadmin -transitionToActive nn1  --forcemanual
hdfs haadmin -transitionToActive nn2  --forcemanual

猜你喜欢

转载自blog.csdn.net/u010719917/article/details/79789210
今日推荐
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
周排行
计算机组成与设计（七）—— 除法器
Integer Approximation(分治+枚举)
大话数据库索引
windows10系统JDK的配置及下载地址
mysql实现秒值转换中原六仔平台搭建
Codeforces Round #556 (Div. 1)
百练1064 网线主管
Codeforces 995F Cowmpany Cowmpensation
子集生成之增量构造法，位向量法，二进制法
ERROR: cmd.exe failed with args /c "/APK\gradle\rungradle.bat...
每日归档
更多
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022