hadoop 全分布式安装

一、前提

本教程使⽤hadoop3.1.1版本
   ⼀共四台机器作为集群，centos128作为master，centos129,centos130,centos131作为worker。（主从节点）
   安装Java8并设置JAVA_HOME环境变量【四台】（之后发教程）
   安装SSH【四台】 (默认安装好)

二、配置hosts⽂件【四台】--- 主机名与IP对应

vi /etc/hosts 中进行配置 (前篇文章很清晰)

添加内容:

192.168.52.128 centos128
192.168.52.129 centos129
192.168.52.130 centos130
192.168.52.131 centos131

三、免密码登录⾃⾝【四台]

ssh-keygen -t rsa //在自己主机上生成公钥, 一路回车 (在centos130)
ssh-copy-id -i ~/.ssh/id_rsa.pub localhost

四、centos1免密码登录其他机器（master免录worker）【单台，只需在centos1上执]

// 在配置免密的主机上(centos128)执行该代码
ssh-copy-id -i ~/.ssh/id_rsa.pub 目标主机(centos129)
ssh-copy-id -i ~/.ssh/id_rsa.pub 目标主机(centos130)  
ssh-copy-id -i ~/.ssh/id_rsa.pub 目标主机(centos131)

五、 hadoop3+配置⽂件

共需要配置 /usr/local/hadoop/hadoop-3.1.1/etc/hadoop 下的六个个⽂件，分别是
hadoop-env.sh、core-site.xml、hdfs-site.xml、yarn-site.xml、mapred-site.xml、workers

1. vi hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root

2. core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://centos128:9820</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/hadoopcluster</value>
        </property>
</configuration>

3. hdfs-size.xml

<configuration>

<property>
        <name>dfs.replication</name>
        <value>3</value>
</property>
<property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>centos129:9868</value>
</property>

</configuration>

4. yarn-site.xml

<configuration>
        <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>centos128</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>

5. mapred-site.xml

<configuration>
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>
</configuration>

6. workers

centos129
centos130
centos131

六、复制Hadoop⽂件到其他集群 (将配置好的hadoop⽂件复制到其他同样位置的机器上)

[root@centos128 etc]# scp -r hadoop centos129:`pwd` //注意自己的目录

[root@centos128 etc]# scp -r hadoop centos130:`pwd`

[root@centos128 etc]# scp -r hadoop centos131:`pwd`

七、集群格式化、启动、关闭

1. 第⼀次启动前先格式化集群【只在centos1上执⾏】 : hdfs namenode -format

2. 启动集群 :

start-dfs.sh

start-yarn.sh

3. 关门集群【只在centos1上执⾏】 :

stop-dfs.sh
stop-yarn.sh

4. 记得关闭防火墙 ( 主节点要与从节点进行交互)

关闭： systemctl stop firewalld
开机禁用： systemctl disable firewalld

OK啦~ 为防忘记特写此文，共勉