hadoop-3.0.0集群搭建

下载hadoop包

wget -c http://ftp.jaist.ac.jp/pub/apache/hadoop/common/hadoop-3.0.0/hadoop-3.0.0.tar.gz

解压

tar -zxvf hadoop-3.0.0.tar.gz -C /usr/java/

配置

配置环境变量，打开vim /etc/profile文件

export HADOOP_HOME=/usr/java/hadoop-3.0.0
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

使之立即生效

source /etc/profile

配置/etc/hosts

192.168.56.101 master
192.168.56.102 slave1
192.168.56.103 slave2

关闭防火墙

systemctl stop firewalld

配置vim core-site.xml文件（etc/hadoop目录下），新增如下配置

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://master:9000</value>
  </property>
  <property>
    <name>hadoop.proxyuser.wujinlei.groups</name>
    <value>*</value>
  </property>
  <property>
    <name>hadoop.proxyuser.wujinlei.hosts</name>
    <value>*</value>
  </property>
</configuration>

配置vim hdfs-site.xml文件，新增如下配置。
- master机器上开放端口9870，供外部访问web页面（NameNode HTTP UI），查看集群情况。
- slave机器上开放端口9864，供外部访问web页面（DataNode HTTP UI）。

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>2</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/home/wujinlei/hadoop/dfs/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/home/wujinlei/hadoop/dfs/data</value>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>master:9870</value>
  </property>
  <property>
    <name>dfs.datanode.http.address</name>
    <value>master:9864</value>
  </property>
</configuration>

配置vim yarn-site.xml，新增如下配置，master机器上开放端口8088，供外部访问web页面，查看集群任务调度情况

<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:8088</value>
  </property>
  <property>
    <name>yarn.application.classpath</name>
    <value>
        /usr/java/hadoop-3.0.0/etc/hadoop,
        /usr/java/hadoop-3.0.0/share/hadoop/common/lib/*,
        /usr/java/hadoop-3.0.0/share/hadoop/common/*,
        /usr/java/hadoop-3.0.0/share/hadoop/hdfs,
        /usr/java/hadoop-3.0.0/share/hadoop/hdfs/lib/*,
        /usr/java/hadoop-3.0.0/share/hadoop/hdfs/*,
        /usr/java/hadoop-3.0.0/share/hadoop/mapreduce/*,
        /usr/java/hadoop-3.0.0/share/hadoop/yarn,
        /usr/java/hadoop-3.0.0/share/hadoop/yarn/lib/*,
        /usr/java/hadoop-3.0.0/share/hadoop/yarn/*,
        /usr/java/jdk1.8.0_45/lib/tools.jar
    </value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
</configuration>

配置vim mapred-site.xml，新增如下配置

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>0.0.0.0:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>0.0.0.0:19888</value>
  </property>
</configuration>

在etc/hadoop下编辑works文件，内容如下

slave1
slave2

启动集群
- 将上述配置好的hadoop文件复制到另外两个节点slave1和slave2
- 启动前先格式化，命令hdfs namenode -format
- 单独启动
  - 启动dfs，命令start-dfs.sh
  - 启动yarn，命令start-yarn.sh
- 全部启动
  - 命令start-all.sh
访问web页面
http://192.168.56.101:8088
http://192.168.56.101:9870
启动历史服务

mapred historyserver

此服务用来访问历史任务详情

hadoop-3.0.0集群搭建

hadoop-3.0.0集群搭建

猜你喜欢