3，数据挖掘环境搭建-Hadoop集群搭建与测试

安装hadoop-2.6.5.tar.gz
hdp10 namenode/datanode
hdp11 datanode
hdp12 datanode

0 安装前准备
修改hosts，修改主机名，关闭防火墙，关闭selinux,安装ssh，配置免密码，安装JDK

1,添加hadoop用户名
useradd hadoop
passwd hadoop
vi /etc/sudoers

root       ALL=(ALL)   ALL
hadoop ALL=(ALL)  ALL

2,修改/home/hadoop/apps/hadoop-2.6.5/etc/hadoop下的配置文件

2.1，修改hadoop-env.sh
vi hadoop-env.sh

  export JAVA_HOME=/usr/java/jdk1.8.0_144

2.2 修改core-site.xml
vi core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://hdp10:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hdpdata</value>
    </property>
</configuration>

2.3，修改hdfs-site.xml
vi hdfs-site.xml

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value> 
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/hdpdata</value>
    </property>
  <property>
    <name>dfs.secondary.http.address</name>
    <value>hdp10:50070</value>
  </property>
  <property>
   <name>dfs.http.address</name>
   <value>hdp10:50090</value>
   </property>
  <property>
   <name>dfs.datanode.data.dir</name>
   <value>/home/hadoop/datanode</value>
   </property>
  <property>
   <name>dfs.namenode.data.dir</name>
   <value>/home/hadoop/namenode</value>
   </property>
  <property>
   <name>dfs.permissions</name>
   <value>false</value>
   </property>
</configuration>

2.4 修改mapred-site.xml
vi mapred-site.xml

<configuration>
 <property>
  <name>mapreduce.framework.name</name>
   <value>yarn</value>
  </property>
</configuration>

2.5，修改yarn-site.xml
vi yarn-site.xml

<configuration>
 <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hdp10</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
</configuration>

3，将hadoop文件添加到系统环境变量
vi /etc/profile

export HADOOP_HOME=/home/hadoop/apps/hadoop-2.6.5
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

4，修改slaves，设置datanode与nodemanager
vi /home/hadoop/apps/hadoop-2.6.5/etc/hadoop/slaves
hdp10
hdp11
hdp12

5，将/etc/profile和hadoop文件夹拷贝至hdp11,hdp12相应的位置
scp /etc/profile root@hdp11:/etc
scp /etc/profile root@hdp12:/etc
scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp11:/home/hadoop/apps
scp -r /home/hadoop/apps/hadoop-2.6.5 root@hdp12:/home/hadoop/apps

chown -R hadoop:hadoop /home/hadoop/apps/hadoop-2.6.5

6，去除HDFS WARN util.NativeCodeLoader
vi /home/hadoop/apps/hadoop-2.5.2/etc/hadoop/log4j.properties
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

7，格式化hadoop文件系统
hdfs namenode -format

8，启动hadoop
cd /home/hadoop/apps/hadoop-2.6.5/sbin
      start-dfs.sh，启动HDFS
      stop-dfs.sh，关闭HDFS
      start-yarn.sh，启动YARN
      stop-yarn.sh，关闭YARN

9，查看启动状态
jps

10，范例
vi 1.txt

hello world hello count
hello class
hello rose
hello jerry
hello class
hello rose

hadoop fs -mkdir -p /wordcount/input
hadoop fs -put 1.txt /wordcount/input
cd /home/hadoop/apps/hadoop-2.6.5/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /wordcount/input /wordcount/output0

3，数据挖掘环境搭建-Hadoop集群搭建与测试

猜你喜欢