【分布式】hadoop安装篇（7）Hadoop 3的伪分布模式安装

准备

$ sudo apt-get install ssh
$ sudo apt-get install pdsh

我们还需要设置pdsh的rcmd，执行下面内容。或创建/etc/pdsh/rcmd_default，里面填入ssh

echo "ssh" > /etc/pdsh/rcmd_default

如果这步骤不设置，我们在启动hdfs时，可能会报告

gsta@gsta005:~/wei/hadoop/hadoop-3.1.0/sbin$ start-dfs.sh
Starting namenodes on [gsta005]
pdsh@gsta005: gsta005: rcmd: socket: Permission denied
Starting datanodes
pdsh@gsta005: localhost: rcmd: socket: Permission denied
Starting secondary namenodes [gsta005]
pdsh@gsta005: gsta005: rcmd: socket: Permission denied

设置免密码登录的ssh

$ ssh-keygen -t rsa -P  -f ~/.ssh/id_rsa
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
验证
$ ssh localhost

配置和启动hdfs

etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://191.8.2.45:9000</value>
    </property>
    <!-- 因为我在一台机器内同时安装了2.9.0版本和3.1.0版本，不采用缺省的/tmp目录，采用指定目录方式-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/gsta/wei/hadoop/hadoop-3.1.0/tmp</value>
    </property>
</configuration>

etc/hadoop/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

格式化hdfs

$ hdfs namenode -format

启动hdfs

$ start-hdfs.sh
设置相关的文件夹 /user,/user/<username>，命令同v2版本

开启web监控

http://191.8.2.45:9870

如果启动的过程中报错：

gsta@gsta005:~/wei/hadoop$ start-dfs.sh
Starting namenodes on [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: gsta005: ssh exited with exit code 1
Starting datanodes
localhost: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: localhost: ssh exited with exit code 1
Starting secondary namenodes [gsta005]
gsta005: ERROR: JAVA_HOME is not set and could not be found.
pdsh@gsta005: gsta005: ssh exited with exit code 1

则在hadoop-env.sh中指定JAVA_HOME

export JAVA_HOME=/home/wei/jdk1.8.0_66

配置和启动yarn

etc/hadoop/mapred-site.xml：

扫描二维码关注公众号，回复： 2233069 查看本文章

<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>

        <property>
                <name>mapreduce.application.classpath</name>
                <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
        </property>
</configuration>

etc/hadoop/yarn-site.xml：

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

启动yarn

start-yarn.sh

web监控查看

http://191.8.2.45:8088/