运维项目实训—Hadoop分布式文件系统HDFS

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/Hannah_zh/article/details/81171453

一、Hadoop分布式文件系统HDFS—单数据存储节点

1.安装、部署Hadoop

具体步骤：https://blog.csdn.net/Hannah_zh/article/details/81169416

2.修改配置文件

<1>制定Namenode的地址

[root@server1 ~]# su - hadoop
[hadoop@server1 ~]$ cd hadoop/etc/hadoop/
[hadoop@server1 hadoop]$ vim core-site.xml 
<configuration>
    <property>
            <name>fs.defaultFS</name>
                    <value>hdfs://172.25.51.1:9000</value>
    </property>
</configuration>

<2>指定Datanode地址以及hdfs保存数据的副本数量

[hadoop@server1 hadoop]$ vim slaves 
172.25.51.1
[hadoop@server1 hadoop]$ vim hdfs-site.xml 
<configuration>
      <property>
             <name>dfs.replication</name>
                  <value>1</value>     ##hdfs保存数据的副本数量为1
      </property>
</configuration>

3.设置ssh免密登陆（前提：安装ssh服务）

[hadoop@server1 ~]$ ssh-keygen
[hadoop@server1 ~]$ cd .ssh/
[hadoop@server1 .ssh]$ ls
id_rsa  id_rsa.pub
[hadoop@server1 .ssh]$ cp id_rsa.pub authorized_keys

示图：验证免密登陆

4.格式化元数据节点（Namenode）

[hadoop@server1 ~]$ cd hadoop
[hadoop@server1 hadoop]$ pwd
/home/hadoop/hadoop
[hadoop@server1 hadoop]$ bin/hdfs namenode -format
[hadoop@server1 hadoop]$ ls /tmp/
hadoop-hadoop  hsperfdata_hadoop

示图：格式化后生成的文件

5.开启dfs

[hadoop@server1 hadoop]$ sbin/start-dfs.sh

6.配置环境变量

注意：配置完环境变量后需重新登陆，才可生效

[hadoop@server1 ~]$ vim .bash_profile 
 10 PATH=$PATH:$HOME/bin:~/java/bin
[hadoop@server1 ~]$ logout    

7.jps命令查看java进程

[root@server1 ~]# su - hadoop
[hadoop@server1 ~]$ jps
2082 DataNode              ##数据节点
2239 SecondaryNameNode     ##从元数据节点
1989 NameNode              ##元数据节点
2941 Jps

8.创建目录并上传到input

[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir /user
[hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir /user/hadoop
[hadoop@server1 hadoop]$ bin/hdfs dfs -put input/   ##上传到input
[hadoop@server1 hadoop]$ bin/hdfs dfs -ls

9.执行hadoop中自带的wordcount程序

[hadoop@server1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount input output

示图：显示如下结果就说明运行成功

二、Hadoop分布式文件系统HDFS—多数据存储节点

实验环境：Redhat6.5版本

Namenode：内存1G

server1：172.25.51.1

Datanode：内存1G

server2：172.25.51.2
server3：172.25.51.3

具体部署过程如下：

对于Namenode节点，删除上一实验部署中的操作并停止dfs

[hadoop@server1 ~]$ cd hadoop
[hadoop@server1 hadoop]$ rm -fr input/ output/
[hadoop@server1 hadoop]$ bin/hdfs dfs -get output
[hadoop@server1 hadoop]$ rm -fr output/
[hadoop@server1 hadoop]$ sbin/stop-dfs.sh 

1.Namenode

[root@server1 ~]# yum install -y nfs-utils
[root@server1 ~]# /etc/init.d/rpcbind start   ##开启nfs服务前。必须开启此服务
[root@server1 ~]# vim /etc/exports 
/home/hadoop      *(rw,anonuid=800,anongid=800)
[root@server1 ~]# /etc/init.d/nfs start
[root@server1 ~]# exportfs -v
[root@server1 ~]# exportfs -rv

2.Datanode（172.25.51.3和172.25.51.2操作一致）

[root@server2 ~]# useradd -u 800 hadoop
[root@server2 ~]# id hadoop
uid=800(hadoop) gid=800(hadoop) groups=800(hadoop)
[root@server2 ~]# yum install -y nfs-utils
[root@server2 ~]# /etc/init.d/rpcbind start
[root@server2 ~]# showmount -e 172.25.51.1
[root@server2 ~]# mount 172.25.51.1:/home/hadoop/ /home/hadoop/
[root@server2 ~]# df
Filesystem                   1K-blocks    Used Available Use% Mounted on
/dev/mapper/VolGroup-lv_root  19134332  925180  17237172   6% /
tmpfs                           510188       0    510188   0% /dev/shm
/dev/vda1                       495844   33451    436793   8% /boot
172.25.51.1:/home/hadoop/     19134336 1962240  16200192  11% /home/hadoop

3.配置hdfs

[root@server1 ~]# su - hadoop
[hadoop@server1 ~]$ cd hadoop/etc/hadoop/
[hadoop@server1 hadoop]$ vim slaves     ##设定Datanode
172.25.51.2
172.25.51.3
[hadoop@server1 hadoop]$ vim hdfs-site.xml 
 19 <configuration>
 20       <property>
 21              <name>dfs.replication</name>
 22                   <value>2</value>     ##复制两份文件
 23       </property>
 24 </configuration>
[hadoop@server1 hadoop]$ cd /tmp/
[hadoop@server1 tmp]$ rm -fr *

4.测试ssh免密服务

[hadoop@server1 tmp]$ ssh server2
[hadoop@server2 ~]$ logout
[hadoop@server1 tmp]$ ssh server3
[hadoop@server3 ~]$ logout
[hadoop@server1 tmp]$ ssh 172.25.120.2
[hadoop@server2 ~]$ logout
[hadoop@server1 tmp]$ ssh 172.25.120.3
[hadoop@server2 ~]$ logout

5.重新格式化

[hadoop@server1 ~]$ cd hadoop
[hadoop@server1 hadoop]$ bin/hdfs namenode -format
[hadoop@server1 hadoop]$ ls /tmp/
hadoop-hadoop  hsperfdata_hadoop

6.启动dfs

[hadoop@server1 hadoop]$ sbin/start-dfs.sh

示图：jps查看java进程