Hadoop集群的搭建(虚拟机准备,JDK和Hadoop安装,Hadoop目录结构)

版权声明:Please make the source marked https://blog.csdn.net/qq_31807385/article/details/84069561

目录

虚拟机准备

JDK和Hadoop安装

Hadoop的目录结构:


虚拟机准备

环境:一台刚装好的CentOS,操作如下:

[root@hadoop100 ~]# ifconfig

获取当前主机的ip地址,然后使用shell登录,使用shell操作,更加便捷。在我们使用shell登录之后,我们可以先查看一下相关的关于网络方面的信息:

# 查看一下关于ip的信息
[root@hadoop100 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 

DEVICE=eth0
HWADDR=00:0C:29:22:E8:45
TYPE=Ethernet
UUID=bd07f61f-25b4-4899-810d-91046e2b145a
ONBOOT=no
NM_CONTROLLED=yes
BOOTPROTO=dhcp


#查看一个关于网卡的配置信息:
[root@hadoop100 ~]# vim /etc/udev/rules.d/70-persistent-net.rules 

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:22:e8:45", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:66:02:e5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
~                 

接下来我们开始配置

#1,关闭防火墙
[root@hadoop100 ~]# chkconfig iptables off
[root@hadoop100 ~]# chkconfig --list iptables
iptables       	0:关闭	1:关闭	2:关闭	3:关闭	4:关闭	5:关闭	6:关闭

#2,设置静态ip
[root@hadoop100 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
NAME="eth0"
IPADDR=192.168.1.101
PREFIX=24
GATEWAY=192.168.1.2
DNS=192.168.1.2

#3,修改主机名
[root@hadoop100 ~]# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop101

#4,修改网卡的信息,删0改1
[root@hadoop100 ~]# vim /etc/udev/rules.d/70-persistent-net.rules 

# PCI device 0x8086:0x100f (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:66:02:e5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
~              


#5,配置/etc/hosts文件,尽量配置的多一点,可以备用,另外在Windows下的hosts文件需要配置一下(此处不演示)。
[root@hadoop100 ~]# vim /etc/hosts

192.168.1.100 hadoop100

192.168.1.101 hadoop101
192.168.1.102 hadoop102
192.168.1.103 hadoop103
192.168.1.104 hadoop104
192.168.1.105 hadoop105
192.168.1.106 hadoop106
192.168.1.107 hadoop107
192.168.1.108 hadoop108
192.168.1.109 hadoop109

192.168.1.110 hadoop110
192.168.1.111 hadoop111
192.168.1.112 hadoop112
192.168.1.113 hadoop113
192.168.1.114 hadoop114

#6,创建一个一般用户,并设置密码
[root@hadoop100 ~]# useradd isea
[root@hadoop100 ~]# passwd isea

#7,配置该用户的root权限,91 shift + g 直接定位到改行
[root@hadoop100 ~]# vim /etc/sudoers
isea    ALL=(ALL)       NOPASSWD:ALL

#8,创建opt目录下创建module文件夹和software文件夹,并将所有权赋给一般用户
[root@hadoop100 ~]# mkdir /opt/module /opt/software
[root@hadoop100 ~]# chown isea:isea /opt/software/ /opt/module/
[root@hadoop100 ~]# ll /opt/
总用量 12
drwxr-xr-x. 2 isea isea 4096 11月 14 17:12 module
drwxr-xr-x. 2 root root 4096 3月  26 2015 rh
drwxr-xr-x. 2 isea isea 4096 11月 14 17:12 software

#9,重新启动,普通用户登录,查看机器的ip地址是否正确。
[isea@hadoop101 opt]$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0C:29:82:89:7B  
          inet addr:192.168.1.101  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe82:897b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:549 errors:0 dropped:0 overruns:0 frame:0
          TX packets:194 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:43261 (42.2 KiB)  TX bytes:23769 (23.2 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1248 (1.2 KiB)  TX bytes:1248 (1.2 KiB)


以上的过程,我们就完成了一台虚拟机的配置。接下来,我们需要安装一下jdk 和 hadoop。

JDK和Hadoop安装

#1,将准备的jdk的源码包和hadoop源码包导入software目录,并检查是否成功
[isea@hadoop101 software]$ ll
总用量 374196
-rw-rw-r--. 1 isea isea 197657687 11月 14 17:55 hadoop-2.7.2.tar.gz
-rw-rw-r--. 1 isea isea 185515842 11月 14 17:55 jdk-8u144-linux-x64.tar.gz


#2,将jar包解压缩到module目录,并查看是否解压缩成功
[isea@hadoop101 software]$ tar -zxvf jdk-8u144-linux-x64.tar.gz -C /opt/module/
*
*
*
[isea@hadoop101 software]$ tar -zxvf hadoop-2.7.2.tar.gz -C /opt/module/
*
*
*
[isea@hadoop101 module]$ ll
总用量 8
drwxr-xr-x. 9 isea isea 4096 5月  22 2017 hadoop-2.7.2
drwxr-xr-x. 8 isea isea 4096 7月  22 2017 jdk1.8.0_144


#3,进入jdk和hadoop的安装目录获取各自的路径,准备配置环境变量
[isea@hadoop101 module]$ cd jdk1.8.0_144/
[isea@hadoop101 jdk1.8.0_144]$ pwd
/opt/module/jdk1.8.0_144
[isea@hadoop101 jdk1.8.0_144]$ cd ..
[isea@hadoop101 module]$ cd hadoop-2.7.2/
[isea@hadoop101 hadoop-2.7.2]$ pwd
/opt/module/hadoop-2.7.2

#4,配置jdk 和hadoop的环境变量,编辑/etc/profile文件(注意sudo),在文末(shift + g)分别添加
[isea@hadoop101 module]$ sudo vim /etc/profile

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-2.7.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

#5,source profile文件,检查是否安装成功
[isea@hadoop101 module]$ hadoop version
Hadoop 2.7.2
Subversion Unknown -r Unknown
Compiled by root on 2017-05-22T10:49Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /opt/module/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar
[isea@hadoop101 module]$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

如此一来,我们就安装好了Hadoop和JDK,我们在将这台虚拟机,克隆两份,在分别修改一下IP地址,就可以愉快的开始hadoop之旅了。

Hadoop的目录结构:

[isea@hadoop102 hadoop-2.7.2]$ ll
总用量 52
drwxr-xr-x. 2 isea isea  4096 5月  22 2017 bin
drwxr-xr-x. 3 isea isea  4096 5月  22 2017 etc
drwxr-xr-x. 2 isea isea  4096 5月  22 2017 include
drwxr-xr-x. 3 isea isea  4096 5月  22 2017 lib
drwxr-xr-x. 2 isea isea  4096 5月  22 2017 libexec
-rw-r--r--. 1 isea isea 15429 5月  22 2017 LICENSE.txt
-rw-r--r--. 1 isea isea   101 5月  22 2017 NOTICE.txt
-rw-r--r--. 1 isea isea  1366 5月  22 2017 README.txt
drwxr-xr-x. 2 isea isea  4096 5月  22 2017 sbin
drwxr-xr-x. 4 isea isea  4096 5月  22 2017 share

(1)bin目录:存放对Hadoop相关服务(HDFS,YARN)进行操作的脚本

(2)etc目录:Hadoop的配置文件目录,存放Hadoop的配置文件

(3)lib目录:存放Hadoop的本地库(对数据进行压缩解压缩功能)

(4)sbin目录:存放启动或停止Hadoop相关服务的脚本

(5)share目录:存放Hadoop的依赖jar包、文档、和官方案例

猜你喜欢

转载自blog.csdn.net/qq_31807385/article/details/84069561