Note: In the actual production and development in both the use of fully distributed
1, build step
1) Preparation 3 clients (turn off the firewall, static ip, host name)
2) Install JDK
3) configuration environment variable
4) install Hadoop
5) configuration environment variable
6) cluster configuration
7) a single point of start
8) Configure ssh
9) rallied and tested cluster
2, scp: (secure cope) secure copy
1, scp definition:
scp data copy may be implemented between the server and the server. (From server1 to server2)
2, the basic grammar
scp -r pdir / pdir / pdir / fname user @ hadoopuser @ hadoopuser @ hadoophost: pdir / pdir / pdir / fname
command recursively to copy the file path / name of the destination user @ host: destination path / name
3, case practical operation
On hadoop1, the hadoop1 in / opt / copy it into the module directory under the hadoop2.
[root@hadoop1 /]$ scp -r /opt/module root@hadoop1:/opt/module
On hadoop3, under software / opt on hadoop1 server / module directory copy on hadoop3.
[root@hadoop3 opt]$scp -r root@hadoop1:/opt/module·root@hadoop3:/opt/module
Software operating on hadoop3 hadoop1 in / opt in / module to copy the directory hadoop4.
[root@hadoop3 opt]$ scp -r root@hadoop1:/opt/module root@hadoop4:/opt/module
Note: The copied the / opt / module directory, do not forget hadoop102, hadoop103, modify all the files on hadoop104
Owner and owner group. sudo chown root: root -R / opt / module
Hadoop1 in the / etc / profile hadoop2 files are copied to the / etc / profile on.
[root@hadoop1 ~]$ scp /etc/profile root@hadoop2:/etc/profile
Note: Do not forget to copy over the configuration file at the source / etc / profile ,.
3, rsync remote synchronization tool
rsync mainly used for backup and mirroring. Has the speed, avoid copying the contents of the same advantages and supports symbolic links.
rsync and scp difference:
do use rsync to copy files faster than scp, rsync only difference file to do the update .
scp is to copy all the files in the past
The basic syntax
pdir / pdir / pdir / fname: rsync -rvl pdir / pdir / pdir / fname user @ hadoopuser @ hadoopuser @ hadoophost
file path command options parameters to be copied / name of the destination user @ host: destination path / name
Options Parameter Description
Case practical operation
/ opt / directory under the / opt / software hadoop1 directory on the server machine synchronized to the root user hadoop2
[root@hadoop1 opt]$ rsync -rvl /opt/software/ root@hadoop2:/opt/software
4, xsync cluster distribution script
Requirements: cycling copy the file to the same directory for all nodes
Requirements Analysis:
(A) the rsync original copy command:
the rsync -rvl / opt / Module1 hadoop103 the root @: / opt /
(B) expected script:
file name you want to synchronize xsync
(C) Note: / usr / local / bin directory of this script is stored, other users can execute directly anywhere in the system.
Script to achieve
(a) in the / usr / local / bin directory, and xsync create a file in the bin directory, file reads as follows:
[root@hadoop2 bin]$ touch xsync
[root@hadoop2 bin]$ vi xsync
Write the following code in the file
! # / bin / the bash
# . 1 acquires the number of input parameters, without parameters, exit
Pcount $ # =
IF ((Pcount == 0 )); the then
echo NO args;
Exit;
Fi
# 2 acquires the file name
p1 = $ 1 fname = `basename
$ p1` echo fname = $ fname
# 3 acquired the parent directory to the absolute path
pdir =` cd -P $ (dirname $ p1); pwd`
echo = $ pdir pdir
# 4 to get the current user name
user = ` whoami`
# . 5 cycles
for ((= Host . 3 ; Host < . 5 ; Host ++)); do
echo ------------------- Hadoop $ Host ------ --------
rsync -rvl $pdir/$fname $user@hadoop$host:$pdir
done
(B) modify the script has execute permissions xsync
[root@hadoop2 bin]$ chmod 777 xsync
(C) calls the script form: xsync file name
[root@hadoop2 bin]$ xsync /usr/local/bin
Note: If you xsync put / home / user / bin directory still can not achieve global use
xsync can be moved to the / usr / local / bin directory.