ubuntu14使用hdp安装hadoop

1. 准备
1）安装java
copy jdk-7u80-linux-x64.tar.gz到本机，解压/test
vi /etc/profile
export JAVA_HOME=/test/jdk1.7.0_80
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
source /etc/profile
2）更改主机名
vi /etc/hostname
vi /etc/hosts
127.0.1.1       hdp01.test 改为173.168.0.121 hdp01.test 
173.168.0.121 hdp01.test
173.168.0.122 hdp02.test
3）sudo apt-get install ntp
service ntp start
4）安装postgres，mysql
第一步：安装 PostgreSQL, phpPgAdmin 和 Apache2
sudo apt-get -y install postgresql postgresql-contrib phppgadmin


第二步：配置 PostgreSQL 用户
su - postgres
psql
\password
\q


第3步：配置Postgresql
vim /etc/postgresql/9.1/main/postgresql.conf


 
Ubuntu上安装MySQL非常简单只需要几条命令就可以完成。
sudo apt-get install mysql-server
sudo apt-get install mysql-client libmysql-java
sudo apt-get install libmysqlclient-dev
sudo apt-get install  libmysql-java


2. 切换到root，设置ssh
1） 安装ssh
dpkg --list | grep ssh
sudo apt-get install openssh-server
2） 生成rsa密钥
ssh-keygen -t rsa -P ''
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3）更改root、ssh及文件权限
chmod o+w ~/ 
chmod 700 -R ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 /root
4）更改sshd_config
vi /etc/ssh/sshd_config
将/etc/ssh/sshd_config文件中的  PermitRootLogin prohibit-password改为PermitRootLogin yes（或者删掉PermitRootLogin prohibit-password也可以，为了安全可以备份，注释掉就可以）
sudo /etc/init.d/ssh restart
5）
vi ~/.ssh/config
host 173.168.0.121
user [YOUR_USER_NAME]
6）在slave机器上:
在root目录下
mkdir .ssh
复制master上的id_rsa.pub到.ssh
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod o+w ~/ 
chmod 700 -R ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 /root


3.从以下三个位置下载包，可以直接在windows上用http下载
nohup wget -c http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari-2.2.1.0-ubuntu14.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.0.0/HDP-2.4.0.0-ubuntu14-deb.tar.gz > 1.log 2>&1 &
nohup wget -c http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14/HDP-UTILS-1.1.0.20-ubuntu14.tar.gz > 2.log 2>&1 &
1) copy这三个文件到/var/www/html/hadoop
cd /var/www/html
mkdir hadoop
2) 运行http服务，解压三个文件到/var/www/html/hadoop


4. 在内网其它机器，添加官方源
sudo wget -nv http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0/ambari.list -O /etc/apt/sources.list.d/ambari.list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
5. 修改官方源配置文件（dist上级目录）
sudo vi /etc/apt/sources.list.d/ambari.list


#VERSION_NUMBER=2.2.1.0-161
#deb http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.2.1.0 Ambari main


deb http://173.168.0.121/hadoop/AMBARI-2.2.1.0/ubuntu14/2.2.1.0-161/ Ambari main


6. 更新源并安装程序
sudo apt-get update
master：sudo apt-get install ambari-server
slave:  sudo apt-get install ambari-agent


7. 配置ambari服务
1）sudo ambari-server setup
输入JAVA_HOME的值，其他默认
2）ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql.jar


8. 启动服务
master：sudo ambari-server start
slave:  sudo ambari-agent start
sudo vi /etc/ambari-agent/conf/ambari-agent.ini 
hostname=hdp01.test


9. 配置hive，ambari数据库
1）
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
CREATE USER 'hive'@'hdp01.test'IDENTIFIED BY 'hive';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE hive;


CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost';
CREATE DATABASE ambari;
USE ambari;
SOURCE /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;


CREATE USER 'oozie'@'localhost' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'localhost';
CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
CREATE USER 'oozie'@'hdp01.test'IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'hdp01.test';
FLUSH PRIVILEGES;
create DATABASE oozie;


2）
sudo vi /etc/mysql/my.cnf 
添加'#'注释掉其中的"bind-address = 127.0.0.1"
sudo service mysql restart


10. 访问web界面
http://ip:8080
使用默认的admin/admin登录。
1）点击左上角图标或者选择 Launch Install Wizard
2）第二步Select Stack，在选择源的地方，填写路径（dist上级目录）
http://173.168.0.121/hadoop/HDP/ubuntu14/2.x/updates/2.4.0.0/
http://173.168.0.121/hadoop/HDP-UTILS-1.1.0.20/repos/ubuntu14/


11. 在第3步


1）只填hdp01.test
sudo cat id_rsa
把private key复制到框里，点击下一步，看到sucess后，点击Back
2）把hdp02.test也填上
选择手动安装ambari-agent
看到两个都sucess后，点击下一步


12. 禁止HDP（Transparent HugePages）
vi /etc/default/grub，在GRUB_CMDLINE_LINUX加入选项 transparent_hugepage=never
grub-mkconfig -o /boot/grub/grub.cfg

重启

13. hive、oozie选择已有的mysql数据库，设置数据库密码