分布式的数据仓库,结构化的数据文件映射为一张表,并提供类SQL查询功能。Hive的设计目的是让精通SQL技能但Java编程技能相对较弱的分析师能够对存放在HDFS中的大规模数据集执行查询
本质是:将HQL转化成MapReduce程序
1)Hive处理的数据存储在HDFS
2)Hive分析数据底层的实现是MapReduce
3)执行程序运行在Yarn上
下载:
http://archive.apache.org/dist/hive/
Hive集群安装(基于Mysql)
1)安装jdk(略)
2)安装hadoop(略)
3)安装mysql数据库(略)
yum install wget
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
[root@hadoopNode1 soft]# rpm -ivh mysql-community-release-el7-5.noarch.rpm
[root@hadoopNode1 soft]# yum install mysql-server
[root@hadoopNode1 soft]# systemctl start mysqld
[root@hadoopNode1 soft]# systemctl enable mysqld #启用系统服务
初始化root用户
mysql -u root
mysql>use mysql;
mysql> update user set password=password('123456') where user='root'; //新密码为123456
mysql> select host,user from user;
mysql> GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456";
mysql> flush privileges;
mysql> exit;
4)mysql数据库增加hive用户,并授权
mysql> create user 'hive' identified by '123456';
mysql> CREATE DATABASE hive;
mysql> use hive;
mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' IDENTIFIED BY '123456';
mysql>GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'master' IDENTIFIED BY '123456'; //写为自己的集群的mysql所在节点
mysql>GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'localhost' IDENTIFIED BY '123456';
mysql> flush privileges;
mysql>exit
systemctl restart mysqld
5)验证MYSQL配置正确
在windows下登陆MySQL
- tar包:
[ambow@hadoopNode1 hive-2.3.2]$ tar -zxvf apache-hive-2.3.2-bin.tar.gz -C ~/app/
7)配置环境变量
HIVE_HOME
PATH
HIVE_HOME=/home/ambow/app/hive-2.3.2
HBASE_HOME=/home/ambow/app/hbase-1.3.2
JAVA_HOME=/home/ambow/app/jdk1.8.0_121
HADOOP_HOME=/home/ambow/app/hadoop-2.7.3
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin
export PATH
export JAVA_HOME
export HADOOP_HOME
export HBASE_HOME
export HIVE_HOME
8)修改配置文件:
(1)进入{HIVE_HOME/conf}目录下,复制hive-env.sh.template
$>cp hive-env.sh.template hive-env.sh
$>vi hive-env.sh
添加:【 export HADOOP_HOME=/home/ambow/app/hadoop 】
(2)进入{HIVE_HOME/conf}目录下,复制hive-default.xml.sh.template
[ambow@hadoopNode1 conf]$ cp hive-default.xml.template hive-site.xml
[ambow@hadoopNode1 conf]$ vi hive-site.xml
Hive2.3.2:
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive</value>
<description>HDFS路径,用于存储不同 map/reduce 阶段的执行计划和这些阶段的中间输出结果 会自动创建</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
<description>连接HIVE元数据存放的mysql使用用户名</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://hadoopNode1:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description> Hive 默认的数据文件存储路径,通常为 HDFS 可写的路径 自动创建</description>
</property>
<property>
<name>datanucleus.readOnlyDatastore</name>
<value>false</value>
</property>
<property>
<name>datanucleus.fixedDatastore</name>
<value>false</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateColumns</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
<description>creates necessary schema on a startup if one doesn't exist. set this to false, after creating it once
</description>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
9)将MySQL的连接驱动包拷贝至{HIVE_HOME/lib}下
ambow@hadoopNode1 lib]$ cp mysql-connector-java-5.1.34.jar $HIVE_HOME/lib
10)启动hadoop集群
zkServer.sh start 各节点
hadoop-demon.sh start zkfc 两个NN
start-all.sh
11)启动hive客户端 验证
$> hive
show databases;