pig on tez测试
测试环境
- pig-0.14.0
- hadoop-2.5.2 ()1+2)
hive on tez 测试后,很好奇,pig是否可以在tez上运行呢?从官网上可以看到pig on tez的描述,就想应该是可以pig on tez的.
pig安装过程略过…
准备数据集
[hadoop@mymaster ~]$ wget http://hortonassets.s3.amazonaws.com/pig/lahman591-csv.zip
[hadoop@mymaster ~]$ unzip lahman591-csv.zip
[hadoop@mymaster ~]$ hadoop fs mkdir /test
[hadoop@mymaster ~]$ hadoop fs -put lahman591-csv/Batting.csv /test // 将Batting.csv上传到hdfs编写pig测试脚本
[hadoop@mymaster ~]$ mkdir pig
[hadoop@mymaster ~]$ vim pig/test.pig
batting = LOAD '/test/Batting.csv' USING PigStorage(',');
raw_runs = FILTER batting BY $1>0;
runs = FOREACH raw_runs GENERATE $0 AS playerID, $1 AS year, $8 AS runs;
grp_data = GROUP runs BY (year);
max_runs = FOREACH grp_data GENERATE group as grp, MAX(runs.runs) AS max_runs;
join_max_runs = JOIN max_runs BY ($0, max_runs), runs BY (year, runs);
join_data = FOREACH join_max_runs GENERATE $0 AS year, $2 AS playerID, $1 AS runs;
DUMP join_data;mr方式运行 test.pig
[hadoop@mymaster ~]$ /usr/local/pig-0.14.0/bin/pig -x mr pig/test.pig
// 片段
Input(s):
Successfully read 95195 records (6399268 bytes) from: "/test/Batting.csv"
Output(s):
Successfully stored 151 records (4507 bytes) in: "hdfs://10.128.17.21:9000/tmp/temp1552838877/tmp1249909816"
Counters:
Total records written : 151
Total bytes written : 4507
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
-------
(1988,boggswa01,128.0)
(1989,boggswa01,113.0)
(1990,henderi01,119.0)
(1991,molitpa01,133.0)
(1992,phillto02,114.0)
(1993,dykstle01,143.0)
(1994,thomafr04,106.0)
(1995,biggicr01,123.0)
(1996,burksel01,142.0)
(1997,biggicr01,146.0)
(1998,sosasa01,134.0)
(1999,bagweje01,143.0)
(2000,bagweje01,152.0)
(2001,sosasa01,146.0)
(2002,soriaal01,128.0)
(2003,pujolal01,137.0)
(2004,pujolal01,133.0)
(2005,pujolal01,129.0)
(2006,sizemgr01,134.0)
(2007,rodrial01,143.0)
(2008,ramirha01,125.0)
(2009,pujolal01,124.0)
(2010,pujolal01,115.0)
(2011,grandcu01,136.0)
2015-06-02 13:49:39,574 [main] INFO org.apache.pig.Main - Pig script completed in 1 minute, 10 seconds and 20 milliseconds (70020 ms)运行情况:
输入:95195 records
输出:151 records
耗时:70s
tez 方式运行test.pig
[hadoop@mymaster pig]$ /usr/local/pig-0.14.0/bin/pig -x tez pig/test.pig
HadoopVersion: 2.5.2
PigVersion: 0.14.0
TezVersion: 0.5.2
UserId: hadoop
FileName: 1.pig
StartedAt: 2015-06-02 13:50:03
FinishedAt: 2015-06-02 13:50:34
Features: HASH_JOIN,GROUP_BY,FILTER
Success!
DAG PigLatin:1.pig-0_scope-0:
ApplicationId: job_1432693876849_0008
TotalLaunchedTasks: 3
FileBytesRead: 3494886
FileBytesWritten: 5509316
HdfsBytesRead: 6398886
HdfsBytesWritten: 4507
Input(s):
Successfully read 95195 records (6398886 bytes) from: "/test/lahman591-csv/Batting.csv"
Output(s):
Successfully stored 151 records (4507 bytes) in: "hdfs://10.128.17.21:9000/tmp/temp-1130777030/tmp93164502"
(1994,thomafr04,106.0)
(1995,biggicr01,123.0)
(1996,burksel01,142.0)
(1997,biggicr01,146.0)
(1998,sosasa01,134.0)
(1999,bagweje01,143.0)
(2000,bagweje01,152.0)
(2001,sosasa01,146.0)
(2002,soriaal01,128.0)
(2003,pujolal01,137.0)
(2004,pujolal01,133.0)
(2005,pujolal01,129.0)
(2006,sizemgr01,134.0)
(2007,rodrial01,143.0)
(2008,ramirha01,125.0)
(2009,pujolal01,124.0)
(2010,pujolal01,115.0)
(2011,grandcu01,136.0)
2015-06-02 13:50:34,623 [main] INFO org.apache.pig.Main - Pig script completed in 34 seconds and 350 milliseconds (34350 ms)
2015-06-02 13:50:34,634 [main] INFO org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down thread pool
运行情况:
输入:95195 records
输出:151 records
耗时:34s
测试结果 tez比yarn快2倍多
根据本轮测试效果差异不大,要根据mr的串联数和数据大小 进行严格的测试 才能达到理想的官网测试性能指标,但是可以肯定的是,mr任务串联越多,tez的性能越显著.
hive on tez 的配置稍嫌麻烦,相对来说pig on tez的测试环境相当容易
参考:http://zh.hortonworks.com/hadoop-tutorial/faster-pig-tez/