MapReduce程序在Hadoop安装包里面本身就有.
测试 Word Count 统计词数
1.编写txt文件:
a.txt
I believe, for every drop of rain
that falls, A flower grows...
I believe that somewhere in the
darkest night, A candle glows...
I believe for everyone who
goes astray, Someone will come,
to show the way, I believe, I believe...
I believe, above the storm the
smallest prayer, Will still be heard...
I believe, that someone in that
great somewhere, Hears every word...
Every time I hear a newborn baby cry,
Or touch a leaf,
or see the sky,
Then I know why, I believe!
Every time I hear a newborn baby cry,
Or touch a leaf,
or see the sky,
Then I know why, I believe!
b.txt:
If you were a teardrop;In my eye
For fear of losing you,I would never cry
And if the golden sun,Should cease to shine its light
Just one smile from you,Would make my whole world bright
2.在hafs文件系统上新建一个子文件夹
要建层级文件夹需要加上字母 -p
hadoop fs -mkdir -p /wordcount/input
3.将测试文件上传到在hdfs文件系统上新建的文件夹上
hadoop fs -put a.txt b.txt /wordcount/input
4.运行MapReduce程序的jar包
hadoop-mapreduce-examples-2.6.4.jar
上面这个jar包是一个例子jar包,里面有很多例子,可以随便运行一个就可以了
hadoop jar hadoop-mapreduce-examples-2.6.4.jar wordcount /wordcount/input/ /wordcount/output
第一个参数是jar里面的哪一个主程序,第二个参数是输入的文件的位置,第三个参数是运行程序之后,输出的结果存放的位置,这个文件夹不能存在,如果存在是会报错的.
我启动的时候出现了问题,原因就是我没有启动yarn集群,我只是启动了hdfs集群,所以这样是无法运行MapReduce程序的,所以可以先start-all.sh将所有的集群启动,然后使用jps查看每一台机器:
bd1:
bd2,bd3,bd4:
正常启动之后的效果:
查看生成的文件
A 2
And 3
Every 2
For 1
Hears 1
Hold 1
I 13
If 1
Just 1
Or 2
Someone 1
Then 2
To 1
Will 1
a 9
above 1
an 1
astray, 1
baby 2
be 1
believe 2
believe! 2
believe, 4
believe... 1
bright 1
candle 1
cease 1
come, 1
cry 1
cry, 2
darkest 1
drop 1
eternity 1
every 2
everyone 1
eye 1
falls, 1
fear 1
flower 2
for 2
from 1
glows... 1
goes 1
golden 1
grain 1
great 1
grows... 1
hand 1
hear 2
heard... 1
heaven 1
hour 1
if 1
in 6
infinity 1
its 1
know 2
leaf, 2
light 1
losing 1
make 1
my 2
never 1
newborn 2
night, 1
of 4
one 1
or 2
palm 1
prayer, 1
rain 1
sand 1
see 3
shine 1
show 1
sky, 2
smallest 1
smile 1
someone 1
somewhere 1
somewhere, 1
still 1
storm 1
sun,Should 1
teardrop;In 1
that 4
the 8
time 2
to 2
touch 2
way, 1
were 1
who 1
whole 1
why, 2
wild 1
will 1
word... 1
world 2
would 1
you 1
you,I 1
you,Would 1
your 1