hadoop hdfs环境搭建

参考：

http://wiki.apache.org/hadoop/GettingStartedWithHadoop

http://blog.csdn.net/warmspace2008/article/details/8540159

hadoop环境搭建参考官方wiki，首先下载hadoop官方包，我选择的是hadoop-1.2.1。conf目录新增配置文件：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/duitang/tmp/hadoop-${user.name}</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
</property>
<property>
  <name>mapred.job.tracker</name>
  <value>hdfs://localhost:54311</value>
</property>
<property> 
  <name>dfs.replication</name>
  <value>8</value>
</property>
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx512m</value>
</property>
</configuration>

通过start-dfs.sh启动，

写道

yunpeng@yunpeng-duitang:/duitang/dist/sys/hadoop-1.2.1$ ./bin/start-dfs.sh
starting namenode, logging to /duitang/dist/sys/hadoop-1.2.1/libexec/../logs/hadoop-yunpeng-namenode-yunpeng-duitang.out
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused

结果日志报错：

写道

2013-12-23 16:18:05,195 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /duitang/tmp/hadoop-yunpeng/dfs/name does not exist
2013-12-23 16:18:05,197 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /duitang/tmp/hadoop-yunpeng/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible.

分析错误，应该是没有没有创建目录，创建之后再次启动，还是报错：

2013-12-23 16:20:42,797 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: NameNode is not formatted.

at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:331)

原来NameNode需要format,执行hadoop namenode -format 

写道

yunpeng@yunpeng-duitang:/duitang/dist/sys/hadoop-1.2.1$ ./bin/hadoop namenode -format
13/12/23 16:22:27 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
13/12/23 16:22:27 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = yunpeng-duitang/127.0.0.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.2.1
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG: java = 1.6.0_29
************************************************************/
Re-format filesystem in /duitang/tmp/hadoop-yunpeng/dfs/name ? (Y or N) Y
13/12/23 16:22:29 INFO util.GSet: Computing capacity for map BlocksMap
13/12/23 16:22:29 INFO util.GSet: VM type = 64-bit
13/12/23 16:22:29 INFO util.GSet: 2.0% max memory = 932118528
13/12/23 16:22:29 INFO util.GSet: capacity = 2^21 = 2097152 entries
13/12/23 16:22:29 INFO util.GSet: recommended=2097152, actual=2097152
13/12/23 16:22:29 INFO namenode.FSNamesystem: fsOwner=yunpeng
13/12/23 16:22:29 INFO namenode.FSNamesystem: supergroup=supergroup
13/12/23 16:22:29 INFO namenode.FSNamesystem: isPermissionEnabled=true
13/12/23 16:22:29 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
13/12/23 16:22:29 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
13/12/23 16:22:29 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
13/12/23 16:22:29 INFO namenode.NameNode: Caching file names occuring more than 10 times
13/12/23 16:22:29 INFO common.Storage: Image file /duitang/tmp/hadoop-yunpeng/dfs/name/current/fsimage of size 113 bytes saved in 0 seconds.
13/12/23 16:22:30 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/duitang/tmp/hadoop-yunpeng/dfs/name/current/edits
13/12/23 16:22:30 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/duitang/tmp/hadoop-yunpeng/dfs/name/current/edits
13/12/23 16:22:30 INFO common.Storage: Storage directory /duitang/tmp/hadoop-yunpeng/dfs/name has been successfully formatted.
13/12/23 16:22:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at yunpeng-duitang/127.0.0.1
************************************************************/

启动成功，通过http://localhost:50070/ 可以查看。

hadoop-site.xml 写道

fs.default.name
mapred.job.tracker
hadoop.job.ugi
mapred.job.queue.name
mapred.used.genericoptionsparser true
mapred.map.max.attempts 5
mapred.reduce.max.attempts 5
mapred.child.java.opts -Xmx1024m
mapred.output.compress true
mapred.output.compression.codec org.apache.hadoop.io.compress.DefaultCodec
mapred.compress.map.output true
mapred.map.output.compression.codec org.apache.hadoop.io.compress.LzoCodec
io.compression.codecs org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.LzoCodec
fs.trash.interval 1440
fs.hdfs.impl org.apache.hadoop.hdfs.Yunti3FileSystem
fs.viewfs.impl org.apache.hadoop.fs.viewfs.ViewFileSystem
fs.yunti.prefix.lists hdfs://xxxx:9000,hdfs://xxxx:9000
fs.viewfs.zk.addr

客户端测试：

Exception in thread "main" java.io.IOException: Call to localhost/127.0.0.1:9000 failed on local exception: java.io.EOFException

报此错说明客户端hadoop版本和服务端hadoop版本不一致。修改客户端引用hadoop的版本