hadoop本地环境测试搭建（windows）

 
  hadoop本地环境测试搭建（windows） 
 

   1.hadoop环境变量设置 
 

   解压hadoop压缩包 
 

   变量名：HADOOP_HOME 变量值：D:\soft\hadoop-2.5.1 
 

   path追加：%HADOOP_HOME%\bin;%HADOOP_HOME%\sbin; 
 

   添加变量名：HADOOP_USER_NAME 变量值：root 
 

 
  注：要重启机器才能使环境变量生效 
 

 
安装hadoop eclipse插件(hadoop-eclipse-plugin-2.5.1.jar导入eclipse安装陆慕plugins文件 夹下)，重启eclipse。 
 

   windows-preferences设置hadoop路径 
 

   window-show view-others找到Map/Reduce，New Hadoop location，填写你自己hadoop信息 
 

   把hadoop解压缩bin下，添加winutils.exe，C:\Windows\System32添加hadoop.dll文件。 
 

   3.更改hadoop源码hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\io\nativeio/NativeIO.java修改 
 

   return true，并导入项目中src下。 
 

   测试类WordCount.java 
 

   package com.xiaolong.wc; 
 

   import java.io.IOException; 
 

   import java.util.StringTokenizer; 
 

   import org.apache.hadoop.conf.Configuration; 
 

   import org.apache.hadoop.fs.Path; 
 

   import org.apache.hadoop.io.IntWritable; 
 

   import org.apache.hadoop.io.Text; 
 

   import org.apache.hadoop.mapreduce.Job; 
 

   import org.apache.hadoop.mapreduce.Mapper; 
 

   import org.apache.hadoop.mapreduce.Reducer; 
 

   import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
 

   import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; 
 

   public class WordCount { 
 

   public static class TokenizerMapper 
 

   extends Mapper<Object, Text, Text, IntWritable>{ 
 

   private final static IntWritable one = new IntWritable(1); 
 

   private Text word = new Text(); 
 

   public void map(Object key, Text value, Context context 
 

   ) throws IOException, InterruptedException { 
 

   StringTokenizer itr = new StringTokenizer(value.toString()); 
 

   while (itr.hasMoreTokens()) { 
 

   word.set(itr.nextToken()); 
 

   context.write(word, one); 
 

}

}

}

   public static class IntSumReducer 
 

   extends Reducer<Text,IntWritable,Text,IntWritable> { 
 

   private IntWritable result = new IntWritable(); 
 

   public void reduce(Text key, Iterable<IntWritable> values, 
 

   Context context 
 

   ) throws IOException, InterruptedException { 
 

   int sum = 0; 
 

   for (IntWritable val : values) { 
 

   sum += val.get(); 
 

}

   result.set(sum); 
 

   context.write(key, result); 
 

}

}

   public static void main(String[] args) throws Exception { 
 

   Configuration conf = new Configuration(); 
 

   conf.set("fs.defaultFS", "hdfs://node1:8020"); 
 

   conf.set("yarn.resourcemanager.hostname", "node3"); 
 

   Job job = Job.getInstance(conf, "word count"); 
 

   job.setJarByClass(WordCount.class); 
 

   job.setMapperClass(TokenizerMapper.class); 
 

   job.setCombinerClass(IntSumReducer.class); 
 

   job.setReducerClass(IntSumReducer.class); 
 

   job.setOutputKeyClass(Text.class); 
 

   job.setOutputValueClass(IntWritable.class); 
 

   FileInputFormat.addInputPath(job, new Path("hdfs://node1:8020/usr/input/test1.txt")); 
 

   FileOutputFormat.setOutputPath(job, new Path("hdfs://node1:8020/usr/output")); 
 

   System.exit(job.waitForCompletion(true) ? 0 : 1); 
 

}

}

   若无任何报错，说明已搭建本地测试环境成功，在dfs中/usr/output会生成以下文件_SUCCESS文件 
 

   以上工具包下载地址：http://download.csdn.net/download/qq_24811513/10198793 
 