hadoop mr

先编写MapReduce程序（对文本数据统计）。

//4个泛型中，前两个是指定mapper输入数据的类型，KEYIN是输入的key的类型，VALUEIN是输入的value的类型
//map 和 reduce 的数据输入输出都是以 key-value对的形式封装的
//默认情况下，框架传递给我们的mapper的输入数据中，key是要处理的文本中一行的起始偏移量，这一行的内容作为value
public class WCMapper extends Mapper<LongWritable, Text, Text, LongWritable>{
    //mapreduce框架每读一行数据就调用一次该方法
   @Override
   protected void map(LongWritable key, Text value,Context context)
         throws IOException, InterruptedException {
        //具体业务逻辑就写在这个方法体中，而且我们业务要处理的数据已经被框架传递进来，在方法的参数中 key-value
        //key 是这一行数据的起始偏移量     value 是这一行的文本内容
        //将这一行的内容转换成string类型
      String line = value.toString();
        //对这一行的文本按特定分隔符切分
      String[] words = StringUtils.split(line, " ");
        //遍历这个单词数组输出为kv形式  k：单词   v ： 1
      for(String word : words){
         context.write(new Text(word), new LongWritable(1));
      }
   }
}

public class WCReducer extends Reducer<Text, LongWritable, Text, LongWritable>{
   //框架在map处理完成之后，将所有kv对缓存起来，进行分组，然后传递一个组<key,valus{}>，调用一次reduce方法
   //<hello,{1,1,1,1,1,1.....}>
   @Override
   protected void reduce(Text key, Iterable<LongWritable> values,Context context)
         throws IOException, InterruptedException {
      long count = 0;
      for(LongWritable value:values){
         //遍历value的list，进行累加求和
         count += value.get();
      }
      //输出这一个单词的统计结果
      context.write(key, new LongWritable(count));
   }
}

/**
 * 用来描述一个特定的作业
 * 比如，该作业使用哪个类作为逻辑处理中的map，哪个作为reduce
 * 还可以指定该作业要处理的数据所在的路径
 * 还可以指定改作业输出的结果放到哪个路径
 *
 */
public class WCRunner {

   public static void main(String[] args) throws Exception {
      
      Configuration conf = new Configuration();
      
      Job wcjob = Job.getInstance(conf);

      //设置整个job所用的那些类在哪个jar包
      wcjob.setJarByClass(WCRunner.class);

      //本job使用的mapper和reducer的类
      wcjob.setMapperClass(WCMapper.class);
      wcjob.setReducerClass(WCReducer.class);

      //指定reduce的输出数据kv类型
      wcjob.setOutputKeyClass(Text.class);
      wcjob.setOutputValueClass(LongWritable.class);

      //指定mapper的输出数据kv类型
      wcjob.setMapOutputKeyClass(Text.class);
      wcjob.setMapOutputValueClass(LongWritable.class);

      //指定要处理的输入数据存放路径
      FileInputFormat.setInputPaths(wcjob, new Path("hdfs://192.168.93.132:9000/wc/srcdata/"));
      //指定处理结果的输出数据存放路径
      FileOutputFormat.setOutputPath(wcjob, new Path("hdfs://192.168.93.132:9000/wc/resultOutput/"));

      //将job提交给集群运行
      wcjob.waitForCompletion(true);  
   }

}

之后将程序打包提交到yarn集群。

我是在win上使用maven构建的项目，引用的包为自己导入的。打包的加入以下配置。

<build>
    <plugins>
        <plugin>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
                <encoding>UTF-8</encoding>
                <compilerArguments>
                    <extdirs>lib</extdirs>
                </compilerArguments>
            </configuration>
        </plugin>
    </plugins>
</build>

要不有可能自己建立的lib下的jar包打包的时候找不到路径。

在hdfs上建立好对应的文件夹

将要统计的文件上传到程序中指定的文件夹下

之后运行编译好的jar就可以了

hadoop jar root-1.0-SNAPSHOT.jar mr.WCRunner

[root@hadoop ~]# hadoop jar root-1.0-SNAPSHOT.jar mr.WCRunner

// 先建立连接到ResourceManager 的client
19/04/20 22:29:08 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.93.132:8032
19/04/20 22:29:09 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/04/20 22:29:10 INFO input.FileInputFormat: Total input paths to process : 1 总共需要处理的数量
19/04/20 22:29:10 INFO mapreduce.JobSubmitter: number of splits:1 1个split
19/04/20 22:29:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1555769992184_0001 -- jobId
19/04/20 22:29:12 INFO impl.YarnClientImpl: Submitted application application_1555769992184_0001 -- 提交应用
19/04/20 22:29:12 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1555769992184_0001/ --url跟踪地址
19/04/20 22:29:12 INFO mapreduce.Job: Running job: job_1555769992184_0001 -- 开始运行这个job
19/04/20 22:29:23 INFO mapreduce.Job: Job job_1555769992184_0001 running in uber mode : false
19/04/20 22:29:23 INFO mapreduce.Job: map 0% reduce 0%
19/04/20 22:29:30 INFO mapreduce.Job: map 100% reduce 0%
19/04/20 22:29:37 INFO mapreduce.Job: map 100% reduce 100%
19/04/20 22:29:37 INFO mapreduce.Job: Job job_1555769992184_0001 completed successfully -- 运行成功
19/04/20 22:29:37 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=268
FILE: Number of bytes written=231203
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=193
HDFS: Number of bytes written=58
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4903
Total time spent by all reduces in occupied slots (ms)=3350
Total time spent by all map tasks (ms)=4903
Total time spent by all reduce tasks (ms)=3350
Total vcore-seconds taken by all map tasks=4903
Total vcore-seconds taken by all reduce tasks=3350
Total megabyte-seconds taken by all map tasks=5020672
Total megabyte-seconds taken by all reduce tasks=3430400
Map-Reduce Framework
Map input records=9
Map output records=18
Map output bytes=226
Map output materialized bytes=268
Input split bytes=111
Combine input records=0
Combine output records=0
Reduce input groups=9
Reduce shuffle bytes=268
Reduce input records=18
Reduce output records=9
Spilled Records=36
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=320
CPU time spent (ms)=2200
Physical memory (bytes) snapshot=421101568
Virtual memory (bytes) snapshot=4199149568
Total committed heap usage (bytes)=293076992
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=82
File Output Format Counters
Bytes Written=58

hadoop 自动生成了对应的输出结果目录

结果

MR执行流程
   (1).客户端提交一个mr的jar包给JobClient(提交方式：hadoop jar ...)
   (2).JobClient通过RPC和JobTracker进行通信，返回一个存放jar包的地址（HDFS）和jobId
   (3).client将jar包写入到HDFS当中(path = hdfs上的地址 + jobId)
   (4).开始提交任务(任务的描述信息，不是jar, 包括jobid，jar存放的位置，配置信息等等)
   (5).JobTracker进行初始化任务
   (6).读取HDFS上的要处理的文件，开始计算输入分片，每一个分片对应一个MapperTask
   (7).TaskTracker通过心跳机制领取任务（任务的描述信息）
   (8).下载所需的jar，配置文件等
   (9).TaskTracker启动一个java child子进程，用来执行具体的任务（MapperTask或ReducerTask）
   (10).将结果写入到HDFS当中

猜你喜欢