Flink1.10实战：自定义聚合函数AggregateFunction

Flink 的AggregateFunction是一个基于中间计算结果状态进行增量计算的函数。由于是迭代计算方式，所以，在窗口处理过程中，不用缓存整个窗口的数据，所以效率执行比较高。

@PublicEvolving

public interface AggregateFunction<IN, ACC, OUT> extends Function, Serializable {

...............................

}

自定义聚合函数需要实现AggregateFunction接口类，它有四个接口实现方法：

a.创建一个新的累加器，启动一个新的聚合,负责迭代状态的初始化

ACC createAccumulator();
b.对于数据的每条数据，和迭代数据的聚合的具体实现

ACC add(IN value, ACC accumulator);
c.合并两个累加器，返回一个具有合并状态的累加器

ACC merge(ACC a, ACC b);
d.从累加器获取聚合的结果

OUT getResult(ACC accumulator);

3.自定义聚合函数MyCountAggregate

package com.hadoop.ljs.flink110.aggreagate;
import org.apache.flink.api.common.functions.AggregateFunction;
/**
 * @author: Created By lujisen
 * @company ChinaUnicom Software JiNan
 * @date: 2020-04-15 22:00
 * @version: v1.0
 * @description: com.hadoop.ljs.flink110.aggreagate
 * 输入类型(IN)、累加器类型(ACC)和输出类型(OUT)。
 */
public class MyCountAggregate implements AggregateFunction<ProductViewData, Long, Long> {
    @Override
    public Long createAccumulator() {
        /*访问量初始化为0*/
        return 0L;
    }
    @Override
    public Long add(ProductViewData value, Long accumulator) {
        /*访问量直接+1 即可*/
        return accumulator+1;
    }
    @Override
    public Long getResult(Long accumulator) {
        return accumulator;
    }
    /*合并两个统计量*/
    @Override
    public Long merge(Long a, Long b) {
        return a+b;
    }
}

4.自定义窗口函数

package com.hadoop.ljs.flink110.aggreagate;
import org.apache.flink.streaming.api.functions.windowing.WindowFunction;
import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
import org.apache.flink.util.Collector;
/**
 * @author: Created By lujisen
 * @company ChinaUnicom Software JiNan
 * @date: 2020-04-15 21:56
 * @version: v1.0
 * @description: com.hadoop.ljs.flink110.aggreagate
 *  *自定义窗口函数，封装成字符串
 *  *第一个参数是上面MyCountAggregate的输出，就是商品的访问量统计
 *  * 第二个参数 输出 这里为了演示 简单输出字符串
 *  * 第三个就是 窗口类 能获取窗口结束时间
 */
public class MyCountWindowFunction2 implements WindowFunction<Long,String,String, TimeWindow> {
@Override
public void apply(String productId, TimeWindow window, Iterable<Long> input, Collector<String> out) throws Exception {
     /*商品访问统计输出*/
    /*out.collect("productId"productId,window.getEnd(),input.iterator().next()));*/
    out.collect("----------------窗口时间："+window.getEnd());
    out.collect("商品ID: "+productId+"  浏览量: "+input.iterator().next());
    }

5.主函数,代码如下：

import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.functions.KeySelector;
import org.apache.flink.streaming.api.TimeCharacteristic;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.timestamps.AscendingTimestampExtractor;
import org.apache.flink.streaming.api.windowing.time.Time;
 
/**
 * @author: Created By lujisen
 * @company ChinaUnicom Software JiNan
 * @date: 2020-04-14 11:28
 * @version: v1.0
 * @description: com.hadoop.ljs.flink110.aggreagate
 * 自定义聚合函数类和窗口类，进行商品访问量的统计，根据滑动时间窗口，按照访问量排序输出
 */
public class AggregateFunctionMain2 {
 
    public  static int windowSize=6000;/*滑动窗口大小*/
    public  static int windowSlider=3000;/*滑动窗口滑动间隔*/
    
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment senv = StreamExecutionEnvironment.getExecutionEnvironment();
        senv.setParallelism(1);
        senv.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
        /*DataStream<String> sourceData = senv.socketTextStream("localhost",9000);*/ 
        //从文件读取数据，也可以从socket读取数据       
        DataStream<String> sourceData = senv.readTextFile("D:\\projectData\\ProductViewData2.txt");
        DataStream<ProductViewData> productViewData = sourceData.map(new MapFunction<String, ProductViewData>() {
            @Override
            public ProductViewData map(String value) throws Exception {
                String[] record = value.split(",");
                return new ProductViewData(record[0], record[1], Long.valueOf(record[2]), Long.valueOf(record[3]));
            }
        }).assignTimestampsAndWatermarks(new AscendingTimestampExtractor<ProductViewData>(){
            @Override
            public long extractAscendingTimestamp(ProductViewData element) {
                return element.timestamp;
            }
        });
        /*过滤操作类型为1  点击查看的操作*/
        DataStream<String> productViewCount = productViewData.filter(new FilterFunction<ProductViewData>() {
            @Override
            public boolean filter(ProductViewData value) throws Exception {
                if(value.operationType==1){
                    return true;
                }
                return false;
            }
        }).keyBy(new KeySelector<ProductViewData, String>() {
            @Override
            public String getKey(ProductViewData value) throws Exception {
                return value.productId;
            }
            //时间窗口 6秒  滑动间隔3秒
        }).timeWindow(Time.milliseconds(windowSize), Time.milliseconds(windowSlider))
        /*这里按照窗口进行聚合*/
        .aggregate(new MyCountAggregate(), new MyCountWindowFunction2());
        //聚合结果输出
        productViewCount.print();
 
        senv.execute("AggregateFunctionMain2");
    }
}

这里自定义聚合函数演示完毕，感谢关注！！！

Flink1.10实战：自定义聚合函数AggregateFunction

猜你喜欢