flink-core

基础的内部数据定义。集群端和用户端共用

一些函数（function）

The base interface for all user-defined functions.
This interface is empty in order to allow extending interfaces to be SAM (single abstract method) interfaces that can be implemented via Java 8 lambdas.

org.apache.flink.api.common.functions.MapFunction
org.apache.flink.api.common.functions.JoinFunction

一些操作符（operator）

An operator is a source, sink, or it applies an operation to one or more inputs, producing a result.

DualInputOperator，对应两个输入，一个输出的function。

累计器（Accumulator）

我们以典型的IntCounter为例

聚合器（Aggregator）

LongSumAggregator为例子

事件事件定义类

Watermark，内部维护一个 private final long timestamp;

sink

source

Tuple

Tuples have a fix length and contain a set of fields, which may all
be of different types. Because Tuples are strongly typed, each distinct tuple length is
represented by its own class. Tuples exists with up to 25 fields and are described in the classes

flink-runtime

集群端的运行包

JobGraph

The JobGraph represents a Flink dataflow program, at the low level that the JobManager accepts.
All programs from higher level APIs are transformed into JobGraphs.

扫描二维码关注公众号，回复： 12469314 查看本文章

The JobGraph is a graph of vertices and intermediate results that are connected together to
form a DAG. Note that iterations (feedback edges) are currently not encoded inside the JobGraph
but inside certain special vertices that establish the feedback channel amongst themselves.

The JobGraph defines the job-wide configuration settings, while each vertex and intermediate result
define the characteristics of the concrete operation and intermediate data.
内部维护一个节点图

	private final Map<JobVertexID, JobVertex> taskVertices = new LinkedHashMap<JobVertexID, JobVertex>();

每个节点维护自己的结果和输入

//JobVertex.java
	/** List of produced data sets, one per writer */
	private final ArrayList<IntermediateDataSet> results = new ArrayList<IntermediateDataSet>();

	/** List of edges with incoming data. One per Reader. */
	private final ArrayList<JobEdge> inputs = new ArrayList<JobEdge>();

ResourceManager

一般有三种RM。

• StandaloneResourceManager

public class StandaloneResourceManager extends ResourceManager<ResourceID> {
    
    

• YarnResourceManager

public class YarnResourceManager extends ResourceManager<YarnWorkerNode> implements AMRMClientAsync.CallbackHandler, NMClientAsync.CallbackHandler {
    
    

• MesosResourceManager

public class MesosResourceManager extends ResourceManager<RegisteredMesosWorkerNode> {
    
    

会启动jobLeaderIdService和leaderElectionService

Dispatcher

是 一个FencedRpcEndpoint。

JobManagerRunner

启动JobManager，每个任务一个。

JobMaster（其实就是新版本的JobManager）

JobMaster implementation. The job master is responsible for the execution of a single JobGraph
是 一个FencedRpcEndpoint

维护了如下字段

	private final Scheduler scheduler;
    private final SlotPool slotPool;
   	private final JobGraph jobGraph;

Scheduler

Slot

Slot是flink进行资源管理的基本单位。

SlotManager

IO.NetWork

IO.Disk

flink-java

给java开发中提供的。

DataSet

flink提供DataSet Api用户处理批量数据。flink先将接入数据转换成DataSet数据集，并行分布在集群的每个节点上；然后将DataSet数据集进行各种转换操作(map，filter等)，最后通过DataSink操作将结果数据集输出到外部系统。

public <R> MapOperator<T, R> map(MapFunction<T, R> mapper) 
public <R> MapPartitionOperator<T, R> mapPartition(MapPartitionFunction<T, R> mapPartition)
public <R> FlatMapOperator<T, R> flatMap(FlatMapFunction<T, R> flatMapper)
public FilterOperator<T> filter(FilterFunction<T> filter)
public <OUT extends Tuple> ProjectOperator<?, OUT> project(int... fieldIndexes) 
public AggregateOperator<T> aggregate(Aggregations agg, int field)
public AggregateOperator<T> sum(int field)
public <K> DistinctOperator<T> distinct(KeySelector<T, K> keyExtractor)
public UnionOperator<T> union(DataSet<T> other)

注意，dataSet中的API

operators

flink-streaming-java

给java流处理提供的

DataStream

用于处理流数据

    public <R> SingleOutputStreamOperator<R> map(MapFunction<T, R> mapper) {
    public <T2> JoinedStreams<T, T2> join(DataStream<T2> otherStream) 
    public <R> SingleOutputStreamOperator<R> flatMap(FlatMapFunction<T, R> flatMapper) 

StreamGraph

维护流处理的状态

	private Set<Integer> sources;
    private Set<Integer> sinks;
	private Map<Integer, StreamNode> streamNodes;

streamGraph会转化为JobGraph进行执行。

	public JobGraph getJobGraph(@Nullable JobID jobID) {
    
    

其中，每一个StreamNode维护一个

    //StreamNode.java
	private List<StreamEdge> inEdges = new ArrayList<StreamEdge>();
	private List<StreamEdge> outEdges = new ArrayList<StreamEdge>();

flink-client

向集群提交工作的过程

flink-libraries

包含CEP等内容

flink-yarn

Per-Job-Cluster模式
一个任务会对应一个Job，每提交一个作业会根据自身的情况，都会单独向yarn申请资源，直到作业执行完成，一个作业的失败与否并不会影响下一个作业的正常提交和运行。独享Dispatcher和ResourceManager，按需接受资源申请；适合规模大长时间运行的作业。

flink-table

tabkle API的相关功能。

flink-connectors