flink1.11. Read Kafka data
The core essence of Watermark can be understood as a delayed trigger mechanism.
In the window processing process of Flink, if it is determined that all the data has arrived, you can perform window calculation operations (such as summary, grouping, etc.) on all the data of the Window. If all the data has not arrived, continue to wait for all the data in the window to arrive. Start processing. In this case, you need to use the WaterMarks mechanism, which can measure the progress of data processing (express the completeness of data arrival), and ensure that event data (all) arrives at the Flink system, or when it arrives out of order and delayed. Able to calculate correct and continuous results as expected. When any Event enters the Flink system, the Watermarks timestamp will be generated according to the current maximum event time.
So how does Flink calculate the value of Watermak?
Watermark = Maximum event time (mxtEventTime) entering Flink-specified delay time (t)
So how does the Window with Watermark trigger the window function?
If the stop time of a window is equal to or less than maxEventTime-t (warkmark at the time), then this window is triggered to execute.