[Flink] # 15_Source and Sink DESCRIPTION

Data Source

StreamExecutionEnvironment.addSource(sourceFunction)
There are the following categories:

• Based on the set

  • fromCollection (Collection) - to create a data stream from a Java Java.util.Collection. All set of element types must be the same.
  • fromCollection (Iterator, Class) - Create a data stream from the iterator. Class specifies the type of the iterator returned element.
  • fromElements (T ...) - Create a data stream from a given target sequence. All object types must be the same.
  • fromParallelCollection (SplittableIterator, Class) - Create a parallel data stream from the iterator. Class specifies the type of the iterator returned element.
  • generateSequence (from, to) - Create a parallel data stream to generate a sequence of numbers within the specified range interval.

• Based on documents

  • readTextFile (path) - to read a text file, the file TextInputFormat that meet specifications, and returns it as a string.
  • readFile (fi leInputFormat, path) - The input format specified file to read the file (once).
  • readFile (fi leInputFormat, path, watchType, interval, pathFilter, typeInfo) - This is a method called the internal above two methods. It reads the given files and fi leInputFormat read path. The watchType provided, this source may be periodically (every millisecond interval) to a new monitoring data given path (FileProcessingMode.PROCESSCONTINUOUSLY FileProcessingMode.PROCESSONCE). You can exclude files need to be addressed further by pathFilter.

• Based on Socket

  • socketTextStream (String hostname, int port) - read from the socket. The separator element can be sliced.

• Custom
  addSource - add a new source function. For example, you can read data from Apache Kafka with addSource (new FlinkKafkaConsumer011 <> (... )).

Data Sink