Flume docks spark streaming

1. There are two ways for flume to connect to kafka: the pull and push processes are as follows:

 

 

2. The difference between the two

 

3. Problems that may be encountered when using the pull method:

Local test environment:

• spark 2.3
• flume 1.8

   Using the pull method, flume reports the following error:

  (This bug exists on the official website and has not been fixed yet: https://issues.apache.org/jira/browse/SPARK-17152 )

 

java.lang.IllegalStateException: begin() called when transaction is OPEN!

 

 

Online solution:

This error is caused by the scala version problem, just replace the scala-library version under flume lib (pro test, easy to use)