[spark streaming] ReceiverTracker data generation and storage

foreword

In Spark Streaming, the overall responsibility for the dynamic scheduling of tasks is JobScheduler, and JobSchedulerthere are two very important members: JobGeneratorand ReceiverTracker. JobGeneratorResponsible for generating a specific RDD DAG for each batch, and ReceiverTrackerresponsible for the source of data.

receiverAll receivers that need to run on the executor InputDStreamneed to inherit ReceiverInputDStream. ReceiverInputDStream has a def getReceiver(): Receiver[T]method, and all subclasses need to implement this method. Such as KafkaInputDStreamcorrespond KafkaReceiver, FlumeInputDStreamcorrespond FlumeReceiver, TwitterInputDStreamcorrespond , TwitterReceiveretc.

Process overview:

• ReceiverTrackerStart, get all InputDStreamsthe corresponding receivers
• Determine the priority position of each Receiver according to the scheduling policy (which executors can be executed on)
• Wrap the Receiver into an RDD and submit a job through sc. The execution function is to create a supervisor instance and call the start() method, that is, the onStart() method of the Receiver is called.
• The onStart of the Receiver continuously receives data, and finally calls the supervisor through the store method to store the block
• Notify ReceiverTrackerthis Block's information after storage
• ReceiverTrackerHand over the Block message to the ReceivedBlockTrackermanagement

Start Receiver

First look at the startup process of receiverTracker:

ssc.start()
    scheduler.start()
        receiverTracker.start()
        jobGenerator.start()
----
 def start(): Unit = synchronized {
    if (isTrackerStarted) {
      throw new SparkException("ReceiverTracker already started")
    }

    if (!receiverInputStreams.isEmpty) {
      endpoint = ssc.env.rpcEnv.setupEndpoint(
        "ReceiverTracker", new ReceiverTrackerEndpoint(ssc.env.rpcEnv))
      if (!skipReceiverLaunch) launchReceivers()
      logInfo("ReceiverTracker started")
      trackerState = Started
    }
  }

In the start method, the Endpoint of ReceiverTracker is created first, and then the launchReceivers() method is called to start Recivers:

 private def launchReceivers(): Unit = {
    val receivers = receiverInputStreams.map { nis =>
      val rcvr = nis.getReceiver()
      rcvr.setReceiverId(nis.id)
      rcvr
    }

    runDummySparkJob()

    logInfo("Starting " + receivers.length + " receivers")
    endpoint.send(StartAllReceivers(receivers))
  }

Traverse all InputStreams and get the corresponding Receiver collection receivers. And send a StartAllReceivers message to ReceiverTrackerEndpoint to see how it is handled after receiving the message:

 case StartAllReceivers(receivers) =>
        val scheduledLocations = schedulingPolicy.scheduleReceivers(receivers, getExecutors)
        for (receiver <- receivers) {
          val executors = scheduledLocations(receiver.streamId)
          updateReceiverScheduledExecutors(receiver.streamId, executors)
          receiverPreferredLocations(receiver.streamId) = receiver.preferredLocation
          startReceiver(receiver, executors)
        }

The scheduling strategy is used to calculate and determine a set of priority positions for each receiver, that is, on which executor node a Receiver should be started. The main principles of scheduling are:

• Satisfy the Receiver's preferredLocation.
• Secondly, ensure that the Receiver is distributed as evenly as possible.

Then traverse all receivers and call the startReceiver(receiver, executors) method to start the receiver:

 private def startReceiver(
        receiver: Receiver[_],
        scheduledLocations: Seq[TaskLocation]): Unit = {
      def shouldStartReceiver: Boolean = {
        // It's okay to start when trackerState is Initialized or Started
        !(isTrackerStopping || isTrackerStopped)
      }

      val receiverId = receiver.streamId
      if (!shouldStartReceiver) {
        onReceiverJobFinish(receiverId)
        return
      }

      val checkpointDirOption = Option(ssc.checkpointDir)
      val serializableHadoopConf =
        new SerializableConfiguration(ssc.sparkContext.hadoopConfiguration)

      // Function to start the receiver on the worker node
      val startReceiverFunc: Iterator[Receiver[_]] => Unit =
        (iterator: Iterator[Receiver[_]]) => {
          if (!iterator.hasNext) {
            throw new SparkException(
              "Could not start receiver as object not found.")
          }
          if (TaskContext.get().attemptNumber() == 0) {
            val receiver = iterator.next()
            assert(iterator.hasNext == false)
            val supervisor = new ReceiverSupervisorImpl(
              receiver, SparkEnv.get, serializableHadoopConf.value, checkpointDirOption)
            supervisor.start()
            supervisor.awaitTermination()
          } else {
            // It's restarted by TaskScheduler, but we want to reschedule it again. So exit it.
          }
        }

      // Create the RDD using the scheduledLocations to run the receiver in a Spark job
      val receiverRDD: RDD[Receiver[_]] =
        if (scheduledLocations.isEmpty) {
          ssc.sc.makeRDD(Seq(receiver), 1)
        } else {
          val preferredLocations = scheduledLocations.map(_.toString).distinct
          ssc.sc.makeRDD(Seq(receiver -> preferredLocations))
        }
      receiverRDD.setName(s"Receiver $receiverId")
      ssc.sparkContext.setJobDescription(s"Streaming job running receiver $receiverId")
      ssc.sparkContext.setCallSite(Option(ssc.getStartSite()).getOrElse(Utils.getCallSite()))

      val future = ssc.sparkContext.submitJob[Receiver[_], Unit, Unit](
        receiverRDD, startReceiverFunc, Seq(0), (_, _) => Unit, ())
      // We will keep restarting the receiver job until ReceiverTracker is stopped
      future.onComplete {
        case Success(_) =>
          if (!shouldStartReceiver) {
            onReceiverJobFinish(receiverId)
          } else {
            logInfo(s"Restarting Receiver $receiverId")
            self.send(RestartReceiver(receiver))
          }
        case Failure(e) =>
          if (!shouldStartReceiver) {
            onReceiverJobFinish(receiverId)
          } else {
            logError("Receiver has been stopped. Try to restart it.", e)
            logInfo(s"Restarting Receiver $receiverId")
            self.send(RestartReceiver(receiver))
          }
      }(ThreadUtils.sameThread)
      logInfo(s"Receiver ${receiver.streamId} started")
    }

Note that the receiver is cleverly packaged into an RDD, and scheduledLocations is used as the RDD's preferred location locationPrefs.

Then submit a Spark Core Job through sc, the execution function is startReceiverFunc (that is, to be executed on the executor), create a ReceiverSupervisorImpl object in this method, and call the start() method, in which the receiver's method will be called. Returns immediately after onStart.

The receiver's onStart method generally creates a new thread or thread pool to receive data. For example, in KafkaReceiver, a new thread pool is created to receive the data of topics in the thread pool.

After supervisor.start() returns, the thread is blocked by supervisor.awaitTermination() so that this task does not exit, so that data can be continuously received.

Receiver data processing

As mentioned earlier, the receiver's onStart() method will create a new thread or thread pool to receive data. How to deal with the received data? Will call the receiver's store(), and the store method calls the supervisor's method. The corresponding store method has many forms:

• pushSingle: Corresponding to a single piece of small data, it is necessary to aggregate multiple pieces of data through BlockGenerator and then store them in blocks
• pushArrayBuffer: corresponds to the data in the form of an array
• pushIterator: corresponding to iterator form data
• pushBytes: corresponds to block data in the form of ByteBuffer

Except for pushSingle, which needs to be stored when the data is aggregated into a block through BlockGenerator, other methods are directly stored in blocks.

See how pushSingle stores blocks in an aggregated way:

def pushSingle(data: Any) {
    defaultBlockGenerator.addData(data)
  }
------
def addData(data: Any): Unit = {
    if (state == Active) {
      waitToPush()
      synchronized {
        if (state == Active) {
          currentBuffer += data
        } else {
          throw new SparkException(
            "Cannot add data as BlockGenerator has not been started or has been stopped")
        }
      }
    } else {
      throw new SparkException(
        "Cannot add data as BlockGenerator has not been started or has been stopped")
    }
  }

The first call here waitToPush()will check the rate by rateLimiter to prevent the addition too fast. If it is too fast, it will block and wait until the next second before adding. The number of records that can be added in one second is spark.streaming.receiver.maxRatecontrolled, that is, the number of records that a Receiver can add per second.
After checking, the data will be added to a variable-length array currentBuffer.

In addition, a timer is created when the BlockGenerator is initialized:

private val blockIntervalMs = conf.getTimeAsMs("spark.streaming.blockInterval", "200ms")
  require(blockIntervalMs > 0, s"'spark.streaming.blockInterval' should be a positive value")

  private val blockIntervalTimer =
    new RecurringTimer(clock, blockIntervalMs, updateCurrentBuffer, "BlockGenerator")

The default timing interval is 200ms, which can be spark.streaming.blockIntervalconfigured. The updateCurrentBuffer method is executed every time:

private def updateCurrentBuffer(time: Long): Unit = {
    try {
      var newBlock: Block = null
      synchronized {
        if (currentBuffer.nonEmpty) {
          val newBlockBuffer = currentBuffer
          currentBuffer = new ArrayBuffer[Any]
          val blockId = StreamBlockId(receiverId, time - blockIntervalMs)
          listener.onGenerateBlock(blockId)
          newBlock = new Block(blockId, newBlockBuffer)
        }
      }

      if (newBlock != null) {
        blocksForPushing.put(newBlock)  // put is blocking when queue is full
      }
    } catch {
      case ie: InterruptedException =>
        logInfo("Block updating timer thread was interrupted")
      case e: Exception =>
        reportError("Error in block updating thread", e)
    }
  }

• Assign currentBuffer to newBlockBuffer
• Re-allocate a new object for currentBuffer to store new data
• After encapsulating the currentBuffer as a Block, add it to blocksForPushing. blocksForPushing is a Queue with a default length of 10, which can be spark.streaming.blockQueueSizeconfigured by

When the BlockGenerator is initialized, it also starts a thread to take out the Block from the blocksForPushing queue and store the block through the supervisor:

private val blockPushingThread = new Thread() { override def run() { keepPushingBlocks() } }

supervisor storage data block

Store first and then report up:

#pushAndReportBlock
val blockStoreResult = receivedBlockHandler.storeBlock(blockId, receivedBlock)
logDebug(s"Pushed block $blockId in ${(System.currentTimeMillis - time)} ms")
val numRecords = blockStoreResult.numRecords
val blockInfo = ReceivedBlockInfo(streamId, numRecords, metadataOption, blockStoreResult) trackerEndpoint.askWithRetry[Boolean](AddBlock(blockInfo))

There is a corresponding receivedBlockHandler for storing data blocks. When WAL spark.streaming.receiver.writeAheadLog.enableis enabled (true), the corresponding is WriteAheadLogBasedBlockHandler (when WAL is enabled, data can be recovered from WAL after the application hangs), and when it is not enabled, the corresponding is BlockManagerBasedBlockHandler.

private val receivedBlockHandler: ReceivedBlockHandler = {
    if (WriteAheadLogUtils.enableReceiverLog(env.conf)) {
      if (checkpointDirOption.isEmpty) {
        throw new SparkException(
          "Cannot enable receiver write-ahead log without checkpoint directory set. " +
            "Please use streamingContext.checkpoint() to set the checkpoint directory. " +
            "See documentation for more details.")
      }
      new WriteAheadLogBasedBlockHandler(env.blockManager, env.serializerManager, receiver.streamId,
        receiver.storageLevel, env.conf, hadoopConf, checkpointDirOption.get)
    } else {
      new BlockManagerBasedBlockHandler(env.blockManager, receiver.storageLevel)
    }

Part of the code of the storeBlock method:

case ArrayBufferBlock(arrayBuffer) =>
    numRecords = Some(arrayBuffer.size.toLong)
    blockManager.putIterator(blockId, arrayBuffer.iterator, storageLevel,tellMaster = true)
case IteratorBlock(iterator) =>
    val countIterator = new CountingIterator(iterator)
    val putResult = blockManager.putIterator(blockId, countIterator, storageLevel,tellMaster = true)
    numRecords = countIterator.count
    putResult
case ByteBufferBlock(byteBuffer) =>
    blockManager.putBytes(blockId, new ChunkedByteBuffer(byteBuffer.duplicate()), storageLevel, tellMaster = true)

Both handlers store blocks to memory or disk through blockManager. Details of storage can be found in BlockManager analysis .

Notify ReceiverTracker

After the block is stored, an instance of ReceivedBlockInfo is created, which corresponds to some information about the block, including streamId (one InputDStream corresponds to one Receiver, and one Receiver corresponds to one streamId), the number of pieces of data in the block, storeResult and other information.

Then use the receivedBlockInfo as a parameter to communicate with the ReceiverTracker to send an AddBlock message. After the ReceiverTracker receives the message, the processing is as follows:

 case AddBlock(receivedBlockInfo) =>
        if (WriteAheadLogUtils.isBatchingEnabled(ssc.conf, isDriver = true)) {
          walBatchingThreadPool.execute(new Runnable {
            override def run(): Unit = Utils.tryLogNonFatalError {
              if (active) {
                context.reply(addBlock(receivedBlockInfo))
              } else {
                throw new IllegalStateException("ReceiverTracker RpcEndpoint shut down.")
              }
            }
          })
        } else {
          context.reply(addBlock(receivedBlockInfo))
        }

The addBlock(receivedBlockInfo) method will be called:

private def addBlock(receivedBlockInfo: ReceivedBlockInfo): Boolean = {
    receivedBlockTracker.addBlock(receivedBlockInfo)
  }

ReceiverTracker has a member receivedBlockTracker that specializes in managing blocks, and add block information through addBlock(receivedBlockInfo):

def addBlock(receivedBlockInfo: ReceivedBlockInfo): Boolean = {
    try {
      val writeResult = writeToLog(BlockAdditionEvent(receivedBlockInfo))
      if (writeResult) {
        synchronized {
          getReceivedBlockQueue(receivedBlockInfo.streamId) += receivedBlockInfo
        }
        logDebug(s"Stream ${receivedBlockInfo.streamId} received " +
          s"block ${receivedBlockInfo.blockStoreResult.blockId}")
      } else {
        logDebug(s"Failed to acknowledge stream ${receivedBlockInfo.streamId} receiving " +
          s"block ${receivedBlockInfo.blockStoreResult.blockId} in the Write Ahead Log.")
      }
      writeResult
    } catch {
      case NonFatal(e) =>
        logError(s"Error adding block $receivedBlockInfo", e)
        false
    }
  }

If WAL is enabled, the block information will be saved in WAL first, and then the block information will be saved streamIdToUnallocatedBlockQueuesmutable.HashMap[Int, ReceivedBlockQueue]in , where the key is the unique id of the InputDStream, and the value is the stored but unassigned block information. After assigning blocks to the batch, this structure is accessed to get the unconsumed blocks corresponding to each InputDStream.