NIO network communication model Kafka broker terminal

First, the overall framework outlined Kafka network communication model

Kafka network communication model is Reactor multithreaded model based NIO designed. Mainly the 1 (Acceptor thread 1) + N (N-threaded Processor number) + M (M a service processing threads) . The following table lists briefly below:

Threads	Thread name	Thread specify
1	kafka-socket-acceptor_%x	Acceptor threads, is responsible for monitoring Client-initiated request
N	kafka-network-thread_%d	Processor thread is responsible for reading and writing Socket
M	kafka-request-handler-_%d	Worker threads, processes, and generates a specific business logic returns Response

FIG complete framework Kafka network communication layers as shown below:

 

Communication layer model Kafka message queue -1 + N + M Model .png

Just beginning to see the top of the frame chart may be some do not understand, and it does not matter, I'll let you communicate on the network layer frame structure Kafka have a general understanding. Section later in this article important source Kafka binding to the above process in detail. Here you can briefly summarize a few important concepts that network communication model:
(1), Acceptor : 1 receiving a thread, is responsible for monitoring new connection requests, and registration OP_ACCEPT event, the new connection in accordance with the "round robin" way of the cross processor corresponding to threading;
(2), processor : N processor threads, each of which has its own Selector processor, which is assigned to the corresponding registered Acceptor SocketChannel OP_READ event, N is the size of the "num.networker .threads " decision;
(. 3), KafkaRequestHandler : M request processing thread within the thread pool comprising -KafkaRequestHandlerPool, -requestQueue request queue and to the data acquisition request from the global RequestChannel KafkaApis process, the size of M by " num. io.threads " decision;
(. 4), RequestChannel: Kafka channel which requested service is terminated, the data structure includes a global request queue and a plurality of local requestQueue Processor processor, responsive to the queue corresponding responseQueue, with the request supplied to the thread KafkaRequestHandler Processor KafkaApis and exchange data.
(. 5), NetworkClient : Is the underlying Java NIO for respective packages, the network interface layer located Kafka. Method Kafka send a message producer objects -KafkaProducer main NetworkClient call completion message;
(. 6), the SocketServer : NIO which is a service, it also starts a thread and a plurality of receiving Acceptor Processor processor thread. Reactor is provided a typical multi-thread mode, receives client request and a processing request phase separation;
(. 7), KafkaServer : Representative example of a Kafka Broker; which startup method is an instance of boot inlet;
(. 8), KafkaApis : Kafka business logic processing of Api, responsible for processing different types of requests; such as "send message" , "Get message -offset offset" and the "heartbeat process request" and the like;

Second, the design and realization of network communication layer Kafka

This section Kafka binding source network communication layer to analyze the design and implementation details herein introduces several important elements of the network communication layers -SocketServer, Acceptor, Processor, RequestChannel and KafkaRequestHandler. Source part of this analysis are based on the 0.11.0 version of Kafka.

1、SocketServer

Socket SocketServer receiving client requests a connection, the core handling the request and returns the result of the processing, the initialization of Acceptor and Processor, the processing logic is implemented here. When KafkaServer instance initialization method will start calling its startup, it will initialize an Acceptor and N Processor threads (EndPoint will initialize each, in general only a Server to set up a port), which is implemented as follows:

 

def startup() {
    this.synchronized {

      connectionQuotas = new ConnectionQuotas(maxConnectionsPerIp, maxConnectionsPerIpOverrides)

      val sendBufferSize = config.socketSendBufferBytes
      val recvBufferSize = config.socketReceiveBufferBytes
      val brokerId = config.brokerId

      var processorBeginIndex = 0
      // 一个broker一般只设置一个端口
      config.listeners.foreach { endpoint =>
        val listenerName = endpoint.listenerName
        val securityProtocol = endpoint.securityProtocol
        val processorEndIndex = processorBeginIndex + numProcessorThreads
        //N 个 processor
        for (i <- processorBeginIndex until processorEndIndex)
          processors(i) = newProcessor(i, connectionQuotas, listenerName, securityProtocol, memoryPool)
        //1个 Acceptor
        val acceptor = new Acceptor(endpoint, sendBufferSize, recvBufferSize, brokerId,
          processors.slice(processorBeginIndex, processorEndIndex), connectionQuotas)
        acceptors.put(endpoint, acceptor)
        KafkaThread.nonDaemon(s"kafka-socket-acceptor-$listenerName-$securityProtocol-${endpoint.port}", acceptor).start()
        acceptor.awaitStartup()

        processorBeginIndex = processorEndIndex
      }
    }

2、Acceptor

Acceptor is a thread class inherits from the abstract class AbstractServerThread. Acceptor main task is to monitor and receiving request from the client, while establishing a data transmission path -SocketChannel, then round-robin fashion to a rear end of the threading Processor (specific socketChannel is added to the queue and the concurrent processing thread wake Processor) .
In the main thread class can follow two important variables:
(. 1), nioSelector : by NSelector.open () method to create a variable that encapsulates the JAVA NIO Selector related operation;
(2), ServerChannel : for monitoring socket socket object server port;
Let's look at the main run method Acceptor source:

 

def run() {
    //首先注册OP_ACCEPT事件
    serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT)
    startupComplete()
    try {
      var currentProcessor = 0
      //以轮询方式查询并等待关注的事件发生
      while (isRunning) {
        try {
          val ready = nioSelector.select(500)
          if (ready > 0) {
            val keys = nioSelector.selectedKeys()
            val iter = keys.iterator()
            while (iter.hasNext && isRunning) {
              try {
                val key = iter.next
                iter.remove()
                if (key.isAcceptable)
                  //如果事件发生则调用accept方法对OP_ACCEPT事件处理
                  accept(key, processors(currentProcessor))
                else
                  throw new IllegalStateException("Unrecognized key state for acceptor thread.")
                //轮询算法
                // round robin to the next processor thread
                currentProcessor = (currentProcessor + 1) % processors.length
              } catch {
                case e: Throwable => error("Error while accepting connection", e)
              }
            }
          }
        }
       //代码省略
  }

  def accept(key: SelectionKey, processor: Processor) {
    val serverSocketChannel = key.channel().asInstanceOf[ServerSocketChannel]
    val socketChannel = serverSocketChannel.accept()
    try {
      connectionQuotas.inc(socketChannel.socket().getInetAddress)
      socketChannel.configureBlocking(false)
      socketChannel.socket().setTcpNoDelay(true)
      socketChannel.socket().setKeepAlive(true)
      if (sendBufferSize != Selectable.USE_DEFAULT_BUFFER_SIZE)
        socketChannel.socket().setSendBufferSize(sendBufferSize)

      processor.accept(socketChannel)
    } catch {
        //省略部分代码
    }
  }

  def accept(socketChannel: SocketChannel) {
    newConnections.add(socketChannel)
    wakeup()
  }

You can see in the source above, the Acceptor thread starts, the first event to be registered OP_ACCEPT on the server socket object -ServerSocketChannel for monitoring port. Then wait for round-robin fashion event of interest occurs. If the event occurs, the call accept () method OP_ACCEPT handle the event. Here, Processor by round robin selection process, this can ensure that load behind a plurality of substantially uniform Processor threads.
Acceptor action of accept () method is mainly as follows:
(1) acquired by the corresponding SelectionKey serverSocketChannel example, and call it the accept () method to establish a connection with the client;
(2) call connectionQuotas.inc () method for increasing the connection counting statistics; and also sets the first (1) created in step socketChannel property returns (e.g. sendBufferSize, KeepAlive, TcpNoDelay, configureBlocking, etc.)
(3) to the socketChannel processor.accept () method for processing. Here are the main socketChannel join Processor Processor concurrent queue newConnections queue, and then wake Processor thread gets socketChannel from the queue and processed. Which, newConnections will be concurrent threads and Processor Acceptor threads access operations, so newConnections is ConcurrentLinkedQueue queue (a unbounded thread-safe queue based on linked nodes)

3、Processor

Processor with Acceptor, just as a thread class that inherits the abstract class AbstractServerThread. The main data is read from the client's request and the response KafkaRequestHandler processed results back to the client. In the main thread class following several important variables:
(. 1), newConnections : above Acceptor one already mentioned, it is a type of queue ConcurrentLinkedQueue [SocketChannel], hold the new cross-connection socketChannel handled by the Processor;
(2), inflightResponses : Map is a collection [String, RequestChannel.Response] type, for recording the response has not been sent;
(. 3), Selector : KSelector type is a variable for managing network connection;
flowchart processor processor threads executing the run method is given to the following:

Thread processing flowchart .png Kafk_Processor

从上面的流程图中能够可以看出Processor处理器线程在其主流程中主要完成了这样子几步操作：
（1），处理newConnections队列中的socketChannel。遍历取出队列中的每个socketChannel并将其在selector上注册OP_READ事件；
（2），处理RequestChannel中与当前Processor对应响应队列中的Response。在这一步中会根据responseAction的类型（NoOpAction/SendAction/CloseConnectionAction）进行判断，若为“NoOpAction”，表示该连接对应的请求无需响应；若为“SendAction”，表示该Response需要发送给客户端，则会通过“selector.send”注册OP_WRITE事件，并且将该Response从responseQueue响应队列中移至inflightResponses集合中；“CloseConnectionAction”，表示该连接是要关闭的；
（3），调用selector.poll()方法进行处理。该方法底层即为调用nioSelector.select()方法进行处理。
（4），处理已接受完成的数据包队列—completedReceives。在processCompletedReceives方法中调用“requestChannel.sendRequest”方法将请求Request添加至requestChannel的全局请求队列—requestQueue中，等待KafkaRequestHandler来处理。同时，调用“selector.mute”方法取消与该请求对应的连接通道上的OP_READ事件；
（5），处理已发送完的队列—completedSends。当已经完成将response发送给客户端，则将其从inflightResponses移除，同时通过调用“selector.unmute”方法为对应的连接通道重新注册OP_READ事件；
（6），处理断开连接的队列。将该response从inflightResponses集合中移除，同时将connectionQuotas统计计数减1；

 

4、RequestChannel

在Kafka的网络通信层中，RequestChannel为Processor处理器线程与KafkaRequestHandler线程之间的数据交换提供了一个数据缓冲区，是通信过程中Request和Response缓存的地方。因此，其作用就是在通信中起到了一个数据缓冲队列的作用。Processor线程将读取到的请求添加至RequestChannel的全局请求队列—requestQueue中；KafkaRequestHandler线程从请求队列中获取并处理，处理完以后将Response添加至RequestChannel的响应队列—responseQueue中，并通过responseListeners唤醒对应的Processor线程，最后Processor线程从响应队列中取出后发送至客户端。

5、KafkaRequestHandler

KafkaRequestHandler也是一种线程类，在KafkaServer实例启动时候会实例化一个线程池—KafkaRequestHandlerPool对象（包含了若干个KafkaRequestHandler线程），这些线程以守护线程的方式在后台运行。在KafkaRequestHandler的run方法中会循环地从RequestChannel中阻塞式读取request，读取后再交由KafkaApis来具体处理。

6、KafkaApis

KafkaApis是用于处理对通信网络传输过来的业务消息请求的中心转发组件。该组件反映出Kafka Broker Server可以提供哪些服务