前言

在 Netty源码分析(3)-服务端新建连接处理的梳理中我们已经分析到新连接在 SubReactor 上的注册，众所周知，建立连接的目的是完成通信，而这个过程无非是将本地的数据发送到对端，或者接收对端的数据。因此本文将关注点聚集在 Netty 框架中业务数据的处理流程，整个分析过程分为了两部分：

读取数据的流程

写出数据的流程

在这里插入图片描述

1. 读取数据的处理

事件循环 NioEventLoop 启动后会处在空循环中不断轮询事件，对于 IO 就绪的事件会交给 NioEventLoop#processSelectedKey() 方法处理，这个方法中根据事件的不同又有不同的操作。我们已经知道新连接在SubReactor上注册后会监听 SelectionKey.OP_READ事件，则此处进入了 Unsafe#read()接口方法的调用

private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
      
      
     final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
    
     ......

     try {
      
      
         int readyOps = k.readyOps();
         // We first need to call finishConnect() before try to trigger a read(...) or write(...) as otherwise
         // the NIO JDK channel implementation may throw a NotYetConnectedException.
         if ((readyOps & SelectionKey.OP_CONNECT) != 0) {
      
      
             // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking
             // See https://github.com/netty/netty/issues/924
             int ops = k.interestOps();
             ops &= ~SelectionKey.OP_CONNECT;
             k.interestOps(ops);

             unsafe.finishConnect();
         }

         // Process OP_WRITE first as we may be able to write some queued buffers and so free memory.
         if ((readyOps & SelectionKey.OP_WRITE) != 0) {
      
      
             // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write
             ch.unsafe().forceFlush();
         }

         // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
         // to a spin loop
         if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
      
      
             unsafe.read();
         }
     } catch (CancelledKeyException ignored) {
      
      
         unsafe.close(unsafe.voidPromise());
     }
 }

Unsafe#read() 的实现有两个，因为在 SubReactor 上注册的是 NioSocketChannel，故此时调用的是 NioByteUnsafe#read() 方法。这个方法的核心是以下 3 个步骤：

doReadBytes(byteBuf) 从 Socket 读取数据到 ByteBuf 缓冲中

pipeline.fireChannelRead(byteBuf) 将 ByteBuf 缓冲对象入参，通知处理器处理读取到的数据

while 循环结束，pipeline.fireChannelReadComplete() 通知处理器读取数据完成事件

public final void read() {
      
      
        final ChannelConfig config = config();
        if (shouldBreakReadReady(config)) {
      
      
            clearReadPending();
            return;
        }
        final ChannelPipeline pipeline = pipeline();
        final ByteBufAllocator allocator = config.getAllocator();
        final RecvByteBufAllocator.Handle allocHandle = recvBufAllocHandle();
        allocHandle.reset(config);

        ByteBuf byteBuf = null;
        boolean close = false;
        try {
      
      
            do {
      
      
                byteBuf = allocHandle.allocate(allocator);
                allocHandle.lastBytesRead(doReadBytes(byteBuf));
                if (allocHandle.lastBytesRead() <= 0) {
      
      
                    // nothing was read. release the buffer.
                    byteBuf.release();
                    byteBuf = null;
                    close = allocHandle.lastBytesRead() < 0;
                    if (close) {
      
      
                        // There is nothing left to read as we received an EOF.
                        readPending = false;
                    }
                    break;
                }

                allocHandle.incMessagesRead(1);
                readPending = false;
                pipeline.fireChannelRead(byteBuf);
                byteBuf = null;
            } while (allocHandle.continueReading());

            allocHandle.readComplete();
            pipeline.fireChannelReadComplete();

            ......
            
        }
    }

NioSocketChannel#doReadBytes()方法的实现很简单，主要通过 ByteBuf#writeBytes() 方法把 Channel 和数据长度入参，从 Channel 中读取对应长度的数据到 ByteBuf 对象

protected int doReadBytes(ByteBuf byteBuf) throws Exception {
      
      
     final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
     allocHandle.attemptedBytesRead(byteBuf.writableBytes());
     return byteBuf.writeBytes(javaChannel(), allocHandle.attemptedBytesRead());
 }

DefaultPipeline#fireChannelRead() 方法逻辑也很简单，主要是从处理器双向链表的 head 头节点开始，从前往后传播读取事件，将读取到的数据交给入站处理器处理
```
public final ChannelPipeline fireChannelRead(Object msg) {
      
      
     AbstractChannelHandlerContext.invokeChannelRead(head, msg);
     return this;
 }
```

所有处理器都被封装在一个 ChannelHandlerContext中，当读取事件开始传播时AbstractChannelHandlerContext#invokeChannelRead()方法将被调用，方法体内把读取到的数据交给处理器相应的方法处理，并由其决定是否将事件继续往下传播，此后的流程不再具体分析

 private void invokeChannelRead(Object msg) {
      
      
     if (invokeHandler()) {
      
      
         try {
      
      
             ((ChannelInboundHandler) handler()).channelRead(this, msg);
         } catch (Throwable t) {
      
      
             notifyHandlerException(t);
         }
     } else {
      
      
         fireChannelRead(msg);
     }
 }

2. 写出数据的处理

写出数据可由 AbstractChannelHandlerContext#writeAndFlush()触发，这个方法其实最后调用到了AbstractChannelHandlerContext#write() 方法，这个方法的执行有两个关键步骤：

findContextOutbound() 从处理器双向链表中当前处理器的位置由后往前查找处理出站事件的处理器

next.invokeWriteAndFlush(m, promise) 将写出的数据交给查找到的处理器处理。可以看到 invokeWriteAndFlush() 方法其实也分为了两步：

invokeWrite0(msg, promise) 将需要写出的数据缓存在 ChannelOutboundBuffer 维护的单向链表中

invokeFlush0() 将 ChannelOutboundBuffer 中缓存的数据写出到 Socket

private void write(Object msg, boolean flush, ChannelPromise promise) {
      
      
     AbstractChannelHandlerContext next = findContextOutbound();
     final Object m = pipeline.touch(msg, next);
     EventExecutor executor = next.executor();
     if (executor.inEventLoop()) {
      
      
         if (flush) {
      
      
             next.invokeWriteAndFlush(m, promise);
         } else {
      
      
             next.invokeWrite(m, promise);
         }
     } else {
      
      
         final AbstractWriteTask task;
         if (flush) {
      
      
             task = WriteAndFlushTask.newInstance(next, m, promise);
         }  else {
      
      
             task = WriteTask.newInstance(next, m, promise);
         }
         if (!safeExecute(executor, task, promise, m)) {
      
      
             // We failed to submit the AbstractWriteTask. We need to cancel it so we decrement the pending bytes
             // and put it back in the Recycler for re-use later.
             //
             // See https://github.com/netty/netty/issues/8343.
             task.cancel();
         }
     }
 }

private void invokeWriteAndFlush(Object msg, ChannelPromise promise) {
      
      
     if (invokeHandler()) {
      
      
         invokeWrite0(msg, promise);
         invokeFlush0();
     } else {
      
      
         writeAndFlush(msg, promise);
     }
 }

查找出站处理器最后肯定会找到 HeadContext，则invokeWrite0()写数据操作最后会调用到 HeadContext#write() 方法

public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
      
      
         unsafe.write(msg, promise);
     }

AbstractUnsafe#write() 方法实现中可以看到需要写出的数据其实是被 ChannelOutboundBuffer#addMessage() 方法缓存了起来

public final void write(Object msg, ChannelPromise promise) {
      
      
         assertEventLoop();

         ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
         
         ......

         outboundBuffer.addMessage(msg, size, promise);
     }

ChannelOutboundBuffer#addMessage() 方法把写出数据封装成一个 Entry 对象，并将原来的 tailEntry 对象替换为新建的 Entry 对象，形成一个单向连接

此处需要说明 ChannelOutboundBuffer 对写出数据的缓存策略，写出的数据会被包装为一个 Entry 对象，并通过尾指针连接成一个单向链表。ChannelOutboundBuffer 对象通过三个关键指针操作这个单向链表

flushedEntry 指向已经被 flush 的节点中的最后一个节点

unflushedEntry 指向未被 flush 的节点中的第一个节点

tailEntry 指向单向链表尾部

public void addMessage(Object msg, int size, ChannelPromise promise) {
      
      
     Entry entry = Entry.newInstance(msg, size, total(msg), promise);
     if (tailEntry == null) {
      
      
         flushedEntry = null;
     } else {
      
      
         Entry tail = tailEntry;
         tail.next = entry;
     }
     tailEntry = entry;
     if (unflushedEntry == null) {
      
      
         unflushedEntry = entry;
     }

     // increment pending bytes after adding message to the unflushed arrays.
     // See https://github.com/netty/netty/issues/1619
     incrementPendingOutboundBytes(entry.pendingSize, false);
 }

invokeFlush0()刷出数据最后也会调用到 HeadContext#flush()方法

public void flush(ChannelHandlerContext ctx) throws Exception {
      
      
         unsafe.flush();
     }

AbstractUnsafe#flush() 方法实现中主要包含两个步骤：

outboundBuffer.addFlush() 将 flushedEntry 指针指向 unflushedEntry 所在的 Entry 对象，并对刷出的 Entry 对象计数

flush0() 开始将数据写出到 Socket

public final void flush() {
      
      
         assertEventLoop();

         ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
         if (outboundBuffer == null) {
      
      
             return;
         }

         outboundBuffer.addFlush();
         flush0();
     }

AbstractUnsafe#flush0() 方法实际完成写出数据的操作，这个过程主要借助了 AbstractUnsafe#doWrite() 方法，而这个抽象方法的实现是NioSocketChannel#doWrite()。如下代码可以看出来，Netty 根据写出数据的不同也会有不同的写出数据处理：

ChannelOutboundBuffer 对象中没有 ByteBuffer 对象类型的数据时，调用 doWrite0() 写出数据

ChannelOutboundBuffer 对象中有 ByteBuffer 对象类型的数据是，直接调用 SocketChannel#write() 写出数据

如果一个自旋周期内没有把数据写完，则调用 incompleteWrite() 设置 Channel 上的操作监听位为 SelectionKey.OP_WRITE，监听到可写事件之后可以继续写

protected void doWrite(ChannelOutboundBuffer in) throws Exception {
      
      
     SocketChannel ch = javaChannel();
     int writeSpinCount = config().getWriteSpinCount();
     do {
      
      
         if (in.isEmpty()) {
      
      
             // All written so clear OP_WRITE
             clearOpWrite();
             // Directly return here so incompleteWrite(...) is not called.
             return;
         }

         // Ensure the pending writes are made of ByteBufs only.
         int maxBytesPerGatheringWrite = ((NioSocketChannelConfig) config).getMaxBytesPerGatheringWrite();
         ByteBuffer[] nioBuffers = in.nioBuffers(1024, maxBytesPerGatheringWrite);
         int nioBufferCnt = in.nioBufferCount();

         // Always us nioBuffers() to workaround data-corruption.
         // See https://github.com/netty/netty/issues/2761
         switch (nioBufferCnt) {
      
      
             case 0:
                 // We have something else beside ByteBuffers to write so fallback to normal writes.
                 writeSpinCount -= doWrite0(in);
                 break;
             case 1: {
      
      
                 // Only one ByteBuf so use non-gathering write
                 // Zero length buffers are not added to nioBuffers by ChannelOutboundBuffer, so there is no need
                 // to check if the total size of all the buffers is non-zero.
                 ByteBuffer buffer = nioBuffers[0];
                 int attemptedBytes = buffer.remaining();
                 final int localWrittenBytes = ch.write(buffer);
                 if (localWrittenBytes <= 0) {
      
      
                     incompleteWrite(true);
                     return;
                 }
                 adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
                 in.removeBytes(localWrittenBytes);
                 --writeSpinCount;
                 break;
             }
             default: {
      
      
                 // Zero length buffers are not added to nioBuffers by ChannelOutboundBuffer, so there is no need
                 // to check if the total size of all the buffers is non-zero.
                 // We limit the max amount to int above so cast is safe
                 long attemptedBytes = in.nioBufferSize();
                 final long localWrittenBytes = ch.write(nioBuffers, 0, nioBufferCnt);
                 if (localWrittenBytes <= 0) {
      
      
                     incompleteWrite(true);
                     return;
                 }
                 // Casting to int is safe because we limit the total amount of data in the nioBuffers to int above.
                 adjustMaxBytesPerGatheringWrite((int) attemptedBytes, (int) localWrittenBytes,
                         maxBytesPerGatheringWrite);
                 in.removeBytes(localWrittenBytes);
                 --writeSpinCount;
                 break;
             }
         }
     } while (writeSpinCount > 0);

     incompleteWrite(writeSpinCount < 0);
 }

AbstractNioByteChannel#doWrite0() 最终调用 AbstractNioByteChannel#doWriteInternal() 来完成写出数据，其关键过程分为两步：

doWriteBytes(buf) 将数据写到 Socket

in.remove() 将 ChannelOutboundBuffer 维护的单向链表中的 flushedEntry 往后移动，flushedEntry 指针之前的链表节点即为已经刷出的节点

private int doWriteInternal(ChannelOutboundBuffer in, Object msg) throws Exception {
      
      
     if (msg instanceof ByteBuf) {
      
      
         ByteBuf buf = (ByteBuf) msg;
         if (!buf.isReadable()) {
      
      
             in.remove();
             return 0;
         }

         final int localFlushedAmount = doWriteBytes(buf);
         if (localFlushedAmount > 0) {
      
      
             in.progress(localFlushedAmount);
             if (!buf.isReadable()) {
      
      
                 in.remove();
             }
             return 1;
         }
     } else if (msg instanceof FileRegion) {
      
      
         FileRegion region = (FileRegion) msg;
         if (region.transferred() >= region.count()) {
      
      
             in.remove();
             return 0;
         }

         long localFlushedAmount = doWriteFileRegion(region);
         if (localFlushedAmount > 0) {
      
      
             in.progress(localFlushedAmount);
             if (region.transferred() >= region.count()) {
      
      
                 in.remove();
             }
             return 1;
         }
     } else {
      
      
         // Should not reach here.
         throw new Error();
     }
     return WRITE_STATUS_SNDBUF_FULL;
 }

NioSocketChannel#doWriteBytes() 实现如下，可以看到其通过 ByteBuff#readBytes() 将 Channel 和数据长度入参，Channel 会将 ByteBuff 对象中的数据写入 Socket，至此写出数据流程结束

 protected int doWriteBytes(ByteBuf buf) throws Exception {
      
      
     final int expectedWrittenBytes = buf.readableBytes();
     return buf.readBytes(javaChannel(), expectedWrittenBytes);
 }

Netty源码分析(4)- SubReactor 读写数据流程解析

文章目录

前言

1. 读取数据的处理

2. 写出数据的处理

猜你喜欢