Netty版本4.1.22
前面在分析NioEventLoop，重点在run方法，将其分为三部分来讲解：select，processSelectedKeys，runAllTasks。在第二部分里实际处理SelectionKey的操作processSelectedKey并没有分析，这篇文章讲连接，其切入点就在该方法。
在第一篇文章分析服务端启动时，ServerBootStrap.bind创建了NioServerSocketChannel，经过初始化；注册；绑定。关于绑定：从bossGroup中选择了一个NioEventLoop与其绑并绑定端口，Netty将实际绑定端口操作交给NioEventLoop的线程来执行，在这里第一次调用了channel.eventLoop().execute()，该方法在NioEventLoop线程未创建时创建线程，并开始在NioEventLoop的run方法里不断循环执行，也就是第二，三篇文章分析的主要内容。这就是连接到来前的情况
channel注册到Selector，监听其感兴趣的事件，当事件发生，其SelectionKey会被放入到sun.nio.ch.SelectorImpl的selectedKeys与publicSelectedKeys，而这两个set已经被Netty里的SelectedSelectionKeySet 代替，NioEventLoop的selectedKeys成员变量指向它，所以事件的SelectionKey就通过该字段来获取，处理SelectionKey的方法就是run的第二步processSelectedKeys，前面文章之分析了部分，实际处理操作processSelectedKey并没有分析。
processSelectedKey
private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {
final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();
......
try {
int readyOps = k.readyOps();
......
// Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead
// to a spin loop
if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {
unsafe.read();
}
} catch (CancelledKeyException ignored) {
unsafe.close(unsafe.voidPromise());
}
}
当连接到来NioServerSocketChannel监听的SelectionKey.OP_ACCEPT事件发生，执行unsafe.read();
关于unsafe，实际的I/O操作基本上都是由Unsafe功能类负责实现的，在AbstractChannel中创建Unsafe，具体实现由子类决定，比如NioServerSocketChannel创建的就是NioMessageUnsafe。关于Unsafe之后文章再分析。
NioMessageUnsafe : AbstractNioMessageChannel的内部类
private final List<Object> readBuf = new ArrayList<Object>();
public void read() {
assert eventLoop().inEventLoop(); // 必须是NioEventLoop线程
// config ，pipeline 都是NioServerSocketChannel的组件
final ChannelConfig config = config();
final ChannelPipeline pipeline = pipeline();
final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();
allocHandle.reset(config);
boolean closed = false;
Throwable exception = null;
try {
try {
do {
int localRead = doReadMessages(readBuf);
if (localRead == 0) {
break;
}
if (localRead < 0) {
closed = true;
break;
}
allocHandle.incMessagesRead(localRead);
} while (allocHandle.continueReading());
} catch (Throwable t) {
exception = t;
}
int size = readBuf.size();
for (int i = 0; i < size; i ++) {
readPending = false;
pipeline.fireChannelRead(readBuf.get(i));
}
readBuf.clear(); // 清理容器
allocHandle.readComplete();
pipeline.fireChannelReadComplete();
......
RecvByteBufAllocator.Handle之后分析，
doReadMessages 创建NioSocketChannel加入readBuf，readBuf作为容器存储NioSocketChannel。
pipeline.fireChannelRead主要是调用ServerBootstrapAcceptor的channelRead方法。
接下来就doReadMessages与fireChannelRead进行分析
1， doReadMessages
protected int doReadMessages(List<Object> buf) throws Exception {
SocketChannel ch = SocketUtils.accept(javaChannel());
try {
if (ch != null) {
buf.add(new NioSocketChannel(this, ch));
return 1;
}
......
第一行就是调用java.nio.channels.ServerSocketChannel.accept()，这里立即返回，因为已经监听到连接请求，得到java.nio.channels.SocketChannel后将其封装成Netty的NioSocketChannel，并加入List<Object> readBuf。
接下来看看NioSocketChannel的初始化过程
public NioSocketChannel(Channel parent, SocketChannel socket) {
super(parent, socket);
config = new NioSocketChannelConfig(this, socket.socket());
}
config 之后分析
protected AbstractNioByteChannel(Channel parent, SelectableChannel ch) {
super(parent, ch, SelectionKey.OP_READ);
}
监听事件为SelectionKey.OP_READ
protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {
super(parent);
this.ch = ch;
this.readInterestOp = readInterestOp;
try {
ch.configureBlocking(false);
......
protected AbstractChannel(Channel parent) {
this.parent = parent;
id = newId();
unsafe = newUnsafe();
pipeline = newChannelPipeline();
}
创建channel组件，config, unsage, id, pipeline。
来看看Channel的部分继承图
*之后的问题就是什么时候注册与绑定？*往下看
2，pipeline.fireChannelRead
该pipeline是NioServerSocketChannel的pipeline，fireChannelRead触发pipeline里handler的channelRead方法。
pipeline可以看作是ChannelHandler的实例链，该方法让Channel流经这条链。关于它之后文章在介绍。
那么此时NioServerSocketChannel的pipeline上的有哪些handler？
还有对于包装了java.nio.channels.SocketChannel的NioSocketChannel，它的初始化，注册过程在哪发生？对应的就是你的代码里对SocketChannel设置的options，attrs，childHandler什么时候关联？
回到NioServerSocketChannel的初始化init阶段，定位到ServerBootStrap的init()方法
......
ChannelPipeline p = channel.pipeline();
final EventLoopGroup currentChildGroup = childGroup;
final ChannelHandler currentChildHandler = childHandler;
final Entry<ChannelOption<?>, Object>[] currentChildOptions;
final Entry<AttributeKey<?>, Object>[] currentChildAttrs;
synchronized (childOptions) {
currentChildOptions = childOptions.entrySet()
.toArray(newOptionArray(childOptions.size()));
}
synchronized (childAttrs) {
currentChildAttrs = childAttrs.entrySet()
.toArray(newAttrArray(childAttrs.size()));
}
p.addLast(new ChannelInitializer<Channel>() {
@Override
public void initChannel(final Channel ch) throws Exception {
final ChannelPipeline pipeline = ch.pipeline();
ChannelHandler handler = config.handler();
if (handler != null) {
pipeline.addLast(handler);
}
ch.eventLoop().execute(new Runnable() {
@Override
public void run() {
pipeline.addLast(new ServerBootstrapAcceptor(
ch, currentChildGroup, currentChildHandler,
currentChildOptions, currentChildAttrs));
}
});
}
});
当服务端启动逻辑运行到这NioServerSocketChannel的pipeline此时会有3个handler，除了head，tail（pipeline初始化时创建的两个handlerContext），中间的就是ChannelInitializer，head->ChannelInitializer->tail。
ChannelInitializer extends ChannelInboundHandlerAdapter，当它的initChannel被触发后首先将NioServerSocketChannel的handler加入到NioServerSocketChannel的pipeline(如果你设置了的话)；
ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter保存NioSocketChannel的workerGroup，ChildHandler，ChildOptions和ChildAttrs，它被加到NioServerSocketChannel的pipline里的操作被封装成一个Runnable交由NioServerSocketChannel所属NioEventLoop的线程执行，
为什么？为了确保线程安全，NioEventLoop可能处理多个channel，每个channel都有自己的组件，如pipeline，NioEventLoop是单线程的，即这些channel的pipeline的变化只由该NioEventLoop的线程来处理，单线程便没有了多线程并发的问题。
当initChannel被触发后NioServerSocketChannel的pipeline会有三个或四个handler，
head->[handler->]ServerBootstrapAcceptor->tail
什么时候触发initChannel？
pipeline.invokeHandlerAddedIfNeeded()会触发ChannelInitializer的initChannel。
pipeline.invokeHandlerAddedIfNeeded()两个地方被调用，一个是
DefaultChannelPipeline：
public void channelRegistered(ChannelHandlerContext ctx) throws Exception {
invokeHandlerAddedIfNeeded();
ctx.fireChannelRegistered();
}
在channel注册完成后调用pipeline.fireChannelRegistered();触发channelRegistered
另一个就是在channel的注册阶段，定位到AbstractChannel的register0里
private void register0(ChannelPromise promise) {
try {
......
boolean firstRegistration = neverRegistered;
doRegister();
neverRegistered = false;
registered = true;
pipeline.invokeHandlerAddedIfNeeded();
safeSetSuccess(promise);
//触发上面的channelRegistered
pipeline.fireChannelRegistered();
......
对于invokeHandlerAddedIfNeeded，只在第一次被调用时调用callHandlerAddedForAllHandlers
DefaultChannelPipeline：
private boolean firstRegistration = true; //用来确保只被调用一次
final void invokeHandlerAddedIfNeeded() {
assert channel.eventLoop().inEventLoop();
if (firstRegistration) {
firstRegistration = false;
callHandlerAddedForAllHandlers();
}
}
firstRegistration：用来确保callHandlerAddedForAllHandlers只被调用一次
private void callHandlerAddedForAllHandlers() {
final PendingHandlerCallback pendingHandlerCallbackHead;
synchronized (this) {
// 能到此说明与pipeline关联的channel之前未注册过
assert !registered;
// 用来标识与pipeline关联的channel已注册
registered = true;
pendingHandlerCallbackHead = this.pendingHandlerCallbackHead;
// Null out so it can be GC'ed.
this.pendingHandlerCallbackHead = null;
}
// This must happen outside of the synchronized(...) block as otherwise handlerAdded(...) may be called while
// holding the lock and so produce a deadlock if handlerAdded(...) will try to add another handler from outside
// the EventLoop.
PendingHandlerCallback task = pendingHandlerCallbackHead;
while (task != null) {
task.execute();
task = task.next;
}
}
registered ：用来标识与pipeline关联的channel已注册，更改为true，之后取得PendingHandlerCallback任务链表的头，循环执行链表上的任务。
上面的设计防止了死锁也保证了this.pendingHandlerCallbackHead与registered的线程安全。
这个任务队列是怎么回事？
this.pendingHandlerCallbackHead是PendingHandlerCallback类型链表的头节点
/**
* This is the head of a linked list that is processed by {@link #callHandlerAddedForAllHandlers()} and so process
* all the pending {@link #callHandlerAdded0(AbstractChannelHandlerContext)}.
*
* We only keep the head because it is expected that the list is used infrequently and its size is small.
* Thus full iterations to do insertions is assumed to be a good compromised to saving memory and tail management
* complexity.
*/
private PendingHandlerCallback pendingHandlerCallbackHead;
PendingHandlerCallback
private abstract static class PendingHandlerCallback implements Runnable {
final AbstractChannelHandlerContext ctx;
PendingHandlerCallback next;
PendingHandlerCallback(AbstractChannelHandlerContext ctx) {
this.ctx = ctx;
}
abstract void execute();
}
两个实现PendingHandlerAddedTask和PendingHandlerRemovedTask分别对应pipeline的handlerContext节点（pipeline可以看作是ChannelHandler的实例链，但其时该链的节点类型为AbstractChannelHandlerContext）的添加和删除。
以添加addLast方法为例：在handler添加到pipeline后会执行callHandlerAdded0(newCtx)调用该handler的handlerAdded方法，如ChannelInitializer它的handlerAdded方法会调用initChannel，但前提时该channel已经注册到其关联的pipeline上，即registered标识为true，而这个DefaultChannelPipeline.registered变量只在callHandlerAddedForAllHandlers方法里被改为true，如上分析该方法在channel注册后调用。不满足则调用callHandlerCallbackLater将newCtx封装成PendingHandlerAddedTask，待注册成功后再调用callHandlerAdded0(newCtx)
来从代码层面看上面这段：
DefaultChannelPipeline的addFirst，addLast，addBefore，addAfter，remove，replace在registered为false时会调用callHandlerCallbackLater方法
DefaultChannelPipeline：
private void callHandlerCallbackLater(AbstractChannelHandlerContext ctx,
boolean added) {
assert !registered;
PendingHandlerCallback task = added ? new PendingHandlerAddedTask(ctx)
: new PendingHandlerRemovedTask(ctx);
PendingHandlerCallback pending = pendingHandlerCallbackHead;
if (pending == null) {
pendingHandlerCallbackHead = task;
} else {
// Find the tail of the linked-list.
while (pending.next != null) {
pending = pending.next;
}
pending.next = task;
}
}
boolean added参数为true表明是添加任务，false是删除任务。该方法就是封装AbstractChannelHandlerContext为PendingHandlerCallback再添加到链表尾部。
以addLast为例
public final ChannelPipeline addLast(EventExecutorGroup group,
String name, ChannelHandler handler) {
final AbstractChannelHandlerContext newCtx;
synchronized (this) {
checkMultiplicity(handler);
newCtx = newContext(group, filterName(name, handler), handler);
addLast0(newCtx);
// registered 为false表明channel并没有注册到它的pipeline上，
// 那么就将newCtx封装成一个任务加到任务链表，当channel被注册后触发
// DefaultChannelPipeline.channelRegistered，
// ChannelHandler.handlerAdded被调用
if (!registered) {
//将AbstractChannelHandlerContext的handlerState
// 由INIT改为ADD_PENDING
newCtx.setAddPending();
callHandlerCallbackLater(newCtx, true);
return this;
}
调用callHandlerCallbackLater后将newCtx封装成PendingHandlerAddedTask
private final class PendingHandlerAddedTask extends PendingHandlerCallback {
PendingHandlerAddedTask(AbstractChannelHandlerContext ctx) {
super(ctx);
}
@Override
public void run() {
callHandlerAdded0(ctx);
}
@Override
void execute() {
EventExecutor executor = ctx.executor();
if (executor.inEventLoop()) {
callHandlerAdded0(ctx);
} else {
try {
executor.execute(this);
} catch (RejectedExecutionException e) {
if (logger.isWarnEnabled()) {
logger.warn(
"Can't invoke handlerAdded() as the EventExecutor {} rejected it, removing handler {}.",
executor, ctx.name(), e);
}
remove0(ctx);
ctx.setRemoved();
}
}
}
}
其execute执行的是callHandlerAdded0(ctx)方法。channel注册后该execute被调用，callHandlerAdded0(ctx);得以执行。
callHandlerAdded0(ctx)触发ChannelInitializer的initChannel，过程如下
private void callHandlerAdded0(final AbstractChannelHandlerContext ctx) {
try {
// We must call setAddComplete before calling handlerAdded.
// Otherwise if the handlerAdded method generates
// any pipeline events ctx.handler() will miss them
// because the state will not allow it.
ctx.setAddComplete();
ctx.handler().handlerAdded(ctx);
} catch (Throwable t) {
ctx.setAddComplete更改该handler的handlerContext的状态handlerState
AbstractChannelHandlerContext：
final void setAddComplete() {
for (;;) {
int oldState = handlerState;
// Ensure we never update when the handlerState is REMOVE_COMPLETE already.
// oldState is usually ADD_PENDING but can also be REMOVE_COMPLETE when an EventExecutor is used that is not
// exposing ordering guarantees.
if (oldState == REMOVE_COMPLETE || HANDLER_STATE_UPDATER.compareAndSet(this, oldState, ADD_COMPLETE)) {
return;
}
}
}
ctx.handler().handlerAdded(ctx)方法调用handler的handlerAdded，如ChannelInitializer
ChannelInitializer：
public void handlerAdded(ChannelHandlerContext ctx) throws Exception {
if (ctx.channel().isRegistered()) {
// This should always be true with our current DefaultChannelPipeline implementation.
// The good thing about calling initChannel(...) in handlerAdded(...) is that there will be no ordering
// surprises if a ChannelInitializer will add another ChannelInitializer. This is as all handlers
// will be added in the expected order.
initChannel(ctx);
}
}
private boolean initChannel(ChannelHandlerContext ctx) throws Exception {
// 防止重入，确保initChannel只被调用一次
if (initMap.putIfAbsent(ctx, Boolean.TRUE) == null) { // Guard against re-entrance.
try {
initChannel((C) ctx.channel());
} catch (Throwable cause) {
// Explicitly call exceptionCaught(...) as we removed the handler before calling initChannel(...).
// We do so to prevent multiple calls to initChannel(...).
exceptionCaught(ctx, cause);
} finally {
remove(ctx);
}
return true;
}
return false;
}
protected abstract void initChannel(C ch) throws Exception;
总结一下NioServerSocketChannel的pipeline的情况，在NioServerSocketChannel初始化init阶段有三个handler，head->ChannelInitializer->tail，在之后的注册阶段 ChannelInitializer的initChannel被调用，此时pipeline变为head->[handler->]ServerBootstrapAcceptor->tail，这里ServerBootstrapAcceptor加入操作是交给NioServerSocketChannel所属NioEventLoop的线程来执行，这里NioEventLoop的execute第一次执行，线程启动，注册后channel所属的NioEventLoop的线程被启动，若之前未创建的话。
前面是铺垫，把思绪拉回到pipeline.fireChannelRead(NioSocketChannel)
pipeline.fireChannelRead(NioSocketChannel)调用，会触发pipeline里各个handler的channelRead，ServerBootstrapAcceptor 的channelRead被调用，这里会执行NioSocketChannel的注册与绑定，pipeline指的是与NioServerSocketChannel关联的那个。
下面来分析下ServerBootstrapAcceptor
private static class ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter {
private final EventLoopGroup childGroup;
private final ChannelHandler childHandler;
private final Entry<ChannelOption<?>, Object>[] childOptions;
private final Entry<AttributeKey<?>, Object>[] childAttrs;
private final Runnable enableAutoReadTask;
ServerBootstrapAcceptor(
final Channel channel, EventLoopGroup childGroup, ChannelHandler childHandler,
Entry<ChannelOption<?>, Object>[] childOptions, Entry<AttributeKey<?>, Object>[] childAttrs) {
this.childGroup = childGroup;
this.childHandler = childHandler;
this.childOptions = childOptions;
this.childAttrs = childAttrs;
// Task which is scheduled to re-enable auto-read.
// It's important to create this Runnable before we try to submit it as otherwise the URLClassLoader may
// not be able to load the class because of the file limit it already reached.
//
// See https://github.com/netty/netty/issues/1328
enableAutoReadTask = new Runnable() {
@Override
public void run() {
channel.config().setAutoRead(true);
}
};
}
@Override
@SuppressWarnings("unchecked")
public void channelRead(ChannelHandlerContext ctx, Object msg) {
final Channel child = (Channel) msg;
child.pipeline().addLast(childHandler);
setChannelOptions(child, childOptions, logger);
for (Entry<AttributeKey<?>, Object> e: childAttrs) {
child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());
}
try {
childGroup.register(child).addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
if (!future.isSuccess()) {
forceClose(child, future.cause());
}
}
});
} catch (Throwable t) {
forceClose(child, t);
}
}
......
该方法首先将childHandler加到NioSocketChannel的pipeline里，这里的addLast操作并不会触发childHandler的initChannel，因为NioSocketChannel还未注册；之后处理options，attrs，之后调用childGroup.register(child)，该注册流程与之前分析的NioServerSocketChannel的注册流程相同。
来跟下注册过程：
首先childGroup.register(child)就是从workerGroup所管理的NioEventLoop数组中利用chooser选择一个NioEventLoop，调用其register方法将其绑定在该NioSocketChannel上，绑定指的是将其赋给该channel，赋值语句是AbstractUnsafe的register方法AbstractChannel.this.eventLoop = eventLoop;
下面从eventLoop的register开始跟
SingleThreadEventLoop:
public ChannelFuture register(Channel channel) {
return register(new DefaultChannelPromise(channel, this));
}
@Override
public ChannelFuture register(final ChannelPromise promise) {
ObjectUtil.checkNotNull(promise, "promise");
promise.channel().unsafe().register(this, promise);
return promise;
}
promise.channel()指的是NioSocketChannel，其unsafe在AbstractChannel构造器里里调用newUnsafe()方法得到，该方法在NioSocketChannel中重写
protected AbstractNioUnsafe newUnsafe() {
return new NioSocketChannelUnsafe();
}
private final class NioSocketChannelUnsafe extends NioByteUnsafe {
其register方法实现在AbstractUnsafe
public final void register(EventLoop eventLoop, final ChannelPromise promise) {
......
AbstractChannel.this.eventLoop = eventLoop;
if (eventLoop.inEventLoop()) {
register0(promise);
} else {
try {
eventLoop.execute(new Runnable() {
@Override
public void run() {
register0(promise);
}
});
} catch (Throwable t) {
logger.warn(
"Force-closing a channel whose registration
task was not accepted by an event loop: {}",
AbstractChannel.this, t);
closeForcibly();
closeFuture.setClosed();
safeSetFailure(promise, t);
}
}
}
上面代码eventLoop.inEventLoop()会先判断下运行中的线程是否是NioSocketChannel所属的NioEventLoop的线程。之前文章分析说NioEventLoop的线程是在eventLoop.execute第一次被调用时创建，具体实现在SingleThreadEventExecutor。那么问题来了是谁的线程再执行？
顺一遍代码到此处前发生了什么：NioServerSocketChannel所属的NioEventLoop的线程在NioEventLoop的run方法里不断循环着，当连接事件发生，该SelectionKey会被放入到NioEventLoop的SelectedSelectionKeySet selectedKeys的keys数组中，processSelectedKey处理SelectionKey，此处是SelectionKey.OP_ACCEPT，调用unsafe.read();
这里unsafe指的是？ 之前分析过，与SelectionKey的attachment有关，NioServerSocketChannel在注册到Selector时是调用AbstractNioChannel的doRegister()：重点在javaChannel().register(eventLoop().unwrappedSelector(), 0, this);this就是说连接事件SelectionKey的attachment是NioServerSocketChannel，所以unsafe指的是NioMessageUnsafe。
顺到这里一直都是NioServerSocketChannel所属的NioEventLoop的线程在执行这些，以及之后的NioSocketChannel的注册操作，所以inEventLoop方法返回false，因为并非NioSocketChannel所属的NioEventLoop的线程在执行，它还没有被创建。由于返回false则代码进入else执行eventLoop.execute，就像之前文章分析的，这里第一次调用会启动线程，这里NioSocketChannel所属的NioEventLoop的线程启动，所属指的是线程启动后SingleThreadEventExecutor的thread变量指向该线程。
接下来register0封装成的Runnable加入到taskQueue，由NioSocketChannel所属NioEventLoop的线程执行。
private void register0(ChannelPromise promise) {
try {
......
boolean firstRegistration = neverRegistered;
doRegister(); // 调用底层JDK注册
neverRegistered = false;
registered = true;
pipeline.invokeHandlerAddedIfNeeded();
safeSetSuccess(promise);
pipeline.fireChannelRegistered();
if (isActive()) {
if (firstRegistration) {
pipeline.fireChannelActive();
......
注册完成后(并没有设置监听SelectionKey.OP_READ事件)触发pipeline.invokeHandlerAddedIfNeeded();，此时NioSocketChannel关联的pipeline有三个处理器head->ChannelInitializer(即childHandler)->tail，如上面分析的，最终ChannelInitializer的initChannel执行，用户自定义的处理链将被添加到NioSocketChannel的pipeline中。
fireChannelRegistered调用各个handler的channelRegistered方法。
对于NioSocketChannel实现的isActive()
public boolean isActive() {
SocketChannel ch = javaChannel();
return ch.isOpen() && ch.isConnected();
}
nio.ServerSocketChannel的accept返回的SocketChannel，该SocketChannel的状态是ST_CONNECTED。
channel创建时即open即isOpen()为true，那么isConnected()呢？跟踪源码
public SocketChannel accept() throws IOException {
......
SocketChannelImpl var2 = null;
......
var2 = new SocketChannelImpl(this.provider(), var4, var6);
.......
return var2;
private static final int ST_CONNECTED = 2;
SocketChannelImpl(SelectorProvider var1, FileDescriptor var2, InetSocketAddress var3) throws IOException {
super(var1);
this.fd = var2;
this.fdVal = IOUtil.fdVal(var2);
this.state = 2;
this.localAddress = Net.localAddress(var2);
this.remoteAddress = var3;
}
所以ch.isConnected()返回true，isActice()返回true，fireChannelActive得以执行，该方法会给NioSocketChannel注册OP_READ事件。到此NioSocketChannel准备就绪。