版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/Jorocco/article/details/86490480
Spark1.6推出的RpcEnv、RpcEndPoint、RpcEndpointRef为核心的新型架构下的RPC通信方式，在底层封装了Akka和Netty，为未来扩充更多的通信系统提供了可能。RpcEnv是一个更宏观的Env，是Spark集群Rpc通信的基础服务环境,因此在集群启动时候所有的节点(无论Master还是Worker)都会创建一个RpcEnv，然后将该节点注册到RpcEnv中。RpcEnv是RPC的环境，所有的RpcEndpoint都需要注册到RpcEnv实例对象中，管理着这些注册的RpcEndpoint的生命周期：
- 根据name或者uri注册RpcEndpoint；
- 管理各种消息的处理；
- 停止RpcEndpoint
Spark RPC中最为重要的三个抽象（“三剑客”）为：RpcEnv、RpcEndpoint、RpcEndpointRef，这样做的好处有：
- 对上层的API来说，屏蔽了底层的具体实现，使用方便
- 可以通过不同的实现来完成指定的功能，方便扩展
- 促进了底层实现层的良性竞争，Spark 1.6.3中默认使用了Netty作为底层的实现，但Akka的依赖依然存在；而Spark2.2.0中的底层实现只有Netty，这样用户可以方便的使用不同版本的Akka或者将来某种更好的底层实现
创建RpcEnv的代码：
/*
变量声明
包名：org.apache.spark
类名：SparkEnv
*/
private[spark] val driverSystemName = "sparkDriver"
private[spark] val executorSystemName = "sparkExecutor"
val isDriver = executorId == SparkContext.DRIVER_IDENTIFIER
val systemName = if (isDriver) driverSystemName else executorSystemName
val rpcEnv = RpcEnv.create(systemName, bindAddress, advertiseAddress, port.getOrElse(-1), conf,
securityManager, numUsableCores, !isDriver)
/*
变量处理
第一步
包名：org.apache.spark.rpc
类名：RpcEnv
*/
def create(
name: String,
bindAddress: String,
advertiseAddress: String,
port: Int,
conf: SparkConf,
securityManager: SecurityManager,
numUsableCores: Int,
clientMode: Boolean): RpcEnv = {
val config = RpcEnvConfig(conf, name, bindAddress, advertiseAddress, port, securityManager,
numUsableCores, clientMode)
new NettyRpcEnvFactory().create(config)
}
/*
第二步
包名：org.apache.spark.rpc.netty
类名：NettyRpcEnv
*/
private[rpc] class NettyRpcEnvFactory extends RpcEnvFactory with Logging {
/**
* NettyRpcEnvFactory创建了NettyRpcEnv之后，如果clientMode为false，即服务端（Driver端Rpc通讯），则使用创建出
* 的NettyRpcEnv的函数startServer定义一个函数变量startNettyRpcEnv（(nettyEnv, nettyEnv.address.port)为函
* 数的返回值），将该函数作为参数传递给函数Utils.startServiceOnPort，即在Driver端启动服务。
*
* 这里可以进入Utils.startServiceOnPort这个函数看看源代码，可以看出为什么不直接调用nettyEnv.startServer，而要把它封装起来
* 传递给工具类来调用：在这个端口启动服务不一定一次就能成功，工具类里对失败的情况做最大次数的尝试，直到启动成功并返回启
* 动成功后的端口。
*/
def create(config: RpcEnvConfig): RpcEnv = {
val sparkConf = config.conf
// Use JavaSerializerInstance in multiple threads is safe. However, if we plan to support
// KryoSerializer in future, we have to use ThreadLocal to store SerializerInstance
// 在多个线程中使用JavaSerializerInstance是安全的。然而，如果我们计划将来支持KryoSerializer，
// 我们必须使用ThreadLocal来存储SerializerInstance
// Netty的通讯都是基于Jav序列化，暂时不支持Kryo
// 1.初始化JavaSerializer，初始化NettyRpcEnv，如果是 非客户端模式就启动netty服务
val javaSerializerInstance =
new JavaSerializer(sparkConf).newInstance().asInstanceOf[JavaSerializerInstance]
// 2.初始化NettyRpcEnv
val nettyEnv =
new NettyRpcEnv(sparkConf, javaSerializerInstance, config.advertiseAddress,
config.securityManager)
// 3.判断是否在Driver端,如果在，则构建TransportServer并注册dispatcher，否则直接返回nettyEnv
if (!config.clientMode) {
// startNettyRpcEnv作为一个函数变量将在下面的startServiceOnPort中被调用
// 简单解释一下这段代码
// 声明一个函数变量，参数是int(actuslPort)，=>后面是实现体，最终返回的是2元组（NettyRpcEnv,int）
val startNettyRpcEnv: Int => (NettyRpcEnv, Int) = { actualPort =>
/** 主要是构建TransportServer和注册dispatcher */
nettyEnv.startServer(config.bindAddress, actualPort)
(nettyEnv, nettyEnv.address.port)
}
try {
// 其实内部实现还是调用startNettyRpcEnv在指定的端口实例化，并且返回nettyEnv对象
Utils.startServiceOnPort(config.port, startNettyRpcEnv, sparkConf, config.name)._1
} catch {
case NonFatal(e) =>
nettyEnv.shutdown()
throw e
}
}
nettyEnv
}
}
/*
包名：org.apache.spark.rpc.netty
类名：NettyRpcEnv
*/
def startServer(bindAddress: String, port: Int): Unit = {
val bootstraps: java.util.List[TransportServerBootstrap] =
// 检查是否启用了Spark通信协议的身份验证。
if (securityManager.isAuthenticationEnabled()) {
// Spark的auth协议进行身份验证
java.util.Arrays.asList(new AuthServerBootstrap(transportConf, securityManager))
} else {
java.util.Collections.emptyList()
}
// 创建TransportServer
server = transportContext.createServer(bindAddress, port, bootstraps)
// 创建RpcEndpointVerifier,然后注册自己到NettyRpcEnv上并发回自己的Ref的实现
dispatcher.registerRpcEndpoint(
RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))
}
/**
包名：org.apache.spark.util
类名：Utils
/**
启动的时候master at spark://biluos.com:7079 后面的端口号开始从7077开始 一直到成功
*/
def startServiceOnPort[T](
startPort: Int,
startService: Int => (T, Int),
conf: SparkConf,
serviceName: String = ""): (T, Int) = {
/**端口号必须1024 and 65535 之间*/
require(startPort == 0 || (1024 <= startPort && startPort < 65536),
"startPort should be between 1024 and 65535 (inclusive), or 0 for a random free port.")
val serviceString = if (serviceName.isEmpty) "" else s" '$serviceName'"
val maxRetries = portMaxRetries(conf)
for (offset <- 0 to maxRetries) {
// Do not increment port if startPort is 0, which is treated as a special port
val tryPort = if (startPort == 0) {
startPort
} else {
userPort(startPort, offset)
}
try {
val (service, port) = startService(tryPort)
// 17/12/05 11:56:50 INFO Utils: Successfully started service 'sparkDriver' on port 55271.
//22=> 17/12/05 11:56:50 INFO Utils: Successfully started service 'SparkUI' on port 4040.
//25=>17/12/05 11:56:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 55290.
logInfo(s"Successfully started service$serviceString on port $port.")
return (service, port)
} catch {
case e: Exception if isBindCollision(e) =>
if (offset >= maxRetries) {
val exceptionMessage = if (startPort == 0) {
s"${e.getMessage}: Service$serviceString failed after " +
s"$maxRetries retries (on a random free port)! " +
s"Consider explicitly setting the appropriate binding address for " +
s"the service$serviceString (for example spark.driver.bindAddress " +
s"for SparkDriver) to the correct binding address."
} else {
s"${e.getMessage}: Service$serviceString failed after " +
s"$maxRetries retries (starting from $startPort)! Consider explicitly setting " +
s"the appropriate port for the service$serviceString (for example spark.ui.port " +
s"for SparkUI) to an available port or increasing spark.port.maxRetries."
}
val exception = new BindException(exceptionMessage)
// restore original stack trace
exception.setStackTrace(e.getStackTrace)
throw exception
}
if (startPort == 0) {
// As startPort 0 is for a random free port, it is most possibly binding address is
// not correct.
logWarning(s"Service$serviceString could not bind on a random free port. " +
"You may check whether configuring an appropriate binding address.")
} else {
logWarning(s"Service$serviceString could not bind on port $tryPort. " +
s"Attempting port ${tryPort + 1}.")
}
}
}
// Should never happen
throw new SparkException(s"Failed to start service$serviceString on port $startPort")
}
RpcEnv、RpcEndPoint、RpcEndpointRef更深入的剖析详见《Spark2.2——RpcEnv（二）》