目录
2.2.2 Option(self).foreach(_.send(ReregisterWithMaster))
Worker和Master一样,在Spark通信架构中都是一个EndPoint,所以他们的启动过程类似,主要区别在于Worker和Master的实现不同,其onStart方法做的事情不同。
一、Worker的main方法
下面我们来看下worker的main方法:
我们看一下第五步:
第3步创建rpcEnv调用的也是RpcEnv.create(),这个和Master的rpcEnv创建过程一样,会初始化通信需要的一堆组件。
第5步,注册Worker,向Dispatcher的receivers中添加一个EndPointData,Inbox的nessages中添加一个Onstart,用于启动Worker.
二、Worker初始化过程
2.1 参数初始化
同样的,这里只列一些相对重要(面试or装逼时重要)的参数。
// Send a heartbeat every (heartbeat timeout) / 4 milliseconds
// 心跳间隔,每个间隔发送一个心跳。默认是worker超时时间的四分之一(超时时间默认1min,所以默认心跳间隔是15s)。
private val HEARTBEAT_MILLIS = conf.getLong("spark.worker.timeout", 60) * 1000 / 4
// Model retries to connect to the master, after Hadoop's model.
// The first six attempts to reconnect are in shorter intervals (between 5 and 15 seconds)
// Afterwards, the next 10 attempts are between 30 and 90 seconds.
// A bit of randomness is introduced so that not all of the workers attempt to reconnect at
// the same time.
// 初始尝试注册次数(这6次尝试的间隔在5~15秒之间)。
private val INITIAL_REGISTRATION_RETRIES = 6
// 总共尝试注册次数(后10次的间隔在30~90s之间)。引入随机数是为了避免所有的worker同时尝试连接
private val TOTAL_REGISTRATION_RETRIES = INITIAL_REGISTRATION_RETRIES + 10
/**
* The master address to connect in case of failure. When the connection is broken, worker will
* use this address to connect. This is usually just one of `masterRpcAddresses`. However, when
* a master is restarted or takes over leadership, it will be an address sent from master, which
* may not be in `masterRpcAddresses`.
*/
// master重启后,发来的ip
private var masterAddressToConnect: Option[RpcAddress] = None
// 活动master的地址
private var activeMasterUrl: String = ""
// 活动master webUi
private[worker] var activeMasterWebUiUrl : String = ""
// worker WebUi
private var workerWebUiUrl: String = ""
// worker uri
private val workerUri = RpcEndpointAddress(rpcEnv.address, endpointName).toString
// worker是否已注册
private var registered = false
// worker是否已连接
private var connected = false
// worker id
private val workerId = generateWorkerId()
// sparkHome
private val sparkHome =
if (testing) {
assert(sys.props.contains("spark.test.home"), "spark.test.home is not set!")
new File(sys.props("spark.test.home"))
} else {
new File(sys.env.get("SPARK_HOME").getOrElse("."))
}
// worker工作目录
var workDir: File = null
// 已完成Executor
val finishedExecutors = new LinkedHashMap[String, ExecutorRunner]
// 驱动映射
val drivers = new HashMap[String, DriverRunner]
// executor
val executors = new HashMap[String, ExecutorRunner]
// 已完成driver
val finishedDrivers = new LinkedHashMap[String, DriverRunner]
// app地址
val appDirectories = new HashMap[String, Seq[String]]
// 已完成apps
val finishedApps = new HashSet[String]
// 以获取的executor
val retainedExecutors = conf.getInt("spark.worker.ui.retainedExecutors",
WorkerWebUI.DEFAULT_RETAINED_EXECUTORS)
// 以获取的driver
val retainedDrivers = conf.getInt("spark.worker.ui.retainedDrivers",
WorkerWebUI.DEFAULT_RETAINED_DRIVERS)
// The shuffle service is not actually started unless configured.
private val shuffleService = if (externalShuffleServiceSupplier != null) {
externalShuffleServiceSupplier.get()
} else {
new ExternalShuffleService(conf, securityMgr)
}
// 公开的地址
private val publicAddress = {
val envVar = conf.getenv("SPARK_PUBLIC_DNS")
if (envVar != null) envVar else host
}
// webUi
private var webUi: WorkerWebUI = null
// 链接尝试次数
private var connectionAttemptCount = 0
// 指标系统
private val metricsSystem = MetricsSystem.createMetricsSystem("worker", conf, securityMgr)
// 虽然知道字面是资源,但是暂时不知道是干啥用的
private val workerSource = new WorkerSource(this)
// 是否反向代理
val reverseProxy = conf.getBoolean("spark.ui.reverseProxy", false)
private var registerMasterFutures: Array[JFuture[_]] = null
private var registrationRetryTimer: Option[JScheduledFuture[_]] = None
// A thread pool for registering with masters. Because registering with a master is a blocking
// action, this thread pool must be able to create "masterRpcAddresses.size" threads at the same
// time so that we can register with all masters.
// 注册master的线程池,线程数与master的个数相同
private val registerMasterThreadPool = ThreadUtils.newDaemonCachedThreadPool(
"worker-register-master-threadpool",
masterRpcAddresses.length // Make sure we can register with all masters at the same time
)
var coresUsed = 0
var memoryUsed = 0
2.2 onStart
没有什么好说的,我们看下registerWithMaster()
2.2.1 tryRegisterAllMasters()
我们先来看下tryRegisterAllMasters().
这里以多线程的方式向每一个master发出注册请求。我们接着往下看:
可见,使用master的引用向master发送了一个RegisterWorker。Master将会用receive()方法接收处理。我们来看下master是怎么处理这个请求的。
org/apache/spark/deploy/master/Master.scala:266
master的返回结果有四种:
1. 如果Master处于StandBy状态,返回MasterInStandBy。
2. 如果Master缓存信息中已经有了该worker的ID,返回RegisterWorkerFailed(重复的worker ID)
3. 注册成功,返回RegisteredWorker。
4. 如果该worker已经注册,返回RegisterWorkerFailed(重复注册worker)
我们看一下worker对这四种情况是如何相应的(上述三个类都是RegisterWorkerResponse的子类):
org/apache/spark/deploy/worker/Worker.scala:477
我们接着往下看:
org/apache/spark/deploy/worker/Worker.scala:428
这里再贴一下1.2的changeMaster
org.apache.spark.deploy.worker.Worker#changeMaster
2.2.2 Option(self).foreach(_.send(ReregisterWithMaster))
我们再来看看注册重试过程。ReregisterWithMaster是发给自己的,我们看下worker是在哪处理的。
org.apache.spark.deploy.worker.Worker#receive
我们接着往下看:
都在截图里,不多说了。
三、 总结
worker启动过程值得关注的就是向Master的注册过程。worker初始化后调用onStart方法,在onStart方法内调用registerWithMaster方法开始向Master注册。
注册次数最多有17次,1次初始注册,16次重试,其中前6次尝试的间隔在5~15秒之间,后10次重试的间隔在30~90秒之间。超出17次仍未注册成功会报注册失败,退出程序。
Master接收到Worker发来的RegisterWorker后根据实际情况有三种返回结果:
MasterInStandBy:Master处于StandBy状态,注册失败,worker忽略这个信息。
RegisterWorkerFailed(有两种原因:重复worker ID和重复注册):注册失败,Worker会重试注册。
RegisteredWorker:注册成功,worker会修改自己的master信息并取消注册重试、向Master发送心跳(默认15秒一次),Master接收到心跳后会更新这个worker的心跳信息、向Master汇报自身最新的executor和driver信息,master会审核这些信息,通知worker杀死未知的executor和driver
当然这里还有一些别的细节,不觉得重要这里就不谈了。