Spark-based Yarn submit jobs in two ways

• yarn-client submit the job the way

1. Client submits a Application, to start a process on the client Driver

2. Driver process will (ResourceManager) RS sends a request to start the resource AM (ApplicationMaster) of

3. RS receiving the request, a random selection NM (NodeManager) Start AM. NM where the node corresponds to the Worker Standalone

4. AM After the start, it will request a number of container resources to the RS, to start Executor

5. RS will find a number of NM return to AM, to start Executor

6. AM Executor will start to send commands NM

7. After Executor starts, the reverse is registered to Driver, Driver is sent to the Executor task, implementation and results back to the end Driver

　　

 

• yarn-cluster task submit way

1. Application submission client application, sends a request to the RS (ResourceManager), a request to start AM (ApplicationMaster)

2. RS receives the request on a random NM (NodeManager) starts AM (corresponding to end Driver)

3. Start AM, AM sends a request to the RS, a request for starting a batch container Executor

4. RS node to return a number of NM AM

5. AM is connected to the NM, NM sends a request to start Executor

6. Executor reverse registration to AM Driver node is located. Driver task sent to Executor