用spark-submit提交任务给集群时涉及的参数

用spark-submit提交任务给集群时涉及的参数:
用bin/spark-submit提交,查看spark-submit用法

bin/spark-submit --help
Usage:spark-submit [options] <app jar | python file> [app arguments]

如local模式时:

 #将程序运行在local mode,启动2个Thread
 bin/spark-submit --master local[2] 1.py

将脚本运行在集群上:

/bin/spark-submit\       
         --master spark://master:7077\     (指定运行在哪里;运行在集群上)
         --deploy-mode client\     (部署模式,客户端还是集群)
         --driver-memory 512M\     
         --executor-memory 1G\     (每个executor的内存)
         --executor-cores 1\          (每个executor的核数)
         --total-executor-cores 2\  (所有executor的核数)
         1.py

猜你喜欢

转载自blog.csdn.net/leitouguan8655/article/details/82804842
今日推荐