YARN 配置文件 capacity-scheduler.xml 参数解读

capacity-scheduler.xml 文件中配置了yarn资源调度器运行中的各项参数,位于hadoop安装目录/hadoop-2.7.2/etc/hadoop目录下。
打开后可以看到默认配置。所有的配置有注有英文注释,考虑到可能对部分朋友不太友好,所以笔者结合自己的理解对这些注释进行了翻译,如有错误,恳请指正:

<configuration>
# 最多可以运行的应用数,默认10000
  <property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>10000</value>
    <description>
      Maximum number of applications that can be pending and running.
    </description>
  </property>
# application master最多能够占用的内存数,可以从另一个维度上控制并行运行的job数量,默认0.1
  <property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.1</value>
    <description>
      Maximum percent of resources in the cluster which can be used to run
      application masters i.e. controls number of concurrent running
      applications.
    </description>
  </property>
# 资源计算方式,默认已占用内存量作为度量,除此之外还能使用多维度计算方式,例如内存、CPU
  <property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
    <description>
      The ResourceCalculator implementation to be used to compare
      Resources in the scheduler.
      The default i.e. DefaultResourceCalculator only uses Memory while
 	  DominantResourceCalculator uses dominant-resource to compare
      multi-dimensional resources such as Memory, CPU etc.
    </description>
  </property>
# 默认队列数量为1,且队列名称为default
  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>default</value>
    <description>
      The queues at the this level (root is the root queue).
    </description>
  </property>
# default队列最多可以使用的资源量限制,默认为100
# 若配置了其他队列,则所有队列该项之和必须为100(************特别注意****************)
  <property>
    <name>yarn.scheduler.capacity.root.default.capacity</name>
    <value>100</value>
    <description>Default queue target capacity.</description>
  </property>
# default队列的每个用户最多可以使用的资源量,默认为1.0(100%)
  <property>
    <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
    <value>1</value>
    <description>
      Default queue user limit a percentage from 0.0 to 1.0.
    </description>
  </property>
# default队列最多可以提交的job数量,默认为100
  <property>
    <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
    <value>100</value>
    <description>
      The maximum capacity of the default queue.
    </description>
  </property>
# default 队列的启用状态,默认为running
  <property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
      The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>
# default 队列的用户权限,默认对所有用户开放
  <property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
    <description>
      The ACL of who can submit jobs to the default queue.
    </description>
  </property>
# default 队列的管理员设置,默认为所有用户
<property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
      The ACL of who can administer jobs on the default queue.
    </description>
  </property>
# 队列本地延迟设置,根据机架感知,对于有nodemanager的客户端,会优先考虑向本地的nodemanager派发task,该值用于配置,默认尝试次数为40次。
  <property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
    <description>
      Number of missed scheduling opportunities after which the CapacityScheduler
      attempts to schedule rack-local containers.
      Typically this should be set to number of nodes in the cluster, By default is setting
      approximately number of nodes in one rack which is 40.
    </description>
  </property>
# 用于配置任务与队列的映射,一种典型的使用场景是为每个用户配置各自的队列,默认为空。
  <property>
    <name>yarn.scheduler.capacity.queue-mappings</name>
    <value></value>
    <description>
      A list of mappings that will be used to assign jobs to queues
      The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
      Typically this list will be used to map users to queues,
      for example, u:%user:%user maps all users to queues with the same name
      as the user.
    </description>
  </property>
# 若设置了上一条,并且设置与用户的设置相冲突,是否覆盖用户的设置,默认为否
  <property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
    <description>
      If a queue mapping is present, will it override the value specified
      by the user? This can be used by administrators to place jobs in queues
      that are different than the one specified by the user.
      The default is false.
    </description>
  </property>

</configuration>

发布了5 篇原创文章 · 获赞 8 · 访问量 218

猜你喜欢

转载自blog.csdn.net/weixin_44512041/article/details/104640151
今日推荐