Article Directory
- Integration Yarn error or can not view the log solution
- The first step: Modify hadoop of yarn-site.xml
- Step Two: distribute the configuration file to each node
- The third step: restart the Hadoop services
- YARN server consolidation history and the history of server Spark
- 1. Turn YARN history server configuration and start
- 2. Spark configure the log server [reference to the following chapters]
- 3. Modify the spark-defaults.conf
- 4. Start Spark HistoryServer Service
- 5. If more dependent Jar package can be uploaded to the HDFS and told to fetch YARN
- Spark history log server configuration
- 1. Modify the file name
- 2. modify the configuration
- 3. Note: The directory on the need to manually create hdfs
- 4. Modify spark-env.sh
- 5. Synchronize Files
- 6. Restart the cluster
- 7. Start a log server on the master
- 8. View the history log port in 4000 (if the load does not come out to try to change the browser)
- 9. If you have write access to Hadoop HDFS questions:
Integration Yarn error or can not view the log solution
The first step: Modify hadoop of yarn-site.xml
vim /export/servers/hadoop/etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>node01</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 关闭YARN内存检查 -->
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<!-- 如果开启如下配置则需要开启Spark历史服务器
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://node01:19888/jobhistory/logs</value>
</property>
-->
Step Two: distribute the configuration file to each node
The third step: restart the Hadoop services
/export/servers/hadoop/sbin/stop-dfs.sh
/export/servers/hadoop/sbin/stop-yarn.sh
/export/servers/hadoop/sbin/start-dfs.sh
/export/servers/hadoop/sbin/start-yarn.sh
YARN server consolidation history and the history of server Spark
1. Turn YARN history server configuration and start
/export/servers/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver
2. Spark configure the log server [reference to the following chapters]
3. Modify the spark-defaults.conf
vim /export/servers/spark/conf/spark-defaults.conf
spark.yarn.historyServer.address node01:4000
4. Start Spark HistoryServer Service
/export/servers/spark/sbin/start-history-server.sh
5. If more dependent Jar package can be uploaded to the HDFS and told to fetch YARN
Disposed in the spark-default.conf
spark.yarn.jars = hdfs://node01:8020/sparkjars/*
After you configure each node will go on HDFS downloaded and cached
each node if you do not configure Spark program will start to distribute packaged Spark_HOME
Spark history log server configuration
By default, the Spark program has finished running, close the window, you can not see the log of the Web UI (4040), and by HistoryServer can provide a service by reading the log file so that we can run at the end of the program, still can View during operation
1. Modify the file name
cd /export/servers/spark/conf
cp spark-defaults.conf.template spark-defaults.conf
2. modify the configuration
vim spark-defaults.conf
spark.eventLog.enabled true
spark.eventLog.dir hdfs://node01:8020/sparklog
3. Note: The directory on the need to manually create hdfs
hadoop fs -mkdir -p /sparklog
4. Modify spark-env.sh
vim spark-env.sh
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://node01:8020/sparklog"
5. Synchronize Files
scp -r /export/servers/spark/conf/ @node02:/export/servers/spark/conf/
scp -r /export/servers/spark/conf/ @node03:/export/servers/spark/conf/
6. Restart the cluster
/export/servers/spark/sbin/stop-all.sh
/export/servers/spark/sbin/start-all.sh
7. Start a log server on the master
/export/servers/spark/sbin/start-history-server.sh
8. View the history log port in 4000 (if the load does not come out to try to change the browser)
http://node01:4000/
9. If you have write access to Hadoop HDFS questions:
org.apache.hadoop.security.AccessControlException
Solution: Add the following configuration hdfs-site.xml, the closure authority verification
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>