kylin遇见的错误

kylin遇见的错误

0.普通问题

0.1

java.net.ConnectException: Call From MyDis/192.168.182.86 to MyDis:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

10020:这是配置的mr的jobhistory的端口：明显history server没开，或者出了问题
	/mr-jobhistory-daemon.sh start historyserver

0.2

运行ERROR，点击日志发现java.io.IOException: OS command error exit with return code: 64, error message: SLF4J: Class path contains multiple SLF4J bindings.

原因：第一个可能是：hive元数据没有开启，开启hive元数据服务即可
第二个是SLF4Jar冲突

1.kylin-mr模式

1.1 org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Exception while unregistering

kylin的日志文件报错(

解决思路:kylinUi日志->聚合日志->yarn的日志( ${HADOOP_HOME}/logs/uselogs/任务编号对应的日志文件)):

我的问题是:hadoop之前的模式mr换成了tez(这个在kylin的模式下也有问题),
从tez换成了mr,kylin在中间表使用hive时mr出错。

(解决：hadoop集群mr的运行)（基本都是配置文件的问题。。。）

ERROR [Thread-65] org.apache.hadoop.mapreduce.
v2.app.rm.RMCommunicator: Exception while unregistering
java.lang.NullPointerException
        at org.apache.hadoop.mapreduce.v2.util.MRWebAppUtil.
        getApplicationWebURLOnJHSWithoutScheme(MRWebAppUtil.java:140)
      

1.2

tez运行时运行的问题

注：如果不是这个问题，我还没发现我在之前的yarn-site.xml配置了yarn-tez这个（之前将mr换成tez）

这个其实很搞：使用yarn-tez执行的目录是hdfs:/kylin/kylin_metadata/kylin-6faa47ea-3a5b-4020-976f-c9fcf9d93bd2/kylin_sales_cube/fact_distinct_columns

少了一层目录；

这就是tez换成mr的原因

error：java.io.IOException: fail to find the statistics file 
in base dir: hdfs:/kylin/kylin_metadata/
kylin-6faa47ea-3a5b-4020-976f-c9fcf9d93bd2/
kylin_sales_cube/fact_distinct_columns/statistics

直接将tez换成mr

<property>
	<name>mapreduce.framework.name</name>
	<value>yarn</value>
</property>

2.kylin-spark模式

yarn的配置文件
<property>
	<name>yarn.nodemanager.aux-services</name>
	<value>spark_shuffle,mapreduce_shuffle</value>
</property>
<property>
	<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
	<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
同时将spark-2.4.4-yarn-shuffle.jar放置到${HADOOP_HOME}/share/hadoop/yarn/lib中
spark-2.4.4-yarn-shuffle.jar位置：${SPARK_HOME}/yarn/

org.apache.spark.SparkException: Exception while starting container container_1575360418223_0002_02_000005 on host MyDis
	at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:125)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:65)
	at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:534)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


<--------------------------------注意这里-------------------------------------->
Caused by: org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:spark_shuffle does not exist
	
	
	
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
	at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
	at org.apache.hadoop.yarn.client.api.impl.NMClientImpl.startContainer(NMClientImpl.java:205)
	at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:122)
	... 5 more

参考链接：https://blog.csdn.net/qq_43008162/article/details/103355122