[Spark] on the use of scala idea to edit the program uses spark

If you want to use the idea, then please download spark-2.4.4-bin-hadoop2.7.tgz version, rather than the other versions, or will be when using the idea of ​​being given the lack of a lot of class

Such as:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
    at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:76)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:71)
    at org.apache.spark.SparkConf.<init>(SparkConf.scala:58)
    at com.hadoop.sparkPi$.main(sparkPi.scala:9)
    at com.hadoop.sparkPi.main(sparkPi.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 5 more

 

 

Guess you like

Origin www.cnblogs.com/CQ-LQJ/p/11621255.html