abstract,spark can be compiled with:
maven,
sbt,
intellj ideal
also,if u want to load spark-project into eclipse ,then it is necessary to make a 'eclipse project' first by one of below solutions:
1.mvn eclipse:eclipse [optional]
2. ./sbt/sbt clean compile package
or sbt/sbt then input 'eclipse'
e.g. if u want build a hadoop 2.5.2 jar u can run with :
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.2 -DskipTests clean package
yarn's/hadoop profile:2.4
yarn/hadoop version:2.5.2
hive:enable default
[INFO] Building jar: /home/user/build-spark/spark-1.6.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.6.1-test-sources.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [01:11 min] [INFO] Spark Project Test Tags ............................ SUCCESS [ 33.845 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 33.728 s] [INFO] Spark Project Networking ........................... SUCCESS [ 13.160 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 10.966 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.310 s] [INFO] Spark Project Core ................................. SUCCESS [04:29 min] [INFO] Spark Project Bagel ................................ SUCCESS [ 6.629 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 23.603 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 49.411 s] [INFO] Spark Project Catalyst ............................. SUCCESS [01:15 min] [INFO] Spark Project SQL .................................. SUCCESS [01:53 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:02 min] [INFO] Spark Project Tools ................................ SUCCESS [ 2.954 s] [INFO] Spark Project Hive ................................. SUCCESS [01:39 min] [INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 23.671 s] [INFO] Spark Project REPL ................................. SUCCESS [ 12.683 s] [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 7.344 s] [INFO] Spark Project YARN ................................. SUCCESS [ 21.928 s] [INFO] Spark Project Assembly ............................. SUCCESS [02:05 min] [INFO] Spark Project External Twitter ..................... SUCCESS [ 12.928 s] [INFO] Spark Project External Flume Sink .................. SUCCESS [ 20.712 s] [INFO] Spark Project External Flume ....................... SUCCESS [ 10.053 s] [INFO] Spark Project External Flume Assembly .............. SUCCESS [ 3.930 s] [INFO] Spark Project External MQTT ........................ SUCCESS [01:49 min] [INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 8.549 s] [INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 11.317 s] [INFO] Spark Project External Kafka ....................... SUCCESS [ 24.706 s] [INFO] Spark Project Examples ............................. SUCCESS [03:14 min] [INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 9.263 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 25:38 min [INFO] Finished at: 2016-03-24T15:58:43+08:00 [INFO] Final Memory: 207M/6072M
./bin/spark-submit --master local --class org.apache.spark.examples.JavaWordCount examples/target/spark-examples_2.10-1.6.1.jar CHANGES.txt
note:
-the spark.ui.port is used by 'spark-shell'(or spark-submit?no) to show info about jobs,stages,executors,environments(these will be shown also by any completed apps) etc.related logs will be printed when it starts up:
15/11/11 15:12:45 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/11/11 15:12:45 INFO SparkUI: Started SparkUI at http://192.168.1.138:4040 15/11/11 15:12:45 INFO Executor: Starting executor ID driver on host localhost
also ,if u want to make a dist tar ,below is a feasible demo:
./make-distribution.sh -Phadoop-2.4 -Dhadoop.version=2.5.2 -DskipTests
if u want to build a spark/import project,some steps are listed here[1]
ref:
http://blog.csdn.net/yunlong34574/article/details/39213503
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IDESetup
http://blog.csdn.net/chenxingzhen001/article/details/25901237
[1] ide setup