Liunx builds Spark development environment
1.Spark
2. Build the Spark development environment
[1] The Spark development environment needs the support of Hadoop, Java, and Scala environments. Since the Java and Hadoop development environments are already installed by default on this machine, I will not repeat them here.
[2] Build the Scala development environment and the SBT development environment:
(2.1) Download the Scala and SBT installation packages:
Scala official website:
http://www.scala-lang.org/
Sbt official website:
https://www.scala-sbt.org/download.html
(2.2) The environment installs Scala and Sbt:
Scala installation:
Use the Xftp5 tool to upload [scala-2.12.5.tgz] to the Liunx server: /usr/local/scala
Use the Xshell5 tool to log in to the server, go to the /usr/local/scala directory to decompress Scala, and enter: tar -xvf scala-2.12.5.tgz
Last login: Sat Apr 7 07:22:36 2018 from 192.168.3.4
[root@marklin ~]# cd /usr/local/scala
[root@marklin scala]# ll
total 19832
-rw-r--r--. 1 root root 20303983 Apr 7 10:10 scala-2.12.5.tgz
[root @ marklin scala] # tar -xvf scala-2.12.5.tgz
To configure environment variables, enter: vim /etc/profile
#Setting SCALA_HOME PATH
export SCALA_HOME=/usr/local/scala/scala-2.12.5
export PATH=${PATH}:${SCALA_HOME}/bin
Enter: source /etc/profile to make environment variables take effect
[3] Download the installation package from the official website [
http://spark.apache.org/
]: spark-2.3.0-bin-hadoop2.7.tgz
[4] Upload the installation package: spark-2.3.0-bin-hadoop2.7.tgz to: /usr/local/spark
[5] Enter: /usr/local/spark, decompress
spark-2.3.0-bin-hadoop2.7.tgz,
enter:
tar -xvf spark-2.3.0-bin-hadoop2.7.tgz
[root@marklin scala]# cd /usr/local/spark
[root@marklin spark]# ll
total 220832
-rw-r--r--. 1 root root 226128401 Apr 7 10:38 spark-2.3.0-bin-hadoop2.7.tgz
[root@marklin spark]# tar -xvf spark-2.3.0-bin-hadoop2.7.tgz
[6] Configure environment variables: vim /etc/profile
#Setting SPARK_HOME PATH
export SPARK_HOME=/usr/local/spark/spark-2.3.0
export PATH=${PATH}:${SPARK_HOME}/bin
Enter: source /etc/profile to make environment variables take effect
[7] Modify the configuration file:
Go to: cd /usr/local/spark/spark-2.3.0/conf, modify the slaves file in the Spark conf directory,
Before modifying, back up and rename cp slaves.template slaves, and change the localhost in the slaves file to the host name, mine is marklin.com:
[root@marklin conf]# cp slaves.template slaves
[root@marklin conf]# ll
total 40
-rw-r--r--. 1 1311767953 1876110778 996 Feb 22 14:42 docker.properties.template
-rw-r--r--. 1 1311767953 1876110778 1105 Feb 22 14:42 fairscheduler.xml.template
-rw-r--r--. 1 1311767953 1876110778 2025 Feb 22 14:42 log4j.properties.template
-rw-r--r--. 1 1311767953 1876110778 7801 Feb 22 14:42 metrics.properties.template
-rw-r--r--. 1 root root 865 Apr 7 10:54 slaves
-rw-r--r--. 1 1311767953 1876110778 865 Feb 22 14:42 slaves.template
-rw-r--r--. 1 1311767953 1876110778 1292 Feb 22 14:42 spark-defaults.conf.template
-rwxr-xr-x. 1 1311767953 1876110778 4221 Feb 22 14:42 spark-env.sh.template
[root@marklin conf]# chmod +x slaves
[root@marklin conf]# ll
total 40
-rw-r--r--. 1 1311767953 1876110778 996 Feb 22 14:42 docker.properties.template
-rw-r--r--. 1 1311767953 1876110778 1105 Feb 22 14:42 fairscheduler.xml.template
-rw-r--r--. 1 1311767953 1876110778 2025 Feb 22 14:42 log4j.properties.template
-rw-r--r--. 1 1311767953 1876110778 7801 Feb 22 14:42 metrics.properties.template
-rwxr-xr-x. 1 root root 865 Apr 7 10:54 slaves
-rw-r--r--. 1 1311767953 1876110778 865 Feb 22 14:42 slaves.template
-rw-r--r--. 1 1311767953 1876110778 1292 Feb 22 14:42 spark-defaults.conf.template
-rwxr-xr-x. 1 1311767953 1876110778 4221 Feb 22 14:42 spark-env.sh.template
[root@marklin conf]# vim slaves
[root@marklin conf]#
Modify spark-env.sh file: backup and rename cp spark-env.sh.tempalte spark-env.sh before modification
[root@marklin conf]# cp spark-env.sh.template spark-env.sh
[root@marklin conf]#
Then open the spark-env.sh file and append the content:
export JAVA_HOME=/usr/local/java/jdk1.8.0_162
export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.5
export SCALA_HOME=/usr/local/scala/scala-2.12.5
export SPARK_HOME=/usr/local/spark/spark-2.3.0
export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_HOME}/etc/hadoop
export SPARK_LOCAL_IP=marklin.com
export SPARK_MASTER_HOST=marklin.com
export SPARK_WORKER_MEMORY=512M
export SPARK_CONF_DIR=${SPARK_HOME}/conf
export SPARK_LOG_DIR=/usr/local/spark/repository/logs
export SPARK_PID_DIR=/usr/local/spark/repository/pids
export SPARK_LIBARY_PATH=.:${JAVA_HOME}/lib:${JAVA_HOME}/jre/lib:${HADOOP_HOME}/lib/native
export SPARK_WORKER_DIR=/usr/local/spark/repository/worker
export SPARK_MASTER_PORT=8188
export SPARK_MASTER_WEBUI_PORT=8180
export SPARK_WORKER_PORT=8181
export SPARK_WORKER_WEBUI_PORT=8182
开放端口：
[root@marklin ~]# systemctl start firewalld.service
[root@marklin ~]# firewall-cmd --zone=public --add-port=8180/tcp --permanent
success
[root@marklin ~]# firewall-cmd --zone=public --add-port=8188/tcp --permanent
success
[root@marklin ~]# firewall-cmd --zone=public --add-port=8181/tcp --permanent
success
[root@marklin ~]# firewall-cmd --zone=public --add-port=8182/tcp --permanent
success
[root@marklin ~]# firewall-cmd --reload
success
[root@marklin ~]# systemctl stop firewalld.service
[root@marklin ~]#
【8】启动测试
进入到：cd /usr/local/spark/spark-2.3.0/sbin ,输入：start-master.sh
[root@marklin sbin]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/repository/logs/
spark-root-org.apache.spark.deploy.master.Master-1-marklin.com.out
输入：http://192.168.3.4:8180/#running-app
输入：
cd bin ，输入：spark-shell
[root@marklin sbin]# cd ..
[root@marklin spark-2.3.0]# cd bin
[root@marklin bin]# spark-shell
2018-04-07 11:43:08 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at
http://marklin.com:4040
Spark context available as 'sc' (master = local[*], app id = local-1523115824100).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162)
Type in expressions to have them evaluated.
Type :help for more information.
scala>