Table of contents
1. Use IDEA to develop WordCount program... 3
Second, the purpose of the experiment... 3
3. Experimental requirements... 3
4. Experimental environment... 3
4.1.1 Start IDEA and create a new project WordCount 3
4.1.2 Add Scala framework support for WordCount project... 7
4.1.4 Set the project directory... 9
4.1.5 Create a new Scala code file... 10
4.1.6 Configuring the pom.xml file... 12
4.1.7 Updating Maven's dependency files... 12
4.1.8 Running the WordCount program... 13
4.1.9 Packaging WordCount program to generate JAR package... 14
4.10 Submit to Spark to run... 17
6. Use IDEA to develop read and write MySQL database programs... 18
Seven, the purpose of the experiment... 18
8. Experimental requirements... 18
9. Experimental environment... 19
4.1.1 Creating a MySQL database... 19
4.1.2 Read and write MySQL database in spark-shell interactive environment... 20
4.1.3 Write a stand-alone application to read and write MySQL database... 23
4.1.4 Create a new project... 23
4.1.5 Set dependent packages... 24
4.1.6 Create a new Scala code file... 25
4.1.8 Configuring the pom.xml file... 25
4.1.9 Compile and run the program... 27
4.2.0 Generate application JAR package... 27
Eleven. Problems and solutions... 29
1. Use IDEA to develop WordCount program
2. The purpose of the experiment
Master the development of Spark applications using IntelliJ IDEA.
3. Experimental requirements
Develop native Spark applications using IntelliJ IDEA.
Deploy distributed Spark applications.
4. Experimental environment
x86_64 ubuntu 16.04
JDK1.8
Spark-2.1.0
Hadoop-2.7.1
IntelliJ IDEA-3.7
scala-2.11.8
5. Experimental steps
4.1.1 Start IDEA and create a new project WordCount
Start IntelliJ IDEA
cd /usr/local/idea
./bin/idea.sh
Open a new project dialog box through the menu " File-->New-->Project "
Click the " Maven " item on the left , and the " Create from_archetype " check box will appear on the right , don't select it, just click the " Next " button at the bottom of the window
In the pop-up window, fill in " dblab " in the "GroupId" dialog box, fill in " WordCount " in the "ArtifactId" dialog box , and then click the "Next" button
When a prompt box appears, click " Enable Auto-Import " in the " Maven projects need to be imported " area at the bottom . In this way, IDEA can automatically connect to the network to download Maven-related dependent files. Every time the pom.xml content in the project is modified in the future, IDEA will automatically connect to the network to download related dependent files.
Select the following Maven configuration file
4.1.2 Add Scala framework support for WordCount project
The development and operation of Spark programs need to rely on Spark-related jar packages. Select in order as shown in the figure below, and manually import the spark jar package into the project.
4.1.3 Data preparation
Open the terminal and create the file word.txt in the /usr/local/spark/mycode/ directory
cd /usr/local/spark/mycode/wordcount/
sudo vi word.txt
4.1.4 Set the project directory
In the project bar on the left side of the interface, click the right mouse button on the "main" subdirectory of the "src" directory, select the "New" menu item in the pop-up menu, and then select the "Directory" menu item in the submenu , creating a new directory.
In the pop-up dialog box, enter the directory name "Scala", and click the "OK" button
On the scala subdirectory, click the right mouse button, select the "Mark Directory as" menu item in the pop-up menu, and then select the " Sources Root " menu item in the submenu, and set the "scala" directory as the source code directory.
On the "java" subdirectory, click the right mouse button and click the "Delete..." menu item in the pop-up menu to delete this directory
4.1.5 Create a new Scala code file
Right-click the scala directory, click "New" in the pop-up menu, and then select the "Scala Class" menu item in the submenu to create a new Scala code file.
In the pop-up window, enter " WordCount " in the "Name" dialog box, select "Object" in the drop-down option box of "Kind", and click the "OK" button
4.1.6 Configure the pom.xml file
For details, please click the link below to view the document:
https://download.csdn.net/download/qq_53142796/87866506