win10搭建hadoop和spark

一、Java环境变量配置

Windows x64：https://www.oracle.com/java/technologies/javase-jdk8-downloads.html

二、hadoop环境变量配置

hadoop-2.7.7.tar.gz：https://mirrors.aliyun.com/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

三、spark环境变量配置

spark-2.4.0-bin-hadoop2.7.tgz：https://mirrors.huaweicloud.com/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz

四、hdfs配置

1、配置hadoop-env.cmd文件里的JAVA_HOME

修改 hadoop-2.7.7\etc\hadoop 下的 hadoop-env.cmd 文件

2、直接启动，发现报错

 

3、将报错的关键信息记录下来

1. 20/03/17 18:23:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2. 20/03/17 18:23:31 WARN nodemanager.DirectoryCollection: Unable to create directory /tmp/hadoop-com/nm-local-dir error (null) entry in command string: null chmod 0755 C:\tmp, removing from the list of valid directories.
3. 20/03/17 18:23:31 WARN nodemanager.DirectoryCollection: Unable to create directory D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs error (null) entry in command string: null chmod 0755 D:\Hadoop\HDFS\hadoop-2.7.7\logs\userlogs, removing from the list of valid directories.
4. 20/03/17 18:23:31 ERROR nodemanager.LocalDirsHandlerService: Most of the disks failed. 1/1 local-dirs are bad: /tmp/hadoop-com/nm-local-dir; 1/1 log-dirs are bad: D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs
5. 20/03/17 18:23:29 ERROR namenode.NameNode: Failed to start namenode.
6. java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
7. 20/03/17 18:23:38 WARN datanode.DataNode: Invalid dfs.datanode.data.dir \tmp\hadoop-com\dfs\data :
8. java.io.IOException: (null) entry in command string: null chmod 0700 C:\tmp\hadoop-com\dfs\data
9. java.io.IOException: All directories in dfs.datanode.data.dir are invalid: "/C:/tmp/hadoop-com/dfs/data"

4、问题描述与解决办法

问题描述：

1、无法加载hadoop library下的jar到Windows平台

2、无法根据命令创建目录 /tmp/hadoop-com/nm-local-dir

3、无法根据命令创建目录 D:/Hadoop/HDFS/hadoop-2.7.7/logs/userlogs

4、初始化的磁盘报错 local-dirs、log-dirs

5、namenode启动失败

6、namenode url 参数错误，fs.defaultFS

7、无效的配置 dfs.datanode.data.dir \tmp\hadoop-com\dfs\data

8、输入 chmod 0700 C:\tmp\hadoop-com\dfs\data 命令为空，需要手动创建文件夹或者配置dfs\data

9、DataNode的目录无效：dfs.datanode.data.dir are invalid: "/C:/tmp/hadoop-com/dfs/data"

解决办法：

1、上github上下载winutils.exe，放到hadoop/bin目录下 https://github.com/steveloughran/winutils

2、配置 /tmp/hadoop-com/nm-local-dir

3、配置 /logs/userlogs

4、手动创建 local-dirs、log-dirs 目录，或者配置 local-dirs、log-dirs

下班了，坐地铁中，未完待续。。。