hadoop:
1.安装java,配置环境变量:export JAVA_HOME=///
export PATH=/////
注意:这个配置有效期不是很久的,要把这配置写到自己的配置文件(~/.bash_profile)
export PATH=$PATH:////////
2.下载hadoop-1.2.1, 解压:
>sudo tar -vxf hadoop-1.2.1.tar
>export HADOOP_HOME=///////
>export PATH=/////
>cd hadoop-1.2.1/conf
配置一个环境文件(hadoop-env.sh):
把”JAVA_HOME=“注解打开,把java的路径加进去; export JAVA_HOME=$(/usr/libexec/java_home)
配置三个文件,hadoop-site.xml, hdfs-site.xml, mapred-site.xml;
其中:
注意fs.default.name这个字段,有可能要把主机名加入去的,mac就是这样,要不,后面的namenode运行不起来。
hadoop-site.xml,
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>~/hadoop_tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost(******.local):8020</value> </property> </configuration>
hdfs-site.xml,
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> </configuration>
mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>liangr.local:8021</value> </property> <property> <name>mapred.tasktracker.map.tasks.maximum</name> <value>2</value> </property> <property> <name>mapred.tasktracker.reduce.tasks.maximum</name> <value>2</value> </property> </configuration>
配置完之后,格式化文件系统:
>hadoop namenode -format
显示为:
最后,运行:
>start-all.sh
查看:
>jps
显示:
1411 SecondaryNameNode
6740 Jps
1317 DataNode
1223 NameNode
1480 JobTracker
1576 TaskTracker
把开web浏览器:
http://localhost:50030/jobtracker.jsp
http://localhost:50070/dfshealth.jsp
NameNode '172-10-22-141.lightspeed.clmasc.sbcglobal.net:8020' Started: Mon Sep 14 11:04:32 CST 2015 Version: 1.2.1, r1503152 Compiled: Mon Jul 22 15:23:09 PDT 2013 by mattf Upgrades: There are no upgrades in progress. Browse the filesystem Namenode Logs Cluster Summary 9 files and directories, 2 blocks = 11 total. Heap Size is 123 MB / 889 MB (13%) Configured Capacity : 232.62 GB DFS Used : 28 KB Non DFS Used : 44.1 GB DFS Remaining : 188.52 GB DFS Used% : 0 % DFS Remaining% : 81.04 % Live Nodes : 1 Dead Nodes : 0 Decommissioning Nodes : 0 Number of Under-Replicated Blocks : 0
liangr Hadoop Map/Reduce Administration Quick Links State: RUNNING Started: Mon Sep 14 11:04:33 CST 2015 Version: 1.2.1, r1503152 Compiled: Mon Jul 22 15:23:09 PDT 2013 by mattf Identifier: 201509141104 SafeMode: OFF Cluster Summary (Heap Size is 115 MB/889 MB) Running Map Tasks Running Reduce Tasks Total Submissions Nodes Occupied Map Slots Occupied Reduce Slots Reserved Map Slots Reserved Reduce Slots Map Task Capacity Reduce Task Capacity Avg. Tasks/Node Blacklisted Nodes Graylisted Nodes Excluded Nodes 0 0 0 1 0 0 0 0 2 2 4.00 0 0 0 Scheduling Information Queue Name State Scheduling Information default running N/A Filter (Jobid, Priority, User, Name) Example: 'user:smith 3200' will filter by 'smith' only in the user field and '3200' in all fields Running Jobs none Retired Jobs none Local Logs Log directory, Job Tracker History This is Apache Hadoop release 1.2.1