131做master ,91,133做salve 131操作 192.168.0.131 chupiao 192.168.0.91 chupiao2 192.168.0.133 rycdb useradd hadoop 一.ssh免密码登陆 ssh-keygen -t rsa -P '' Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: b0:b0:84:bb:ad:27:15:c4:0f:56:4c:1e:68:cb:73:53 hadoop@chupiao [hadoop@chupiao ~]$ cd .ssh/ [hadoop@chupiao ~]$ll [hadoop@chupiao .ssh]$ cp id_rsa.pub authorized_keys [hadoop@chupiao .ssh]$ ssh chupiao The authenticity of host 'chupiao (192.168.0.131)' can't be established. RSA key fingerprint is 3a:a1:ae:c8:1c:a0:15:ef:9e:3d:87:dd:89:9f:f1:93. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'chupiao,192.168.0.131' (RSA) to the list of known hosts. Last login: Thu Jan 10 10:03:41 2013 from localhost.localdomain 输入 yes 来继续。这会把该服务器添加到你的已知主机的列表中 发现链接成功,并且无需密码。 在 91,133机器上分别操作 [root@chupiao2 ~]# useradd hadoop [root@chupiao2 hadoop]# su - hadoop [hadoop@chupiao2 ~]$ ssh-keygen -t rsa -P '' Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 06:88:7e:7c:c8:64:25:4c:09:b9:04:b2:03:86:ab:df hadoop@chupiao2 cd .ssh [hadoop@chupiao .ssh]$ chmod 644 authorized_keys 在131机器上执行,将authorized_keys传输到91,133 [hadoop@chupiao .ssh]$ scp authorized_keys 192.168.0.133:/home/hadoop/.ssh 测试:ssh 91或者ssh 133(第一次需要输入yes)。 如果不须要输入密码则配置成功,如果还须要请检查上面的配置能不能正确。 在三台机器上,下载hadoop并解压 vi hadoop-env.sh export JAVA_HOME=/usr/local/jdk1.6.0_32 export HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha vi /home/hadoop/hadoop-2.0.2-alpha/etc/hadoop/core-site.xml <configuration> <property> <!-- 用于dfs命令模块中指定默认的文件系统协议 --> <name>fs.default.name</name> <value>hdfs://192.168.0.131:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp/var</value> <!-- hadoop错误INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1 --> </property> </configuration> 1)fs.default.name是NameNode的URI。hdfs://主机名:端口/ 2)hadoop.tmp.dir :Hadoop的默认临时路径,这个最好配置, 如果在新增节点或者其他情况下莫名其妙的DataNode启动不了, 就删除此文件中的tmp目录即可。不过如果删除了NameNode机器的此目录, 那么就需要重新执行NameNode格式化的命令。 conf/mapred-site.xml <?xmlversion="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>mapred.job.tracker</name> <value>192.168.0.131:9001</value> </property> <property> <name>mapred.local.dir</name> <value>/home/hadoop/tmp/var</value> </property> </configuration> 1)mapred.job.tracker是JobTracker的主机(或者IP)和端口。主机:端口。 conf/hdfs-site.xml <?xmlversion="1.0"?> <?xml-stylesheettype="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>dfs.name.dir</name> <value>/home/hadoop/name1, /home/hadoop/name2</value> #hadoop的name目录路径 </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/data1, /home/hadoop/data2</value> </property> <property> <name>dfs.replication</name> <!-- 我们的集群又两个结点,所以rep两份 --> <value>2</value> </property> </configuration> 1) dfs.name.dir是NameNode持久存储名字空间及事务日志的本地文件系统路径。 当这个值是一个逗号分割的目录列表时,nametable数据将会被复制到所有目录中做冗余备份。 2) dfs.data.dir是DataNode存放块数据的本地文件系统路径,逗号分割的列表。 当这个值是逗号分割的目录列表时,数据将被存储在所有目录下,通常分布在不同设备上。 3)dfs.replication是数据需要备份的数量,默认是3,如果此数大于集群的机器数会出错。 注意:此处的name1、name2、data1、data2目录不能预先创建,hadoop格式化时会自动创建,如果预先创建反而会有问题。 vi Masters 192.168.0.131 vi Slaves 192.168.0.133 192.168.0.91 启动 sbin/ start-all.sh 查看是否启动 /bin/ ./hadoop dfs -ls 如果没有错误提示或者出现文件列表,那么恭喜你,Hadoop成功启动了 http://www.cnblogs.com/aniuer/archive/2012/07/16/2594448.html 最新版本的安装文档 [hadoop@hadoop ~]$ vi /home/hadoop/HadoopInstall/hadoop-conf/hadoop-env.sh 指定JAVA_HOME和HADOOP_HOME export JAVA_HOME=/usr/local/jdk1.6.0_32 export HADOOP_HOME=/home/hadoop/hadoop-2.0.2-alpha export PATH=$PATH:/home/hadoop/hadoop-0.22.0/bin 由于Hadoop用ssh 通信,因此先进行免密码登录设定, $ apt-get install ssh $ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa $ cat ~/.ssh/id_rsa.pub>> ~/.ssh/authorized_keys chmod 600 authorized_keys 或者 cp id_rsa.pub authorized_keys $ ssh localhost 完成后请登入确认不用输入密码,(第一次登入需按enter键,第二次就可以直接登入到系统。 ~$ ssh localhost ~$ exit ~$ ssh localhost ~$ exit 如果mapred启动不起来 用 start-mapred.sh试一下,请查看start-all 里面的各环境变量 HADOOP_MAPRED_HOME http://blog.csdn.net/hguisu/article/details/7237395 http://www.cnblogs.com/maczpc/archive/2010/01/17/1650191.htm ./hadoop dfs -mkdir /input ./hadoop fs -ls / ./hadoop dfs -put ~/test-txt /input ./hadoop dfs -ls /input/test-txt/file2.txt ./hadoop fs -cat /input/test-txt/file2.txt ./hadoop dfs -cat /input/test-txt/file2.txt hadoop 命令 http://blog.csdn.net/cheungmine/article/details/7744487 zookeeper 安装 cp zoo_sample.cfg zoo.cfg http://www.linuxidc.com/Linux/2012-07/65607.htm vi zoo.cfg dataDir=/home/hadoop/zookeeper-3.4.3/data server.1=192.168.0.131:2888:3888 server.2=192.168.0.91:2888:3888 server.3=192.168.0.133:2888:3888 #dataLogDir=/home/hadoop/zookeeper/log # 日志保存路径 /home/hadoop/zookeeper-3.4.3/data vi myid 1 在各节点上分别启动 ./zkServer.sh start 各节点测试测试 ./zkCli.sh 192.168.0.91:2181 http://www.linuxidc.com/Linux/2012-07/65607.htm export ZOOKEEPER_INSTALL=/home/hadoop/zookeeper-3.4.3 export PATH=$PATH:$ZOOKEEPER_INSTALL/bin hbase 安装 http://blog.csdn.net/shi1122/article/details/7676488 1、修改conf/hbase-env.sh export JAVA_HOME=/usr/local/jdk1.6.0_32 export HBASE_CLASSPATH=/home/hadoop/hbase-0.94.0/conf export HBASE_MANAGES_ZK=false 2、修改hbase-site.xml,增加以下内容 <property> <name>hbase.rootdir</name> <value>hdfs://192.168.0.131:9000/hbase</value>(注:这里须hadoop-config/core-site.xml中的fs.default.name保持一致) </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value>(注:这里的端口需要和zookeeper的端口配置值一样) <description>Property from ZooKeeper's config zoo.cfg.</description> </property> vi regionservers 192.168.0.131 192.168.0.91 192.168.0.133 hbase 启动出现版本不一样的错误: org.apache.Hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown 将HBase解压缩包中lib的Hadoop Core jar文件替换为当前你所使用的Hadoop版本即可 2.2 版本是 /home/hadoop/hadoop-2.0.2-alpha/share/hadoop/common hadoop-comm 2013-01-11 15:55:41,202 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster 3、把/home/frank/HadoopInstall/hadoop-config/hdfs-site.xml文件拷贝至hbase的conf文件夹下 4、把${ZOOKEEPER_HOME}/conf/zoo.cfg拷贝至hbase的conf文件夹下 5、在conf/regionservers中添加hadoop-config/conf/slaves中所有的datanode节点。 6、删除/hbase-0.90.2/lib/hadoop-core-0.20-append-r1056497.jar 拷贝/hadoop-0.20.2/hadoop-0.20.0-core.jar到/hbase-0.90.2/lib/ java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1740) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1754) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.authentication.util.KerberosName at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:212) at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:275) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) at org.apache.hadoop.hbase.security.User.call(User.java:586) at org.apache.hadoop.hbase.security.User.callStatic(User.java:576) at org.apache.hadoop.hbase.security.User.access$400(User.java:50) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.isSecurityEnabled(User.java:564) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:535) at org.apache.hadoop.hbase.security.User.login(User.java:198) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:267) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1735) ... 5 more Server IPC version 5 cannot communicate with client version 4 版本不同引起的,可以将 hadoop 里面的相关jar包放在hbase的lib里面 cd bin hbase shell hbase(main):001:0> help hbase(main):002:0> status hbase(main):003:0> version java 程序如何操作hbase http://blog.csdn.net/shi1122/article/details/7676488 http://blog.csdn.net/karen_wang/article/details/6283729
hadoop+hbase
猜你喜欢
转载自stevenfeng.iteye.com/blog/1768508
今日推荐
周排行