http://2hei.net/setup-multi-hbase-master-on-hadoop-cluster.html
Setup Multi Hbase master on Hadoop Cluster to avoid single point failure. When active master failed/down for some reason exceed timeout we expected, backup master will be active and take over the role of master, see the value of zookeeper.session.timeout.
$cat /usr/lib/hbase/conf/hbase-site.xml
<name>zookeeper.session.timeout</name>
<value>180000</value>
<description>ZooKeeper session timeout. HBase passes this to the zk quorum as suggested maximum time for a session.In milliseconds.</description>
</property>
- Test scenarios:
hadoop01.cluster.2hei.net (Hbase master)
hadoop02.cluster.2hei.net (Hbase backup master)
- Configuration Steps:
1. add backup master hostname to /usr/lib/hbase/conf/backup-masters
$cat /usr/lib/hbase/conf/backup-masters
hadoop02.cluster.2hei.net
2. start hbase (hbase not manage zookeeper):
#start zookeeper
/usr/lib/zookeeper/bin/zkServer.sh start#start master
sh /usr/lib/hbase/bin/hbase-daemon.sh start master#start backup master
sh /usr/lib/hbase/bin/hbase-daemon.sh start master –backup
- Test steps:
1. stop hbase master
sh /usr/lib/hbase/bin/hbase-daemon.sh stop master
2. check hbase master(hadoop01) logs:
2012-07-23 08:44:53,058 INFO org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for -ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=, dest=hadoop07.cluster.2hei.net,60020,1338437695217; 5 (online=5, exclude=null) available servers
2012-07-23 08:44:53,058 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning region -ROOT-,,0.70236052 to hadoop07.cluster.2hei.net,60020,1338437695217
2012-07-23 08:44:53,058 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to hadoop07.cluster.2hei.net,60020,1338437695217
— stop master —
Mon July 23 08:45:47 UTC 2012 Killing master
3. check hbase backup master(hadoop02) logs:
2012-07-23 04:14:56,348 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-07-23 04:14:56,356 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster started in backup mode. Stalling until master znode is written.
2012-07-23 04:14:56,389 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the active master, 192.168.0.100:60000; waiting to become the next active master
2012-07-23 05:34:15,361 WARN org.apache.hadoop.ipc.HBaseServer: Incorrect header or version mismatch from 192.168.0.201:44928 got version 47 expected version 3
— stop master —
2012-07-23 08:47:58,003 DEBUG org.apache.hadoop.hbase.master.LogCleaner: Add log cleaner in chain: org.apache.hadoop.hbase.master.TimeToLiveLogCleaner
2012-07-23 08:47:58,257 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-07-23 08:47:58,334 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60010
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 60010 webServer.getConnectors()[0].getLocalPort() returned 60010
2012-07-23 08:47:58,344 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 60010
2012-07-23 08:47:58,344 INFO org.mortbay.log: jetty-6.1.26
2012-07-23 08:47:58,879 INFO org.mortbay.log: Started [email protected]:60010
We can see hadoop02 is listening 60010 and to be the active hbase master now.
reference: http://wiki.apache.org/hadoop/Hbase/MultipleMasters