今天在HBase群里碰到一个客户端死锁问题，堆栈如下
Java stack information for the threads listed above:
===================================================
"KVQueueService-Handler-2-EventThread":
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.stop(ZooKeeperNodeTracker.java:98)
- waiting to lock <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:605)
- locked <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:374)
at org.apache.hadoop.hbase.zookee
per.ZooKeeperWatcher.process(ZooKeeperWatcher.java:271)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497)
"TableDataQueueService-Handler-4":
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:600)
- waiting to lock <0x0000000760c88e00> (a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1698)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:132)
- locked <0x0000000760ca2680> (a org.apache.hadoop.hbase.zookeeper.RootRegionTracker)
at org.
apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:830)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:942)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.j
ava:942)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:845)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1520)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1376)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:936)
at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:792)
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:775)
at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
at com.alibaba.alimonitor.store.hbase.TableDataDaoService.writeData(TableDataDaoService.java:111)
at com.alibaba.alimonitor.store.service.Tab
leDataQueueService.handle(TableDataQueueService.java:28)
at com.alibaba.alimonitor.store.service.TableDataQueueService.handle(TableDataQueueService.java:14)
at com.alibaba.alimonitor.store.service.AbstractQueueService$HandleThread.run(AbstractQueueService.java:115)
Found 1 deadlock.
可见发生死锁的业务线程和zookeeper的eventThread线程。EventThread是zookeeper用来通知客户端watcher的线程，它会回调使用者注册的watcher。
在这里，首先是eventThread处理connectionEvent的Expired事件
case Expired:
if (ZKUtil.isSecureZooKeeper(this.conf)) {
// We consider Expired equivalent to AuthFailed for this
// connection. Authentication is never going to complete. The
// client should proceed to do cleanup.
saslLatch.countDown();
}
String msg = prefix(this.identifier + " received expired from " +
"ZooKeeper, aborting");
// TODO: One thought is to add call to ZooKeeperListener so say,
// ZooKeeperNodeTracker can zero out its data values.
if (this.abortable != null) this.abortable.abort(msg,
new KeeperException.SessionExpiredException());
break;
}
这里会abort调用，其实是调用的HConnectionImplementation的abort方法
public void abort(final String msg, Throwable t) {
if (t instanceof KeeperException) {
LOG.info("This client just lost it's session with ZooKeeper, will"
+ " automatically reconnect when needed.");
if (t instanceof KeeperException.SessionExpiredException) {
LOG.info("ZK session expired. This disconnect could have been" +
" caused by a network partition or a long-running GC pause," +
" either way it's recommended that you verify your environment.");
resetZooKeeperTrackers();
}
return;
}
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.aborted = true;
close();
}
再看resetZooKeeperTrackers方法，是一个同步方法，EventThread在这里拿到HConnectionImplementation对象锁
private synchronized void resetZooKeeperTrackers() {
if (masterAddressTracker != null) {
masterAddressTracker.stop();
masterAddressTracker = null;
}
if (rootRegionTracker != null) {
rootRegionTracker.stop();
rootRegionTracker = null;
}
clusterId = null;
if (zooKeeper != null) {
zooKeeper.close();
zooKeeper = null;
}
}
这个时候业务线程开始region location，进入RootRegionTrack的waitRootRegionLocation
public ServerName waitRootRegionLocation(long timeout)
throws InterruptedException {
if (false == checkIfBaseNodeAvailable()) {
String errorMsg = "Check the value configured in 'zookeeper.znode.parent'. "
+ "There could be a mismatch with the one configured in the master.";
LOG.error(errorMsg);
throw new IllegalArgumentException(errorMsg);
}
return dataToServerName(super.blockUntilAvailable(timeout, true));
}
业务线程进入blockUntilAvailable，拿到父类ZookeeperNodeTracker的对象锁
public synchronized byte [] blockUntilAvailable(long timeout, boolean refresh)
throws InterruptedException {
if (timeout < 0) throw new IllegalArgumentException();
boolean notimeout = timeout == 0;
long startTime = System.currentTimeMillis();
long remaining = timeout;
if (refresh) {
try {
//这里正好抛出异常
this.data = ZKUtil.getDataAndWatch(watcher, node);
} catch(KeeperException e) {
abortable.abort("Unexpected exception handling blockUntilAvailable", e);
}
}
while (!this.stopped && (notimeout || remaining > 0) && this.data == null) {
// We expect a notification; but we wait with a
// a timeout to lower the impact of a race condition if any
wait(100);
remaining = timeout - (System.currentTimeMillis() - startTime);
}
return this.data;
}
由于这里zookeeper调用正好抛出异常导致业务线程去abort，这个时候因为EventThread已经先abort了，所以业务线程需要等待HConnectionImplementation对象锁的释放。而这个时候EventThread继续执行，resetZooKeeperTrackers中继续调用 rootRegionTracker.stop()方法
public synchronized void stop() {
this.stopped = true;
notifyAll();
}
这个方法需要拿到父类ZookeeperNodeTracker的对象锁，而此时这个锁已经被业务线程拿到了，这就发生了死锁。根本原因abort操作没有做并发控制。最新版本的hbase已经修复了这个bug，修复后的abort方法，增加了resetting同步状态，后续abort请求直接return了。
public void abort(final String msg, Throwable t) {
if (t instanceof KeeperException) {
LOG.info("This client just lost it's session with ZooKeeper, will"
+ " automatically reconnect when needed.");
if (t instanceof KeeperException.SessionExpiredException) {
LOG.info("ZK session expired. This disconnect could have been" +
" caused by a network partition or a long-running GC pause," +
" either way it's recommended that you verify your environment.");
//同步状态
synchronized (resetLock) {
if (resetting) return;
this.resetting = true;
}
resetZooKeeperTrackers();
this.resetting = false;
}
return;
}
if (t != null) LOG.fatal(msg, t);
else LOG.fatal(msg);
this.aborted = true;
close();
}
官方issue地址 https://issues.apache.org/jira/browse/HBASE-7259