Kudu: delete the data does not exist error status = Not found: key not found (error 0)

1. Mito

2. Background

A real-time synchronization of tasks, suddenly found written Kudu is time to implement

An error

020-01-17 09:32:22,067 ERROR com.dtwave.meteor.connector.core.service.WorkerSubTask -[deliverRecords][590]:  Sub task T_6676615507350773-0 threw an uncaught exception. Sub task is being killed and will not recover util manually restarted.
com.dtwave.meteor.connector.common.exception.SinkException: Sub task T_6676615507350773-0 flush error: Row error for primary key="1000", tablet=null, server=1a65f2ced6b84c2096b0fd39097bfe5e, status=Not found: key not found (error 0)
        at com.dtwave.meteor.connector.kudu.util.BufferedRecords.flush(BufferedRecords.java:106)
        at com.dtwave.meteor.connector.kudu.KuduWriter.write(KuduWriter.java:86)
        at com.dtwave.meteor.connector.kudu.KuduSinkTask.sink(KuduSinkTask.java:73)
        at com.dtwave.meteor.connector.core.service.WorkerSubTask.deliverRecords(WorkerSubTask.java:576)
        at com.dtwave.meteor.connector.core.service.WorkerSubTask.poll(WorkerSubTask.java:470)
        at com.dtwave.meteor.connector.core.service.WorkerSubTask.iteration(WorkerSubTask.java:360)
        at com.dtwave.meteor.connector.core.service.WorkerSubTask.execute(WorkerSubTask.java:291)
        at com.dtwave.meteor.connector.core.service.WorkerSubTask.doRun(WorkerSubTask.java:252)

3. Analysis

Apache Kudu can not delete the data does not exist

Use Apache Kudu client, to KafkaConnect Sink extended. Use of Apache Kudu Java client. Suddenly one day find a job can not be submitted until the error.

Kudu later found that this is a self-checking mechanism. To ignore this checking mechanism, more in line with our SQL habit, I did the code transformation.

In the submission of the configuration Kudu., Submitted using the manual configuration. And I also recommend using a manual configuration submitted, so that better efficiency after treatment for abnormal submit more complete data.
Configuration is as follows:

session . setFlushMode( SessionConf iguration. FlushMode . MANUAL_ FLUSH);

The code was originally flush

/**
  * 最终flush操作，主要解决删除的时候，删除了不存在的数据
  */
 private void terminalFlush() throws KuduException {
     final java.util.List<OperationResponse> responses = session.flush();
     for (OperationResponse response : responses) {
        
         if (response.hasRowError()) {
             throw new SinkException("encounter key not found error.More.detail " +
                     "=> table : "+ response.getRowError().getOperation().getTable().getName() +"" +
                     "=> row : "+response.getRowError().getOperation().getRow().stringifyRowKey());
         }
     }
     LOG.debug("Sub task {} flushed sink records", id);
 }

change into

/**
 * 最终flush操作，主要解决删除的时候，删除了不存在的数据
 */
private void terminalFlush() throws KuduException {
    final java.util.List<OperationResponse> responses = session.flush();
    for (OperationResponse response : responses) {
        if (!response.hasRowError()) {
            continue;
        }
        String errorString = response.getRowError().toString();
        // 主要过滤kudu 删除的时候，删除到了不存在的数据
        if(errorString.contains("key not found")){
            LOG.warn("encounter key not found error.More.detail " +
                    "=> table : "+ response.getRowError().getOperation().getTable().getName() +"" +
                    "=> row : "+response.getRowError().getOperation().getRow().stringifyRowKey());
            continue;
        }
        if (response.hasRowError()) {
            throw new SinkException("encounter key not found error.More.detail " +
                    "=> table : "+ response.getRowError().getOperation().getTable().getName() +"" +
                    "=> row : "+response.getRowError().getOperation().getRow().stringifyRowKey());
        }
    }
    LOG.debug("Sub task {} flushed sink records", id);
}

Reference: http: //www.rengongzineng.com.cn/post/5131.html