Apache Kudu cannot delete nonexistent data

Keywords: Big Data Apache Session Java SQL

Extend KafkaConnect Sink with Apache Kudu client.

Java client of Apache Kudu used. Suddenly one day, I found that the job could not be submitted, and I kept reporting an error.

Later, it was found that this is a verification mechanism of Kudu itself. In order to ignore this verification mechanism, and more in line with our SQL habits, I have made changes to the code.

 

In Kudu's submission configuration, manual submission is used. And I also recommend using the configuration of manual submission, which is more efficient and more complete for the processing of exception data after submission.

The configuration mode is as follows:

session.setFlushMode(SessionConfiguration.FlushMode.MANUAL_FLUSH);

 

public void flush() throws KuduException {
 final List<OperationResponse> responses = session.flush();
            for (OperationResponse response : responses) {
                if(!response.hasRowError()){ // No errors continue to run
                    continue;
                }

                String errorStr = response.getRowError().toString();
                if(errorStr.contains("key not found")){
                    log.warn("encounter key not found error.More details =>"
                            + " table: " + response.getRowError().getOperation().getTable().getName()
                            + " row:" + response.getRowError().getOperation().getRow().stringifyRowKey());
                    continue;
                }
                throw new ConnectException("Failed to flush one or more changes. " +
                        "Transaction rolled back: "
                        + response.getRowError().toString() + " oper info ->"
                        + " table: " + response.getRowError().getOperation().getTable().getName()
                        + " row:" + response.getRowError().getOperation().getRow().stringifyRowKey()
                );
            }
}

Posted by simmosn on Wed, 23 Oct 2019 07:02:25 -0700