Extend KafkaConnect Sink with Apache Kudu client.
Java client of Apache Kudu used. Suddenly one day, I found that the job could not be submitted, and I kept reporting an error.
Later, it was found that this is a verification mechanism of Kudu itself. In order to ignore this verification mechanism, and more in line with our SQL habits, I have made changes to the code.
In Kudu's submission configuration, manual submission is used. And I also recommend using the configuration of manual submission, which is more efficient and more complete for the processing of exception data after submission.
The configuration mode is as follows:
session.setFlushMode(SessionConfiguration.FlushMode.MANUAL_FLUSH);
public void flush() throws KuduException { final List<OperationResponse> responses = session.flush(); for (OperationResponse response : responses) { if(!response.hasRowError()){ // No errors continue to run continue; } String errorStr = response.getRowError().toString(); if(errorStr.contains("key not found")){ log.warn("encounter key not found error.More details =>" + " table: " + response.getRowError().getOperation().getTable().getName() + " row:" + response.getRowError().getOperation().getRow().stringifyRowKey()); continue; } throw new ConnectException("Failed to flush one or more changes. " + "Transaction rolled back: " + response.getRowError().toString() + " oper info ->" + " table: " + response.getRowError().getOperation().getTable().getName() + " row:" + response.getRowError().getOperation().getRow().stringifyRowKey() ); } }