Kafka's producer interceptor

1, Interceptor principle

The Producer interceptor was introduced in Kafka version 0.10, which is mainly used to realize the customized control logic on the clients side.
For the producer, the interceptor gives the user the opportunity to make some customization requirements for the message, such as modifying the message, before the message is sent and before the producer callback logic. At the same time, producer allows users to specify multiple interceptors to act on the same message in order to form an interceptor chain. The implementation interface of intercetor is org.apache.kafka.clients.producer.produceinterceptor, and its defined methods include:
　　(1)configure(configs)
Called when getting configuration information and initialization data.
　　(2)onSend(ProducerRecord):
This method is encapsulated in the KafkaProducer.send method, that is, it runs in the user's main thread. Producer ensures that the method is called before the message is serialized and calculated. You can do anything with the message in this method, but you'd better not modify the topic and partition to which the message belongs, otherwise the calculation of the target partition will be affected
　　(3)onAcknowledgement(RecordMetadata, Exception):
This method is called when the message is answered or the message sending fails, and usually before the producer callback logic is triggered. onAcknowledgement runs in the IO thread of the producer, so do not put heavy logic in this method, otherwise it will slow down the message sending efficiency of the producer
　　(4)close:
Close the interceptor, which is mainly used to clean up some resources
As mentioned earlier, the interceptor may be run in multiple threads, so the user needs to ensure thread safety during specific implementation. In addition, if multiple interceptors are specified, producer will call them in the specified order, and only catch the exceptions that may be thrown by each interceptor and record them in the error log instead of passing them up. This should be paid special attention during use.

2, Interceptor case

1) Requirements:
Implement a simple interception chain composed of double interceptors. The first interceptor will add the timestamp information to the front of the message value before sending the message; The second interceptor will update the number of messages sent successfully or failed after the message is sent.

2) Case practice
(1) Add timestamp interceptor

package com.libt.kafka.interceptor;
import java.util.Map;
import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

public class TimeInterceptor implements ProducerInterceptor<String, String> {

    @Override
    public void configure(Map<String, ?> configs) {

    }

    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        // Create a new record，Writes the timestamp to the front of the message body
        return new ProducerRecord(record.topic(), record.partition(), record.timestamp(), record.key(),
                System.currentTimeMillis() + "," + record.value().toString());
    }

    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {

    }

    @Override
    public void close() {

    }
}

(2) Count the number of successful and failed messages sent, and print these two counters when producer is closed

package com.libt.kafka.interceptor;
import java.util.Map;
import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

public class CounterInterceptor implements ProducerInterceptor<String, String>{
    private int errorCounter = 0;
    private int successCounter = 0;

    @Override
    public void configure(Map<String, ?> configs) {
        
    }

    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
         return record;
    }

    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
        // Count the number of successes and failures
        if (exception == null) {
            successCounter++;
        } else {
            errorCounter++;
        }
    }

    @Override
    public void close() {
        // Save results
        System.out.println("Successful sent: " + successCounter);
        System.out.println("Failed sent: " + errorCounter);
    }
}

(3) producer main program

package com.libt.kafka.interceptor;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;

public class InterceptorProducer {

    public static void main(String[] args) throws Exception {
        // 1 Set configuration information
        Properties props = new Properties();
        props.put("bootstrap.servers", "hadoop1:9092");
        props.put("acks", "all");
        props.put("retries", 0);
        props.put("batch.size", 16384);
        props.put("linger.ms", 1);
        props.put("buffer.memory", 33554432);
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        
        // 2 Build interception chain
        List<String> interceptors = new ArrayList<>();
        interceptors.add("com.atguigu.kafka.interceptor.TimeInterceptor");     interceptors.add("com.atguigu.kafka.interceptor.CounterInterceptor"); 
        props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, interceptors);
         
        String topic = "first";
        Producer<String, String> producer = new KafkaProducer<>(props);
        
        // 3 send message
        for (int i = 0; i < 10; i++) {
            
            ProducerRecord<String, String> record = new ProducerRecord<>(topic, "message" + i);
            producer.send(record);
        }
         
        // 4 Be sure to turn it off producer，This will call interceptor of close method
        producer.close();
    }
}

3) Testing
(1) Start the consumer on kafka and run the client java program.

[hadoop1 kafka]$ bin/kafka-console-consumer.sh \
--zookeeper hadoop1:2181 --from-beginning --topic first

1501904047034,message0
1501904047225,message1
1501904047230,message2
1501904047234,message3
1501904047236,message4
1501904047240,message5
1501904047243,message6
1501904047246,message7
1501904047249,message8
1501904047252,message9

(2) Observe the output data of the java platform console as follows:

Successful sent: 10
Failed sent: 0