Message filtering

Message filtering means that consumers do not want to consume all the messages in the topic, but only some specific messages. Selecting these specific messages from topic is called message filtering. Message shunting can be achieved through message filtering, such as producer-produced messages. Headers may be different, so we can write two or more consumers to process different header s'messages pertinently.

Spring Cloud Stream implements message filtering in three ways. One is to specify conditional expressions by using the condition attribute of @StreamListener annotation, the other is to set TAGS in message header, and the third is to use SQL92 grammar to filter messages.

1,condition

This approach is applicable to all MQ and is the simplest and easiest to use. First, we set a header for the producer's message. The specific header name and value can be set according to the actual business needs. As an example, the code is as follows:

package com.zj.node.contentcenter.controller.content;

import lombok.RequiredArgsConstructor;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * Producer
 *
 * @author 01
 * @date 2019-08-03
 **/
@RestController
@RequiredArgsConstructor
public class TestProducerController {

    private final Source source;

    @GetMapping("/stream-send-msg")
    public String streamSendMsg(String flagHeader) {
        source.output().send(
                MessageBuilder.withPayload("Message Body")
                        // Setting the header for filtering messages
                        .setHeader("flag-header", flagHeader)
                        .build()
        );

        return "send message success!";
    }
}

Then the condition attribute is specified in the consumer's @StreamListener annotation, and the conditional expression is written according to the specific header. The following example:

package com.zj.node.usercenter.rocketmq;

import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.cloud.stream.messaging.Sink;
import org.springframework.stereotype.Service;

/**
 * Consumer
 *
 * @author 01
 * @date 2019-08-10
 **/
@Slf4j
@Service
public class TestStreamConsumer {

    @StreamListener(
            value = Sink.INPUT,
            // The conditional expression of message filtering, which only meets the expression, will execute the method
            condition = "headers['flag-header']=='UP'"
    )
    public void receive(String messageBody) {
        log.info("adopt stream Received the message, messageBody = {}", messageBody);
    }
}

Now let's do a simple test where the producer delivers two messages, one flagHeader=UP and the other flagHeader=DOWN. The log information output by consumers is as follows:

2,TAGS

This approach is supported only by RocketMQ, but not by Kafka and RabbitMQ. This approach also requires headers for messages produced by producers, but this header is a little special. We modify the code based on the previous section. The modified code is as follows:

package com.zj.node.contentcenter.controller.content;

import lombok.RequiredArgsConstructor;
import org.apache.rocketmq.spring.support.RocketMQHeaders;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * Producer
 *
 * @author 01
 * @date 2019-08-03
 **/
@RestController
@RequiredArgsConstructor
public class TestProducerController {

    private final Source source;

    @GetMapping("/stream-send-msg")
    public String streamSendMsg() {
        source.output().send(
                MessageBuilder.withPayload("Message Body")
                        // tag can only be set one, not multiple
                        .setHeader(RocketMQHeaders.TAGS, "tag-1")
                        .build()
        );

        return "send message success!";
    }
}

Rebuilding Consumers

Since stream only provides an input channel in the Ink interface by default, in order to more intuitively see messages from different tag s entering different channels, we need to use a custom interface to define multiple "input". The code is as follows:

public interface MySink {

    String T1_INPUT = "t1-input";
    String T2_INPUT = "t2-input";

    @Input(T1_INPUT)
    SubscribableChannel input1();

    @Input(T2_INPUT)
    SubscribableChannel input2();
}

Modify the configuration file as follows:

spring:
  cloud:
    stream:
      rocketmq:
        binder:
          name-server: 192.168.190.129:9876
        bindings:
          t1-input:
            consumer:
              # Represents that t1-input consumes only messages with tag-1
              tags: tag-1
          t2-input:
            consumer:
              # Indicates that t2-input can consume messages with tag-2 or tag-3 (| | for separating different tags)
              tags: tag-2||tag-3
      bindings:
        t1-input:
          destination: stream-test-topic
          group: binder-group1
        t2-input:
          destination: stream-test-topic
          group: binder-group2

Modify the annotations for the startup class as follows:

import com.zj.node.usercenter.rocketmq.MySink;
import org.springframework.cloud.stream.annotation.EnableBinding;

@EnableBinding({MySink.class})
...

Modify the consumer code as follows:

package com.zj.node.usercenter.rocketmq;

import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.stereotype.Service;

/**
 * Consumer
 *
 * @author 01
 * @date 2019-08-10
 **/
@Slf4j
@Service
public class TestStreamConsumer {

    @StreamListener(MySink.T1_INPUT)
    public void receive1(String messageBody) {
        log.info("Consumption tag-1 News, messageBody = {}", messageBody);
    }

    @StreamListener(MySink.T2_INPUT)
    public void receive2(String messageBody) {
        log.info("Consumption tag-2 or tag-3 News, messageBody = {}", messageBody);
    }
}

After completing the above modifications, the producer is used to deliver the message, and the console output is as follows:

2019-08-11 18:06:28.491 INFO 16172 - - [MessageThread_1] c.z.n.u.rocketmq.TestStream Consumer: Consumption of tag-1 messages, message body = message body

3,SQL92

This method is only supported by RocketMQ, but not by Kafka and RabbitMQ. If you use this method, do not use TAGS. This method uses SQL grammar to filter messages, referring specifically to official documents:

http://rocketmq.apache.org/rocketmq/filter-messages-by-sql92-in-rocketmq/

By default, RocketMQ's SQL filtering support is turned off, so we need to turn it on by adding some configuration items. First enter the installation directory of RocketMQ, and then edit the conf/broker.conf file:

[root@study-01 ~]# cd /usr/local/rocketmq-4.5.1/
[root@study-01 /usr/local/rocketmq-4.5.1]# vim conf/broker.conf

Add the following configuration items at the end of the file:

enablePropertyFilter = true

Note: After adding the configuration, the Broker of RocketMQ will need to be restarted to take effect.

nohup sh $ROCKETMQ_HOME/bin/mqbroker -n localhost:9876 -c $ROCKETMQ_HOME/conf/broker.conf &

Rebuilding Producers

Also based on the code in the previous section, the producer's code is as follows:

package com.zj.node.contentcenter.controller.content;

import lombok.RequiredArgsConstructor;
import org.springframework.cloud.stream.messaging.Source;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * Producer
 *
 * @author 01
 * @date 2019-08-03
 **/
@RestController
@RequiredArgsConstructor
public class TestProducerController {

    private final Source source;

    @GetMapping("/stream-send-msg")
    public String streamSendMsg() {
        source.output().send(
                MessageBuilder.withPayload("Message Body")
                        .setHeader("amount", "100")
                        .build()
        );

        return "send message success!";
    }
}

Rebuilding Consumers

Because of the foreshadowing of the previous section, there is not much need to be changed here. First, modify the consumer profile, as follows:

spring:
  cloud:
    stream:
      rocketmq:
        binder:
          name-server: 192.168.190.129:9876
        bindings:
          t1-input:
            consumer:
              # Messages indicating that t1-input consumes only amount less than or equal to 100
              sql: 'amount <= 100'
          t2-input:
            consumer:
              # Messages indicating that t2-input consumes only amount greater than 100
              sql: 'amount > 100'
      bindings:
        t1-input:
          destination: stream-test-topic
          group: binder-group1
        t2-input:
          destination: stream-test-topic
          group: binder-group2

Finally, the consumer code is modified as follows:

package com.zj.node.usercenter.rocketmq;

import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.stream.annotation.StreamListener;
import org.springframework.stereotype.Service;

/**
 * Consumer
 *
 * @author 01
 * @date 2019-08-10
 **/
@Slf4j
@Service
public class TestStreamConsumer {

    @StreamListener(MySink.T1_INPUT)
    public void receive1(String messageBody) {
        log.info("Consumption amount <= 100 News, messageBody = {}", messageBody);
    }

    @StreamListener(MySink.T2_INPUT)
    public void receive2(String messageBody) {
        log.info("Consumption amount > 100 News, messageBody = {}", messageBody);
    }
}

After completing the above modifications, the producer is used to deliver the message, and the console output is as follows:

2019-08-11 18:40:30.019 INFO 9928 - - [MessageThread_1] c.z.n.u. rocketmq.TestStream Consumer: consumed amount <= 100 messages, message body = message body

Source code for tag and sql configuration items:

• com.alibaba.cloud.stream.binder.rocketmq.properties.RocketMQConsumerProperties