springcloud
Stream message driven
Message driven overview
What is Spring Cloud Stream: officially, Spring Cloud Stream is a framework for building message driven microservices.
- The application interacts with the binder object in Spring Cloud Stream through inputs or outputs.
- We configure binding, and the binder object of Spring Cloud Stream is responsible for interacting with the message middleware. Therefore, we only need to figure out how to interact with Spring Cloud Stream to facilitate the use of message driven methods.
- By using Spring Integration to connect the message broker middleware to realize message event driven.
- Spring Cloud Stream provides personalized automated configuration for some vendors' message oriented middleware products,
- Three core concepts of publish subscribe, consumption group and partition are cited.
Currently, only RabbitMQ and Kafka are supported.
- Three core concepts of publish subscribe, consumption group and partition are cited.
Shield the differences of underlying message middleware, reduce the switching version, and unify the programming model of message
Official website: https://spring.io/projects/spring-cloud-stream#overview
Chinese Instruction Manual: https://m.wang1314.com/doc/webapp/topic/20971999.html
design idea
Standard mq
- Producers / consumers transmit information through Message media: Message
- Messages must go through a specific channel message channel
- How are messages in the message channel consumed? Who is responsible for receiving and processing: the sub interface SubscribableChannel of the message channel MessageChannel is subscribed by the MessageHandler message processor
Why Cloud Stream
The differences of these middleware lead to some problems in our actual project development. If we use one of the two message queues and the business requirements behind it, I want to migrate to another message queue. At this time, it is undoubtedly disastrous. A lot of things have to be pushed down and redone because it is coupled with our system, At this time, springcloud Stream provides us with a decoupling method.
Why can stream unify the underlying differences?
Without the concept of binder, when our SpringBoot application wants to directly interact with message middleware, there will be great differences in the implementation details due to the different original intentions of each message middleware
- By defining the binder as the middle layer, the isolation between application and message middleware details is perfectly realized.
- By exposing the unified Channel channel to the application, the application does not need to consider a variety of different message middleware implementations.
By default, the RabbitMQ binder implementation maps each target to TopicExchange. For each consumer group.
Binder binder
Without the concept of binder, when our SpringBoot application wants to directly interact with the message middleware, there will be great differences in the implementation details due to the different original intentions of each message middleware. By defining the binder as the middle layer, the isolation between the application and the message middleware details is perfectly realized.
The further encapsulation of message oriented middleware by Stream can make the code layer insensitive to the middleware, and even dynamically switch the middleware (rabbitmq to kafka), which makes the development of micro services highly decoupled, and services can pay more attention to their own business processes
By defining Binder as the middle layer, the isolation between application and message middleware details is realized.
- INPUT corresponds to the consumer
- OUTPUT corresponds to the producer
The message communication mode in Stream follows the publish subscribe mode
Broadcast Topic
- RabbitMQ is Exchange
- In kafka, it is Topic
Spring Cloud Stream standard process routine
- Binder is convenient to connect middleware and shield differences
- Channel channel is an abstraction of Queue. In the message communication system, it is the medium for storage and forwarding. The Queue is configured through the channel
- Source and Sink can be simply understood as the reference object is the Spring Cloud Stream itself. Publishing messages from the Stream is output, and receiving messages is input
Coding API and common annotations
Case description
RabbitMQ environment is OK
Three new sub modules are built in the project
- Cloud stream rabbitmq provider 8801, as a producer, sends messages
- Cloud stream rabbitmq consumer 8802 as a message receiving module
- Cloud stream rabbitmq consumer 8803 as a message receiving module
Message driven producer
cloud-stream-rabbitmq-provider8801
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-eureka-server -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>com.atguigu.springcloud</groupId>
<artifactId>cloud-api-commons</artifactId>
<version>${project.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-devtools -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
configuration file
server:
port: 8801
spring:
application:
name: cloud-stream-provider
cloud:
stream:
binders: # Configure the service information of rabbitmq to be bound here;
defaultRabbit: # Represents the name of the definition, which is used for binding integration
type: rabbit # Message component type
environment: # Set the environment configuration related to rabbitmq
spring:
rabbitmq:
host: localhost
port: 5672
username: guest
password: guest
bindings: # Integration of services
output: # This name is the name of a channel
destination: studyExchange # Represents the Exchange name definition to use
content-type: application/json # Set the message type, json this time, and "text/plain" for text
binder: defaultRabbit # Set the specific settings of the message service to be bound
eureka:
client: # Configure Eureka registration on the client
service-url:
defaultZone: http://localhost:7001/eureka
instance:
lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (30 seconds by default)
lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
instance-id: send-8801.com # Displays the host name in the information list
prefer-ip-address: true # The access path becomes an IP address
Main startup class streamqmain8801
@SpringBootApplication
public class StreamMQMain8801 {
public static void main(String[] args) {
SpringApplication.run(StreamMQMain8801.class, args);
}
}
Business class
Send message interface
public interface IMessageProvider
{
public String send();
}
Send message interface implementation class
@EnableBinding(Source.class) //Defines the push pipeline for messages
public class MessageProviderImpl implements IMessageProvider
{
@Resource
private MessageChannel output; // Message sending pipeline
@Override
public String send()
{
String serial = UUID.randomUUID().toString();
output.send(MessageBuilder.withPayload(serial).build());
System.out.println("*****serial: "+serial);
return null;
}
}
Controller
@RestController
public class SendMessageController
{
@Resource
private IMessageProvider messageProvider;
@GetMapping(value = "/sendMessage")
public String sendMessage()
{
return messageProvider.send();
}
}
Test access: http://localhost:8801/sendMessage
See the serial number and port number of the control belt output
Message driven consumers
cloud-stream-rabbitmq-consumer8802
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.cloud/spring-cloud-starter-eureka-server -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>com.atguigu.springcloud</groupId>
<artifactId>cloud-api-commons</artifactId>
<version>${project.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-web -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-devtools -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
<optional>true</optional>
</dependency>
<!-- https://mvnrepository.com/artifact/org.projectlombok/lombok -->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<!-- https://mvnrepository.com/artifact/org.springframework.boot/spring-boot-starter-test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
configuration file
server:
port: 8802
spring:
application:
name: cloud-stream-consumer
cloud:
stream:
binders: # Configure the service information of rabbitmq to be bound here;
defaultRabbit: # Represents the name of the definition, which is used for binding integration
type: rabbit # Message component type
environment: # Set the environment configuration related to rabbitmq
spring:
rabbitmq:
host: localhost
port: 5672
username: guest
password: guest
bindings: # Integration of services
input: # This name is the name of a channel
destination: studyExchange # Represents the Exchange name definition to use
content-type: application/json # Set the message type, json this time, and "text/plain" for text
binder: defaultRabbit # Set the specific settings of the message service to be bound
eureka:
client: # Configure Eureka registration on the client
service-url:
defaultZone: http://localhost:7001/eureka
instance:
lease-renewal-interval-in-seconds: 2 # Set the heartbeat interval (30 seconds by default)
lease-expiration-duration-in-seconds: 5 # If the interval of 5 seconds is exceeded now (the default is 90 seconds)
instance-id: receive-8802.com # Displays the host name in the information list
prefer-ip-address: true # The access path becomes an IP address
Main startup class streamqmain8802
@SpringBootApplication
public class StreamMQMain8802 {
public static void main(String[] args) {
SpringApplication.run(StreamMQMain8802.class, args);
}
}
Business class
@Component
@EnableBinding(Sink.class)
public class ReceiveMessageListenerController {
@Value("${server.port}")
private String serverPort;
@StreamListener(Sink.INPUT)
public void input(Message<String> message) {
System.out.println("Consumer 1, accept:"+message.getPayload()+"\t port:"+serverPort);
}
}
Test 8801 sends 8802 receives messages
http://localhost:8801/sendMessage
At this time, 8802 will display the received message
According to 8802, clone will run 8803
Two problems after operation
There is the problem of repeated consumption
At present, 8802 / 8803 have been received at the same time, and there is a problem of repeated consumption
How to solve it?
The same group of consumers is a competitive relationship, and only one can consume
principle
When microservice applications are placed in the same group, it can ensure that messages will only be consumed once by one of them. Different groups can be consumed. There will be competition in the same group, and only one of them can be consumed.
The 8802 / 8803 implements polling packets. Only one consumer at a time. Messages sent by the 8801 module can only be received by one of 8802 or 8803, so as to avoid repeated consumption
Message persistence problem
Through the above, we have solved the problem of repeated consumption, and then look at persistence
- The group of 8803: atguigua is not removed
- The group of 8803: atguigua is not removed
8801 sends 4 messages to rabbitmq first
- Start 8802 first, there is no grouping attribute configuration, and there is no message printed in the background
- Start 8803 first, with grouping attribute configuration, and the message on MQ is printed in the background
Summary: consumers with groups can read the information of the group after startup