Basic concepts of MQ

1.1 MQ overview

The full name of MQ is Message Queue, which is a container for saving messages during message transmission. It is mostly used for communication between distributed systems

We call from system A to system B directly and remotely. Although this is A simple call, it also has limitations, such as excessive system coupling, slow synchronization request and peak traffic. Therefore, MQ also complies with it.

1.2 MQ advantages

The three main functions of MQ: application decoupling, asynchronous speed-up and flow cutting

1.2.1 application decoupling

The higher the coupling of the system, the lower the fault tolerance and Maintainability:

If one of the system services goes down, the consistency of other system services cannot be guaranteed and can only fail. MQ can be used for decoupling to improve fault tolerance and maintainability.

After the introduction of MQ, if one of the system services goes down, the other services will not be affected, and MQ has a retry mechanism, you can consume messages after the service is restarted.

1.2.2 asynchronous speed increase

Improve user experience and system throughput (number of requests processed per unit time):

How long does it take for a user to place an order to access the order system? We can see the need to access DB and various related systems: 20 + 300 + 300 + 300 = 920ms
This efficiency is not fast. If the QPS is high, there are more orders, many users can't wait in the follow-up queue for feedback, and the experience is not good.

After the user places an order request, he can get the order response by waiting for 25ms (20 + 5 = 25ms). Because the order only needs to interact with MQ, and then MQ interacts with inventory and other systems. MQ is asynchronous and does not need to wait for feedback from other systems. The throughput is naturally improved.

1.2.2 peak shaving and valley filling

After using MQ, the speed of consuming messages is limited to 1000. In this way, the data generated during the peak period is bound to be overstocked in MQ, and the peak period is "cut". However, due to the message backlog, the speed of consuming messages will remain at 1000 for a period of time after the peak period until the overstocked messages are consumed, which is called "filling the valley". It can improve the stability of the system.

1.3 MQ disadvantages

No matter what kind of product, it has both its advantages and disadvantages. There is no perfect solution.

• Reduced system availability
  The more external dependencies the system introduces, the worse the stability of the system. Once MQ goes down, it will have an impact on the business. How to ensure high availability of MQ?
• Increased system complexity
  The addition of MQ greatly increases the complexity of the system. In the past, synchronous remote calls were made between systems, but now asynchronous calls are made through MQ. How to ensure that messages are not lost?

1.4 common MQ products

Introduction to RabbitMQ

AMQP, namely Advanced Message Queuing Protocol, is a network protocol and an open standard of application layer protocol. It is designed for message oriented middleware. The client and message middleware based on this protocol can deliver messages, which is not limited by different products and different development languages of the client / middleware. In 2006, AMQP specification was issued. Analogy HTTP.

In 2007, Rabbit MQ 1.0 developed by Rabbit technology based on AMQP standard was released. RabbitMQ is developed in Erlang. Erlang language is designed by Ericson. It is a language specially for developing high concurrency and distributed systems. It is widely used in the field of telecommunications.

RabbitMQ infrastructure is as follows:

Related concepts in RabbitMQ

• Broker: an application that receives and distributes messages. RabbitMQ Server is the Message Broker
• Virtual host: designed for multi tenancy and security factors, the basic components of AMQP are divided into a virtual group, which is similar to the concept of namespace in the network. When multiple different users use the services provided by the same RabbitMQ server, they can be divided into multiple vhosts. Each user creates an exchange / queue in its own vhost
• Connection: TCP connection between publisher / consumer and broker
• Channel: if a Connection is established every time RabbitMQ is accessed, the overhead of establishing a TCP connection when the message volume is large will be huge and the efficiency will be low. A channel is a logical Connection established within a Connection. If the application supports multithreading, each thread usually creates a separate channel for communication. The AMQP method contains a channel id to help the client and message broker identify the channel, so the channels are completely separated. As a lightweight Connection, channel greatly reduces the overhead of establishing TCP connection by the operating system
• Exchange: message arrives at the first stop of the broker, matches the routing key in the query table according to the distribution rules, and distributes the message to the queue. Common types are: direct (point-to-point), topic (publish subscribe) and fan out (multicast)
• Queue: the message is finally sent here to wait for the consumer to pick it up
• Binding: the virtual connection between exchange and queue. The binding can contain routing key s. The binding information is saved in the query table in exchange for the distribution basis of message s

Working mode of RabbitMQ

RabbitMQ provides six working modes: simple mode, work queues, Publish/Subscribe publish and subscribe mode, Routing mode, Topics topic mode, and RPC remote call mode (remote call, not MQ; not introduced for the moment).
Introduction to the corresponding mode on the official website: https://www.rabbitmq.com/getstarted.html

rely on

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.1.4.RELEASE</version>
  </parent>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-amqp</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

configuration file

server.port=8080
#These are the default configurations. You don't deserve it.
#spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.addresses=81.71.140.7
spring.rabbitmq.virtual-host=/baiqi

Simple mode

In the model above, there are the following concepts:

• P: The producer is the program that sends the message
• C: Consumer: the receiver of the message, who will always wait for the message to arrive
• Queue: message queue, shown in red. Similar to a mailbox, messages can be cached; The producer delivers the message to it, and the consumer takes the message out of it

Here, the code is directly displayed in the form of spring boot

/**
 * HelloWorld rabbitmq，The direct connection mode only needs to declare the queue, and all messages are forwarded through the queue. There is no need to set up a switch
 */
@Configuration
public class HelloWorldConfig {

	@Bean
	public Queue setQueue() {
		return new Queue("helloWorldqueue");
	}
}

// producer
@RestController
public class ProducerController {
	@Autowired
	private RabbitTemplate rabbitTemplate;

	//helloWorld direct mode
	@ApiOperation(value="helloWorld Sending interface",notes="Send directly to queue")
	@GetMapping(value="/helloWorldSend")
	public Object helloWorldSend(String message) throws AmqpException, UnsupportedEncodingException {
		//Set some request parameters
		MessageProperties messageProperties = new MessageProperties();
		messageProperties.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN);

		//Send a message
		rabbitTemplate.send("helloWorldqueue",new Message(message.getBytes("UTF-8"),messageProperties));
		return "message sended : "+message;
	}
}	
//consumer
@Component
public class ConcumerReceiver {
	//Multiple consumers in the direct connection mode will be allocated to one of them for consumption. Similar to task mode
	//Set some properties by injecting RabbitContainerFactory object, which is equivalent to channel.basicQos in task
	@RabbitListener(queues="helloWorldqueue")
	public void helloWorldReceive(String message) {

	     System.out.println("helloWorld pattern received message : " +message);
	}
}

All information about the declared Queue and the messages it processes can be seen in the following display

Output results:

work mode

• Work Queues: compared with the simple mode of the entry program, there are one or more consumers, and multiple consumers consume messages in the same queue together.
• Application scenario: when tasks are too heavy or there are many tasks, using work queue can improve the speed of task processing.

Work Queues is almost the same as the code for the simple mode of the entry program. You can completely copy and copy one more consumer to test the consumption messages of multiple consumers at the same time.

@Configuration
public class WorkConfig {

    //Declaration queue
    @Bean
    public Queue workQ1() {
        return new Queue("work_sb_mq_q");
    }

}


// producer
@RestController
public class ProducerController {
	@Autowired
	private RabbitTemplate rabbitTemplate;

	//Work queue mode
	@ApiOperation(value="workqueue Sending interface",notes="Sent to all consumers listening to the queue")
	@GetMapping(value="/workqueueSend")
	public Object workqueueSend(String message) throws AmqpException, UnsupportedEncodingException {
		MessageProperties messageProperties = new MessageProperties();
		messageProperties.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN);
		//Create multiple messages for sending operation
		for (int i = 0; i <10 ; i++) {
			rabbitTemplate.send("work_sb_mq_q",  new Message(message.getBytes("UTF-8"),messageProperties));
		}
		return "message sended : "+message;
	}
}	
//consumer
@Component
public class ConcumerReceiver {
	//Work queue mode
    @RabbitListener(queues="work_sb_mq_q")
    public void wordQueueReceiveq1(String message) {

		System.out.println("Work queue mode 1 received message : " +message);
    }

    @RabbitListener(queues="work_sb_mq_q")
    public void wordQueueReceiveq2(String message) {

		System.out.println("Work queue mode 2 received message : " +message);
    }
}

Send message: workqueueSend output result:

Summary

1. If there are multiple consumers in a queue, the relationship between consumers for the same message is a competitive relationship.
2. Work Queues when tasks are too heavy or there are many tasks, using Work Queues can improve the speed of task processing. For example, if multiple SMS services are deployed, only one node needs to be sent successfully.

Pub/Sub subscription mode

In the subscription model, an Exchange role is added, and the process changes slightly:

• P: The producer, that is, the program that sends the message, but it is no longer sent to the queue, but to X (switch)
• C: The consumer, the receiver of the message, will always wait for the message to arrive
• Queue: message queue, receiving messages and caching messages
• Exchange: switch. On the one hand, it receives messages sent by producers. On the other hand, you know how to process messages, such as delivering to a special queue, delivering to all queues, or discarding messages. How to operate depends on the type of exchange. There are three common types of exchange:
  Fanout: broadcast and deliver messages to all queues bound to the switch
  Direct: direct, which sends messages to queues that match the specified routing key
  Topic: wildcard, which gives the message to the queue conforming to the routing pattern
  Exchange (switch) is only responsible for forwarding messages and does not have the ability to store messages. Therefore, if there is no queue bound to exchange or no queue that meets the routing rules, the messages will be lost!

/**
 * Fanout The schema needs to declare exchange and bind the queue, and exchange is responsible for forwarding to the queue.
 * The broadcast mode switch type is set to fanout
 */
@Configuration
public class FanoutConfig {

	//Declaration queue
	@Bean
	public Queue fanoutQ1() {
		return new Queue("fanout.q1");
	}
	@Bean
	public Queue fanoutQ2() {
		return new Queue("fanout.q2");
	}

	//Declare exchange
	@Bean
	public FanoutExchange setFanoutExchange() {
		return new FanoutExchange("fanoutExchange");
	}
	
	//Declare the binding relationship between Binding,exchange and queue
	@Bean
	public Binding bindQ1() {
		return BindingBuilder.bind(fanoutQ1()).to(setFanoutExchange());
	}
	@Bean
	public Binding bindQ2() {
		return BindingBuilder.bind(fanoutQ2()).to(setFanoutExchange());
	}
}

// producer
@RestController
public class ProducerController {
	@Autowired
	private RabbitTemplate rabbitTemplate;

	// pub/sub publish subscribe mode switch type fanout
	@ApiOperation(value="fanout Sending interface",notes="Send to fanoutExchange. The message will go to the exchange All under queue forward")
	@GetMapping(value="/fanoutSend")
	public Object fanoutSend(String message) throws AmqpException, UnsupportedEncodingException {
		MessageProperties messageProperties = new MessageProperties();
		messageProperties.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN);
		//fanout mode only sends messages to exchange. Distribute to all queue s under exchange
		rabbitTemplate.send("fanoutExchange", "", new Message(message.getBytes("UTF-8"),messageProperties));
		return "message sended : "+message;
	}
}	

//consumer
@Component
public class ConcumerReceiver {
	//Message listening in pub/sub mode
	@RabbitListener(queues="fanout.q1")
	public void fanoutReceiveq1(String message) {

	    System.out.println("Publish subscribe mode 1 received message : " +message);
	}
	@RabbitListener(queues="fanout.q2")
	public void fanoutReceiveq2(String message) {

	    System.out.println("Publish subscribe mode 2 received message : " +message);
	}
}

Send message: fanoutExchange output result:

Summary

1. The switch needs to be bound to the queue. After binding; A message can be received by multiple consumers.
2. Differences between publish subscribe mode and work queue mode:
   1. The work queue mode does not need to define switches, but the publish / subscribe mode needs to define switches
   2. The producer of publish / subscribe mode sends messages to the switch, and the producer of work queue mode sends messages to the queue (the default switch is used at the bottom)
   3. The publish / subscribe mode needs to set the binding between the queue and the switch. The work queue mode does not need to be set. In fact, the work queue mode will bind the queue to the default switch

Routing mode

Mode description:

• The binding between the queue and the switch cannot be arbitrary, but a routing key should be specified
• When sending a message to Exchange, the sender of the message must also specify the RoutingKey of the message
• Exchange will no longer deliver messages to each bound queue, but will judge according to the routing key of the message. Messages will be received only if the routing key of the queue is completely consistent with the routing key of the message

Illustration:

• P: The producer, that is, the program that sends the message, but it is no longer sent to the queue, but to X (switch)
• 10: Exchange (exchange) receives the producer's message, and then submits the message to the queue that exactly matches the routing key
• C1: consumer, whose queue specifies the message whose routing key is error
• C2: consumer, whose queue specifies the messages whose routing key needs to be info, error and warning

/*
   Routing mode | routing mode switch type: direct
*/
@Configuration
public class DirectConfig {

	//Declaration queue
	@Bean
	public Queue directQ1() {
		return new Queue("direct_sb_mq_q1");
	}
	@Bean
	public Queue directQ2() {
		return new Queue("direct_sb_mq_q2");
	}


	//Declare exchange
	@Bean
	public DirectExchange setDirectExchange() {
		return new DirectExchange("directExchange");
	}

	//To declare binding, you need to declare a routingKey
	@Bean
	public Binding bindDirectBind1() {
		return BindingBuilder.bind(directQ1()).to(setDirectExchange()).with("china.changsha");
	}
	@Bean
	public Binding bindDirectBind2() {
			return BindingBuilder.bind(directQ2()).to(setDirectExchange()).with("china.beijing");
	}
}

// producer
@RestController
public class ProducerController {
	@Autowired
	private RabbitTemplate rabbitTemplate;

	//routing working mode switch type direct
	@ApiOperation(value="direct Sending interface",notes="Send to directExchange. exchange When forwarding a message, it will go to routingKey Matching queue send out")
	@GetMapping(value="/directSend")
	public Object routingSend(String routingKey,String message) throws AmqpException, UnsupportedEncodingException {

		if(null == routingKey) {
			routingKey="china.changsha";
		}
		MessageProperties messageProperties = new MessageProperties();
		messageProperties.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN);
		//fanout mode only sends messages to exchange. Distribute to all queue s under exchange
		rabbitTemplate.send("directExchange", routingKey, new Message(message.getBytes("UTF-8"),messageProperties));
		return "message sended : routingKey >"+routingKey+";message > "+message;
	}
}	

//consumer
@Component
public class ConcumerReceiver {
    //Routing mode
    @RabbitListener(queues="direct_sb_mq_q1")
    public void routingReceiveq1(String message) {

	    System.out.println("Routing Routing mode routingReceiveq11111 received message : " +message);
    }

    @RabbitListener(queues="direct_sb_mq_q2")
    public void routingReceiveq2(String message) {

	    System.out.println("Routing Routing mode routingReceiveq22222 received message : " +message);
    }
}

Send different keys and messages: routingkey=china.changsha message = Shanghai
china.beijing Beijing
Output results:

Summary

1. The Routing mode requires the queue to specify the routing key when binding the switch, and the message will be forwarded to the queue that meets the routing key

Topics wildcard pattern

Mode description:

• Compared with Direct, Topic can route messages to different queues according to RoutingKey. However, Topic Exchange allows the queue to use wildcards when Binding routing keys!
• Routingkey s are generally composed of one or more words, which are separated by ".", such as item.insert
• Wildcard rule: # match one or more words, * match exactly one word, for example: item. # can match item.insert.abc or item.insert, and item. * can only match item.insert
  

Illustration:

• Red Queue: the bound is usa. #, so all routing key s starting with usa. will be matched
• Yellow Queue: bound to #. news, so all routing key s ending in. news will be matched

/*
Topics Mode switch type topic
* */
@Configuration
public class TopicConfig {

	//Declaration queue
	@Bean
	public Queue topicQ1() {
		return new Queue("topic_sb_mq_q1");
	}
	@Bean
	public Queue topicQ2() {
		return new Queue("topic_sb_mq_q2");
	}


	//Declare exchange
	@Bean
	public TopicExchange setTopicExchange() {
		return new TopicExchange("topicExchange");
	}

	//To declare binding, you need to declare a roytingKey
	@Bean
	public Binding bindTopicHebei1() {
		return BindingBuilder.bind(topicQ1()).to(setTopicExchange()).with("changsha.*");
	}
	@Bean
	public Binding bindTopicHebei2() {
		return BindingBuilder.bind(topicQ2()).to(setTopicExchange()).with("#.beijing");
	}

}

// producer
@RestController
public class ProducerController {
	@Autowired
	private RabbitTemplate rabbitTemplate;

	//Topic working mode switch type topic
	@ApiOperation(value="topic Sending interface",notes="Send to topicExchange. exchange When forwarding a message, it will go to routingKey Matching queue send out,*Represents a word,#Represents 0 or more words. ")
	@GetMapping(value="/topicSend")
	public Object topicSend(String routingKey,String message) throws AmqpException, UnsupportedEncodingException {

		if(null == routingKey) {
			routingKey="changsha.kf";
		}
		MessageProperties messageProperties = new MessageProperties();
		messageProperties.setContentType(MessageProperties.CONTENT_TYPE_TEXT_PLAIN);
		//The fan out mode only sends messages to the exchange. It is distributed to all queue s under the exchange
		rabbitTemplate.send("topicExchange", routingKey, new Message(message.getBytes("UTF-8"),messageProperties));
		return "message sended : routingKey >"+routingKey+";message > "+message;
	}
}	

//consumer
@Component
public class ConcumerReceiver {
    //topic mode
	//Note that this pattern has a priority matching principle. For example, if routingKey=hunan.IT is sent, it will match hunan.*(hunan.IT,hunan.eco), and then *. ITd will not be matched
	@RabbitListener(queues="topic_sb_mq_q1")
	public void topicReceiveq1(String message) {
		System.out.println("Topic pattern topic_sb_mq_q1 received message : " +message);
	}

	@RabbitListener(queues="topic_sb_mq_q2")
	public void topicReceiveq2(String message) {
		System.out.println("Topic pattern topic_sb_mq_q2 received  message : " +message);
	}
}

Send different keys and messages: routingKey=changsha.hp message = Huangpu District, Shanghai
china.zxs.beijing Beijing, China
Output results:

Summary

1. Topic topic mode can realize the functions of Pub/Sub publish and subscribe mode and Routing mode, but topic can use wildcards when configuring Routing keys, which is more flexible.

Working mode summary

1. Simple mode HelloWorld
   One producer and one consumer do not need to set the switch (use the default switch).
2. Work Queue mode
   One producer and multiple consumers (competitive relationship) do not need to set the switch (use the default switch).
3. Publish/subscribe mode
   You need to set the switch of type fanout and bind the switch to the queue. After sending a message to the switch, the exchange opportunity sends the message to the bound queue.
4. Routing mode
   You need to set the switch with the type of direct, bind the switch to the queue, and specify the routing key. When a message is sent to the switch, the switch will send the message to the corresponding queue according to the routing key.
5. Wildcard pattern Topic
   It is necessary to set the switch with the type of topic, bind the switch to the queue, and specify the routing key in the wildcard mode. When a message is sent to the switch, the switch will send the message to the corresponding queue according to the routing key.

All great actions and thoughts have a trivial beginning.