The following is the RocketMQ - Producer series article index:
1: Start the process (this article)
2: Routing dynamic updates
3: Send Mode Analysis + Message Type Difference
4: Sending process of producer message
5: Routing queue selection, client redundancy policy (QA)
Introduction to Producer
Producer is the sender of the RocketMQ message and is responsible for producing the message.
It establishes a Keep-alive connection with one of the nodes in the NameServer cluster, periodically reads Topic routing information from NameServer, and stores the routing information in local memory.
It establishes a long connection to the Master Broker that provides Topic services and periodically sends a heartbeat to the Master Broker;
It only sends messages to Master Broker, and chooses the appropriate Queue from the Message Queue list to send messages for load balancing.
It supports sending messages of various types, such as ordinary messages, transaction messages, timed messages, etc.
It sends messages in three ways: synchronous, asynchronous, one-way, and so on.
Simple interactive diagrams of the production side with Master Broker and NameServer can be viewed:
Note: Producers can also interact with other features such as broker queries for messages.
2. The producer starts the process:
Before we understand the specific production start-up process, we first ask a few questions to analyze the source code with questions:
-
What exactly did the message producer do when it started?
-
An application needs to send multiple topics, different topics need to send broker s to different clusters. What should I do?
We can first understand and analyze class diagram relationships related to producers:
As you can see from the class diagram, there are two implementations of MQProducer.
One is DefaultMQProducer (non-transactional message producer);
One is TransactionMQProducer, which supports transactional messages;
Next, a simple analysis of the docking class core parameters or methods is performed:
2.1 MqAdmin
MqAdmin: Core Method Resolution (Mq Management Foundation Interface)
//Create a theme void createTopic(final String key, final String newTopic, final int queueNum) throws MQClientException; //Find offset from queue based on Timestamp long searchOffset(final MessageQueue mq, final long timestamp) throws MQClientException; //Find the maximum physical offset in the message queue long maxOffset(final MessageQueue mq) throws MQClientException; //Find the minimum physical offset in the message queue. long minOffset(final MessageQueue mq) throws MQClientException; //Get the earliest stored message time long earliestMsgStoreTime(final MessageQueue mq) throws MQClientException; //Find messages based on message offset MessageExt viewMessage(final String offsetMsgId) throws RemotingException, MQBrokerException, InterruptedException, MQClientException; //Query messages based on conditions QueryResult queryMessage(final String topic, final String key, final int maxNum, final long begin, final long end) throws MQClientException, InterruptedException; //Find messages based on subject and message ID. MessageExt viewMessage(String topic, String msgId) throws RemotingException, MQBrokerException, InterruptedException, MQClientException;
2.2 Core Method Resolution
MQProducer: Core Method Resolution (Producer Base Interface):
//start-up void start() throws MQClientException; //Close void shutdown(); //Get corresponding queue information from top List<MessageQueue> fetchPublishMessageQueues(final String topic) throws MQClientException; //Sync-Message Sending SendResult send(final Message msg, final MessageQueue mq) throws MQClientException,RemotingException, MQBrokerException, InterruptedException; //Asynchronous-Message Sending void send(final Message msg, final MessageQueueSelector selector, final Object arg, final SendCallback sendCallback) throws MQClientException, RemotingException, InterruptedException; //Sync-Select Queue Message Sending SendResult send(final Message msg, final MessageQueueSelector selector, final Object arg, final long timeout) throws MQClientException, RemotingException, MQBrokerException,InterruptedException; //One-way Message Sending void sendOneway(final Message msg, final MessageQueue mq) throws MQClientException, RemotingException, InterruptedException; //Transaction Message-Send TransactionSendResult sendMessageInTransaction(final Message msg, final Object arg) throws MQClientException; //Batch Message-Send SendResult send(final Collection<Message> msgs) throws MQClientException, RemotingException, MQBrokerException, InterruptedException;
Note: where start() and shutdown() are started and closed by the producer,
2.3 clientConfig
clientConfig: Core Property Method Resolution (Client Configuration)
//nameServer-Address, default from: System property: rocketmq.namesrv.addr or environment variable: NAMESRV_ Get in ADDR
private String namesrvAddr = System.getProperty(MixAll.NAMESRV_ADDR_PROPERTY, System.getenv(MixAll.NAMESRV_ADDR_ENV));
//Instance name, default: DEFAULT or system property - rocketmq.client.name
private String instanceName = System.getProperty("rocketmq.client.name", "DEFAULT");
//Build an id for the mq client, for example: ip@instanceName @unitName: 172.16.62. 75@19312 @unitName
public String buildMQClientId() {
StringBuilder sb = new StringBuilder();
sb.append(this.getClientIP());
sb.append("@");
sb.append(this.getInstanceName());
if (!UtilAll.isBlank(this.unitName)) {
sb.append("@");
sb.append(this.unitName);
}
return sb.toString();
}
//Set namesrv address
public void setNamesrvAddr(String namesrvAddr) {
this.namesrvAddr = namesrvAddr;
}Note: namesrvAddr denotes the nameServer address and can be set by calling the setNamesrvAddr method or by environment variables or system properties. BuilMQClientId means set producer Id.
TransactionMQProducer: (Transaction message, explained separately later, ignored in this chapter)
(omitted)
4. DefaultMQProducer core attribute method parsing: (non-transactional message producer)
// constructor
public DefaultMQProducer(final String producerGroup, RPCHook rpcHook) {
this.producerGroup = producerGroup;
defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
}
// Various Sending Messages
public SendResult send(
Message msg) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
return this.defaultMQProducerImpl.send(msg);
}
// Startup Method
public void start() throws MQClientException {
this.defaultMQProducerImpl.start();
if (null != traceDispatcher) {
try {
traceDispatcher.start(this.getNamesrvAddr());
} catch (MQClientException e) {
log.warn("trace dispatcher start failed ", e);
}
}
}Note: The constructor of DefaultMQProducer, send and start related methods are all around DefaultMQProducerImpl, defaultMQProducerImpl: DefaultMQProducerImpl: Default producer's implementation class, whose start method is the core method for producer startup. Next, we will analyze the implementation of its start method.
DefaultMQProducerImpl#start
/**
* mq-producer start-up
* @param startFactory
* @throws MQClientException
*/
public void start(final boolean startFactory) throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
// 0-Service Status Settings
this.serviceState = ServiceState.START_FAILED;
//1-Detection Configuration
this.checkConfig();
//2-and change the producer's instanceName to the process ID.
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
this.defaultMQProducer.changeInstanceNameToPID();
}
//3-Create an instance of MQClientlnstance
this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQProducer, rpcHook);
//4-Register the producer with MQClientlnstance.
boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
//5-Default topic Information Cache (this.defaultMQProducer.getCreateTopicKey() ='TBW102')
this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
//6-Whether to Start-mQClientFactory
if (startFactory) {
mQClientFactory.start();
}
log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
this.defaultMQProducer.isSendMessageWithVIPChannel());
this.serviceState = ServiceState.RUNNING;//Set state to running
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The producer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
//7-Send heartbeat to all broker s
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
}The analysis is as follows:
0-Service Status Settings:
The meaning of setting the state value is to prevent duplicate startup, and its enumeration class is: ServiceState; If the initialization state is not equal to: CREATE_JUST, then run out abnormally
1-Detect configuration:
private void checkConfig() throws MQClientException {
Validators.checkGroup(this.defaultMQProducer.getProducerGroup());
if (null == this.defaultMQProducer.getProducerGroup()) {
throw new MQClientException("producerGroup is null", null);
}
//The production group cannot be equal to DEFAULT_PRODUCER
if (this.defaultMQProducer.getProducerGroup().equals(MixAll.DEFAULT_PRODUCER_GROUP)) {
throw new MQClientException("producerGroup can not equal " + MixAll.DEFAULT_PRODUCER_GROUP + ", please specify another one.",
null);
}
}Note: To detect the legality of -producerGroup
2-and change the producer's instanceName to the process ID.
// Determine if producerGroup equals CLIENT_INNER_PRODUCER
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
this.defaultMQProducer.changeInstanceNameToPID();
}
Is Call ClientConfig#changeInstanceNameToPID
public void changeInstanceNameToPID() {
if (this.instanceName.equals("DEFAULT")) {
this.instanceName = String.valueOf(UtilAll.getPid());
}
}Note: instanceName == DEFAULT, change it to the process ID of the startup for the purpose of MQClientInstance building
3-Create an instance of MQClientlnstance
MQClientManager manages MQClientInstance, its internal maintained data structure is: ConcurrentHashMap, key:clientId, and MQClientManager itself is a singleton mode, the core method analysis is as follows:
MQClientManager
private static MQClientManager instance = new MQClientManager();//-Single column mode
private AtomicInteger factoryIndexGenerator = new AtomicInteger();//index's factory
// MQClientInstance Cache
private ConcurrentMap<String/* clientId */, MQClientInstance> factoryTable = new ConcurrentHashMap<String, MQClientInstance>();
//Build Return MQClientInstance
public MQClientInstance getAndCreateMQClientInstance(final ClientConfig clientConfig, RPCHook rpcHook) {
String clientId = clientConfig.buildMQClientId();//id of building mq client
MQClientInstance instance = this.factoryTable.get(clientId);
if (null == instance) {
instance = new MQClientInstance(clientConfig.cloneClientConfig(),
this.factoryIndexGenerator.getAndIncrement(), clientId, rpcHook);
MQClientInstance prev = this.factoryTable.putIfAbsent(clientId, instance);
if (prev != null) {
instance = prev;
log.warn("Returned Previous MQClientInstance for clientId:[{}]", clientId);
} else {
log.info("Created new MQClientInstance for clientId:[{}]", clientId);
}
}
return instance;
}Remarks:
ClientConfig.buildMQClientId is analyzed above to build clientId;getAndCreateMQClientInstance This method is designed to build or query MQClientInstance.
MQClientInstance: Encapsulates the RocketMQ network processing API, which is the network channel that message producers (Producers), message consumers (Consumers) interact with NameServer, Broker.
Next, analyze the advantages and disadvantages of using the same MQClientInstance with multiple producers:
-
Benefits: In general, to reduce client resource usage, if all instanceName s and unitName s are set to the same value, only one instance of MQClientInstance will be created (for producer topic sending messages in the same set of broker clusters)
-
Disadvantages: What happens if multiple topic s reuse MQClientInstance? This happens when you start multiple Producers in a JVM without setting instanceName and unitName, then the two Producers will share an MQClientInstance and the messages will be routed to the same cluster.
For example, you started two Producers and configured different Name Server addresses, which meant that the two Producers would allocate messages to different clusters, but because they shared an MQClientInstance, this MQClientInstance was built on a prior Producer configuration, the second Producer was considered the same instance and the configuration was the same when it was shared. Messages are routed the same way and do not achieve the desired results.
4-Register the producer with MQClientInstance.
//Key:group, value:producer
private final ConcurrentMap<String/* group */, MQProducerInner> producerTable = new ConcurrentHashMap<String, MQProducerInner>();
// Add current producer to MQClientlnstance management
public boolean registerProducer(final String group, final DefaultMQProducerImpl producer) {
if (null == group || null == producer) {
return false;
}
MQProducerInner prev = this.producerTable.putIfAbsent(group, producer);
if (prev != null) {
log.warn("the producer group[{}] exist already.", group);
return false;
}
return true;
}Note: The interface class implemented by DefaultMQProducerImpl is: MQProducerInner
5-Add the default topic information cache, here you need to understand the meaning of the topicPublishInfoTable data structure
//key:topic value:TopicPublishInfo-Routing related information for message sending
private final ConcurrentMap<String/* topic */, TopicPublishInfo> topicPublishInfoTable =
new ConcurrentHashMap<String, TopicPublishInfo>();TopicPublishInfo:
Analysis, familiar admiration familiar taste, MessageQueue and TopicRouteData have been analyzed quite cleanly in NameServer, as follows:
public class TopicPublishInfo {
//Is it a sequential message
private boolean orderTopic = false;
//Whether to include routing information
private boolean haveTopicRouterInfo = false;
//Message queue for this topic queue
private List<MessageQueue> messageQueueList = new ArrayList<MessageQueue>();
//This value increases by 1 for each message queue selected, if Integer.MAX_VALUE, then reset to 0 to select the message queue.
private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex();
//Routing information
private TopicRouteData topicRouteData;
//Select the queue method, lastBrokerName actually failed the last time brokerName was sent. If not empty, this brokerName where the queue was sent this time will choose another brokerName
public MessageQueue selectOneMessageQueue(final String lastBrokerName) {
if (lastBrokerName == null) {
return selectOneMessageQueue();
} else {
//If the message is sent again unsuccessfully, avoid the Broker where the last MesageQueue was located the next time you make a message queue selection, otherwise you are likely to fail again.
int index = this.sendWhichQueue.getAndIncrement();
for (int i = 0; i < this.messageQueueList.size(); i++) {
int pos = Math.abs(index++) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
MessageQueue mq = this.messageQueueList.get(pos);
if (!mq.getBrokerName().equals(lastBrokerName)) {
return mq;
}
}
return selectOneMessageQueue();
}
}
//Get the value by increasing sendWhichQueue directly, then modularize it with the number of message queues in the current routing table, and return the MessageQueue at that location (selectOneMessageQueue() method)
public MessageQueue selectOneMessageQueue() {
int index = this.sendWhichQueue.getAndIncrement();
int pos = Math.abs(index) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
return this.messageQueueList.get(pos);
}
//messageQueueList-Length greater than 0
public boolean ok() {
return null != this.messageQueueList && !this.messageQueueList.isEmpty();
}
//Whether to include routing information
public boolean isHaveTopicRouterInfo() {
return haveTopicRouterInfo;
}6-Start-MQClientInstance
MQClientInstance#start
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED; //1>Status-Settings failed to start
// If not specified,looking address from name server
if (null == this.clientConfig.getNamesrvAddr()) { //2>Determine if the nameSrvAddr address is empty, http gets the nameSrvAddr
this.mQClientAPIImpl.fetchNameServerAddr();
}
// Start request-response channel : netty
this.mQClientAPIImpl.start(); // 3>Start netty correlation
// Start various schedule tasks
this.startScheduledTask(); //4> [Important] Start relevant timed tasks
// Start pull service
this.pullMessageService.start(); //5>Consumer-side correlation, follow-up explanation
// Start rebalance service
this.rebalanceService.start(); //6>Consumer-related, follow-up explanation
// Start push service
this.defaultMQProducer.getDefaultMQProducerImpl().start(false); //7>Start an mqProducter internally, startFactory=false
log.info("the client factory [{}] start OK", this.clientId);
this.serviceState = ServiceState.RUNNING; >8 Status Settings Running
break;
case RUNNING:
break;
case SHUTDOWN_ALREADY:
break;
case START_FAILED:
throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
default:
break;
}
}
}Note: A separate explanation follows: this.startScheduledTask();
7-Send heartbeat to all broker s
(this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();)
public void sendHeartbeatToAllBrokerWithLock() {
if (this.lockHeartbeat.tryLock()) {
try {
this.sendHeartbeatToAllBroker(); 1> Send a heartbeat to all broker
this.uploadFilterClassSource(); 2> Update Filtering filterSource Neglectable
} catch (final Exception e) {
log.error("sendHeartbeatToAllBroker exception", e);
} finally {
this.lockHeartbeat.unlock();
}
} else {
log.warn("lock heartBeat, but failed.");
}
}Note: sendHeartbeatToAllBroker, relatively simple,
The brokerVersionTable(ConcurrentHashMap) is maintained for the returned results, and you can't miss it. There are timed tasks that send heartbeats to all brokers at regular intervals
Summary: Through seven steps, we have learned the start-up process of producers, which can be roughly divided into: detecting related configurations, registering and building related classes (e.g. MQClientInstance related, netty related, etc.), and then starting related timed tasks; Summarize briefly the start-up process of the producer as follows:
Article Navigation
| category | Title | Release |
| Redis | Redis(1): Why can a single thread be so fast | Published |
| Redis(2): Memory model and recycling algorithm | Published | |
| Redis(3): Persistence | Published | |
| Redis(4): master-slave synchronization | 2021.11.05 | |
| Redis(5): Cluster building | Upcoming Online | |
| Redis(6): practical applications | Upcoming Online | |
| Elasticsearch | Elasticsearch: Overview | Published |
| Elasticsearch: Core | Published | |
| Elasticsearch: Actual Warfare | Published | |
| Elasticsearch Writing Process Details | Upcoming Online | |
| Elasticsearch Query Process Details | Upcoming Online | |
| Elasticsearch cluster consistency | Upcoming Online | |
| Basic concepts of Lucene | Upcoming Online | |
| Elasticsearch deployment architecture and capacity planning | Upcoming Online | |
| RocketMQ | RocketMQ-NameServer Summary and Core Source Analysis | Published |
| RocketMQ - An in-depth analysis of Producer | Upcoming Online | |
| RocketMQ-Producer(1) Start Process Decryption | This article | |
| RocketMQ-Producer(2) Dynamic Routing Update | 2021.11.08 | |
| RocketMQ-Producer(3) Send Mode and Message Type | 2021.11.10 | |
| RocketMQ-Producer(4) Message Sending Process | 2021.11.12 | |
| RocketMQ-Producer(5) Routing queue selection, client redundancy policy | ||
| Broker - Start Process Source Decryption | Upcoming Online | |
| Broker - Accepts message processing flow decryption | Upcoming Online | |
| Tomcat Source Analysis | Tomcat(1): Project structure and architecture analysis | Upcoming Online |
| Tomcat(2): Start Close Process Analysis | Upcoming Online | |
| Tomcat(3): Applying Loading Principle Analysis | Upcoming Online | |
| Tomcat(4): Analysis of Network Request Principle | Upcoming Online | |
| Tomcat(5): Embedding and performance tuning | Upcoming Online | |
| Nacos | Nacos Project Structure and Architecture Analysis | Upcoming Online |
| Nacos Service Registration Source Resolution | Upcoming Online | |
| Nacos Configuration Management Source Resolution | Upcoming Online | |
| Nacos Version 2.0 Optimization Function Resolution | Upcoming Online | |
| Netty | Computer Network &nio Core Principles | Upcoming Online |
| Explain in detail how kafka encapsulates network communication components based on native nio? | Upcoming Online | |
| Introducing the Core Components of netty Initial Knowledge | Upcoming Online | |
| Source Resolution netty Server, how does the port pull up? What about new connection access? | Upcoming Online | |
| How can events in deep netty be propagated in pipeline s? | Upcoming Online | |
| What is unpacking and gluing in network programming? How is kafka and netty solved? | Upcoming Online | |
| An in-depth explanation of how the most complex cache allocation in netty works? | Upcoming Online | |
| Source analysis netty, kafka, sentinel implementation of different time wheels and details | Upcoming Online | |
| Try to compare the performance optimization in kafka & netty from a God's point of view and enrich the use of various design modes | Upcoming Online | |
| Netty's Practice in Rocketmq and Interpretation of RocketMq's Message Protocol | Upcoming Online | |
Focus on top IT technology, trust authors and get the following PDF information for the 2021 Global Architect Summit.
