RocketMQ-Producer Start Process Decryption

Keywords: Java Distribution architecture Middleware message queue

The following is the RocketMQ - Producer series article index:

1: Start the process (this article)

2: Routing dynamic updates

3: Send Mode Analysis + Message Type Difference

4: Sending process of producer message

5: Routing queue selection, client redundancy policy (QA)

Introduction to Producer

Producer is the sender of the RocketMQ message and is responsible for producing the message.

It establishes a Keep-alive connection with one of the nodes in the NameServer cluster, periodically reads Topic routing information from NameServer, and stores the routing information in local memory.

It establishes a long connection to the Master Broker that provides Topic services and periodically sends a heartbeat to the Master Broker;

It only sends messages to Master Broker, and chooses the appropriate Queue from the Message Queue list to send messages for load balancing.

It supports sending messages of various types, such as ordinary messages, transaction messages, timed messages, etc.

It sends messages in three ways: synchronous, asynchronous, one-way, and so on.

Simple interactive diagrams of the production side with Master Broker and NameServer can be viewed:

Note: Producers can also interact with other features such as broker queries for messages.

2. The producer starts the process:

Before we understand the specific production start-up process, we first ask a few questions to analyze the source code with questions:

  1. What exactly did the message producer do when it started?

  2. An application needs to send multiple topics, different topics need to send broker s to different clusters. What should I do?

We can first understand and analyze class diagram relationships related to producers:

As you can see from the class diagram, there are two implementations of MQProducer.

One is DefaultMQProducer (non-transactional message producer);  

One is TransactionMQProducer, which supports transactional messages;  

Next, a simple analysis of the docking class core parameters or methods is performed:

2.1 MqAdmin

MqAdmin: Core Method Resolution (Mq Management Foundation Interface)     

//Create a theme
void createTopic(final String key, final String newTopic, final int queueNum) throws MQClientException;
//Find offset from queue based on Timestamp
long searchOffset(final MessageQueue mq, final long timestamp) throws MQClientException;
//Find the maximum physical offset in the message queue
long maxOffset(final MessageQueue mq) throws MQClientException;
//Find the minimum physical offset in the message queue.
long minOffset(final MessageQueue mq) throws MQClientException;
//Get the earliest stored message time
long earliestMsgStoreTime(final MessageQueue mq) throws MQClientException;
//Find messages based on message offset
MessageExt viewMessage(final String offsetMsgId) throws RemotingException, MQBrokerException, InterruptedException, MQClientException;
//Query messages based on conditions
QueryResult queryMessage(final String topic, final String key, final int maxNum, final long begin, final long end) throws MQClientException, InterruptedException;
//Find messages based on subject and message ID.
MessageExt viewMessage(String topic, String msgId) throws RemotingException, MQBrokerException, InterruptedException, MQClientException;

2.2 Core Method Resolution

MQProducer: Core Method Resolution (Producer Base Interface):

void start() throws MQClientException;
void shutdown();
//Get corresponding queue information from top
List<MessageQueue> fetchPublishMessageQueues(final String topic) throws MQClientException;
//Sync-Message Sending
SendResult send(final Message msg, final MessageQueue mq) throws MQClientException,RemotingException, MQBrokerException, InterruptedException;
//Asynchronous-Message Sending
void send(final Message msg, final MessageQueueSelector selector, final Object arg, final SendCallback sendCallback) throws MQClientException, RemotingException, InterruptedException;
//Sync-Select Queue Message Sending
SendResult send(final Message msg, final MessageQueueSelector selector, final Object arg, final long timeout) throws MQClientException, RemotingException, MQBrokerException,InterruptedException;
//One-way Message Sending    
void sendOneway(final Message msg, final MessageQueue mq) throws MQClientException, RemotingException, InterruptedException;
//Transaction Message-Send
TransactionSendResult sendMessageInTransaction(final Message msg, final Object arg) throws MQClientException;
//Batch Message-Send   
SendResult send(final Collection<Message> msgs) throws MQClientException, RemotingException, MQBrokerException, InterruptedException;

Note: where start() and shutdown() are started and closed by the producer,

2.3 clientConfig

clientConfig: Core Property Method Resolution (Client Configuration)

//nameServer-Address, default from: System property: rocketmq.namesrv.addr or environment variable: NAMESRV_ Get in ADDR
private String namesrvAddr = System.getProperty(MixAll.NAMESRV_ADDR_PROPERTY, System.getenv(MixAll.NAMESRV_ADDR_ENV));
//Instance name, default: DEFAULT or system property -
private String instanceName = System.getProperty("", "DEFAULT");
//Build an id for the mq client, for example: ip@instanceName @unitName: 172.16.62. 75@19312 @unitName
public String buildMQClientId() {
    StringBuilder sb = new StringBuilder();
    if (!UtilAll.isBlank(this.unitName)) {
    return sb.toString();
//Set namesrv address
public void setNamesrvAddr(String namesrvAddr) {
    this.namesrvAddr = namesrvAddr;

Note: namesrvAddr denotes the nameServer address and can be set by calling the setNamesrvAddr method or by environment variables or system properties. BuilMQClientId means set producer Id.

TransactionMQProducer: (Transaction message, explained separately later, ignored in this chapter)


4. DefaultMQProducer core attribute method parsing: (non-transactional message producer)

// constructor
public DefaultMQProducer(final String producerGroup, RPCHook rpcHook) {
    this.producerGroup = producerGroup;
    defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
// Various Sending Messages
public SendResult send(
    Message msg) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    return this.defaultMQProducerImpl.send(msg);
// Startup Method
public void start() throws MQClientException {
    if (null != traceDispatcher) {
        try {
        } catch (MQClientException e) {
            log.warn("trace dispatcher start failed ", e);

Note: The constructor of DefaultMQProducer, send and start related methods are all around DefaultMQProducerImpl, defaultMQProducerImpl: DefaultMQProducerImpl: Default producer's implementation class, whose start method is the core method for producer startup. Next, we will analyze the implementation of its start method.


 * mq-producer start-up
 * @param startFactory
 * @throws MQClientException
public void start(final boolean startFactory) throws MQClientException {
    switch (this.serviceState) {
        case CREATE_JUST:
            // 0-Service Status Settings
            this.serviceState = ServiceState.START_FAILED;
            //1-Detection Configuration
            //2-and change the producer's instanceName to the process ID.
            if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
            //3-Create an instance of MQClientlnstance
            this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQProducer, rpcHook);
            //4-Register the producer with MQClientlnstance.
            boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
            if (!registerOK) {
                this.serviceState = ServiceState.CREATE_JUST;
                throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
                    + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
            //5-Default topic Information Cache (this.defaultMQProducer.getCreateTopicKey() ='TBW102')
            this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
            //6-Whether to Start-mQClientFactory
            if (startFactory) {
  "the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
            this.serviceState = ServiceState.RUNNING;//Set state to running
        case RUNNING:
        case START_FAILED:
        case SHUTDOWN_ALREADY:
            throw new MQClientException("The producer service state not OK, maybe started once, "
                + this.serviceState
                + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
    //7-Send heartbeat to all broker s

The analysis is as follows:  

0-Service Status Settings:

The meaning of setting the state value is to prevent duplicate startup, and its enumeration class is: ServiceState; If the initialization state is not equal to: CREATE_JUST, then run out abnormally

1-Detect configuration:

private void checkConfig() throws MQClientException {
    if (null == this.defaultMQProducer.getProducerGroup()) {
        throw new MQClientException("producerGroup is null", null);
    //The production group cannot be equal to DEFAULT_PRODUCER
    if (this.defaultMQProducer.getProducerGroup().equals(MixAll.DEFAULT_PRODUCER_GROUP)) {
        throw new MQClientException("producerGroup can not equal " + MixAll.DEFAULT_PRODUCER_GROUP + ", please specify another one.",

Note: To detect the legality of -producerGroup

2-and change the producer's instanceName to the process ID.

// Determine if producerGroup equals CLIENT_INNER_PRODUCER
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
Is Call ClientConfig#changeInstanceNameToPID
public void changeInstanceNameToPID() {
    if (this.instanceName.equals("DEFAULT")) {
        this.instanceName = String.valueOf(UtilAll.getPid());

Note: instanceName == DEFAULT, change it to the process ID of the startup for the purpose of MQClientInstance building

3-Create an instance of MQClientlnstance

MQClientManager manages MQClientInstance, its internal maintained data structure is: ConcurrentHashMap, key:clientId, and MQClientManager itself is a singleton mode, the core method analysis is as follows:  


private static MQClientManager instance = new MQClientManager();//-Single column mode
private AtomicInteger factoryIndexGenerator = new AtomicInteger();//index's factory
// MQClientInstance Cache
private ConcurrentMap<String/* clientId */, MQClientInstance> factoryTable = new ConcurrentHashMap<String, MQClientInstance>();
//Build Return MQClientInstance
public MQClientInstance getAndCreateMQClientInstance(final ClientConfig clientConfig, RPCHook rpcHook) {
    String clientId = clientConfig.buildMQClientId();//id of building mq client
    MQClientInstance instance = this.factoryTable.get(clientId);
    if (null == instance) {
        instance = new MQClientInstance(clientConfig.cloneClientConfig(),
                this.factoryIndexGenerator.getAndIncrement(), clientId, rpcHook);
        MQClientInstance prev = this.factoryTable.putIfAbsent(clientId, instance);
        if (prev != null) {
            instance = prev;
            log.warn("Returned Previous MQClientInstance for clientId:[{}]", clientId);
        } else {
  "Created new MQClientInstance for clientId:[{}]", clientId);
    return instance;


ClientConfig.buildMQClientId is analyzed above to build clientId;getAndCreateMQClientInstance This method is designed to build or query MQClientInstance.   

  MQClientInstance: Encapsulates the RocketMQ network processing API, which is the network channel that message producers (Producers), message consumers (Consumers) interact with NameServer, Broker.

Next, analyze the advantages and disadvantages of using the same MQClientInstance with multiple producers:

  1. Benefits: In general, to reduce client resource usage, if all instanceName s and unitName s are set to the same value, only one instance of MQClientInstance will be created (for producer topic sending messages in the same set of broker clusters)  

  2. Disadvantages: What happens if multiple topic s reuse MQClientInstance? This happens when you start multiple Producers in a JVM without setting instanceName and unitName, then the two Producers will share an MQClientInstance and the messages will be routed to the same cluster.

For example, you started two Producers and configured different Name Server addresses, which meant that the two Producers would allocate messages to different clusters, but because they shared an MQClientInstance, this MQClientInstance was built on a prior Producer configuration, the second Producer was considered the same instance and the configuration was the same when it was shared. Messages are routed the same way and do not achieve the desired results.

4-Register the producer with MQClientInstance.

//Key:group, value:producer
private final ConcurrentMap<String/* group */, MQProducerInner> producerTable = new ConcurrentHashMap<String, MQProducerInner>();
// Add current producer to MQClientlnstance management
public boolean registerProducer(final String group, final DefaultMQProducerImpl producer) {
    if (null == group || null == producer) {
        return false;
    MQProducerInner prev = this.producerTable.putIfAbsent(group, producer);
    if (prev != null) {
        log.warn("the producer group[{}] exist already.", group);
        return false;
    return true;

Note: The interface class implemented by DefaultMQProducerImpl is: MQProducerInner

5-Add the default topic information cache, here you need to understand the meaning of the topicPublishInfoTable data structure

//key:topic value:TopicPublishInfo-Routing related information for message sending
private final ConcurrentMap<String/* topic */, TopicPublishInfo> topicPublishInfoTable =
    new ConcurrentHashMap<String, TopicPublishInfo>();


Analysis, familiar admiration familiar taste, MessageQueue and TopicRouteData have been analyzed quite cleanly in NameServer, as follows:

public class TopicPublishInfo {
    //Is it a sequential message
    private boolean orderTopic = false;
    //Whether to include routing information
    private boolean haveTopicRouterInfo = false;
    //Message queue for this topic queue
    private List<MessageQueue> messageQueueList = new ArrayList<MessageQueue>();
    //This value increases by 1 for each message queue selected, if Integer.MAX_VALUE, then reset to 0 to select the message queue.
    private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex();
    //Routing information
    private TopicRouteData topicRouteData;
    //Select the queue method, lastBrokerName actually failed the last time brokerName was sent. If not empty, this brokerName where the queue was sent this time will choose another brokerName
    public MessageQueue selectOneMessageQueue(final String lastBrokerName) {
        if (lastBrokerName == null) {
            return selectOneMessageQueue();
        } else {
            //If the message is sent again unsuccessfully, avoid the Broker where the last MesageQueue was located the next time you make a message queue selection, otherwise you are likely to fail again.
            int index = this.sendWhichQueue.getAndIncrement();
            for (int i = 0; i < this.messageQueueList.size(); i++) {
                int pos = Math.abs(index++) % this.messageQueueList.size();
                if (pos < 0)
                    pos = 0;
                MessageQueue mq = this.messageQueueList.get(pos);
                if (!mq.getBrokerName().equals(lastBrokerName)) {
                    return mq;
            return selectOneMessageQueue();
    //Get the value by increasing sendWhichQueue directly, then modularize it with the number of message queues in the current routing table, and return the MessageQueue at that location (selectOneMessageQueue() method)
    public MessageQueue selectOneMessageQueue() {
        int index = this.sendWhichQueue.getAndIncrement();
        int pos = Math.abs(index) % this.messageQueueList.size();
        if (pos < 0)
            pos = 0;
        return this.messageQueueList.get(pos);
    //messageQueueList-Length greater than 0
    public boolean ok() {
        return null != this.messageQueueList && !this.messageQueueList.isEmpty();
    //Whether to include routing information
    public boolean isHaveTopicRouterInfo() {
        return haveTopicRouterInfo;



  public void start() throws MQClientException {
        synchronized (this) {
            switch (this.serviceState) {
                case CREATE_JUST:
                    this.serviceState = ServiceState.START_FAILED;   //1>Status-Settings failed to start
                    // If not specified,looking address from name server
                    if (null == this.clientConfig.getNamesrvAddr()) {  //2>Determine if the nameSrvAddr address is empty, http gets the nameSrvAddr
                    // Start request-response channel : netty
                    this.mQClientAPIImpl.start();  // 3>Start netty correlation
                    // Start various schedule tasks
                    this.startScheduledTask();   //4> [Important] Start relevant timed tasks
                    // Start pull service
                    this.pullMessageService.start();    //5>Consumer-side correlation, follow-up explanation
                    // Start rebalance service
                    this.rebalanceService.start();      //6>Consumer-related, follow-up explanation
                    // Start push service
                    this.defaultMQProducer.getDefaultMQProducerImpl().start(false); //7>Start an mqProducter internally, startFactory=false
          "the client factory [{}] start OK", this.clientId);
                    this.serviceState = ServiceState.RUNNING; >8 Status Settings Running
                case RUNNING:
                case SHUTDOWN_ALREADY:
                case START_FAILED:
                    throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);

Note: A separate explanation follows: this.startScheduledTask();

7-Send heartbeat to all broker s


    public void sendHeartbeatToAllBrokerWithLock() {
        if (this.lockHeartbeat.tryLock()) {
            try {
                this.sendHeartbeatToAllBroker(); 1> Send a heartbeat to all broker
                this.uploadFilterClassSource();  2> Update Filtering filterSource Neglectable
            } catch (final Exception e) {
                log.error("sendHeartbeatToAllBroker exception", e);
            } finally {
        } else {
            log.warn("lock heartBeat, but failed.");

Note: sendHeartbeatToAllBroker, relatively simple,

The brokerVersionTable(ConcurrentHashMap) is maintained for the returned results, and you can't miss it. There are timed tasks that send heartbeats to all brokers at regular intervals

Summary: Through seven steps, we have learned the start-up process of producers, which can be roughly divided into: detecting related configurations, registering and building related classes (e.g. MQClientInstance related, netty related, etc.), and then starting related timed tasks; Summarize briefly the start-up process of the producer as follows:

Article Navigation

RedisRedis(1): Why can a single thread be so fastPublished
Redis(2): Memory model and recycling algorithmPublished
Redis(3): PersistencePublished
Redis(4): master-slave synchronization2021.11.05
Redis(5): Cluster buildingUpcoming Online
Redis(6): practical applicationsUpcoming Online
ElasticsearchElasticsearch: OverviewPublished
Elasticsearch: CorePublished
Elasticsearch: Actual WarfarePublished
Elasticsearch Writing Process DetailsUpcoming Online
Elasticsearch Query Process DetailsUpcoming Online
Elasticsearch cluster consistencyUpcoming Online
Basic concepts of LuceneUpcoming Online
Elasticsearch deployment architecture and capacity planningUpcoming Online
RocketMQRocketMQ-NameServer Summary and Core Source AnalysisPublished
RocketMQ - An in-depth analysis of ProducerUpcoming Online
RocketMQ-Producer(1) Start Process DecryptionThis article
RocketMQ-Producer(2) Dynamic Routing Update2021.11.08
RocketMQ-Producer(3) Send Mode and Message Type2021.11.10
RocketMQ-Producer(4) Message Sending Process2021.11.12
RocketMQ-Producer(5) Routing queue selection, client redundancy policy
Broker - Start Process Source DecryptionUpcoming Online
Broker - Accepts message processing flow decryptionUpcoming Online
Tomcat Source AnalysisTomcat(1): Project structure and architecture analysisUpcoming Online
Tomcat(2): Start Close Process AnalysisUpcoming Online
Tomcat(3): Applying Loading Principle AnalysisUpcoming Online
Tomcat(4): Analysis of Network Request PrincipleUpcoming Online
Tomcat(5): Embedding and performance tuningUpcoming Online
NacosNacos Project Structure and Architecture AnalysisUpcoming Online
Nacos Service Registration Source ResolutionUpcoming Online
Nacos Configuration Management Source ResolutionUpcoming Online
Nacos Version 2.0 Optimization Function ResolutionUpcoming Online
NettyComputer Network &nio Core PrinciplesUpcoming Online
Explain in detail how kafka encapsulates network communication components based on native nio?Upcoming Online
Introducing the Core Components of netty Initial KnowledgeUpcoming Online
Source Resolution netty Server, how does the port pull up? What about new connection access?Upcoming Online
How can events in deep netty be propagated in pipeline s?Upcoming Online
What is unpacking and gluing in network programming? How is kafka and netty solved?Upcoming Online
An in-depth explanation of how the most complex cache allocation in netty works?Upcoming Online
Source analysis netty, kafka, sentinel implementation of different time wheels and detailsUpcoming Online
Try to compare the performance optimization in kafka & netty from a God's point of view and enrich the use of various design modesUpcoming Online
Netty's Practice in Rocketmq and Interpretation of RocketMq's Message ProtocolUpcoming Online


Focus on top IT technology, trust authors and get the following PDF information for the 2021 Global Architect Summit.




Posted by Pi_Mastuh on Tue, 02 Nov 2021 09:18:26 -0700