RocketMq cluster construction steps teaching package teaching package meeting

Keywords: Apache vim ssh SELinux

;

mq cluster construction

I haven't written these technical articles for a while. I'll spare some time to write them today, or I'll forget them
This article records the process of building rocketmq cluster, and it's also a half day's achievement record! Interested friends like to go!

Well, there's not much bullshit. Let's start.

This article is written with reference to https://blog.csdn.net/qq/article/details/82467562

preparation

Step 1: turn off the firewall of all the machines to be built
Step 2: perform the following steps for each machine

[root@ma01 ~]# vim /etc/sysconfig/selinux
......
SELINUX=disabled
[root@ma01~]# setenforce 0
[root@ma01~]# getenforce

Step 3: install JDK, maven, zip, unzip and SSH password free login on all machines

Configure crt connection: https://blog.csdn.net/cmqwan/article/details/61932792
 To install maven, please refer to my blog: https://www.cnblogs.com/clicli/p/5866390.html
 Install zip,unzip reference: http://www.rpmfind.net/linux/rpm2html/search.php? Query = zip & submit = search +... & system = & arch=
Installation ssh reference: https://blog.csdn.net/m0_/article/details/74275859
 jdk Baidu ha, a lot of reference blog!

Step 4: the following command is used to copy between ssh machines

scp -r /home/administrator/test/ root@192.168.1.100:/root/


Step 5: after downloading, unzip

unzip  rocketmq-all-4.4.0-bin-release.zip 

Step 6: enter the extracted folder rocketmq-bin4.4.0, and create logs, data / store, and data2 / store directories in the folder

Step 7: modify several startup files under bin in installation order. Because the default configuration memory space is too large, local startup will report an error

1. vim runbroker.sh    Change the corresponding place to  -server -Xms512m -Xmx512m -Xmn256m
2. vim runserver.sh (The same thing) -server -Xms512m -Xmx512m -Xmn126m -XX:PermSize=128m -XX:MaxPermSize=320m
3. vim tools.sh           -server -Xms256m -Xmx256m -Xmn128m -XX:PermSize=128m -XX:MaxPermSize=128m

Step 8: modify these four files under rocketmq-bin4.4.0/conf/2m-2s-async

Note: I have used three machines. All of them are configured with 130131 and 132. You can use two machines completely. If you configure one of 131 and 132, you can use it. ip can change it by yourself.

Step 9: modify the following configuration file for the 130 main machine. The contents of the two files are a s follows: broker-a.properties broker-b-s.properties
broker-a.properties

brokerClusterName=RocketMQCluster
brokerName=broker-a
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
##Broker listening port for external services
listenPort=10911
#nameserver address, semicolon split
namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876
#When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default
defaultTopicQueueNums=4
#Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online
autoCreateTopicEnable=true
#Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online
autoCreateSubscriptionGroup=true
brokerIP1=192.168.175.130
storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data/store
storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data/store/commitlog
# Consumption queue storage path storage path
storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data/store/consumequeue
#Message index storage path
storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data/store/index
#checkpoint file storage path
storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data/store/checkpoint
#abort file storage path
abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data/store/abort
#Delete file at 4 a.m. by default
deleteWhen=04
#File retention time, default 48 hours
fileReservedTime=120
# commitLog the size of each file is 1G by default
mapedFileSizeCommitLog=1073741824
#Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation
mapedFileSizeConsumeQueue=3000000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#Detect physical file disk space
#diskMaxUsedSpaceRatio=88

broker-b-s.properties

brokerClusterName=RocketMQCluster
brokerName=broker-b
brokerId=1
deleteWhen=04
fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
listenPort=10921
#nameserver address, semicolon split
namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876
brokerIP1=192.168.175.130
#When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default
defaultTopicQueueNums=4
#Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online
autoCreateTopicEnable=true
#Whether Broker is allowed to automatically create subscription groups? It is recommended to turn them on offline and off online
autoCreateSubscriptionGroup=true
storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data2/store
storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/commitlog2
# Consumption queue storage path storage path
storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/consumequeue2
#Message index storage path
storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/index2
#checkpoint file storage path
storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/checkpoint2
#abort file storage path
abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/abort2
#Delete file at 4 a.m. by default
deleteWhen=04
#File retention time, default 48 hours
fileReservedTime=120
# commitLog the size of each file is 1G by default
mapedFileSizeCommitLog=1073741824
#Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation
mapedFileSizeConsumeQueue=3000000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#Detect physical file disk space
#diskMaxUsedSpaceRatio=88

Step 10: the 131132 machine only modifies the contents of broker-b.properties and broker-a-s.properties as follows:
broker-b.properties

# limitations under the License.
brokerClusterName=RocketMQCluster
brokerName=broker-b
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
listenPort=10911
#nameserver address, semicolon split
namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876
brokerIP1=192.168.175.131
#When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default
defaultTopicQueueNums=4
#Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online
autoCreateTopicEnable=true
#Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online
autoCreateSubscriptionGroup=true

storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data/store
storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data/store/commitlog
# Consumption queue storage path storage path
storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data/store/consumequeue
#Message index storage path
storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data/store/index
#checkpoint file storage path
storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data/store/checkpoint
#abort file storage path
abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data/store/abort
#Message size limit
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Delete file at 4 a.m. by default
deleteWhen=04
#File retention time, default 48 hours
fileReservedTime=120
# commitLog the size of each file is 1G by default
mapedFileSizeCommitLog=1073741824
#Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation
mapedFileSizeConsumeQueue=3000000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#Detect physical file disk space
#diskMaxUsedSpaceRatio=88

broker-a-s.properties

# limitations under the License.
brokerClusterName=RocketMQCluster
brokerName=broker-a
brokerId=1
deleteWhen=04
fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

listenPort=10921
namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876
brokerIP1=192.168.175.131
#When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default
defaultTopicQueueNums=4
#Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online
autoCreateTopicEnable=true
#Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online
autoCreateSubscriptionGroup=true

storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data2/store
storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/commitlog
# Consumption queue storage path storage path
storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/consumequeue
#Message index storage path
storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/index
#checkpoint file storage path
storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/checkpoint
#abort file storage path
abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/abort
#Message size limit
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000

#Delete file at 4 a.m. by default
deleteWhen=04
#File retention time, default 48 hours
fileReservedTime=120
# commitLog the size of each file is 1G by default
mapedFileSizeCommitLog=1073741824
#Each file of ConsumeQueue saves 30W by default, which is adjusted according to the business situation
mapedFileSizeConsumeQueue=300000
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000
#Detect physical file disk space
#diskMaxUsedSpaceRatio=88

Step 11: start


//All three are implemented:

nohup sh bin/mqnamesrv > ./logs/namesrvrun.log 2>&1 &

130 Machine execution: 
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-a.log 2>&1 &

nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b-s.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-b-s.log 2>&1 &

131, 132 Machine execution:
nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b.properties  -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-b.log 2>&1 &

nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a-s.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-a-s.log 2>&1 &

After execution, the result of jps is that there are two brokerstartup s. If the error is reported, look at the logs folder log created by yourself

Now, the basic configuration of rocketmq is set up. Let's talk about the actual code

Import dependency package

	<dependencies>
		<!-- https://mvnrepository.com/artifact/org.apache.rocketmq/rocketmq-client -->
		<dependency>
			<groupId>org.apache.rocketmq</groupId>
			<artifactId>rocketmq-client</artifactId>
			<version>4.4.0</version>
		</dependency>
		<dependency>
			<groupId>com.alibaba.rocketmq</groupId>
			<artifactId>rocketmq-all</artifactId>
			<version>3.5.9</version>
			<type>pom</type>
		</dependency>
		<dependency>
			<groupId>ch.qos.logback</groupId>
			<artifactId>logback-classic</artifactId>
			<version>1.1.1</version>
		</dependency>
		<dependency>
			<groupId>ch.qos.logback</groupId>
			<artifactId>logback-core</artifactId>
			<version>1.1.1</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.10</version>
			<scope>test</scope>
		</dependency>
	</dependencies>

mq message sender


import java.io.UnsupportedEncodingException;

import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingException;

/**
 * message sender 
 * @author LELE
 *
 */
public class Producer {

	public static void main(String[] args) throws MQClientException, InterruptedException {

		// Declare and initialize a producer
		// A producer group name is required as a parameter of the construction method, producer 1 here
		DefaultMQProducer producer = new DefaultMQProducer("producer1");
		producer.setVipChannelEnabled(false);
		// Set the NameServer address, which should be changed to the actual NameServer address, separated by
		// NameServer address must have
		// producer.setClientIP("xxxx");
		// producer.setInstanceName("Producer");
		producer.setNamesrvAddr("192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876");

		// Call the start() method to start a producer instance
		producer.start();

		// Send a message to the Topic as TopicTest, tag as TagA, and the message content as the value of i on the "Hello RocketMQ" splice
		try {
			for(int i=0;i<30000;i++) {
				// Encapsulate message
				Message msg = new Message("TopicTest", // topic
						"TagA", // tag
						("Hello RocketMQ--------"+i).getBytes(RemotingHelper.DEFAULT_CHARSET)// body
				);
				// Call producer's send() method to send a message
				// The synchronization method is called here, so there will be a return result
				SendResult sendResult = producer.send(msg);
				// Print return results
				System.out.println(sendResult);
			}
			
		} catch (RemotingException e) {
			e.printStackTrace();
		} catch (MQBrokerException e) {
			e.printStackTrace();
		} catch (UnsupportedEncodingException e) {
			e.printStackTrace();
		}
		// After sending the message, call the shutdown() method to shut down the producer
		System.out.println("send success");
		producer.shutdown();
	}

}

Message consumer


import java.util.List;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;

/**
 * The message receiver needs the server to start the mq service
 * @author LELE
 *
 */
public class Consumer {

	public static void main(String[] args) throws MQClientException {

		// Declare and initialize a consumer
		// A consumer group name is required as the parameter of the construction method, here is consumer1
		DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer1");
		// consumer.setVipChannelEnabled(false);
		// Also set the NameServer address
		consumer.setNamesrvAddr("192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876");

		// Here is a consumer's consumption strategy
		// Default policy of consume from the end of the queue, i.e. skip historical messages
		// Consume? From? First? Offset starts from the beginning of the queue, that is, all the historical messages (also stored in the broker) are consumed once
		// Consumption starts from a certain time point and is used with setConsumeTimestamp(). The default is half an hour ago
		consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);

		// Set the Topic and Tag subscribed by consumer, * represents all tags
		consumer.subscribe("TopicTest", "*");

		// Set a Listener, mainly for message logical processing
		consumer.registerMessageListener(new MessageListenerConcurrently() {

			@Override
			public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {

				MessageExt msg = msgs.get(0);

				if (msg.getTopic().equals("TopicTest")) {

					// Execute the consumption logic of TopicTest1

					System.out.println("TagA:" + new String(msg.getBody()));

				}

				System.out.println(Thread.currentThread().getName() + " Receive New Messages: " + msgs.size()
						+ "----------------------------------------------------------------------------------");
				// Return to consumption status
				// Success of consumption
				// Reconsume? Later consumption failed, need to be re consumed later
				return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
			}
		});
		// Call the start() method to start consumer
		consumer.start();
		System.out.println("Consumer Started.");

	}
}

Start to play, young man, remember to like it!

Posted by scoppc on Thu, 07 May 2020 00:07:47 -0700