mq cluster construction
I haven't written these technical articles for a while. I'll spare some time to write them today, or I'll forget them
This article records the process of building rocketmq cluster, and it's also a half day's achievement record! Interested friends like to go!
Well, there's not much bullshit. Let's start.
This article is written with reference to https://blog.csdn.net/qq/article/details/82467562
preparation
Step 1: turn off the firewall of all the machines to be built
Step 2: perform the following steps for each machine
[root@ma01 ~]# vim /etc/sysconfig/selinux ...... SELINUX=disabled [root@ma01~]# setenforce 0 [root@ma01~]# getenforce
Step 3: install JDK, maven, zip, unzip and SSH password free login on all machines
Configure crt connection: https://blog.csdn.net/cmqwan/article/details/61932792 To install maven, please refer to my blog: https://www.cnblogs.com/clicli/p/5866390.html Install zip,unzip reference: http://www.rpmfind.net/linux/rpm2html/search.php? Query = zip & submit = search +... & system = & arch= Installation ssh reference: https://blog.csdn.net/m0_/article/details/74275859 jdk Baidu ha, a lot of reference blog!
Step 4: the following command is used to copy between ssh machines
scp -r /home/administrator/test/ root@192.168.1.100:/root/
Step 5: after downloading, unzip
unzip rocketmq-all-4.4.0-bin-release.zip
Step 6: enter the extracted folder rocketmq-bin4.4.0, and create logs, data / store, and data2 / store directories in the folder
Step 7: modify several startup files under bin in installation order. Because the default configuration memory space is too large, local startup will report an error
1. vim runbroker.sh Change the corresponding place to -server -Xms512m -Xmx512m -Xmn256m 2. vim runserver.sh (The same thing) -server -Xms512m -Xmx512m -Xmn126m -XX:PermSize=128m -XX:MaxPermSize=320m 3. vim tools.sh -server -Xms256m -Xmx256m -Xmn128m -XX:PermSize=128m -XX:MaxPermSize=128m
Step 8: modify these four files under rocketmq-bin4.4.0/conf/2m-2s-async
Note: I have used three machines. All of them are configured with 130131 and 132. You can use two machines completely. If you configure one of 131 and 132, you can use it. ip can change it by yourself.
Step 9: modify the following configuration file for the 130 main machine. The contents of the two files are a s follows: broker-a.properties broker-b-s.properties
broker-a.properties
brokerClusterName=RocketMQCluster brokerName=broker-a brokerId=0 deleteWhen=04 fileReservedTime=48 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH ##Broker listening port for external services listenPort=10911 #nameserver address, semicolon split namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876 #When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default defaultTopicQueueNums=4 #Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online autoCreateTopicEnable=true #Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online autoCreateSubscriptionGroup=true brokerIP1=192.168.175.130 storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data/store storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data/store/commitlog # Consumption queue storage path storage path storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data/store/consumequeue #Message index storage path storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data/store/index #checkpoint file storage path storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data/store/checkpoint #abort file storage path abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data/store/abort #Delete file at 4 a.m. by default deleteWhen=04 #File retention time, default 48 hours fileReservedTime=120 # commitLog the size of each file is 1G by default mapedFileSizeCommitLog=1073741824 #Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation mapedFileSizeConsumeQueue=3000000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #Detect physical file disk space #diskMaxUsedSpaceRatio=88
broker-b-s.properties
brokerClusterName=RocketMQCluster brokerName=broker-b brokerId=1 deleteWhen=04 fileReservedTime=48 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH listenPort=10921 #nameserver address, semicolon split namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876 brokerIP1=192.168.175.130 #When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default defaultTopicQueueNums=4 #Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online autoCreateTopicEnable=true #Whether Broker is allowed to automatically create subscription groups? It is recommended to turn them on offline and off online autoCreateSubscriptionGroup=true storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data2/store storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/commitlog2 # Consumption queue storage path storage path storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/consumequeue2 #Message index storage path storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/index2 #checkpoint file storage path storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/checkpoint2 #abort file storage path abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/abort2 #Delete file at 4 a.m. by default deleteWhen=04 #File retention time, default 48 hours fileReservedTime=120 # commitLog the size of each file is 1G by default mapedFileSizeCommitLog=1073741824 #Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation mapedFileSizeConsumeQueue=3000000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #Detect physical file disk space #diskMaxUsedSpaceRatio=88
Step 10: the 131132 machine only modifies the contents of broker-b.properties and broker-a-s.properties as follows:
broker-b.properties
# limitations under the License. brokerClusterName=RocketMQCluster brokerName=broker-b brokerId=0 deleteWhen=04 fileReservedTime=48 brokerRole=ASYNC_MASTER flushDiskType=ASYNC_FLUSH listenPort=10911 #nameserver address, semicolon split namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876 brokerIP1=192.168.175.131 #When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default defaultTopicQueueNums=4 #Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online autoCreateTopicEnable=true #Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online autoCreateSubscriptionGroup=true storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data/store storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data/store/commitlog # Consumption queue storage path storage path storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data/store/consumequeue #Message index storage path storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data/store/index #checkpoint file storage path storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data/store/checkpoint #abort file storage path abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data/store/abort #Message size limit maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Delete file at 4 a.m. by default deleteWhen=04 #File retention time, default 48 hours fileReservedTime=120 # commitLog the size of each file is 1G by default mapedFileSizeCommitLog=1073741824 #Each file of ConsumeQueue saves 300W by default, which is adjusted according to the business situation mapedFileSizeConsumeQueue=3000000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #Detect physical file disk space #diskMaxUsedSpaceRatio=88
broker-a-s.properties
# limitations under the License. brokerClusterName=RocketMQCluster brokerName=broker-a brokerId=1 deleteWhen=04 fileReservedTime=48 brokerRole=SLAVE flushDiskType=ASYNC_FLUSH listenPort=10921 namesrvAddr=192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876 brokerIP1=192.168.175.131 #When sending messages, topic s that do not exist in the server are automatically created. The number of queues created by default defaultTopicQueueNums=4 #Whether Broker is allowed to automatically create Topic? It is recommended to open offline and close Online autoCreateTopicEnable=true #Whether Broker is allowed to automatically create subscription groups. It is recommended to enable offline and close Online autoCreateSubscriptionGroup=true storePathRootDir=/opt/local/data/install/rocketmq-bin4.4.0/data2/store storePathCommitLog=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/commitlog # Consumption queue storage path storage path storePathConsumerQueue=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/consumequeue #Message index storage path storePathIndex=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/index #checkpoint file storage path storeCheckpoint=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/checkpoint #abort file storage path abortFile=/opt/local/data/install/rocketmq-bin4.4.0/data2/store/abort #Message size limit maxMessageSize=65536 #flushCommitLogLeastPages=4 #flushConsumeQueueLeastPages=2 #flushCommitLogThoroughInterval=10000 #flushConsumeQueueThoroughInterval=60000 #Delete file at 4 a.m. by default deleteWhen=04 #File retention time, default 48 hours fileReservedTime=120 # commitLog the size of each file is 1G by default mapedFileSizeCommitLog=1073741824 #Each file of ConsumeQueue saves 30W by default, which is adjusted according to the business situation mapedFileSizeConsumeQueue=300000 #destroyMapedFileIntervalForcibly=120000 #redeleteHangedFileInterval=120000 #Detect physical file disk space #diskMaxUsedSpaceRatio=88
Step 11: start
//All three are implemented: nohup sh bin/mqnamesrv > ./logs/namesrvrun.log 2>&1 & 130 Machine execution: nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-a.log 2>&1 & nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b-s.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-b-s.log 2>&1 & 131， 132 Machine execution: nohup sh bin/mqbroker -c conf/2m-2s-async/broker-b.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-b.log 2>&1 & nohup sh bin/mqbroker -c conf/2m-2s-async/broker-a-s.properties -n"192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876" > ./logs/broker-a-s.log 2>&1 &
After execution, the result of jps is that there are two brokerstartup s. If the error is reported, look at the logs folder log created by yourself
Now, the basic configuration of rocketmq is set up. Let's talk about the actual code
Import dependency package
<dependencies> <!-- https://mvnrepository.com/artifact/org.apache.rocketmq/rocketmq-client --> <dependency> <groupId>org.apache.rocketmq</groupId> <artifactId>rocketmq-client</artifactId> <version>4.4.0</version> </dependency> <dependency> <groupId>com.alibaba.rocketmq</groupId> <artifactId>rocketmq-all</artifactId> <version>3.5.9</version> <type>pom</type> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.1.1</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-core</artifactId> <version>1.1.1</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> <scope>test</scope> </dependency> </dependencies>
mq message sender
import java.io.UnsupportedEncodingException; import org.apache.rocketmq.client.exception.MQBrokerException; import org.apache.rocketmq.client.exception.MQClientException; import org.apache.rocketmq.client.producer.DefaultMQProducer; import org.apache.rocketmq.client.producer.SendResult; import org.apache.rocketmq.common.message.Message; import org.apache.rocketmq.remoting.common.RemotingHelper; import org.apache.rocketmq.remoting.exception.RemotingException; /** * message sender * @author LELE * */ public class Producer { public static void main(String[] args) throws MQClientException, InterruptedException { // Declare and initialize a producer // A producer group name is required as a parameter of the construction method, producer 1 here DefaultMQProducer producer = new DefaultMQProducer("producer1"); producer.setVipChannelEnabled(false); // Set the NameServer address, which should be changed to the actual NameServer address, separated by // NameServer address must have // producer.setClientIP("xxxx"); // producer.setInstanceName("Producer"); producer.setNamesrvAddr("192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876"); // Call the start() method to start a producer instance producer.start(); // Send a message to the Topic as TopicTest, tag as TagA, and the message content as the value of i on the "Hello RocketMQ" splice try { for(int i=0;i<30000;i++) { // Encapsulate message Message msg = new Message("TopicTest", // topic "TagA", // tag ("Hello RocketMQ--------"+i).getBytes(RemotingHelper.DEFAULT_CHARSET)// body ); // Call producer's send() method to send a message // The synchronization method is called here, so there will be a return result SendResult sendResult = producer.send(msg); // Print return results System.out.println(sendResult); } } catch (RemotingException e) { e.printStackTrace(); } catch (MQBrokerException e) { e.printStackTrace(); } catch (UnsupportedEncodingException e) { e.printStackTrace(); } // After sending the message, call the shutdown() method to shut down the producer System.out.println("send success"); producer.shutdown(); } }
Message consumer
import java.util.List; import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer; import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext; import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus; import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently; import org.apache.rocketmq.client.exception.MQClientException; import org.apache.rocketmq.common.consumer.ConsumeFromWhere; import org.apache.rocketmq.common.message.MessageExt; /** * The message receiver needs the server to start the mq service * @author LELE * */ public class Consumer { public static void main(String[] args) throws MQClientException { // Declare and initialize a consumer // A consumer group name is required as the parameter of the construction method, here is consumer1 DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer1"); // consumer.setVipChannelEnabled(false); // Also set the NameServer address consumer.setNamesrvAddr("192.168.175.130:9876;192.168.175.131:9876;192.168.175.132:9876"); // Here is a consumer's consumption strategy // Default policy of consume from the end of the queue, i.e. skip historical messages // Consume? From? First? Offset starts from the beginning of the queue, that is, all the historical messages (also stored in the broker) are consumed once // Consumption starts from a certain time point and is used with setConsumeTimestamp(). The default is half an hour ago consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET); // Set the Topic and Tag subscribed by consumer, * represents all tags consumer.subscribe("TopicTest", "*"); // Set a Listener, mainly for message logical processing consumer.registerMessageListener(new MessageListenerConcurrently() { @Override public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) { MessageExt msg = msgs.get(0); if (msg.getTopic().equals("TopicTest")) { // Execute the consumption logic of TopicTest1 System.out.println("TagA:" + new String(msg.getBody())); } System.out.println(Thread.currentThread().getName() + " Receive New Messages: " + msgs.size() + "----------------------------------------------------------------------------------"); // Return to consumption status // Success of consumption // Reconsume? Later consumption failed, need to be re consumed later return ConsumeConcurrentlyStatus.CONSUME_SUCCESS; } }); // Call the start() method to start consumer consumer.start(); System.out.println("Consumer Started."); } }
Start to play, young man, remember to like it!