🔰 Learning video 🔰
ActiveMQ tutorial (MQ messaging middleware quick start)
Number of episodes: 48-57
🔰 Learning maxims 🔰
If you don't practice boxing, you'll end up empty; If the foundation is not strong, the earth will shake.
🔶 Restore the default configuration file of ActiveMQ to facilitate subsequent testing
Step 1: delete the modified configuration file
Enter the conf directory and delete the modified configuration file
rm activemq.xml # Enter y for OK
Step 2: restore backup configuration
Rename the previous backup file activemq.xml
mv activemq.xml.bk activemq.xml
1, Introduction
Official website: https://activemq.apache.org/persistence
High availability of MQ: transaction, persistence, sign in and persistence.
Transaction, persistence and sign in are MQ's own. Persistence requires external physical backup.
In order to avoid the loss of information after accidental downtime, it is necessary to restore the message queue after restart. The message system generally adopts the persistence mechanism. ActiveMQ's message persistence mechanisms include JDBC, AMQ, KahaDB and LevelDB. No matter which persistence method is used, the message storage logic is consistent.
After the sender sends the message, the message center first stores the message in the local data file, memory database or remote database, and then tries to send the message to the receiver. If successful, the message will be deleted from the storage. If failed, the message center will continue to try to send.
After the message center is started, first check the specified storage location. If there are unsuccessful messages, you need to send them.
2, Persistence mechanism
2.1 AMQ Message Store (understand)
AMQ is a file storage form, which has the characteristics of fast writing speed and easy recovery. Messages are stored in files. The default size of the file is 32M. When all messages in a stored file have been consumed, the file will be marked as removable. In the next cleaning stage, the file will be deleted. AMQ is available for versions prior to ActiveMQ 5.3.
2.2 Kaha DB
Official website description: https://activemq.apache.org/kahadb
Based on the log file, the default persistence plug-in is available from ActiveMQ 5.4.
🔶 see
Open the configuration file activemq.xml to view.
Enter the / activeMQ/apache-activemq-5.15.15/data directory to view
KahaDB is the default storage mode at present, which can be used in any scenario to improve performance and recovery capability. The message store uses a transaction log (body) and only an index file (directory) to store all its addresses. KahaDB is a solution for message persistence, which optimizes typical message usage patterns. Data is appended to data logs. When the data in the log file is no longer needed, the log file will be discarded.
🔶 Storage mechanism
kahadb has only four types of files and a lock in the message storage directory, which is very concise compared with several other file storage engines of ActiveMQ.
Composition = 4 files + a lock
File 1: DB - < number >. Log
KahaDB stores messages in a data record file of predefined size, which is named DB - < number >. Log. When the data file is full, a new file will be created and the number will increase with the increase of the number of messages. For example, for every 32M file, the file name will be numbered according to the number, such as db-1.log, db-2.log and db-3.log. When there are no more messages referenced in the data file, the file is deleted or archived.
File 2: db.data
This file contains the persistent BTree index, which indexes the messages in the message data record. It is an index file of messages. It is essentially a B-Tree (B-Tree). B-Tree is used as an index to point to the messages stored in DB - < number >. Log.
File 3: db.free
Which pages in the current db.data file are free? The specific content of the file is the ID of all free pages
File 4: db.redo
db.redo is used for message recovery. If the KahaDB message store is started after forced exit, it is used to recover the BTree index.
File 5: lock
File lock, which indicates the broker that is currently granted kahadb read / write permission
2.3 levelDB (understand)
This file system was introduced after ActiveMQ 5.8. It is very similar to KahaDB. It is also a file based local database storage form, but it provides faster persistence than KahaDB.
However, instead of using a custom B-Tree implementation to index the pre write logs, it uses a LevelDB based index.
The default configuration is as follows:
<persistenceAdapter> <levelDB directory="activemq-data"/> </persistenceAdapter>
3, JDBC (primary)
🔹 Step 1: Analyze
🔹 Step 2: copy driver
Copy MySQL driver Jar package To the lib directory of activemq.
Authorize lib directory copy permission.
chmod 777 /activeMQ/apache-activemq-5.15.15/lib
🔹 Step 3: modify the configuration file
Modify lines 81-83 of the activemq.xml configuration file.
Before modification:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
After modification:
<persistenceAdapter> <jdbcPersistenceAdapter dataSource="#mysql-ds"/> </persistenceAdapter>
dataSource specifies the bean name of the persistent database to be referenced and whether createTablesOnStartup creates a data table at startup. The default value is true. In this way, the data table will be created every time it is started. Generally, it is set to true at the first startup and then changed to false.
🔹 Step 4: database configuration
Add the database configuration to the configuration file. Generally, the MQ server and the database server are separated, so the MySQL database is created on the Windows system.
The configuration of the official website replication is as follows:
Outside the label of < broker >, before < import resource = "jetty. XML" /.
<broker> ... </broker> <!-- MySql DataSource Sample Setup --> <bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="jdbc:mysql://localhost/activemq?relaxAutoCommit=true"/> <property name="username" value="activemq"/> <property name="password" value="activemq"/> <property name="poolPreparedStatements" value="true"/> </bean> <import resource="jetty.xml"/>
Configure according to your own database.
<!-- MySql DataSource Sample Setup --> <bean id="mysql-ds" class="org.apache.commons.dbcp2.BasicDataSource" destroy-method="close"> <property name="driverClassName" value="com.mysql.jdbc.Driver"/> <property name="url" value="jdbc:mysql://192.168.56.1:3306/activemq?relaxAutoCommit=true&serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8&useSSL=false"/> <property name="username" value="root"/> <property name="password" value="123456"/> <property name="poolPreparedStatements" value="true"/> </bean>
In order to prevent error reporting, so many configurations are added after the url.
jdbc:mysql://192.168.56.1:3306/activemq?relaxAutoCommit=true&serverTimezone=UTC&useUnicode=true&characterEncoding=utf-8&useSSL=false
Special note: remote access to Windows local database needs to be enabled: https://blog.csdn.net/ethan__xu/article/details/89320614 Otherwise, it will fail to start at step 5.
Note: if you want to use druid database connection pool, you need to introduce druid jar package in step 2.
🔹 Step 5: warehouse creation SQL and table creation description
Build a database called activemq
CREATE DATABASE activemq;
Restart activemq and you can see that three tables appear in the database.
I had an error at runtime and couldn't start it. After seeing this comment, I modified it and finally started it successfully~
💨 activemq_msgs
ACTIVEMQ_ The acks table stores the information of the persistent subscription and the message ID received by the last persistent subscription.
- ID: self incremented database PK
- CONTAINER: Destination of the message
- MSGID_PROD: the primary key of the message sender
- MSG_SEQ: is the order of sending messages, msgid_ PROD+MSG_SEQ MessagelD that can form JMS
- EXPIRATION: the EXPIRATION time of the message. It stores the number of milliseconds from 1970-01-01 to the present
- MSG: binary data of Java serialized object of message Ontology
- PRIORITY: PRIORITY, from 0 to 9. The higher the value, the higher the PRIORITY
💨 activemq_acks
ACTIVEMQ_ The acks table is used to store the subscription relationship, the information of the persistent subscription and the message ID received by the last persistent subscription. If it is a persistent Topic, the subscription relationship between the subscriber and the server is saved in this table. The database fields are as follows:
- CONTAINER: Destination of the message
- SUB_DEST: if Static cluster is used, this field will contain information about other systems in the cluster
- CLIENT_ID: each subscriber must have a. - only client ID to distinguish
- SUB_NAME: subscriber name
- SELECTOR: SELECTOR. You can choose to consume only messages that meet the conditions. Conditions can be implemented with user-defined attributes, AND multi-attribute AND and OR operations can be supported.
- LAST_ACKED_ID: the ID of the consumed message recorded.
💨 activemq_lock
ACTIVEMQ_ The lock table is only useful in the cluster environment. Only one Broker can get messages, called Master Broker. Others can only be used as backup and wait for the Master Broker to be unavailable before they can become the next Master Broker. This table is used to record which Broker is the current Master Broker.
🔹 Step 6: code run verification
Note that persistence should be turned on
messageProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
💨 Queue producer
import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
public class JmsProduce {
public static final String ACTIVEMQ_URL = "tcp://192.168.150.101:61616";
public static final String QUEUE_NAME = "jdbc01";
public static void main(String[] args) throws JMSException {
// 1 create a connection factory and use the default user name and password according to the given url address
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
// 2. Through the connection factory, obtain the connection and start the access
Connection connection = activeMQConnectionFactory.createConnection();
connection.start();
// 3 create a session
// Parameter 1: transaction
// Parameter 2: sign in
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// 4. Create destination (queue or subject, select queue here)
Queue queue = session.createQueue(QUEUE_NAME);
// 5 create the producer of the message
MessageProducer messageProducer = session.createProducer(queue);
messageProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
// 6. Send three messages to the MQ queue by using messageProducer
for (int i = 0; i < 3; i++) {
// 7 create a message and write a message as required
TextMessage textMessage = session.createTextMessage("jdbc msg----" + i);//A string
// 8 send to mq through messageProducer
messageProducer.send(textMessage);
}
// 9 close resources
messageProducer.close();
session.close();
connection.close();
System.out.println("---------Publish message to MQ complete--------");
}
}
MQ server console:
mysql database ACTIVEMQ_MSGS table:
💨 Queue consumer
import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
import java.io.IOException;
public class JmsConsumer {
public static final String ACTIVEMQ_URL = "tcp://192.168.150.101:61616";
public static final String QUEUE_NAME = "jdbc01";
public static void main(String[] args) throws JMSException, IOException {
// 1 create a connection factory and use the default user name and password according to the given url address
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
// 2. Through the connection factory, obtain the connection and start the access
Connection connection = activeMQConnectionFactory.createConnection();
connection.start();
// 3 create a session
// Parameter 1: transaction
// Parameter 2: sign in
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
// 4. Create destination (queue or subject, select queue here)
Queue queue = session.createQueue(QUEUE_NAME);
// 5 consumers who create messages
MessageConsumer messageConsumer = session.createConsumer(queue);
// 6 processing messages
/*
Mode 1: synchronous blocking mode (receive())
The subscriber or receiver calls the receive() method of MessageConsumer to receive the message,
receive Methods will be blocked until they can receive messages (or time out).
receive(xxxx),If you add a parameter, it means waiting for a certain time, and it will be closed after the time.
*/
// while (true) {
// TextMessage textMessage = (TextMessage) messageConsumer.receive();
// if (textMessage!=null) {
// System.out.println("-------- message received by consumer:" + textMessage.getText());
// } else {
// break;
// }
// }
/*
Mode 2: asynchronous non blocking mode (listener onMessage())
The subscriber or receiver registers a message listener through the setMessageListener(MessageListener listener) of the MessageConsumer,
When the message arrives, the system automatically calls the onMessage(Message message) method of the listener MessageListener.
*/
messageConsumer.setMessageListener(new MessageListener() {
@Override
public void onMessage(Message message) {
if (message != null && message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
try {
System.out.println("-------Consumer received message: " + textMessage.getText());
} catch (JMSException e) {
e.printStackTrace();
}
}
}
});
System.in.read(); // Ensure that the console does not go out and the consumer will wait after starting. If it is turned off, the consumer will turn it off immediately before listening.
// 7 resource shutdown
messageConsumer.close();
session.close();
connection.close();
}
}
MQ server console:
mysql database ACTIVEMQ_MSGS table. After the message is consumed, the data is cleared:
In point-to-point types:
When DeliveryMode is set to non_ During persistence, the message is saved in memory and will not be saved in the database;
When DeliveryMode is set to PERSISTENCE, the message is saved in the corresponding file or database of the broker.
Moreover, once the message in the peer-to-peer type is consumed by the Consumer, it will be deleted from the broker.
💨 Theme consumer
Start the consumer first
import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
import java.io.IOException;
public class JmsConsumerTopicPersist {
public static final String ACTIVEMQ_URL = "tcp://192.168.150.101:61616";
public static final String TOPIC_NAME = "Topic-jdbc";
public static void main(String[] args) throws JMSException, IOException {
System.out.println("customer z3");
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
Connection connection = activeMQConnectionFactory.createConnection();
connection.setClientID("z3");
connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic(TOPIC_NAME);
TopicSubscriber topicSubscriber = session.createDurableSubscriber(topic, "mq-jdbc");
connection.start();
Message message = topicSubscriber.receive();
while (message != null) {
TextMessage textMessage = (TextMessage) message;
System.out.println("------Received persistent Topic: " + textMessage.getText());
message = topicSubscriber.receive(1000L);
}
session.close();
connection.close();
}
}
MQ server console:
View database activemq_acks:
💨 Subject producer
import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
public class JmsProduceTopicPersist {
public static final String ACTIVEMQ_URL = "tcp://192.168.150.101:61616";
public static final String TOPIC_NAME = "Topic-jdbc";
public static void main(String[] args) throws JMSException {
ActiveMQConnectionFactory activeMQConnectionFactory = new ActiveMQConnectionFactory(ACTIVEMQ_URL);
Connection connection = activeMQConnectionFactory.createConnection();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Topic topic = session.createTopic(TOPIC_NAME);
MessageProducer messageProducer = session.createProducer(topic);
messageProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
connection.start();
for (int i = 0; i < 3; i++) {
TextMessage textMessage = session.createTextMessage("TOPIC_NAME----" + i); //A string
messageProducer.send(textMessage);
}
// 9 close resources
messageProducer.close();
session.close();
connection.close();
System.out.println("---------Publish message to MQ complete--------");
}
}
On the MQ server console, there are three messages that have been processed.
After processing the message, the consumer program ends and goes offline.
View database activemq_acks:
🔹 Small summary
queue: the message will be saved to ActiveMQ without consumer consumption_ In the msgs table, as long as any one of the consumers has consumed, these messages will be deleted immediately after consumption.
topic: generally, when the consumer subscription is started before production, the subscriber will be saved to activemq_acks.
4, JDBC with Journal
This method overcomes the shortcomings of JQBC Store. JDBC needs to write and read the library every time a message comes.
ActiveMQ Journal, which uses cache write technology, greatly improves performance.
When the consumption speed of consumers can keep up with the production speed of producer messages in time, the journal file can greatly reduce the messages that need to be written to the DB.
For example, the producer produces 1000 messages, which will be saved to the journal file. If the consumer consumes more than 90% of the messages before the journal file is synchronized to the DB, then only the remaining 10% of the messages need to be synchronized to the DB. If the consumption speed of consumers is very slow, the journal file can make messages written to dB in batch.
🔹 Modify profile
Note out the original:
<persistenceAdapter> <jdbcPersistenceAdapter dataSource="#mysql-ds"/> </persistenceAdapter>
Replace with:
<persistenceFactory> <journalPersistenceAdapterFactory journalLogFiles="4" journalLogFileSize="32768" useJournal="true" useQuickJournal="true" dataSource="#mysql-ds" dataDirectory="activemq-data"/> </persistenceFactory>
Restart activemq
🔹 test
First, start the queue producer in the previous chapter. You can view three new messages on the MQ console, but in ActiveMQ of mysql database_ There is no data in the msgs table. At this time, when the queue consumer is started, the three newly added messages are processed, and there is no change in the database.
5, Summary
🔹 Persistent messages mainly refer to
When the server where MQ is located is down, the message will not be lost.
🔹 Persistence mechanism evolution process
From the original AMQ Message Store scheme to the High performance journal (high performance transaction support) attachment launched in ActiveMQ V4 version, and synchronously launched the storage scheme for relational databases. In ActiveMQ version 5.3, support for KahaDB was introduced (after version 5.4, it is called the default persistence scheme of ActiveMQ). Later, ActiveMQ version 5.8 began to support LevelDB. Up to now, V5.9 + version provides a standard Zookeeper+LevelDB clustering scheme. We focus on three persistent storage schemes: KahaDB, LevelDB and mysq| database.
🔹 Message persistence mechanism of ActiveMQ
- AMQ: log file based
- KahaDB: the default persistence plug-in from ActiveMQ 5.4 based on log files
- JDBC: Based on third-party database
- LevelDB: file based local database storage. Since ActiveMQ version 5.8, levelidb has launched a persistence engine with higher performance than KahaDB
- Replicated LevelDB Store: provides LevelDB and Zookeeper based data replication from ActiveMQ 5.9, which is the preferred data replication scheme for master slave mode.
🔹 The storage logic of the message is consistent
After the sender sends the message, the message center first stores the message in the local data file, memory database or remote database, and then attempts to send the message to the receiver. If the message is sent successfully, it will delete the message from the storage. If it fails, it will continue to try. After the message center is started, first check the specified storage location. If there are unsuccessful messages, you need to send them.