Introduction to Canal
canal [k] ə' Nl], meaning waterway/pipeline/ditch, primarily used for incremental log resolution based on MySQL databases, providing incremental data subscriptions and consumption
Early Alibaba had the business requirement of synchronizing across computer rooms due to dual-room deployment in Hangzhou and the United States. The main way to achieve this was to acquire incremental changes based on business trigger. Since 2010, business has been trying to synchronize incremental changes from database log resolution, which has resulted in a large number of database incremental subscriptions and consumer businesses.
Incremental subscription and consumer-based businesses include
- database mirroring
- Real-time database backup
- Index building and real-time maintenance (splitting heterogeneous indexes, inverted indexes, etc.)
- Business cache refresh
- Incremental data processing with business logic
Current canal support source MySQL versions include 5.1.x, 5.5.x, 5.6.x, 5.7.x, 8.0.x
2. How Canal works
1. MySQL Primary Replication Principle
- MySQL master writes data changes to a binary log, where records are called binary log events and can be viewed by show binlog events
- MySQL slave copies master's binary log events to its relay log
- MySQL slave replays events in relay log to reflect data changes to its own data
2. How Canal works
-
Canal simulates MySQL slave's interaction protocol, disguises itself as MySQL slave, and sends dump protocol to MySQL master
-
MySQL master receives a dump request and starts pushing binary log s to slave (that is, canal)
-
Canal parses binary log objects (originally byte streams)
III. Canal Actual War
1. Demand Analysis
Items coming youlai-mall With the current progress of using Redis to cache OAuth2 client information, role privilege mapping relationships, and menu routing in the MySQL database, there are two obvious problems:
- Modifying roles, menus, permissions, and either side of the OAuth2 client information in the background management interface requires that the cache be invalidated or updated, with high code coupling.
- The database directly modifies the above information so that the cache cannot be invalidated or updated.
The first scenario has at least one solution, either clearing the cache at the code level or updating the cache, but what if you modify the database directly? Actual work may often encounter scenarios where the database is modified directly. This article integrates Canal + RabbitMQ through SpringBoot to monitor the database and synchronize to invalidate or update the cache. Of course, the introduction of Canal middleware refresh cache in the project is just the beginning, and Canal will be used to synchronize the merchandise tables to ElasticSearch next.
2. MySQL opens the binlog log
MySQL deployment: https://www.cnblogs.com/haoxianrui/p/15488810.html
Open biglog log log
vim /etc/my.cnf
Add Configuration
[mysqld] log-bin=mysql-bin # Open binlog binlog-format=ROW # Select ROW mode server_id=1 # Configuring MySQL replaction requires a definition and does not duplicate the slaveId of anal
Restart MySQL to see if the configuration is valid
show variables like 'log_bin';
3. RabbitMQ Queue Creation
- Add switch canal.exchange
- Add queue canal.queue
- Queue-bound switch
4. Canal Configuration and Startup
Canal Server Download
- Official documents: https://github.com/alibaba/canal/wiki
- Project Address: https://github.com/alibaba/canal
- Download address: https://github.com/alibaba/canal/releases
Enter the download address, select canal.deployer-1.1.5.tar.gz
Unzip the zipped package, and I'll put the last unzipped file into the middleware file with the project coming, in the same way as the previous nacos and sentinel.
Canal Server Configuration
There are two things to configure, one to listen for database configuration and the other to RabbitMQ connection configuration.
The two files altered are the Canal configuration file canal.properties and the instance configuration file instance.properties.
㊙️: A Server can configure multiple instances to listen on. There is an example instance that comes with the Canal function by default. This article uses the example instance. If you add an instance, copy the contents of the example folder to a sibling directory, and specify the name of the added instance in canal.properties.
-
canal.properties
Configure Canal services by configuring RabbitMQ and connections ( 🏷️ List only the places that need to be modified)
# tcp, kafka, rocketMQ, rabbitMQ canal.serverMode = rabbitMQ ################################################## ######### RabbitMQ ############# ################################################## rabbitmq.host = x.youlai.tech rabbitmq.virtual.host =/ rabbitmq.exchange =canal.exchange rabbitmq.username =guest rabbitmq.password =guest rabbitmq.deliveryMode =
-
instance.properties
Listen for database configuration ( 🏷️ List only the places that need to be modified)
# position info canal.instance.master.address=x.youlai.tech:3306 # username/password canal.instance.dbUsername=root canal.instance.dbPassword=root # mq config canal.mq.topic=canal.routing.key
5. SpringBoot Integration Canal + RabbitMQ
🏠 Full Source: https://gitee.com/youlaitech/youlai-mall
Introducing dependencies
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
RabbitMQ Connection Configuration
spring:
rabbitmq:
host: x.youlai.tech
port: 5672
username: guest
password: guest
RabbitMQ Listening Synchronization Cache
/**
* Canal + RabbitMQ Listen for database data changes
*
* @author <a href="mailto:xianrui0365@163.com">haoxr</a>
* @date 2021/11/4 23:14
*/
@Component
@Slf4j
@RequiredArgsConstructor
public class CanalListener {
private final ISysPermissionService permissionService;
private final ISysOauthClientService oauthClientService;
private final ISysMenuService menuService;
@RabbitListener(bindings = {
@QueueBinding(
value = @Queue(value = "canal.queue", durable = "true"),
exchange = @Exchange(value = "canal.exchange"),
key = "canal.routing.key"
)
})
public void handleDataChange(String message) {
CanalMessage canalMessage = JSONUtil.toBean(message, CanalMessage.class);
String tableName = canalMessage.getTable();
log.info("Canal Monitor {} Change; Detailed:{}", tableName, message);
if ("sys_oauth_client".equals(tableName)) {
log.info("======== Clear Client Information Cache ========");
oauthClientService.cleanCache();
} else if (Arrays.asList("sys_permission", "sys_role", "sys_role_permission").contains(tableName)) {
log.info("======== Refresh Role Rights Cache ========");
permissionService.refreshPermRolesRules();
} else if (Arrays.asList("sys_menu", "sys_role", "sys_role_menu").contains(tableName)) {
log.info("======== Clean Menu Routing Cache ========");
menuService.cleanCache();
}
}
}
6. Field Test
🏷️ If you use the RabbitMQ test on the project line, remember that you need to create a new queue. If more than one person consumes the same queue, you will feel that Canal is missing listening data.
Next, we simulate whether Redis's menu routing cache can be invalidated when menu data is modified directly in the database.
Start Canal
Switch to the project's CD. /middleware/canal/deployer/bin startup directory and enter startup to start Canal
Start the youlai-admin application, and the test results are as follows. You can see that the final menu route cache will fail when modifying menu table data directly in the database, and achieve the desired effect.
4. Summary
This article monitors MySQL data changes through Canal + RabbitMQ, which can handle scenarios where the cache fails or refreshes after modifying the database data directly. The introduction of Canal is just the beginning because there are so many scenarios in which Canal can be used. Next, there are projects that use Canal to synchronize commodity data from MySQL database to ElasticSearch index library. I think it will become more and more popular in the future, so I suggest that it is necessary to have a better understanding of this Canal framework.
5. Contact information
Welcome to the group of children's shoes who are interested in joining the group. They belong to the group of learning and communication only. They have no interests. If the QR code expires, you can add the note "Yes" to WeChat. I will pull you into the group. If you are interested in joining the open source project. youlai-mall Welcome to Development Privately Trust Me.
| Youlai Group 2-D Codes | WeChat Applet Experience Code | My Wechat |
|---|---|---|
 |
 |
 |