canal's Origin and Related Introduction

To understand the introduction of canal framework, installation and source code, please refer towiki The basic installation process and demo are very clear on the official git. It is strongly recommended to read the related introduction on git.

Mysql related settings before canal deployment

• For self-built MySQL, you need to turn on the Binlog write function, configure binlog-format as ROW mode, and configure it in my.cnf as follows

    [mysqld]
    log-bin=mysql-bin # Open binlog
    binlog-format=ROW # Select ROW mode
    server_id=1 # Configuring MySQL replacement requires definitions, not duplication with canal's slaveId

• Create a canal account
  The container starts mysql and needs to enter the container internal settings (docker exec-it mysql bash)
  Self-construction is direct execution of commands

mysql -uroot -proot
#Create an account (account: canal; password: canal)
CREATE USER canal IDENTIFIED BY 'canal'; 
#Grant authority
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
#Refresh and apply permissions
FLUSH PRIVILEGES;

docker-compose deployment canal

Based on the existing docker,docker-compose environment;
The docker-compose.yml file is as follows:

version: '2'
 
services: 
  canal-server: 
    image: canal/canal-server:v1.1.3
    container_name: canal-server
    ports: 
      - 11111:11111
    environment: 
      - canal.instance.mysql.slaveId=12
      - canal.auto.scan=false
      - canal.destinations=test
      - canal.instance.master.address=192.168.1.9:3306
      - canal.instance.dbUsername=canal
      - canal.instance.dbPassword=canal
      - canal.mq.topic=test
      - canal.instance.filter.regex=esen_approval.apt_approval
    volumes: 
      - ./canal-server/conf/:/admin/canal-server/conf/
      - ./canal-server/logs/:/admin/canal-server/logs/

• canal.instance.mysql.slaveId: slaveId cannot be the same as mysql's serverId
• canal.instance.master.address: mysql address
• canal.instance.dbUsername: mysql account
• canal.instance.dbPassword: mysql password
• More Configuration Items

Start: docker-compose up-d
New test projects add mvn dependencies:

<dependency>
    <groupId>com.alibaba.otter</groupId>
    <artifactId>canal.client</artifactId>
    <version>1.1.0</version>
</dependency>

Test class demo:

public class ClientSample {


    public static void main(String args[]) {
        String destination = "test";
        String ip = "192.168.1.29";
        int batchSize = 1024;
        int count = 0;
        int sum = 0;
        int perSum = 0;
        long start = System.currentTimeMillis();
        long end = 0;
        final ArrayBlockingQueue<Long> queue = new ArrayBlockingQueue<Long>(100);
        try {
            final CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(ip, 11111),
                    destination,
                    "canal",
                    "canal");

            Thread ackThread = new Thread(new Runnable() {

                @Override
                public void run() {
                    while (true) {
                        try {
                            long batchId = queue.take();
                            connector.ack(batchId);
                        } catch (InterruptedException e) {
                        }
                    }
                }
            });
            ackThread.start();

            ((SimpleCanalConnector) connector).setLazyParseEntry(true);
            connector.connect();
            connector.subscribe();
            while (true) {
                Message message = connector.getWithoutAck(batchSize, 100L, TimeUnit.MILLISECONDS);
                long batchId = message.getId();
                int size = message.getRawEntries().size();
                sum += size;
                perSum += size;
                count++;
                queue.add(batchId);
                if (count % 10 == 0) {
                    end = System.currentTimeMillis();
                    if (end - start != 0) {
                        long tps = (perSum * 1000) / (end - start);
                        System.out.println(" total : " + sum + " , current : " + perSum + " , cost : " + (end - start)
                                + " , tps : " + tps);
                        start = end;
                        perSum = 0;
                    }
                }
            }
        } catch (Throwable e) {
            e.printStackTrace();
        }
    }
}