When we build distributed systems, we often need to control mutually exclusive access to shared resources. At this time, we are involved in the implementation of distributed locks (also known as global locks). Based on the current tools, we have a large number of ways of implementation, such as: Redis-based implementation, Zookeeper-based implementation. This paper introduces a method of implementing distributed lock and semaphore based on Consul Key/Value storage.

Distributed Lock Implementation

Distributed locks based on Constul are mainly implemented by acqui and release operations in the Key/Value storage API. The acquire and release operations are similar to Check-And-Set operations:

• The acquire operation returns true only when the lock does not have a holder, and the Value set, while the session executing the operation holds the lock on the Key, otherwise it returns false.
• The release operation uses the specified session to release a Key lock. If the specified session is invalid, it returns false. Otherwise, it set s the Value and returns true.

The Key/Value API s are mainly used in the implementation.

• create session: https://www.consul.io/api/session.html#session_create
• delete session: https://www.consul.io/api/session.html#delete-session
• KV acquire/release: https://www.consul.io/api/kv.html#create-update-key

Basic process

Specific realization

public class Lock {

    private static final String prefix = "lock/";  // Synchronization lock parameter prefix

    private ConsulClient consulClient;
    private String sessionName;
    private String sessionId = null;
    private String lockKey;

    /**
     *
     * @param consulClient
     * @param sessionName   session name of synchronization lock
     * @param lockKey       Key paths locked synchronously in consul's KV storage will automatically add prefix prefix to facilitate categorization and query
     */
    public Lock(ConsulClient consulClient, String sessionName, String lockKey) {
        this.consulClient = consulClient;
        this.sessionName = sessionName;
        this.lockKey = prefix + lockKey;
    }

    /**
     * Get synchronization lock
     *
     * @param block     Is it blocked until the lock is acquired?
     * @return
     */
    public Boolean lock(boolean block) {
        if (sessionId != null) {
            throw new RuntimeException(sessionId + " - Already locked!");
        }
        sessionId = createSession(sessionName);
        while(true) {
            PutParams putParams = new PutParams();
            putParams.setAcquireSession(sessionId);
            if(consulClient.setKVValue(lockKey, "lock:" + LocalDateTime.now(), putParams).getValue()) {
                return true;
            } else if(block) {
                continue;
            } else {
                return false;
            }
        }
    }

    /**
     * Release synchronization lock
     *
     * @return
     */
    public Boolean unlock() {
        PutParams putParams = new PutParams();
        putParams.setReleaseSession(sessionId);
        boolean result = consulClient.setKVValue(lockKey, "unlock:" + LocalDateTime.now(), putParams).getValue();
        consulClient.sessionDestroy(sessionId, null);
        return result;
    }

    /**
     * Create session
     * @param sessionName
     * @return
     */
    private String createSession(String sessionName) {
        NewSession newSession = new NewSession();
        newSession.setName(sessionName);
        return consulClient.sessionCreate(newSession, null).getValue();
    }

}

unit testing

The logic of the following unit test is to simulate different distributed services to compete for locks in a threaded way. Multiple processing threads simultaneously apply for distributed locks in a blocking manner. When processing threads acquire locks, Sleep uses a random event to simulate business logic and release locks after processing.

public class TestLock {

    private Logger logger = Logger.getLogger(getClass());

    @Test
    public void testLock() throws Exception  {
        new Thread(new LockRunner(1)).start();
        new Thread(new LockRunner(2)).start();
        new Thread(new LockRunner(3)).start();
        new Thread(new LockRunner(4)).start();
        new Thread(new LockRunner(5)).start();
        Thread.sleep(200000L);
    }

    class LockRunner implements Runnable {

        private Logger logger = Logger.getLogger(getClass());
        private int flag;

        public LockRunner(int flag) {
            this.flag = flag;
        }

        @Override
        public void run() {
            Lock lock = new Lock(new ConsulClient(), "lock-session", "lock-key");
            try {
                if (lock.lock(true)) {
                    logger.info("Thread " + flag + " start!");
                    Thread.sleep(new Random().nextInt(3000L));
                    logger.info("Thread " + flag + " end!");
                }
            } catch (Exception e) {
                e.printStackTrace();
            } finally {
                lock.unlock();
            }
        }
    }

}

The results of unit test execution are as follows:

2017-04-12 21:28:09,698 INFO  [Thread-0] LockRunner - Thread 1 start!
2017-04-12 21:28:12,717 INFO  [Thread-0] LockRunner - Thread 1 end!
2017-04-12 21:28:13,219 INFO  [Thread-2] LockRunner - Thread 3 start!
2017-04-12 21:28:15,672 INFO  [Thread-2] LockRunner - Thread 3 end!
2017-04-12 21:28:15,735 INFO  [Thread-1] LockRunner - Thread 2 start!
2017-04-12 21:28:17,788 INFO  [Thread-1] LockRunner - Thread 2 end!
2017-04-12 21:28:18,249 INFO  [Thread-4] LockRunner - Thread 5 start!
2017-04-12 21:28:19,573 INFO  [Thread-4] LockRunner - Thread 5 end!
2017-04-12 21:28:19,757 INFO  [Thread-3] LockRunner - Thread 4 start!
2017-04-12 21:28:21,353 INFO  [Thread-3] LockRunner - Thread 4 end!

From the test results, we can see that when concurrency is controlled by distributed locks, only one operation can be executed for multiple synchronization operations, and other operations can only be executed after the locks are released. So through such distributed locks, we can control shared resources and only one operation can be executed at the same time. Execution is carried out to ensure distributed concurrency in data processing.

Suggestions for optimization

In this paper, we implement a simple distributed lock based on Constul, but in actual operation, unlock operation may not be correctly executed due to various unexpected circumstances, so that the scored distributed lock can not be released. Therefore, in order to use distributed locks more perfectly, we must also realize the control of overtime cleaning of locks to ensure that even if there is an abnormal unlock, it can be automatically repaired to enhance the robustness of the system. So how to achieve it? Please continue to pay attention to my subsequent decomposition!

Reference Documents

The API of Key/Value: https://www.consul.io/api/kv.html

Electoral mechanism: https://www.consul.io/docs/guides/leader-election.html