With the continuous development of Internet technology and the increasing number of users, more and more business scenarios need to use distributed systems. In distributed systems, accessing shared resources requires a mutual exclusion mechanism to prevent mutual interference and ensure consistency. At this time, distributed locks are needed.
Characteristics of distributed locks
- Mutex. At any time, only one client can hold the lock
- Lock timeout. Even if a client crashes while holding a lock without actively releasing the lock, it is necessary to ensure that other subsequent clients can lock successfully
- Locking and unlocking must be the same client. The client cannot release the lock added by others.
Version one
# -*- coding: utf-8 -*- # @DateTime : 2020/3/9 15:36 # @Author : woodenrobot import uuid import math import time from redis import WatchError def acquire_lock_with_timeout(conn, lock_name, acquire_timeout=3, lock_timeout=2): """ be based on Redis Distributed locks implemented :param conn: Redis connect :param lock_name: Lock name :param acquire_timeout: The timeout for obtaining the lock, which is 3 seconds by default :param lock_timeout: Lock timeout, 2 seconds by default :return: """ identifier = str(uuid.uuid4()) lockname = f'lock:{lock_name}' lock_timeout = int(math.ceil(lock_timeout)) end = time.time() + acquire_timeout while time.time() < end: # If the lock does not exist, lock it and set the expiration time to avoid deadlock if conn.setnx(lockname, identifier): conn.expire(lockname, lock_timeout) return identifier # If there is a lock and the lock has no expiration time, set the expiration time for it to avoid deadlock elif conn.ttl(lockname) == -1: conn.expire(lockname, lock_timeout) time.sleep(0.001) return False def release_lock(conn, lockname, identifier): """ Release lock :param conn: Redis connect :param lockname: Lock name :param identifier: Identification of lock :return: """ # python in redis Transaction is through pipeline Packaging implementation of with conn.pipeline() as pipe: lockname = 'lock:' + lockname while True: try: # watch lock, multi If the key Changed by other clients, Transaction operation will throw WatchError abnormal pipe.watch(lockname) iden = pipe.get(lockname) if iden and iden.decode('utf-8') == identifier: # Transaction start pipe.multi() pipe.delete(lockname) pipe.execute() return True pipe.unwatch() break except WatchError: pass return False
Locking process
- First, you need to generate a unique ID for the lock. Here, use uuid;
- Then use setnx Set the lock. If there is no lock of other clients before the lock name, the lock is successfully added. Then set the expiration time of the lock to prevent the birth and death lock and return the unique identification of the lock;
- If the setting fails, first judge whether the lock with the lock name has an expiration time, because setnx and expire The execution of the two commands is not atomic. Locking may succeed, but setting the timeout fails, resulting in deadlock. If it does not exist, reset the expiration time of the lock. If it exists, it will continue to cycle until the locking time expires and the locking fails.
Unlocking process
- First, the entire unlocking operation needs to be performed in a Redis transaction;
- use watch Monitor the lock to prevent deleting other people's locks when unlocking;
- Query whether the ID of the lock name is the same as the ID of this unlocking;
- If it is the same, the lock will be deleted in the transaction. If the lock automatically expires during the deletion process, it will be obtained by other clients after expiration because it is set watch The deletion will fail, so that the deletion of other client locks will not occur.
Version 2
If you use a redis version greater than or equal to 2.6.12 Version, the locking process can be simplified. Because after this version Redis set Operation support EX and NX Parameter is an atomic operation.
- EX seconds: set the expiration time of the key to seconds. Executing SET key value EX seconds has the same effect as executing SETEX key seconds value.
- Nx: set the key only when the key does not exist. Executing SET key value NX has the same effect as executing SETNX key value.
# -*- coding: utf-8 -*- # @DateTime : 2020/3/9 15:36 # @Author : woodenrobot import uuid import math import time from redis import WatchError def acquire_lock_with_timeout(conn, lock_name, acquire_timeout=3, lock_timeout=2): """ be based on Redis Distributed locks implemented :param conn: Redis connect :param lock_name: Lock name :param acquire_timeout: The timeout for obtaining the lock, which is 3 seconds by default :param lock_timeout: Lock timeout, 2 seconds by default :return: """ identifier = str(uuid.uuid4()) lockname = f'lock:{lock_name}' lock_timeout = int(math.ceil(lock_timeout)) end = time.time() + acquire_timeout while time.time() < end: # If the lock does not exist, lock it and set the expiration time to avoid deadlock if conn.set(lockname, identifier, ex=lock_timeout, nx=True): return identifier time.sleep(0.001) return False def release_lock(conn, lockname, identifier): """ Release lock :param conn: Redis connect :param lockname: Lock name :param identifier: Identification of lock :return: """ # python in redis Transaction is through pipeline Packaging implementation of with conn.pipeline() as pipe: lockname = 'lock:' + lockname while True: try: # watch lock, multi If the key Changed by other clients, Transaction operation will throw WatchError abnormal pipe.watch(lockname) iden = pipe.get(lockname) if iden and iden.decode('utf-8') == identifier: # Transaction start pipe.multi() pipe.delete(lockname) pipe.execute() return True pipe.unwatch() break except WatchError: pass return False
Version 3
You may also find that the unlocking process is a little complicated in code logic. Don't worry, we can use it Lua Scripts implement atomic operations to simplify the unlocking process.
# -*- coding: utf-8 -*- # @DateTime : 2020/3/9 15:36 # @Author : woodenrobot import uuid import math import time def acquire_lock_with_timeout(conn, lock_name, acquire_timeout=3, lock_timeout=2): """ be based on Redis Distributed locks implemented :param conn: Redis connect :param lock_name: Lock name :param acquire_timeout: The timeout for obtaining the lock, which is 3 seconds by default :param lock_timeout: Lock timeout, 2 seconds by default :return: """ identifier = str(uuid.uuid4()) lockname = f'lock:{lock_name}' lock_timeout = int(math.ceil(lock_timeout)) end = time.time() + acquire_timeout while time.time() < end: # If the lock does not exist, lock it and set the expiration time to avoid deadlock if conn.set(lockname, identifier, ex=lock_timeout, nx=True): return identifier time.sleep(0.001) return False def release_lock(conn, lock_name, identifier): """ Release lock :param conn: Redis connect :param lockname: Lock name :param identifier: Identification of lock :return: """ unlock_script = """ if redis.call("get",KEYS[1]) == ARGV[1] then return redis.call("del",KEYS[1]) else return 0 end """ lockname = f'lock:{lock_name}' unlock = conn.register_script(unlock_script) result = unlock(keys=[lockname], args=[identifier]) if result: return True else: return False