1. The role of registries
With the registry, service providers can dynamically add and delete services, and service consumers can pull up the latest services to achieve synchronization after receiving update notifications. Unified configuration can be achieved in the registry, and dynamic adjustment of parameters can be automatically notified to all service nodes.
2. Implementation of four Dubbo registries
The implementation of Dubbo registry is in the dubbo-registry module.
2.1 ZooKeeper
Based on Zookeeper. ZooKeeper Learning
2.1.1 Zookeeper Registry Data Structure
Implementation principle of 2.1.2 Zookeeper registry
Zookeeper Registry adopts the implementation of "Event Notification"+ "Client Pull":
- When the client first connects to the registry, it will get the full amount of data in the corresponding directory.
- The client registers a watch on the subscribed node and maintains a long TCP connection between the client and the registry.
- When the node transacts and the version number of the node changes, the watch event will be triggered and the data will be pushed to the subscriber. After the subscriber receives the notification, the data in the corresponding directory will be pulled.
- The Service Governance Center subscribes to all service layer data, and the service is set to "*" to indicate all subscriptions.
2.1.3 Zookeeper Registration and Cancellation
//register protected void doRegister(URL url) { try { //Create directory zkClient.create(toUrlPath(url), url.getParameter(Constants.DYNAMIC_KEY, true)); } catch (Throwable e) { throw new RpcException("Failed to register " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e); } } //Cancellation of registration protected void doUnregister(URL url) { try { //Delete directory zkClient.delete(toUrlPath(url)); } catch (Throwable e) { throw new RpcException("Failed to unregister " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e); } } private String toUrlPath(URL url) { return toCategoryPath(url) + Constants.PATH_SEPARATOR + URL.encode(url.toFullString()); } //Return "/ dubbo / interface name / providers" private String toCategoryPath(URL url) { return toServicePath(url) + Constants.PATH_SEPARATOR + url.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY); } //Returns "/dubbo/interface name" private String toServicePath(URL url) { String name = url.getServiceInterface(); if (Constants.ANY_VALUE.equals(name)) { return toRootPath(); } return toRootDir() + URL.encode(name); } //Return "/" or "/dubbo/" private String toRootDir() { if (root.equals(Constants.PATH_SEPARATOR)) { return root; } return root + Constants.PATH_SEPARATOR; } private String toRootPath() { return root; } private final static String DEFAULT_ROOT = "dubbo"; private final ZookeeperClient zkClient; public ZookeeperRegistry(URL url, ZookeeperTransporter zookeeperTransporter) { //Omit n lines of code String group = url.getParameter(Constants.GROUP_KEY, DEFAULT_ROOT); //Add the prefix "/" if (!group.startsWith(Constants.PATH_SEPARATOR)) { group = Constants.PATH_SEPARATOR + group; } this.root = group; zkClient = zookeeperTransporter.connect(url); //Omit n lines of code }
ZookeeperClient and ZookeeperTransporter are interfaces. There are two implementations in Dubbo. One is Curator Client based on Curator Client and Curator ZookeeperTransporter; the other is ZkclientZookeeperClient and ZkclientZookeeperTransporter based on zkclient Client Client. The default implementation is curator.
@SPI("curator") public interface ZookeeperTransporter { @Adaptive({Constants.CLIENT_KEY, Constants.TRANSPORTER_KEY}) ZookeeperClient connect(URL url); }
When a matching URL cannot be found based on Constants.CLIENT_KEY and Constants.TRANSPORTER_KEY, the default Curator implementation is used.
2.1.4 Implementation of Zookeeper Subscription
First look at the difference between put and put IfAbsent of Concurrent Map
public static final String ANY_VALUE = "*"; public static final String INTERFACE_KEY = "interface"; public static final String CHECK_KEY = "check"; protected void doSubscribe(final URL url, final NotifyListener listener) { try { //When the interface name is * it means subscribing to all services if (Constants.ANY_VALUE.equals(url.getServiceInterface())) { //The default value is "/dubbo" String root = toRootPath(); ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url); if (listeners == null) { //listeners are empty, indicating that they are not in the cache zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>()); listeners = zkListeners.get(url); } ChildListener zkListener = listeners.get(listener); if (zkListener == null) { //zkListener is empty, indicating that it is the first call listeners.putIfAbsent(listener, new ChildListener() { //This internal method will not be executed immediately, but will be executed when a change notification is triggered. @Override public void childChanged(String parentPath, List<String> currentChilds) { //Traversing through all child nodes for (String child : currentChilds) { child = URL.decode(child); //If a child node does not subscribe, subscribe. if (!anyServices.contains(child)) { anyServices.add(child); subscribe(url.setPath(child).addParameters(Constants.INTERFACE_KEY, child, Constants.CHECK_KEY, String.valueOf(false)), listener); } } } }); zkListener = listeners.get(listener); } //Create persistent nodes zkClient.create(root, false); //Traversing the direct child nodes of subscribed persistent nodes List<String> services = zkClient.addChildListener(root, zkListener); if (services != null && !services.isEmpty()) { for (String service : services) { service = URL.decode(service); anyServices.add(service); subscribe(url.setPath(service).addParameters(Constants.INTERFACE_KEY, service, Constants.CHECK_KEY, String.valueOf(false)), listener); } } } else { List<URL> urls = new ArrayList<URL>(); for (String path : toCategoriesPath(url)) { ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url); if (listeners == null) { zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>()); listeners = zkListeners.get(url); } ChildListener zkListener = listeners.get(listener); if (zkListener == null) { listeners.putIfAbsent(listener, new ChildListener() { @Override public void childChanged(String parentPath, List<String> currentChilds) { ZookeeperRegistry.this.notify(url, listener, toUrlsWithEmpty(url, parentPath, currentChilds)); } }); zkListener = listeners.get(listener); } zkClient.create(path, false); List<String> children = zkClient.addChildListener(path, zkListener); if (children != null) { urls.addAll(toUrlsWithEmpty(url, path, children)); } } notify(url, listener, urls); } } catch (Throwable e) { throw new RpcException("Failed to subscribe " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e); } }
private String[] toCategoriesPath(URL url) { String[] categories; //If the category is "*", subscribe to four types of paths (providers,consumers,routers,configurations) if (Constants.ANY_VALUE.equals(url.getParameter(Constants.CATEGORY_KEY))) { categories = new String[]{Constants.PROVIDERS_CATEGORY, Constants.CONSUMERS_CATEGORY, Constants.ROUTERS_CATEGORY, Constants.CONFIGURATORS_CATEGORY}; } else { //The default is providers categories = url.getParameter(Constants.CATEGORY_KEY, new String[]{Constants.DEFAULT_CATEGORY}); } String[] paths = new String[categories.length]; for (int i = 0; i < categories.length; i++) { paths[i] = toServicePath(url) + Constants.PATH_SEPARATOR + categories[i]; } return paths; }
2.2 Redis
Based on Redis. Redis learning
2.2.1 Redis Registry Data Structure
2.2.2 Redis renewal Key and Cleaning Key
org.apache.dubbo.registry.redis.RedisRegistry.deferExpired() defines the expiration time of Redis refresh key and clears the expired key. The service provider publishes a service that first creates a Key in Redis, then publishes a register event, launches the expireExecutor thread pool in the RedisRegistry constructor to periodically call the deferExpired method to refresh the expiration time of the service.
private volatile boolean admin = false; //Get the local cache for all registered key s for (URL url : new HashSet<URL>(getRegistered())) { if (url.getParameter(Constants.DYNAMIC_KEY, true)) { String key = toCategoryPath(url); //Refresh expiration time if (jedis.hset(key, url.toFullString(), String.valueOf(System.currentTimeMillis() + expirePeriod)) == 1) { //Broadcast, redistribute jedis.publish(key, Constants.REGISTER); } } } //If it's a service governance center, you need to clear out expired keys if (admin) { clean(jedis); }
The org.apache.dubbo.common.Constants class defines constants.
public static final String DYNAMIC_KEY = "dynamic";
org.apache.dubbo.registry.support.AbstractRegistry.getRegistered() gets the key of the cache.
private final Set<URL> registered = new ConcurrentHashSet<URL>(); public Set<URL> getRegistered() { return registered; } public void register(URL url) { if (url == null) { throw new IllegalArgumentException("register url == null"); } if (logger.isInfoEnabled()) { logger.info("Register: " + url); } registered.add(url); }
The org.apache.dubbo.registry.redis.RedisRegistry.clean() Service Governance Center deletes expired key s.
private void clean(Jedis jedis) { Set<String> keys = jedis.keys(root + Constants.ANY_VALUE); if (keys != null && !keys.isEmpty()) { for (String key : keys) { //Get jedis to connect all key s Map<String, String> values = jedis.hgetAll(key); if (values != null && values.size() > 0) { boolean delete = false; long now = System.currentTimeMillis(); for (Map.Entry<String, String> entry : values.entrySet()) { URL url = URL.valueOf(entry.getKey()); if (url.getParameter(Constants.DYNAMIC_KEY, true)) { long expire = Long.parseLong(entry.getValue()); //If the key expires earlier than the current time, the key expires. if (expire < now) { jedis.hdel(key, entry.getKey()); delete = true; if (logger.isWarnEnabled()) { logger.warn("Delete expired key: " + key + " -> value: " + entry.getKey() + ", expire: " + new Date(expire) + ", now: " + new Date(now)); } } } } if (delete) { //Broadcast notification cancels which Key to register. jedis.publish(key, Constants.UNREGISTER); } } } } }
2.2.3 Redis registration
(1)org.apache.dubbo.registry.redis.RedisRegistry.doRegister(URL url)
public void doRegister(URL url) { // /dubbo/interface/providers(or consumers or configurations or routes) String key = toCategoryPath(url); //URL String value = url.toFullString(); //Calculate expiration time String expire = String.valueOf(System.currentTimeMillis() + expirePeriod); boolean success = false; RpcException exception = null; //Traversing all nodes in the connection pool for (Map.Entry<String, JedisPool> entry : jedisPools.entrySet()) { JedisPool jedisPool = entry.getValue(); try { Jedis jedis = jedisPool.getResource(); try { //Save key values jedis.hset(key, value, expire); //Release jedis.publish(key, Constants.REGISTER); success = true; //In non-replicate mode, only one node needs to be written, otherwise all nodes need to be written. if (!replicate) { break; // If the server side has synchronized data, just write a single machine } } finally { jedis.close(); } } catch (Throwable t) { exception = new RpcException("Failed to register service to redis registry. registry: " + entry.getKey() + ", service: " + url + ", cause: " + t.getMessage(), t); } } if (exception != null) { if (success) { logger.warn(exception.getMessage(), exception); } else { throw exception; } } }
(2)org.apache.dubbo.registry.redis.RedisRegistry.toCategoryPath(URL url)
private String toCategoryPath(URL url) { //ToService Path (url) gets the root node (default "/ dubbo /") and the interface name //Constants.PATH_SEPARATOR("/") //url.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY) defaults to "providers" return toServicePath(url) + Constants.PATH_SEPARATOR + url.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY); } //Splicing root node and interface name private String toServicePath(URL url) { return root + url.getServiceInterface(); }
The RedisRegistry constructor assigns root:
private final static String DEFAULT_ROOT = "dubbo"; public RedisRegistry(URL url) { ....ellipsis N Line code... String group = url.getParameter(Constants.GROUP_KEY, DEFAULT_ROOT); if (!group.startsWith(Constants.PATH_SEPARATOR)) { group = Constants.PATH_SEPARATOR + group; } if (!group.endsWith(Constants.PATH_SEPARATOR)) { group = group + Constants.PATH_SEPARATOR; } this.root = group; ....ellipsis N Line code... }
Let's look at org. apache. dubbo. common. URL. getService Interface ()
public String getServiceInterface() { return getParameter(Constants.INTERFACE_KEY, path); } public String getParameter(String key, String defaultValue) { String value = getParameter(key); if (value == null || value.length() == 0) { return defaultValue; } return value; } public String getParameter(String key) { String value = parameters.get(key); if (value == null || value.length() == 0) { value = parameters.get(Constants.DEFAULT_KEY_PREFIX + key); } return value; }
Constants are defined in the class org.apache.dubbo.common.Constants:
public final static String PATH_SEPARATOR = "/"; public static final String CATEGORY_KEY = "category"; public static final String DEFAULT_CATEGORY = PROVIDERS_CATEGORY; public static final String PROVIDERS_CATEGORY = "providers"; public static final String GROUP_KEY = "group"; public static final String DEFAULT_KEY_PREFIX = "default."; public static final String INTERFACE_KEY = "interface";
2.2.4 Redis Subscription
For the first subscription, an internal thread class Notifier is created, which asynchronously subscribes to the channel when Notifier is started, and the main thread pulls all transaction information from the registry at once. Notifier subscribes to channel push events to achieve subsequent registry information changes.
The code for the run() method part of the Notifier thread of RedisRegistry is as follows:
if (service.endsWith(Constants.ANY_VALUE)) { //If you end with * if (!first) { //If you have subscribed, get all the key s and update the local cache first = false; Set<String> keys = jedis.keys(service); if (keys != null && !keys.isEmpty()) { for (String s : keys) { doNotify(jedis, s); } } resetSkip();//Reset Failure Counter } //Subscribe jedis.psubscribe(new NotifySub(jedisPool), service); } else { if (!first) { first = false; doNotify(jedis, service); resetSkip(); } jedis.psubscribe(new NotifySub(jedisPool), service + Constants.PATH_SEPARATOR + Constants.ANY_VALUE); //Subscribe to specified categories }
2.3 Simple
Memory-based default implementation. Standard RPC services do not support clustering and may cause a single point of failure.
2.4 Multicast
Mutual discovery is achieved by broadcasting addresses.
3. Caching mechanism
The purpose of caching mechanism is to exchange space for time. If each remote call needs to pull the list of available remote services from the registry, it will cause great pressure on the network. A general cache mechanism is implemented in AbstractRegistry. Dubbo stores a copy in memory, a property object, a disk file, and a file object.
//Local disk cache private final Properties properties = new Properties(); private final ConcurrentMap<URL, Map<String, List<URL>>> notified = new ConcurrentHashMap<URL, Map<String, List<URL>>>(); //Local disk cache file private File file; private void loadProperties() { if (file != null && file.exists()) { InputStream in = null; try { //Read disk files in = new FileInputStream(file); //Loading to properties objects properties.load(in); if (logger.isInfoEnabled()) { logger.info("Load registry store file " + file + ", data: " + properties); } } catch (Throwable e) { logger.warn("Failed to load registry store file " + file, e); } finally { if (in != null) { try { in.close(); } catch (IOException e) { logger.warn(e.getMessage(), e); } } } } }
org.apache.dubbo.registry.support.AbstractRegistry.saveProperties(URL url)
private final boolean syncSaveFile; if (syncSaveFile) { //Synchronous preservation doSaveProperties(version); } else { //Thread pool asynchronous save registryCacheExecutor.execute(new SaveProperties(version)); }
4. Retrial mechanism
org.apache.dubbo.registry.support.FailBackRegistry adds the retry() method.