Nacos Client configuration mechanism

spring Load Remote Configuration

Before we understand the NACOS client configuration, let's see how spring loads the remote configuration. Spring provides an extended interface for loading remote configurations, PropertySourceLocator. Here's a simple example:

Implementing Property Source Locator

public class GreizPropertySourceLocator implements PropertySourceLocator {
    @Override
    public PropertySource<?> locate(Environment environment) {
        // Custom configuration, source from anywhere
        Map<String, Object> source = new HashMap<>();
        source.put("userName", "Greiz");
        source.put("userAge", 18);
        return new MapPropertySource(GreizPropertySource.PROPERTY_NAME, source);
    }
}

Property Source Locator has only one interface where we can load custom configurations, such as getting configurations from databases or files.

springboot Start Configuration Class

@Configuration
public class GreizConfigBootstrapConfiguration {
    @Bean
    public GreizPropertySourceLocator greizPropertySourceLocator() {
        return new GreizPropertySourceLocator();
    }
}

Add Start Specified Load Class in META-INF/spring.factories

org.springframework.cloud.bootstrap.BootstrapConfiguration=\
com.greiz.demo.config.GreizConfigBootstrapConfiguration

Use

@Component
public class Greiz {
    @Value("${userName}")
    private String name;
    @Value("${userAge}")
    private Integer age;
      // Provincial getter/setter
}

Use the same as the local configuration.

spring Starts Loading Remote Configuration Process

During the spring start-up prepareContext phase, all implementation classes loaded with custom configurations are executed by PropertySourceLocator and eventually added to the Environment management.

nacos-client

Draw Remote Configuration

The nacos client loads the remote configuration at startup in the above way. Let's take a look at the specific process according to the source code. NacosPropertySourceLocator implements PropertySourceLocator, so the location method is called when spring starts.

public PropertySource<?> locate(Environment env) {
   // 1. Create an object NacosConfigService that interacts remotely
   ConfigService configService = nacosConfigProperties.configServiceInstance();
   ... Ellipsis code
   // 2. Operating on the NacosPropertySource object, the following three methods will eventually call the object build
   nacosPropertySourceBuilder = new NacosPropertySourceBuilder(configService, timeout);
   // 3. 
   String name = nacosConfigProperties.getName();
   String dataIdPrefix = nacosConfigProperties.getPrefix();
   if (StringUtils.isEmpty(dataIdPrefix)) {
      dataIdPrefix = name;
   }
   if (StringUtils.isEmpty(dataIdPrefix)) {
      dataIdPrefix = env.getProperty("spring.application.name");
   }
   // The properties retrieved remotely are stored in this class and eventually placed in the Environment.
   CompositePropertySource composite = new CompositePropertySource(NACOS_PROPERTY_SOURCE_NAME);
   // Loading Common Module Configuration
   loadSharedConfiguration(composite);
   // Load Extension Configuration
   loadExtConfiguration(composite);
   // Load unique configuration
   loadApplicationConfiguration(composite, dataIdPrefix, nacosConfigProperties, env);
   return composite;
}

The NacosPropertySourceBuilder.build() method is ultimately invoked when three configurations are loaded.

NacosPropertySource build(String dataId, String group, String fileExtension, boolean isRefreshable) {
   // load configuration
   Properties p = loadNacosData(dataId, group, fileExtension);
   NacosPropertySource nacosPropertySource = new NacosPropertySource(group, dataId, propertiesToMap(p), new Date(), isRefreshable);
   // Caching nacosProperty Source
   NacosPropertySourceRepository.collectNacosPropertySources(nacosPropertySource);
   return nacosPropertySource;
}

The main logic is in NacosPropertySourceBuilder.loadNacosData().

private Properties loadNacosData(String dataId, String group, String fileExtension) {
    // Get configuration
    String data = configService.getConfig(dataId, group, timeout);
    ... Ellipsis code
    // The. properties extension
    if (fileExtension.equalsIgnoreCase("properties")) {
        Properties properties = new Properties();
        properties.load(new StringReader(data));
        return properties;
    } else if (fileExtension.equalsIgnoreCase("yaml") || fileExtension.equalsIgnoreCase("yml"))         {// yaml or. yml extension
      YamlPropertiesFactoryBean yamlFactory = new YamlPropertiesFactoryBean();
      yamlFactory.setResources(new ByteArrayResource(data.getBytes()));
      return yamlFactory.getObject();
     }
   return EMPTY_PROPERTIES;
}

What really gets the remote configuration is NacosConfigService.getConfig(), which calls getConfigInner().

private String getConfigInner(String tenant, String dataId, String group, long timeoutMs) throws NacosException {
    group = null2defaultGroup(group);
    ParamUtils.checkKeyParam(dataId, group);
    ConfigResponse cr = new ConfigResponse();
    cr.setDataId(dataId);
    cr.setTenant(tenant);
    cr.setGroup(group);

    // 1. Prefer failvoer configuration
    String content = LocalConfigInfoProcessor.getFailover(agent.getName(), dataId, group, tenant);
    if (content != null) {
        cr.setContent(content);
        configFilterChainManager.doFilter(null, cr);
        content = cr.getContent();
        return content;
    }

    try {
        // 2. Server acquisition configuration
        content = worker.getServerConfig(dataId, group, tenant, timeoutMs);
        cr.setContent(content);
        configFilterChainManager.doFilter(null, cr);
        content = cr.getContent();
        return content;
    } catch (NacosException ioe) {
        if (NacosException.NO_RIGHT == ioe.getErrCode()) {
            throw ioe;
        }
    }

    // 3. Take a local snapshot when the server hangs up
    content = LocalConfigInfoProcessor.getSnapshot(agent.getName(), dataId, group, tenant);
    cr.setContent(content);
    configFilterChainManager.doFilter(null, cr);
    content = cr.getContent();
    return content;
}

Of course, access to the dirty work of the service interface requires a client worker ClientWorker. Here is the call to ClientWorker.getServerConfig() in NacosConfigService.getConfig().

public String getServerConfig(String dataId, String group, String tenant, long readTimeout)
    throws NacosException {
    // That's how simple the configuration of http request acquisition is
    HttpResult result = agent.httpGet(Constants.CONFIG_CONTROLLER_PATH, null, params, agent.getEncode(), readTimeout);
  ... Ellipsis code
    // Write a snapshot of the local file
    LocalConfigInfoProcessor.saveSnapshot(agent.getName(), dataId, group, tenant, result.content);
  ...Ellipsis code
        return result.content;
}

At this point, Nacos Client starts reading the remote configuration and encapsulates it to the end of the Environment.

Long polling for updates

The previous section is an analysis of how Nacos Client loads remote configuration at project startup. This section will analyze how Nacos Client knows about configuration changes during project operation.

As mentioned earlier, NacosConfigService is a bridge between Nacos Client and Nacos Server. Let's see how this class works in the configuration update process. Let's first look at how NacosConfigService is constructed.

public NacosConfigService(Properties properties) throws NacosException {
    ... Ellipsis code
    // Initialize namespace
    initNamespace(properties);
    // Query Service List Change
    agent = new MetricsHttpAgent(new ServerHttpAgent(properties));
    agent.start();
    // Here is the configuration update solution
    worker = new ClientWorker(agent, configFilterChainManager, properties);
}

How does ClientWorker maintain client property updates? Look at what the ClientWorker constructor does.

public ClientWorker(final HttpAgent agent, final ConfigFilterChainManager configFilterChainManager, final Properties properties) {
        ...Ellipsis code
    executor = Executors.newScheduledThreadPool(1, new ThreadFactory() {
        ...Ellipsis code
    });
  
    executorService = Executors.newScheduledThreadPool(Runtime.getRuntime().availableProcessors(), new ThreadFactory() {
        ...Ellipsis code
    });

    // Check configuration every 10 milliseconds
    executor.scheduleWithFixedDelay(new Runnable() {
        @Override
        public void run() {
            try {
                checkConfigInfo();
            } catch (Throwable e) {
                LOGGER.error("[" + agent.getName() + "] [sub-check] rotate check error", e);
            }
        }
    }, 1L, 10L, TimeUnit.MILLISECONDS);
}

public void checkConfigInfo() {
    // Divide tasks up into batches
    int listenerSize = cacheMap.get().size();
    int longingTaskCount = (int) Math.ceil(listenerSize / ParamUtil.getPerTaskConfigSize());
    if (longingTaskCount > currentLongingTaskCount) {
        for (int i = (int) currentLongingTaskCount; i < longingTaskCount; i++) {
            executorService.execute(new LongPollingRunnable(i));
        }
        currentLongingTaskCount = longingTaskCount;
    }
}

LongPolling Runnable is a member class of ClientWorker, which implements the Runnable interface. Take a look at the run() method.

public void run() {
    List<CacheData> cacheDatas = new ArrayList<CacheData>();
    List<String> inInitializingCacheList = new ArrayList<String>();
    try {
        // 1. Processing configurations in this task only and checking failover configurations
        for (CacheData cacheData : cacheMap.get().values()) {
            if (cacheData.getTaskId() == taskId) {
                cacheDatas.add(cacheData);
                try {
                    checkLocalConfig(cacheData);
                    if (cacheData.isUseLocalConfigInfo()) {
                        cacheData.checkListenerMd5();
                    }
                } catch (Exception e) {
                    LOGGER.error("get local config info error", e);
                }
            }
        }
                // 2. Comparing the MD5 value of client with that of server, and returning different configurations in the way of "example.properties+DEFAULT_GROUP"
        List<String> changedGroupKeys = checkUpdateDataIds(cacheDatas, inInitializingCacheList);
       // 3. Retrieve the configuration content from the server with the updated configuration
        for (String groupKey : changedGroupKeys) {
            String[] key = GroupKey.parseKey(groupKey);
            String dataId = key[0];
            String group = key[1];
            String tenant = null;
            if (key.length == 3) {
                tenant = key[2];
            }
            try {
                String content = getServerConfig(dataId, group, tenant, 3000L);
                CacheData cache = cacheMap.get().get(GroupKey.getKeyTenant(dataId, group, tenant));
                // Modify the client local value and recalculate the md5 value of the object
                cache.setContent(content);
            } catch (NacosException ioe) {
                ...Ellipsis code
            }
        }
        for (CacheData cacheData : cacheDatas) {
            if (!cacheData.isInitializing() || inInitializingCacheList.contains(GroupKey.getKeyTenant(cacheData.dataId, cacheData.group, cacheData.tenant))) {
                // 4. Check for updates based on md5 values, if updates notify listener
                cacheData.checkListenerMd5();
                cacheData.setInitializing(false);
            }
        }
        inInitializingCacheList.clear();
        // 5. Put this in the thread pool to form a long poll to check the consistency of client and server configurations.
        executorService.execute(this);
    } catch (Throwable e) {
        executorService.schedule(this, taskPenaltyTime, TimeUnit.MILLISECONDS);
    }
}

This section analyses how the Nacos Client configuration keeps close real-time synchronization with the server. Through long polling + short http connection.

Refresh value

Before we begin this section, let's look at a CacheData-like structure that has appeared several times above.

public class CacheData {
    private final String name;
    private final ConfigFilterChainManager configFilterChainManager;
    public final String dataId;
    public final String group;
    public final String tenant;
    // Listener list
    private final CopyOnWriteArrayList<ManagerListenerWrap> listeners;
    // Content md5 value
    private volatile String md5;
    // Whether to use local configuration or not
    private volatile boolean isUseLocalConfig = false;
    // Local version number
    private volatile long localConfigLastModified;
    private volatile String content;
    // Segmented Task ID in Long Polling
    private int taskId;
    private volatile boolean isInitializing = true;
  
        ...Ellipsis code
}

According to the name, CacheData is the object in the configuration data cache. The listeners attribute is interesting. It has a list of listeners in the BO. When the object md5 changes, listeners are notified by traversing listeners.

The previous section checks md5 after getting the updated configuration from the server and calls the CacheData.checkListenerMd5() method:

void checkListenerMd5() {
   for (ManagerListenerWrap wrap : listeners) {
        if (!md5.equals(wrap.lastCallMd5)) {
            safeNotifyListener(dataId, group, content, md5, wrap);
        }
    }
}

class ManagerListenerWrap {
    final Listener listener;
    String lastCallMd5 = CacheData.getMd5String(null);
        ... Ellipsis code
}

private void safeNotifyListener(final String dataId, final String group, final String content, final String md5, final ManagerListenerWrap listenerWrap) {
    final Listener listener = listenerWrap.listener;
    Runnable job = new Runnable() {
        @Override
        public void run() {
            ... Ellipsis code
                // Calling the listener's method
                listener.receiveConfigInfo(contentTmp);
                listenerWrap.lastCallMd5 = md5;
            ... Ellipsis code
        }
    };
    try {
        if (null != listener.getExecutor()) {
            listener.getExecutor().execute(job);
        } else {
            job.run();
        }
    } catch (Throwable t) {
    }
}

This section analyses the notification to configuration listeners after the update configuration is obtained from the server. But when did the listener register? Next, we continue to analyze the process of registering listeners to CacheData.

NacosContextRefresher implements Application Listener. After the container is ready, the onApplicationEvent() method is called, and finally the registerNacosListener() method is called.

private void registerNacosListener(final String group, final String dataId) {
   Listener listener = listenerMap.computeIfAbsent(dataId, i -> new Listener() {
     // Notify the listener to call this method. 
     @Override
      public void receiveConfigInfo(String configInfo) {
         refreshCountIncrement();
         String md5 = "";
         if (!StringUtils.isEmpty(configInfo)) {
            try {
               MessageDigest md = MessageDigest.getInstance("MD5");
               md5 = new BigInteger(1, md.digest(configInfo.getBytes("UTF-8"))).toString(16);
            }
            catch (NoSuchAlgorithmException | UnsupportedEncodingException e) {
               log.warn("[Nacos] unable to get md5 for dataId: " + dataId, e);
            }
         }
         refreshHistory.add(dataId, md5);
         // spring refresh event notification, refresh listener will be executed
         applicationContext.publishEvent(new RefreshEvent(this, null, "Refresh Nacos config"));
      }
      @Override
      public Executor getExecutor() {
         return null;
      }
   });
  // Registered Book Monitor
  configService.addListener(dataId, group, listener);
  ...Ellipsis code
}

NacosConfigService.addListener():

public void addListener(String dataId, String group, Listener listener) throws NacosException {
    worker.addTenantListeners(dataId, group, Arrays.asList(listener));
}

ClientWorker.addTenantListeners()

public void addTenantListeners(String dataId, String group, List<? extends Listener> listeners) throws NacosException {
    group = null2defaultGroup(group);
    String tenant = agent.getTenant();
    CacheData cache = addCacheDataIfAbsent(dataId, group, tenant);
    for (Listener listener : listeners) {
        cache.addListener(listener);
    }
}

Summarize the process of updating configuration during system operation:

1. Register the local update Listener to CacheData at startup.
2. ClientWorker long polling synchronization server update configuration.
3. Get the updated configuration in 2 and reset the CacheData content.
4. Lister. receiveConfigInfo () registered in CacheData callback 1
5. Listener finally notifies spring to refresh the event and completes Context to refresh the attribute values.

summary

Nacos Config Client and Nacos Config Server adopt fixed-time long polling http request access configuration updates, so the design of Nacos Config Server and Config Client is simple. Server also does not have too much pressure on long connection mode Client.