preface
Through the mid-term report and exchange meeting, the author has a deep understanding of Subset business process; At the same time, I also have an understanding of some misunderstandings in the early stage. This article is to update Subset business analysis and correct misunderstandings.
1. Subset is not load balancing
Briefly describe the misunderstanding of preliminary work;
1.1 task requirements
At the beginning of the project, the author only knew that the Subset routing rule was based on the original load balancing logic, so he spent a lot of time on debt balancing:
1.2 structure diagram of load balancing source code
Through source code analysis (refer to previous articles for details), you can get the source code structure diagram of load balancing in TarsJava (based on TarsJava SpringBoot):
@EnableTarsServer annotation: indicates that this is a Tars service;
- @Import(TarsServerConfiguration.class): import tarsserverconfiguration files related to Tars service;
- Communicator: communicator;
- getServantProxyFactory(): get the agent factory manager;
- getObjectProxyFactory(): get the object proxy factory;
- createLoadBalance(): create a client load balancing caller;
- select(): select the load balancing caller (there are four modes to select);
- Invoker: invoker;
- invoke(): specific execution method;
- doInvokeServant(): the lowest level execution method;
- invoke(): specific execution method;
- Invoker: invoker;
- refresh(): update the load balancing caller;
- select(): select the load balancing caller (there are four modes to select);
- createProtocolInvoker(): creates a protocol caller;
- createLoadBalance(): create a client load balancing caller;
- Communicator: communicator;
1.3 load balancing four callers
Among them, load balancing is strongly related to traffic allocation and routing. In TarsJava, load balancing has four callers to choose from:
- Consistent hashloadbalance: consistent hash selector;
- HashLoadBalance: hash selector;
- Roundrobin loadbalance: polling selector;
- DefaultLoadBalance: the default selector (ConsistentHashLoadBalance, HashLoadBalance and roundrobin loadbalance are known from the source code);
1.4 add two load balancing callers
Combined with the requirements document, the author thinks that Subset is to add two load balancing callers:
- ProportionLoadBalance: proportional routing;
- DyeLoadBalance: route by coloring;
The new business processes are:
- First, judge whether it is proportional / colored routing, and call the corresponding load balancing caller;
- Then carry out the original load balancing logic;
- Encapsulate the routing results into status;
1.5 Subset should be a "filter" node rather than a "select" node
There is nothing wrong with this understanding, because the Subset routing rule is before load balancing; But to be exact, this understanding is actually wrong, because Subset is not load balancing.
Subset is a subset of set, so if the subset field is set, you need to select the active node of subset according to the subset field, which is similar to set, and select the active node of subset according to the rules before being responsible for equalization.
In other words, Subset plays a more important role than selecting nodes (returning one) like load balancing, but filtering nodes (returning multiple) like filters.
Therefore, it is necessary to re analyze the source code, find the location where the client obtains the source code of the service node, and analyze and understand it.
2. Source code analysis from scratch
We need to find the place to get the server node.
Because of the previous source code foundation, we can quickly locate the source code:
@EnableTarsServer annotation: indicates that this is a Tars service;
- @Import(TarsServerConfiguration.class): import tarsserverconfiguration files related to Tars service;
- Communicator: communicator;
- getServantProxyFactory(): get the agent factory manager;
- getObjectProxyFactory(): get the object proxy factory;
- Communicator: communicator;
2.1 getObjectProxyFactory() source code analysis
protected ObjectProxyFactory getObjectProxyFactory() {
return objectProxyFactory;
}
The getObjectProxyFactory() method returns an ObjectProxyFactory object proxy factory. Let's click in to see what the factory does:
public <T> ObjectProxy<T> getObjectProxy(Class<T> api, String objName, String setDivision, ServantProxyConfig servantProxyConfig,
LoadBalance<T> loadBalance, ProtocolInvoker<T> protocolInvoker) throws ClientException {
//Service agent configuration
if (servantProxyConfig == null) {
servantProxyConfig = createServantProxyConfig(objName, setDivision);
} else {
servantProxyConfig.setCommunicatorId(communicator.getId());
servantProxyConfig.setModuleName(communicator.getCommunicatorConfig().getModuleName(), communicator.getCommunicatorConfig().isEnableSet(), communicator.getCommunicatorConfig().getSetDivision());
servantProxyConfig.setLocator(communicator.getCommunicatorConfig().getLocator());
addSetDivisionInfo(servantProxyConfig, setDivision);
servantProxyConfig.setRefreshInterval(communicator.getCommunicatorConfig().getRefreshEndpointInterval());
servantProxyConfig.setReportInterval(communicator.getCommunicatorConfig().getReportInterval());
}
//Update server node
updateServantEndpoints(servantProxyConfig);
//Create load balancing
if (loadBalance == null) {
loadBalance = createLoadBalance(servantProxyConfig);
}
//Create protocol call
if (protocolInvoker == null) {
protocolInvoker = createProtocolInvoker(api, servantProxyConfig);
}
return new ObjectProxy<T>(api, servantProxyConfig, loadBalance, protocolInvoker, communicator);
}
The core function of the factory is to generate proxy objects. Here, first configure the service, update the server nodes, then create load balancing and protocol calls, and finally return the configured proxy objects.
2.2 updateservintendpoints() update server node source code analysis
What we need to pay attention to and is in the updateservintendpoints() update server node method. We find the source code of this method as follows:
private void updateServantEndpoints(ServantProxyConfig cfg) {
CommunicatorConfig communicatorConfig = communicator.getCommunicatorConfig();
String endpoints = null;
if (!ParseTools.hasServerNode(cfg.getObjectName()) && !cfg.isDirectConnection() && !communicatorConfig.getLocator().startsWith(cfg.getSimpleObjectName())) {
try {
/** Query server node from registry server */
if (RegisterManager.getInstance().getHandler() != null) {
//Resolve the server node and isolate it with ":
endpoints = ParseTools.parse(RegisterManager.getInstance().getHandler().query(cfg.getSimpleObjectName()),
cfg.getSimpleObjectName());
} else {
endpoints = communicator.getQueryHelper().getServerNodes(cfg);
}
if (StringUtils.isEmpty(endpoints)) {
throw new CommunicatorConfigException(cfg.getSimpleObjectName(), "servant node is empty on get by registry! communicator id=" + communicator.getId());
}
ServantCacheManager.getInstance().save(communicator.getId(), cfg.getSimpleObjectName(), endpoints, communicatorConfig.getDataPath());
} catch (CommunicatorConfigException e) {
/** If it fails, take it out of the local cache file */
endpoints = ServantCacheManager.getInstance().get(communicator.getId(), cfg.getSimpleObjectName(), communicatorConfig.getDataPath());
logger.error(cfg.getSimpleObjectName() + " error occurred on get by registry, use by local cache=" + endpoints + "|" + e.getLocalizedMessage(), e);
}
if (StringUtils.isEmpty(endpoints)) {
throw new CommunicatorConfigException(cfg.getSimpleObjectName(), "error occurred on create proxy, servant endpoint is empty! locator =" + communicatorConfig.getLocator() + "|communicator id=" + communicator.getId());
}
//Save the server node information into the ObjectName property of the communicator config configuration item
cfg.setObjectName(endpoints);
}
if (StringUtils.isEmpty(cfg.getObjectName())) {
throw new CommunicatorConfigException(cfg.getSimpleObjectName(), "error occurred on create proxy, servant endpoint is empty!");
}
}
The core function of the method is in the try statement, which is to obtain all nodes on the server. The logic of obtaining is:
- If the server is not instantiated, obtain the list of service nodes through getServerNodes() method from the communicator configuration item of communicator config;
- If the server has been instantiated, obtain the list of service nodes according to the attached service name;
- If the above operation fails, obtain the service node list from the cache;
2.3 getServerNodes() to obtain the source code analysis of the server node
It can be seen that the core method of obtaining server nodes is getServerNodes(), and the source code is as follows:
public String getServerNodes(ServantProxyConfig config) {
QueryFPrx queryProxy = getPrx();
String name = config.getSimpleObjectName();
//Surviving nodes
Holder<List<EndpointF>> activeEp = new Holder<List<EndpointF>>(new ArrayList<EndpointF>());
//Hung node
Holder<List<EndpointF>> inactiveEp = new Holder<List<EndpointF>>(new ArrayList<EndpointF>());
int ret = TarsHelper.SERVERSUCCESS;
//Judge whether it is an enabled set
if (config.isEnableSet()) {
ret = queryProxy.findObjectByIdInSameSet(name, config.getSetDivision(), activeEp, inactiveEp);
} else {
ret = queryProxy.findObjectByIdInSameGroup(name, activeEp, inactiveEp);
}
if (ret != TarsHelper.SERVERSUCCESS) {
return null;
}
Collections.sort(activeEp.getValue());
//value is the last node parameter
//Format the obtained node list into a string format
StringBuilder value = new StringBuilder();
if (activeEp.value != null && !activeEp.value.isEmpty()) {
for (EndpointF endpointF : activeEp.value) {
if (value.length() > 0) {
value.append(":");
}
value.append(ParseTools.toFormatString(endpointF, true));
}
}
//A formatted string plus the service name of Tars
if (value.length() < 1) {
return null;
}
value.insert(0, Constants.TARS_AT);
value.insert(0, name);
return value.toString();
}
The processing logic of getServerNodes() is:
- getServerNodes() first creates two Holder objects to save the values of the live node list activeEp and the non live node list inactiveEp;
- Then, judge whether it is an enabled set, and call findObjectByIdInSameSet() or findObjectByIdInSameGroup() methods by using object proxy to obtain the values of the list of surviving and non surviving nodes, which are encapsulated in activeEp and inactiveEp;
- Format the obtained node list into a string format value;
- Perform subsequent formatting operations;
2.4 format of endpoints
Through the following test methods, we can know that the formatted string format is as follows:
abc@tcp -h host1 -p 1 -t 3000 -a 1 -g 4 -s setId1 -v 10 -w 9:tcp -h host2 -p 1 -t 3000 -a 1 -g 4 -s setId2 -v 10 -w 9
3. Where should the subset be added
Subset should be before the node list is formatted.
3.1 obtain the source code structure diagram of the server node
Through the above analysis, we can get the source code structure diagram of the server node getServerNodes():
@EnableTarsServer annotation: indicates that this is a Tars service;
- @Import(TarsServerConfiguration.class): import tarsserverconfiguration files related to Tars service;
- Communicator: communicator;
- getServantProxyFactory(): get the agent factory manager;
- getObjectProxyFactory(): get the object proxy factory;
- Updateservintendpoints(): updates the server node;
- getServerNodes(): get the list of service nodes;
- Updateservintendpoints(): updates the server node;
- Communicator: communicator;
3.2 modify getServerNodes() method
From the above analysis, we can know that the processing logic of getServerNodes() is:
- First, create two Holder objects;
- Then, the values of the list of surviving and non surviving nodes are obtained and encapsulated in activeEp and inactive EP;
- Format the obtained node list into a string format value;
- Perform subsequent formatting operations;
We should filter the nodes in the list before data formatting. Otherwise, if we format the string first and then filter, the string operation will be involved. When there are too many service nodes, the verification and judgment of this part of the string will consume performance. Therefore, we should filter the nodes through the Subset rule before formatting. The modified getServerNodes() method is as follows:
public String getServerNodes(ServantProxyConfig config) {
QueryFPrx queryProxy = getPrx();
String name = config.getSimpleObjectName();
//Surviving nodes
Holder<List<EndpointF>> activeEp = new Holder<List<EndpointF>>(new ArrayList<EndpointF>());
//Hung node
Holder<List<EndpointF>> inactiveEp = new Holder<List<EndpointF>>(new ArrayList<EndpointF>());
int ret = TarsHelper.SERVERSUCCESS;
//Judge whether it is an enabled set
if (config.isEnableSet()) {
ret = queryProxy.findObjectByIdInSameSet(name, config.getSetDivision(), activeEp, inactiveEp);
} else {
ret = queryProxy.findObjectByIdInSameGroup(name, activeEp, inactiveEp);
}
if (ret != TarsHelper.SERVERSUCCESS) {
return null;
}
Collections.sort(activeEp.getValue());
//value is the last node parameter
// //Format the obtained node list into a string format
// StringBuilder value = new StringBuilder();
// if (activeEp.value != null && !activeEp.value.isEmpty()) {
// for (EndpointF endpointF : activeEp.value) {
// if (value.length() > 0) {
// value.append(":");
// }
// value.append(ParseTools.toFormatString(endpointF, true));
// }
// }
//Extract the above annotation codes and add filter nodes according to subset rules
StringBuilder value = filterEndpointsBySubset(activeEp, config);
//A formatted string plus the service name of Tars
if (value.length() < 1) {
return null;
}
value.insert(0, Constants.TARS_AT);
value.insert(0, name);
return value.toString();
}
The logic of modification is:
- Extract the code that formats the node list into a string format value;
- Add filterEndpointsBySubset(activeEp, config) to filter node methods according to Subset rules;
- The parameters of this method are the list of surviving nodes and routing rules;
- The logic of this method is to judge the Subset rule first, and then the data format of the node list;
3.3 added filterEndpointsBySubset() method
The implementation logic code of this method is as follows:
public StringBuilder filterEndpointsBySubset(Holder<List<EndpointF>> activeEp, ServantProxyConfig config){
StringBuilder value = new StringBuilder();
//Non null judgment of config
if(config == null){
return null;
}
String ruleType = config.getRuleType();
Map<String, String> ruleData = config.getRuleData();
String routeKey = config.getRouteKey();
if(ruleData.size() < 1 || ruleType == null){
return null;
}
//Proportional routing
if(Constants.TARS_SUBSET_PROPORTION.equals(ruleType)){
int totalWeight = 0;
int supWeight = 0;
String subset = null;
//Get total weight
for(String weight : ruleData.values()){
totalWeight+=Integer.parseInt(weight);
}
//Get random number
Random random = new Random();
int r = random.nextInt(totalWeight);
//Find subset based on random number
for (Map.Entry<String, String> entry : ruleData.entrySet()){
supWeight+=Integer.parseInt(entry.getValue());
if( r < supWeight){
subset = entry.getKey();
break;
}
}
//Using subset to filter unqualified nodes
if (activeEp.value != null && !activeEp.value.isEmpty()) {
for (EndpointF endpointF : activeEp.value) {
//subset judgment
if(endpointF != null && endpointF.getSubset().equals(subset)){
if (value.length() > 0) {
value.append(":");
}
value.append(ParseTools.toFormatString(endpointF, true));
}
}
}
return value;
}
//Route by request parameters
if(Constants.TARS_SUBSET_PARAMETER.equals(ruleType)){
//Gets the path to route to
String route = ruleData.get("route");
if( route == null ){
return null;
}
//Judge whether the Key "equal" is included; "match" and get the dyeing Key
String key;
if(ruleData.containsKey("equal")){
//Precise routing
key = ruleData.get("equal");
//Non null verification of the dyeing Key
if(key == null || "".equals(key)){
return null;
}
//Using subset to filter unqualified nodes
if (activeEp.value != null && !activeEp.value.isEmpty()) {
for (EndpointF endpointF : activeEp.value) {
//subset judgment
if(endpointF != null && routeKey.equals(key) && route.equals(endpointF.getSubset())){
if (value.length() > 0) {
value.append(":");
}
value.append(ParseTools.toFormatString(endpointF, true));
}
}
}
} else if( ruleData.containsKey("match")){
//Regular routing
key = ruleData.get("match");
//Non null verification of the dyeing Key
if(key == null || "".equals(key)){
return null;
}
//Using subset to filter unqualified nodes
if (activeEp.value != null && !activeEp.value.isEmpty()) {
for (EndpointF endpointF : activeEp.value) {
//subset judgment, regular rule
if(endpointF != null && StringUtils.matches(key, routeKey) && route.equals(endpointF.getSubset())){
if (value.length() > 0) {
value.append(":");
}
value.append(ParseTools.toFormatString(endpointF, true));
}
}
}
} else {
//[error reporting]
return null;
}
return value;
}
//Irregular routing
if(Constants.TARS_SUBSET_DEFAULT.equals(ruleType)){
//Gets the path to route to
String route = ruleData.get("default");
if( route == null ){
return null;
}
//Using subset to filter unqualified nodes
if (activeEp.value != null && !activeEp.value.isEmpty()) {
for (EndpointF endpointF : activeEp.value) {
//subset judgment
if(endpointF != null && endpointF.getSubset().equals(route)){
if (value.length() > 0) {
value.append(":");
}
value.append(ParseTools.toFormatString(endpointF, true));
}
}
}
return value;
}
return value;
}
Because the method is redundant, but the idea is correct, the test runs smoothly, and it needs to be further modified, simplified and optimized in the later stage.
4. Summary
4.1 Subset is not load balancing
Subset traffic routing should be equivalent to a filter before load balancing.
4.2 source code structure diagram of getservernodes()
You can know the ideological logic of obtaining the server node, and obtain the source code structure diagram of the server node getServerNodes():
@EnableTarsServer annotation: indicates that this is a Tars service;
- @Import(TarsServerConfiguration.class): import tarsserverconfiguration files related to Tars service;
- Communicator: communicator;
- getServantProxyFactory(): get the agent factory manager;
- getObjectProxyFactory(): get the object proxy factory;
- Updateservintendpoints(): updates the server node;
- getServerNodes(): get the list of service nodes;
- Updateservintendpoints(): updates the server node;
- Communicator: communicator;
4.3 the core is in the filterEndpointsBySubset() method
The main function of this method is to filter nodes according to Subset rules and format the node list.
last
Newcomer production, if there are mistakes, welcome to point out, thank you very much! Welcome to the official account and share some more everyday things. If you need to reprint, please mark the source!