Typical application scenarios are: N servers provide caching services, need to load balance the server, distribute requests to each server on average, and each machine is responsible for 1/N of the service.
The commonly used algorithm is to take the remainder of hash result (hash() mod N): for machine numbers from 0 to N-1, according to the custom hash() algorithm, the hash() value of each request is modeled by N to get the remainder i, and then the request is distributed to the machine with the number I. However, such an algorithm has a fatal problem. If a machine goes down, the requests that should fall on that machine cannot be processed correctly. At this time, it is necessary to remove the fallen servers from the algorithm. At this time, the cache data of (N-1)/N servers will need to be recalculated. If a new machine is added, the cache data of (N-1)/N servers will need to be recalculated. Cached data for servers with N/(N+1) needs to be recalculated.
Consistent Hashing algorithm is a distributed algorithm, which is often used in load balancing. Memcached client also chooses this algorithm to solve the problem of uniformly distributing key-value to many Memcached servers. It can replace the traditional mode taking operation and solve the problem that the mode taking operation can not cope with adding or deleting Memcached Server (adding or deleting servers will lead to the same key, and the hit rate will drop sharply if the server whose data can not be truly stored is not allocated in get operation).
Simply put, consistent hashing organizes the entire hash space into a virtual ring, such as assuming that the value space of a hash function H is 0 - (2 ^ 32) - 1 (that is, the hash value is a 32-bit unsigned integer).
The following is my own summary:
Consistency hash algorithm flow without virtual nodes
1. Define a server list information;
2. Calculate the hash value from the server list and add it to the map (or redis);
3. Calculate the hash value of key.
4. Take out the list of hash values larger than this in map.
5. If there is one, take out the server closest to the node clockwise.
6. If not, the first node in the map can be removed.
7. Completion;
The code is as follows:
//List of servers to be added to the Hash ring
private static String[] servers = { "192.168.0.1:8080", "192.168.0.2:8080",
"192.168.0.3:8080", "192.168.0.4:8080", "192.168.0.5:8080" };
//key represents the hash value of the server and value represents the server
private static SortedMap<Integer, String> sortedMap = new TreeMap<Integer, String>();
//Program initialization, put all servers into sortedMap
static {
for (int i=0; i<servers.length; i++) {
int hash = getHash(servers[i]);
System.out.println("[" + servers[i] + "]Join the collection, his Hash The value is" + hash);
sortedMap.put(hash, servers[i]);
}
System.out.println();
}
//Get the node that should be routed to
private static String getServer(String key) {
//Get the hash value of the key
int hash = getHash(key);
//Get all Map s that are larger than the Hash value
SortedMap<Integer, String> subMap = sortedMap.tailMap(hash);
if(subMap.isEmpty()){
//If there is no one larger than the hash value of the key, start with the first node
Integer i = sortedMap.firstKey();
//Return to the corresponding server
return sortedMap.get(i);
}else{
//The first Key is the nearest node clockwise past the node.
Integer i = subMap.firstKey();
//Return to the corresponding server
return subMap.get(i);
}
}
//Using FNV1_32_HASH algorithm to calculate the Hash value of the server, there is no need to rewrite hashCode method, the final effect is no difference.
private static int getHash(String str) {
final int p = 16777619;
int hash = (int) 2166136261L;
for (int i = 0; i < str.length(); i++)
hash = (hash ^ str.charAt(i)) * p;
hash += hash << 13;
hash ^= hash >> 7;
hash += hash << 3;
hash ^= hash >> 17;
hash += hash << 5;
// If the calculated value is negative, take its absolute value.
if (hash < 0)
hash = Math.abs(hash);
return hash;
}
public static void main(String[] args) {
String[] keys = {"Banana", "pineapple", "Honey"};
for(int i=0; i<keys.length; i++)
System.out.println("[" + keys[i] + "]Of hash The value is" + getHash(keys[i])
+ ", Routed to Node[" + getServer(keys[i]) + "]");
}
Consistent hash algorithm flow with virtual reception
1. First, the original server is added to the list of real nodes.
2. Adding virtual nodes, traversing LinkedList using foreach loop will be more efficient.
3. Get the hash value of the key.
4. Get all Map s larger than the Hash value;
5. If there is no one larger than the hash value of the key, start with the first node.
6. If so, the first Key is the nearest node clockwise past the node.
7. Return to the corresponding server.
8. The virtual Node virtual node name should be intercepted.
9. End
The code is as follows:
//List of servers to be added to the Hash ring
private static String[] servers = {"192.168.0.1:8080", "192.168.0.2:8080", "192.168.0.3:8080",
"192.168.0.4:8080", "192.168.0.5:8080"};
//Real node list, considering the server on-line and off-line scenarios, that is, adding and deleting scenarios will be more frequent, using LinkedList here will be better.
private static List<String> realNodes = new LinkedList<String>();
//Virtual node, key represents the hash value of virtual node, value represents the name of virtual node
private static SortedMap<Integer, String> virtualNodes = new TreeMap<Integer, String>();
//The number of virtual nodes, written here, for demonstration purposes, a real node corresponds to five virtual nodes
private static final int VIRTUAL_NODES = 5;
static{
//First add the original server to the list of real nodes
for(int i=0; i<servers.length; i++)
realNodes.add(servers[i]);
//Adding virtual nodes makes traversing LinkedList more efficient using foreach loops
for (String str : realNodes){
for(int i=0; i<VIRTUAL_NODES; i++){
String virtualNodeName = str + "&&VN" + String.valueOf(i);
int hash = getHash(virtualNodeName);
System.out.println("Virtual Node[" + virtualNodeName + "]Added, hash The value is" + hash);
virtualNodes.put(hash, virtualNodeName);
}
}
System.out.println();
}
//Using FNV1_32_HASH algorithm to calculate the Hash value of the server, there is no need to rewrite hashCode method, the final effect is no difference.
private static int getHash(String str){
final int p = 16777619;
int hash = (int)2166136261L;
for (int i = 0; i < str.length(); i++)
hash = (hash ^ str.charAt(i)) * p;
hash += hash << 13;
hash ^= hash >> 7;
hash += hash << 3;
hash ^= hash >> 17;
hash += hash << 5;
// If the calculated value is negative, take its absolute value.
if (hash < 0)
hash = Math.abs(hash);
return hash;
}
//Get the node that should be routed to
private static String getServer(String key){
//Get the hash value of the key
int hash = getHash(key);
// Get all Map s that are larger than the Hash value
SortedMap<Integer, String> subMap = virtualNodes.tailMap(hash);
String virtualNode;
if(subMap.isEmpty()){
//If there is no one larger than the hash value of the key, start with the first node
Integer i = virtualNodes.firstKey();
//Return to the corresponding server
virtualNode = virtualNodes.get(i);
}else{
//The first Key is the nearest node clockwise past the node.
Integer i = subMap.firstKey();
//Return to the corresponding server
virtualNode = subMap.get(i);
}
//The virtual Node virtual node name needs to be intercepted
if(!StringUtils.isEmpty(virtualNode)){
return virtualNode.substring(0, virtualNode.indexOf("&&"));
}
return null;
}
public static void main(String[] args){
String[] keys = {"Banana", "pineapple", "Honey"};
for(int i=0; i<keys.length; i++)
System.out.println("[" + keys[i] + "]Of hash The value is" +
getHash(keys[i]) + ", Routed to Node[" + getServer(keys[i]) + "]");
}
If you have any questions, please leave a message and I will answer them one by one.